No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models
Authors: Jean Kaddour
Oscar Key
Piotr Nawrot
Pasquale Minervini
Matt Kusner
Venue: NeurIPS 2023 poster (2023)
Keywords: language models
transformers
efficient training
TL;DR: We investigate efficient training algorithms for Transformer Language models and find many situations in which they have only marginal improvements over standard training.

Abstract: The computation necessary for training Transformer-based language models has skyrocketed in recent years.
This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, layer dropping), batch selection (selective backprop., RHO-loss), and efficient optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed computation budget using such methods, we find that their training, validation, and downstream gains vanish compared to a baseline with a fully-decayed learning rate. We define an evaluation protocol that enables computation to be done on arbitrary machines by mapping all computation time to a reference machine which we call reference system time. We discuss the limitations of our proposed protocol and release our code to encourage rigorous research in efficient training procedures: https://github.com/JeanKaddour/NoTrainNoGain.

No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models
Jean Kaddour Oscar Key!- Piotr Nawrot? Pasquale Minervini
Matt Kusner
Centre for Artificial Intelligence, University College London Schooo of Informatics, University of Edinburgh {jean. kaddour 20, oscar 20,, kusner)@ucll..... acc.aac.uu {piotr. nawrot, p.minervini)@ed ac. uk
Abstract
The computation necessary for training Transformer-base based language models has skyrocketed in recent years. This trend has motivated research on efficient train- ing algorithms designed improve training, validation, and downstream perfor- mance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, layer dropping), batch selection (selective backprop, RHO loss), and efficient optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed computation budget us- ing such methods, we find that their training, validation, and downstream gains vanish compared to baseline with a fully-decayed learning rate. We define an evaluation protocol that enables computation to be done arbitrary machines by mapping all computation time to reference machine which we call refer- ence system time. We discuss the limitations of proposed protocol and lease our code to encourage rigorous research in efficient training procedures: https://gite htt//gtttuuu.... htw.ttthb.cooomaaaee htwp://aaaaaddoroooooaaaallooooa
Introduction
Language models are advancing rapidly, surpassing human-level performances in some experimental setups [11, 1, 47, 99, 19, 12]. These improvements are primarily due to model, data, and training budget scaling [44, 34, 3]. Training a single ttate-of-the-as art language model requires hundreds of thousands of GPU hours, costs millions of dollars, and consumes as much energy as multiple average US family households per year [81, 73, 14, 92, 43].
To remedy this, there is rapidly growing line of work on efficient training algorithms, which modify the training procedure to save computation [81, 8, 1 112. 85, 64, 43J. Specifically, three distinct classes of such algorithms are dynamic architectures (layer stacking [30] and layer dropping [107)) which ignore some of the weights during training, batch selection (selective backprop [39] and RHO loss [66]) which skip irrelevant data, and efficient optimizers (Lion [13] and Sophia [58]) which claim to convergence faster than Adam(W) [48, 62].
However, find that the evaluation methodology is not standardized, and there are inconsistencies in the quantification of a "speed-up"" Two general trends are: comparisons of (1) intermediate performances throughout training instead of final ones within pre-defined training budget, and (2) incomplete speedup metrics, e.g., training progress as a function of the number of iterations (or epochs) instead of wall-clock computation times despite unequal per-iteration costs.
As pointed out by Dehghani et al. [20], Dahl et al. [17], this can be unfair to baselines, which were not tuned for efficiency within the same compute budget.
Equal contribution, alphabetical order.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

GLUE
SuperGLUE
6h
12h
24h
6h
12h
24h
Baseline
77.1 + 0.2
77.8 0.2
78.3 + 1.1
57.8 + 0.9
58.5 1.1
58.6 1.2
Layer stacking
76.8 0.8
78.4 0.4
58.6 0.2
0.7
28.7
Layer dropping
76,8 = 0.3
7.1 7.2
58.6
58,4 5.75
Selective backprop
7
7..
0.3
5..4
0.9
58.5 0.4
RHO loss
0.1
1.3
7.2
$.
0.6
58.7 0.7
Baseline (BF16)
77.0 = 0.3
77.8 + 0.2
77.9
0.5
0.5
57.9 0.6
Lion (BF16)
62.0 + 13.7
72.0 0.5
7.8
2.5
57.2 0.2
2.3
Sophia (BF16)
73.9 1.3
71.1 4.2
7.8
0.6
5.78
57.5 + 0.7
Table 1: Downstream performance, BERT. Results for efficient training methods on the GLUE and SuperGLUE dev sets for three budgets (6 hours, hours, and 24 hours) after pre-training a crammed BERT model [22. 29]. We report average validation of GLUE and SuperGLUE scores across all tasks (standard deviations for three seeds). We use mixed precision training with BF16 for the optimizer comparison we found FP16 precision lead to numerical instabilities (Section 6).
An example for (I) includes learning rate sched- ules, which can be arbitrarily stretched to im- prove the final performance while "sacrific- ing" the quality of intermediate checkpoints [103, 49, 87]. This can make comparisons of intermediate performances unfair to baselines. For (2), additional regularization techniques can improve the per-iteration convergence at higher per-iteration costs [5, 27, 42, 17], rendering wall-clock time comparison more appropriate.
SNI
6h
12h
24h
Baseline
34.2
38.3
0.4
39.5 0.1
Layer stacking
38.9 38.9
39.2
38.9
Layer dropping Lion
31.5
34.4
38.0
1.1
30.7 0.4
$3.7
Sophia
2.87&
31.5 0.8 3.11
Table 2: Downstream performance, T5-Base [77, 69]. SNI [98] test ROUGE-L results (three seeds).
In this paper, we propose simple evaluation protocol for comparing speedups of efficient training algorithms. We use this protocol evaluate language models from scratch. We compare different training budgets (6, 12, and 24 hours), model architectures (BERT-Base [22] and T5- Base [77]), and (for batch selection algorithms) datasets (C4 [77], Wikipedia and BookCorpus [1 and MiniPile [41]). account variability in measured wall-clock time on different hardware and software configurations, we propose simple measure that we call reference system time (RST).
these algorithms for pre-training Transformer-basee
Our key findings are as follows:
Training loss (layer stacking, layer dropping, Lion, Sophia): The only approach to con- sistently outperform the training loss of the fully-decayed learning rate baseline across budgets and models is layer stacking (Lion matches this performance for certain BERT training budgets). This improvement reduces as the budget increases 24 hours.
Validation loss (selective backprop, RHO loss): Across three training datasets, none of the batch selection methods outperform the validation loss of the baseline.
Downstream tasks: For 24-hour budget, none the efficient training algorithms we evaluate improves the downstream performance of the baseline.
Methods with lower per-iteration costs than the baseline (i.e., dynamic architecture methods: layer stacking, layer dropping) can slightly improve downstream performance for lower budgets (6 hours, 12 hours), but the improvement disappears with longer training.
Methods with higher per-iteration costs (i.c., batch selection methods: selective backprop, RHO loss, and some eflicient optimizer methods: Sophia) are significantly worse than the baseline in some downstream tasks (GLUE, SNI), for all budgets.
we ignore the additional per-iteration computations of the three above methods, the downstream performance is still matched by the baseline.

---

2 Comparing Efficient Training Algorithms
What is the fairest way to compare efficient training algorithms? Ideally, each method should be given an identical compute budget, and methods that converge to better solutions than a baseline within this budget achieve a speed-up. Below, we discuss the issues commonly used metrics to specify the compute budget, and propose improved metric.
The Pitfalls of Iterations Specifying a budget in number of training iterations is usually not suitable because the iteration time can vary between methods.
The Pitfalls of FLOPs While FLOPSs count the number of elementary arithmetic operations, they do not account for parallelizability (e.g., RNNs vs Transformers) or hardware-related details that can affect runtime. For example, FLOP counting ignores memory access times (e.g., due to different layouts of the data in memory) and communication overheads, among other things [20, 8].
The Pitfalls of Wall-Clock Time WCT can fluctuate even on the same hardware, for instance, due to the usage of non-deterministic operations 2, hidden background processes, or inconsequential configurations, such as the clock rate. Further, we would like a metric that allows researchers to run on shared compute clusters, where hardware configurations will vary.
Our Proposal: Reference System Time (RST) We propose simple time measure that will allow us standardize any timing result w.r.t a reference hardware system (e.g.. NVIDIA RTX 3090, CUDA runtime 11.8, Py PyTorch 2.0, etc.). We define the time elapsed on a reference system as reference system time (RST). To convert the training run of an arbitrary device RST, we first record the time training iteration on the reference training system. Then, we compute the RST by multiplying the number of iterations run on the arbitrary device by the time per iteration on the reference system. This way, the time is grounded in practical time units but can be applied to any system.
2.1 The Pitfall of Comparing Intermediate Performances
One difficulty with comparing efficient training methods stems from the importance of the learning rate schedule to model performance. Various works have demonstrated that deep learning models tend to reach their maximum performance only once the learning rate has been fully decayed [109, 76]. If the learning rate schedule is stretched, this can delay the number of iterations required for the model to reach given performance threshold since the optimizer keeps "cxploring" the local loss basin, wasting iterations oscillating [33, 42).
This leads to unfair comparisons when the learning rate schedule not set fairly between methods; a non-obvious pitfall that has been discussed previously in the literature [50, 20], but which still observed in several of the works we evaluated [30, 107, 39, 58]. Consider comparing a baseline with an efficient training method X, where both methods are trained for the same number of iterations, and the learning rate is decayed in terms of the number of iterations. B and X reach a similar performance at the end of training. but because X completes the iterations in less WCT, speed-up is declared.
However, this is not a fair comparison because the training budget was measured in iterations, but the
performance of the models was evaluated at a point specified in WCT, specifically the time it took X to complete the budget. Thus, B is evaluated at intermediate point during training when its learning rate schedule has not been fully decayed. If the learning rate schedule of had been shortened so finished the same WCT X, the two methods may have reached similar performances in the same WCT time. The same problem arises when the budget is specified using different units to the point of measurement, including the cases where FLOPs are used for the budget.
To avoid this issue, each method should be trained to a fixed budget with the performance measured at the end of this budget. Importantly, the learning rate and other schedules must be decayed function of the same unit used to measure the budget.
/nttps://pytorch org/ httpaa/aaee/ettteeeaaemmesss html 'We average the time required per iteration across 1000 iterations given some model architecture, batch, sequence length, system configuration, etc.
3

---

3 Experimental Setup
Given this computation measure, we can now evaluate efficient training algorithms under fixed computation budgets. We conduct experiments using two established and widely used Transformer language models: BERT [22] and T5 [77]. We choose single-GPU setting following recent works [29, 36, 69] facilitate reproducibility and access for compute-restricted researchers. However, in principle, our protocol can be run on any hardware and straightforwardly used in distributed setup.
BERT: Encoder-only. We follow the setup and hyperparameters of Geiping & Goldstein [29] with minor modifications. We pre-train a BERT-base like model with 16 layers instead of 12, which consists of 120M parameters, using a masked language modeling (MLM) objectíve. We use the AdamW optimizer [62] with hyperparameters 31 0.9, 82 0.98, 10-12 and weight decay of 10-2 [62]. Further, we use a one-cycle learning rate schedule [87] with a peak learning rate of 10-3 and gradient clipping of 0.5. The batch size 1536 sequences, and the sequence length 128. We fine-tune and evaluate the pre-trained BERT models using the GLUE [94] and SuperGLUE [95] benchmarks, For fine-tuning on GLUE, we use the hyper-parameters from Geiping & Goldstein [29]. On SuperGLUE, we tune them using the validation accuracy of BoolQ [15]. We found inconsistencies in the literature on how aggregate the SuperGLUB scores; here, we average the following scores: for CB, we use the FI score; for the rest (BoolQ, COPA, RTE, WIC, WSC), we use the accuracy.
We pre-train a T5v1.1-Base [77, 82] model using the original span-corrupting MLM objective and SentencePiece [52] tokenizer. We follow Nawrot [69] and use the AdamW optimizer [62] with tensor-wise LR scaling by its root mean square (RMS), base learning rate 0.02, no weight decay, cosine sched- ule with final of 10-5 [63], gradient clipping of 1.0, and 10d warm up steps. We use a batch size of 144 examples, where each example consists of input of 512 tokens and an output of 114 tokens. We evaluate our pre-trained T5 models on the auper-NaturallInstuutttie benchmark. To fine-tune the model on SNI, we strictly follow the hyperparameters of [69, 98]. For both pre- mre-training and fine-tuning, use TF32 precision.
T5: Encoder-Decoded
Algorithm Layer stacking [30]
Input: number of layers I,, number of stacking operations k
Initialize model fó with 1/22 layers fo + Train (.))
for i 0,... k do
f + (ft, f() (Stack layers.)
Train (f!)
[SNI, 98]
Dataset. Unless specified otherwise, we use the C4 [77] dataset for less than one epoch (i.e., without data repetitions) without sentence curriculum and de-duplication [53, 29]. In Section we additionally use two other datasets.
Learning-rate schedule. We adjust the learning rate schedule based on the elapsed time conditional on a time budget (measured in RST), similar to [37, 29], who measure raw WCT.
Hyper-parameter search. For all considered methods, we tune their hyper-parameters based on the pre-training loss. We list all details about each method's 's hyper-parameters, our considered grid search ranges, and the best values we found in Appendix C.
Reference System. We record the RST on two separate systems for BERT and T5. For BERT, we choose single NVIDIA RTX 3090, CUDA 11.7, Py PyTorch 1.13. For T5, we choose an NVIDIA A100, CUDA runtime 11.8, PyTorch 2.0.
4 Case Study 1: Dynamic Architectures
4.1
Layer stacking
Layer stacking [30], as summarized Algorithm 1, replicates a L-layer model into a 2L-layer model by copying its parameters, effectively warm-starting the stacked model with parameters transferred from the smaller model. Thereby, it benefits from faster per-iteration times in the early training phases when using fewer layers. Gong et al. [30] attribute the success of this method to attention

---

3.0
4
baseline
baseline
stacking
stacking
layer drop
layer drop
2.0
2
12 RST (hours)
24
12 RST (hours)
24
(a) BERT
(b) TS
Figure Training losses, dynamic architecture methods (layer stacking, and layer dropping). Results are shown for RST budgets of 6 hours (-...), ), 12 hours (-- and 24 hours (-), on C4.
distributions of bottom layers being very similar to attention distributions of top layers, indicating that their functionalities are similar.
4.2 Layer dropping
Layer dropping exploits the following observation: the layers of network do not contribute equally to the loss reduction throughout training [35, 6, 106]. It does by randomly choosing parts of the model to be skipped during each training step. Specifically, it replaces a subset of Transformer blocks with the identity function. As a result, it reduces the overall computational cost of the model for each training step because it skips these layers during the forward and backward passes.
Algorithm 2 Layer dropping [107]
1: Input: iterations T. layer keep prohability
a, temperature budget " > 0, layers L, functions (self-attention, layer-norm, feed- forward) fArTN.
loss function L. data (No.)). output layer fo
2: 7 7
3: for ( to do
4:
p Keep probability.]
a,
a) expepp[11 -f)
To minimize the impact of layer dropping on the pretrain- ing loss of the model, Zhang He [107] employ time and depth schedule that determines the probability of drop- ping each block. The time schedule begins with zero probability of dropping each block. Then it increases this probability throughout the training process until it reaches maximum of (1-dl (1, where the hyperparameter ā = 0.5 as chosen by Zhang & He [107].
6:
Pa f- 1-o [Layer decay.)
for: 8 L do
7:
8:
Bernoulli(p) (Keep drop.)
9:
if a == 0 then
X + x; (Drop.)
else
x
Larra(Lew(x)1
X c- x; + fevxifen
P
14:
p pa (Decay prob.)
E- ((fo(x)))
The depth schedule ensures that blocks located earlier in the model dropped with lower probability than those located later/deeper. An important hyperparameter of the depth schedule in layer dropping ?y. which controls the rate at which the probability of dropping layers increases. A higher value of y leads to a quicker increase in the probability. Zhang & He [107] set yy 100 in their experiments. We refer the reader to the original work by Zhang & He [107] for more details.
fATTN, SLN, fFEw. Update(()
4.3 Results
Training losses. Figure shows the pre-training losses for the baseline, layer stacking, and layer dropping BERT and T5 when given a budget of 24 hours in RST. In both settings, layer stacking achieves the lowest loss within this budget. The gap between layer stacking and the bascline closes almost completely the budget is increased to 24 hours. However, layer dropping is consistently worse than the baseline throughout training. In both models, the gap between layer dropping and the baseline is larger than between the baseline and layer stacking at the end of training. Further, layer dropping introduces additional fluctuations in the loss when training T5, compared to the baseline.
Downstream performance. Figure 2 shows the SuperGLUU. downstream performance after using the baseline and all dynamic architecture methods (layer stacking and layer dropping) to pre-train
5

---

baseline
stacking
layer drop
selective backprop
ROOLs
Ghrs
58
25
55
80
65
12hrs
59
25
80
65
24hrs
59
25
55
CОРА
WIC
WSC
BoolQ
RTE
avg
Figure 2: BERT models evaluated on SuperGLUE. The black vertical error bars indicate the standard deviation over three seeds. The black horizontal line shows the baseline average performance. For clarity, the individual tasks are plotted against the left-hand axis, while the average accuracy is plotted against the right-hand axis.
BERT. evaluate all methods using three different RST budgets (to get a sense of the Pareto- frontier of budget and performance [75]): 6 hours, 12 hours, and 24 hours. We notice that specific SuperGLUE datasets, the efficiency methods have little or even detrimental effect on performance (CB, WiC, WSC, BoolQ). Across all datasets, COPA appears to have the largest gains from efficient training. Averaged across all datasets, efficient training methods have little effect. We report the exact numbers in Table (right). On average, across all budgets, neither layer stacking and layer dropping significantly improve over the baseline.
We also compare the performance of dynamic architecture methods methods on the T5-base model evaluated on the SNI benchmark and report the results in Table 2. Here layer stacking significantly improves over the baseline for and 12 hour budgets. However, given 24 hours for training, the final accuracy layer stacking reduces back to the accuracy of the 6 hour model. Here the baseline significantly improves over all methods. Notably, for all budgets, layer dropping is consistently outperformed by the baseline.
5
Case Study 2: Batch Selection
Scaling up the size of web-crawled data for pre-training has been of the major drivers to improve the capabilities language models [44, 10]. A line of work has argued that training speed-ups emerge through the selection of informative examples. They argue that these examples can be identified from certain statistics collected throughout training [4, 45, 39, 46, 72]. Here, we focus on two such methods that directly alter the training procedure steer gradient computations towards informative samples by subsampling examples from mega- mega-batch batch to curate the mini-bater bati-batch, called batch selection.
5.1 Selective Backprop
Due to its simplicity, we first examine selective backprop [39], described in Algorithm 3. The high-level idea of selective backprop compute the backward pass only on the training examples with the highest loss. To construct such batches, it first computes the loss of every example in

---

Algorithm 3 Selective backprop [39]
Algorithm 4 RHO loss [66]
I: Input: iterations T. data y.)) loss L, mini-batch size Bm, mega-batch size Ba, num- her of inputs to compute loss CDF R, selectivity
1: Input: iterations train data Puin. validation
data Peal loss small model 0 trained on Duat mini-batch size Bm. mega-batch size Bar. learning rate v). model
8>0 O, model 0. loss history H =
for t to T do
2: for (2s, Ut] E Diain do
Bm < 0 [Initialize mini-batch.)
3: e x- [(r,, yi; 05) [Small model loss train)
4:
== (Select mega-batch.) Bu C {=)
4: for 1 to T do
5: Bas C (x) ) (Select mega-batch.)
{6))=1
L(5ar::) @ [Compute loss.)
6:
for j < to Bar do
6:
(4,) [41)==1 Bu +- L(BM 0) (Compute loss.)
7:
H - ( H)) (Add to history)
7:
R + te f3 j Bar
p CDF(C,; H). (Selection prob.)
8:
Rmp f- max (R) [top B. R)
9:
8 ~ Bernoulli(p) (Select or not.)
9:
Update(6: 00, Rwp) (Backwards pass.)
10:
ifs - then
11:
Bm < Bm U (x)
12:
if [m = Bm, then
13:
0 Update(0 (,, Bm) (Backwards pass.)
14:
Bm
6h
12h
24h
Val. Loss
GLUE
Baseline
2.42 + 0.00
2.20 + 0.00
2.10 # 0.01
Baseline
2.21
77.79 # 0.2
Selective backprop
2.73 0.18
2.44 = 0.06
2.18 0.02
Selective backprop
2.23
77.92 = 0.1
RHO loss
2.61 # 0.00
2.37 # 0.00
2.06
RHO loss
2.19
77.16 # 0.4
Table 3: Validation lasses, batch selection methods Table 4: Batch Selection For Free. Re-
(selective backprop, and RHO loss). Results are shown sults for 12-hour RST budget on C4, re- for RST budgets of 6, 12, and 24 hours, on C4.
moving all costs used to select batches.
uniformly-sampled mega-batch. It then samples high-loss subset of the mega-batch by ranking points based on their loss percentiles w.r.t. historical losses among recently-seen inputs.
5.2 RHO Loss
Mindermann al. [66] argue that solely prioritizing high training loss results in two types of examples that are unwanted: (i) mislabeled and ambiguous data, as commonly found in noisy, web-crawled data; and (ii) outliers, which are less likely appear at test time. The authors propose down-weighting such data via a selection objective called Reducible Holdout (RHO) loss, shown in Algorithm 4.
The authors acknowledge that their method comes with three overhead costs: (1) pre-training proxy model using holdout data, (2) one extra forward pass over the entire training set to store the proxy model's 's training losses, and (3) additional forward passes for the batch selection. In our evaluations we never count the costs of (1) and (2) because, as Mindermann et al. [66] point out, these costs can be amortized over many training runs. For (1), we pre-train model for 6 hours of RST on held-out data, which is enough to reach reasonable performances (Table 1).
Despite Mindermann et al. [66] motivating their method for scenario where additional workers are available to perform batch selection, we wondered it might still provide performance gains even this is not the case. Mindermann et al. [66] do not implement workers are used to select batches. Because this, we evaluate RHO loss under two protocols: the main results, we count the batch selection costs (3) against the training budget, while in Section 5.4, we provide a second set of results where we ignore the cost of batch selection.
evaluate algorithm where extra
5.3 Results
We assume that the effects of selecting better training batches should be largely agnostic to whether we pre-train a BERT or T5 model. Hence, instead of training both architectures, we decide to pre-train only BERT models and instead vary the datasets and budgets as follows.
For the first set of experiments, we fix the budget to 12 hours and consider three different pre-training datasets: (i) C4 [77], consisting only of webpage text which, despite being regularly used for pre-
7

---

Algorithm 5 Sophia [58]
Algorithm 6 Lion [13]
1: Input: 01, learning rate (m ) hyper- parameters {A, 32, p. k), and GNB- Estimator
1: Input: 01, learning rate {n=)=Z1 hyperpa-
rameters {), 31,
2: Set mg 0
2: Set min = 0, 00 = 0, - =
3: for t to T do
3: for I = 1 to 7 do
4: Compute minibach loss (()))..
4: Compute minibach loss (.)).)
5: Compute = (O.).
5: Compute Sr V (0,)
We sign (3 (1 B1 )tt)
6: m. - + (1 81 31)
7: my = 32 + (1 82)
7:
if mod
then Compute fą = (N:(
0++1 0. 4/
8:
9:
14 B214 + (1 B2)hi
10:
else
11:
h h4-1
12:
0,
= 02 Th nh; (weight decay)
13:
0411 0. Nt clip(m+/ max (1y, c).
training, known have quality issues [53], (ii) Bookcorpus and Wikipedia [22], which contain polished, book(-like) text and MiniPile [41]. a subset of the diverse Pile pre-training corpus [28], containing code, mathematics, books, webpages, and other scientific articles.
Validation loss. To rank the pre-training performances, we compare the validation loss, which is
directly comparable as we use the same inputs for all methods (which is not the case for the training data). This is shown in Figure 3, which shows the validation loss every hours throughout training. We find that both batch selection methods do not improve over the baseline.
Downstream performance. Next, investigate down- stream performances, fix the C4 corpus as the pre-training corpus, and vary the budgets (6. 12, and 24 hours). Fig- ures 2 and 16 show the results on SuperGLUE and GLUE. We observe very small differences between the methods and the baseline. On average (Table 1) no batch selection method significantly outperforms the baseline.
C4
WKBC
MP
3.5
3.0
2.5
2.0
123
123
12
5.4 Ablation: Batch Selection For Free
RST (hours)
Figure 3: Validation losses for differ- ent datasets. Results for batch selection methods (selective backprop and RHO loss) for 12-hour RST budget.
We want to disentangle whether batch selection methods fail improve over the baseline because their gains do not compensate for their computational overhead. In the pre- vious section (Section 5.3) and the main results (Table 1), we accounted for this overhead. Therefore, in those experiments, selective backprop and RHO loss effectively update the model for fewer iterations than the bascline within the fixed budgets. Here, we re-run small number of experiments where batch selection is free, and train these models with the same number of iterations as the bascline.
Specifically, we run experiment for 12 hours using the C4 corpus and GLUE downstream tasks. For selective backprop, we choose 1. which resulted in -1.7x of the wall-clock time; while for RHO loss, we choose a mega-batte batch size that is 10x larger (15360) than the mini-batch size (1536), following Mindermann al. [66], which led to ~5.3x of the original pre-training time. Table 4 shows the results. We see that RHO loss reaches slightly better final validation loss but performs worse on the GLUE downstream tasks than baseline and selective backprop, which does not improve over the bascline.

---

baseline
baseline
lion
lion
sophia
sophia
NLL
3
2.3
2
6 12
24
12 RST (hours)
RST (hours)
(a) BERT
(b) T5
Figure 5: Training losses, efficient optimizer methods (Lion, and Sophia). Results are shown for RST budgets of 6 hours (....., 12 hours (---), and 24 hours (-), C4.
6 Case Study 3: Efficient Optimizers
Recently, two efficient optimizers were proposed to replace the ubiquitous Adam(W) optimizers [48, 62] for language models: Lion [13] and Sophia [58].
Lion [13] is an optimizer symbolically discovered in the vast program space of first-order optimization primitives. As such, it does not follow any theory-grounded principle that would justify its acceleration property a priori; however, Chen et al. [13] report empirical speed-ups over Adam W in many settings.
Sophia [58] is scalable stochastic second-order optimizer primarily designed for and evaluated on language model pre-training The authors claim that Sophia achieves a 2x speed-up compared with AdamW in the number of steps, total compute, and wall-clock time. The authors study two Hessian estimators, but as this writing, only open-source the code for the empirically better one Wew---ewwto--ttlettt which
here.
The baseline in this section simply refers to AdamW [62].
Mixed Precision Training with BF16 vs. FP16 A common practice training language models is to use mixed precision training [65]. In initial experiments with BERT, we observed several numerical instabilities (NaN training losses) during papere-ar search after inserting Lion and Sophia into training pipelines as drop-in replacements. Our default mixed-precise FP16, and as noted by other practitioners of Sophia [57], BF16 can sometimes be more stable. Hence; for the optimizer comparisons with BERT, report results in BF16; including the baseline (although we notice that the baseline's 's training curves are essentially identical across both modes). For the T5 model, we use the TF32 precision format.
mode was
RMS-Scaling for T5 Pre-training T5 pre-training [77, 82] typ- ically employs the Adafactor optimizer [83], which relies on tensor- wise learning rate scaling determined by the tensor's root mean square (RMS). This scaling has been identified as critical for conver- gence during pre-training when using A [69]. Initial tests with Lion and Sophia without additional adjustments led to divergence for higher learning rates suboptimal performance. This mirrored the behavior of Adam W without RMS scaling. To address this, we incorporated RMS scaling into Lion and Sophia for subsequent round of experiments, which we outline in Appendix C.6.
Daseline
Sophia
MLM loss
5
10 15
20
RST (hours)
6.1 Results
Figure 4: BERT Validation loss when we do not count Sophia's Hessian update steps.
In the case of BERT downstream performances (Table 1), we find that Lion and Sophia perform about the same as baseline. Fig- ure 6 shows the results in more detail. We note that baseline (AdamW) consistently ranks the highest and has comparatively low standard deviation over random seeds.

---

Analogous to the batch selection for free ablation in Section 5.4, we also experiment with Sophia while not counting for Hessian update steps and running for the same number of iterations, as shown in Figure 4. Surprisingly, we still do not observe any speedup.
baseline
Lion
Sophia
100
80
80
60
6hrs
77
61us
58
40
45
25
100
80
80
60
12hrs
78
12hrs
58
40
65
25
55
100
80
80
65
24hr8
78
24hrs
58
40
65
25
50
MNLI
SST-2
STSB
RTE
QNLI
QQP
MRPC
CoLA
avg
CB
COPA
WiC
WSC
BoolQ
RTE
aNg
(a) GLUE
(b) SuperGLUE
Figure 6: BERT BF16 models evaluated on (Luper-)GLUE, The black vertical error bars indicate the standard deviation over three seeds. The black horizontal line shows the average performance of the baseline. For clarity, the individual tasks are plotted against the left-hand axis, while the average accuracy is plotted against the right-hand axis.
7 Conclusion
In this work, we closely examined efficient training algorithms which promised deliver training speed-ups. First, we clarify that quantifying a training speed-up without specifying an explicit training budget be misleading and that some previous works missed this. To normalize the wall clock time across different hardware, we introduce the reference system time measure. Then, we put three classes of algorithms the test in various pre-training settings of BERT and T5 models. We found only few settings where some of the considered algorithms improved over the baseline.
Acknowledgments
The authors would like to thank Edoardo Ponti for his feedback on an earlier version of this manuscript. JK and OK acknowledge support from the Engineering and Physical Sciences Research Council with grant number EP/S022566// OK was supported by G-Research. PN was supported by the UKRI Centre for Doctoral Training Natural Language Processing, funded by the UKRI (grant EP/S022481/1) and the University of Edinburgh, School of Informatics and School of Philosophy, Psychology Language Sciences. PM was partially funded by the European Union's 's Horizon 2020 research and innovation programme under grant agreement no. 875160. ELIAI (The Edinburgh Laboratory for Integrated Artificial Intelligence) EPSRC (grant no. EP/W002876/1), an industry grant from Cisco, and donation from Accenture LLP; and is grateful to NVIDIA for the GPU donations. This work was supported by the Edinburgh International Data Facility (EIDF) and the Innovation Programme at the University of Edinburgh.
10

---

References
[1] Abbas, A., Tirumala, K., Simig, D., Ganguli. S., and Morcos, S. Semdedup: Data-efficient learning web-scale through semantic deduplication. arXiv preprint arXiv:303..09400 2023.
[2] Ainslie, J., Lei, T., de Jong, M., Ontan' on, S., Brahma, S., Zemlyanskiy, Y., Uthus. D. C.. Guo, M., Lee-Thorp. J.. Tay, Y.. Sung, Y.-H., and Sanghai. S. K. Colt5; Faster long-range transformers with conditional computation. ArXiv, abs/2303.09752, 2023.
[3] Alabdulmohssn, I. M., Neyshabur, B., and Zhai, X. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300-223. 2022.
[4] Alain, G., Lamb, A., Sankar, C., Courville, C., and Bengio, Y. Variance reduction in SGD by distributed importance sampling. CoRR, abs/ 11155110061111 2015.
[5] Anil, R., Gupta, V., Koren, T., Regan, K., and Singer, Y. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.0900 2020.
[6] Arora, S., Cohen, N., and Hazan, E. On the optimization of deep networks: Implicit accel- eration by overparameterizatton. In Dy, J. and Krause, A. (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learn- ing Research, pp. 244-253. PMLR, 10-15 Jul 2018. URL https://proceeding mlr.
press/v80/ arora18a. html.
[7] Bachlechner, T., Majumder, B. P., Mao. H., Cottrell, G., and McAuley, J. Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pp. 1352-1361.
PMLR, 2021.
[8] Bartoldson, B. R., Kailkhura, B., and Blalock, Compute-efficient deep learning: Algorithmic trends and opportunities. Journal of Machine Learning Research, 24:1-77, 2023.
[9] Brock, A., Lim, T., Ritchie, J. M., and Weston, N. FreezeOut: Accelerate Training by Progressively Freezing Layers, June 2017. URL http://arxiiy http:///arxiv.org/abbb/ utXiv:1706.00m 04483 [cs, stat].
04983.
[10] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A.. Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[11] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I. and Amodei, D. Language models are few-shot learners. In NewrlPS, 2020.
[12] Bubeck, S., Chandrasckaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12712, 2023.
[13] Chen, X., Liang C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Lu- ong. T., Hsich, C.-J., et al. Symbolic discovery optimization algorithms. arXiv preprint arXiv:2302.0666 2023.
[14] Chowdhery, A.. Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P.. Chung, H. W. Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y. Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S. Dev, S., Michalewski, H.. Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito. D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowyez, A., Moreira, E., Child, R., Polazov, O., Lee, K., Zhou, Z.,, Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Hellst-e, K., Eck, D., Dean, J., Petrov, S., and Fiedel, Palm: Scaling language
modeling with pathways, 2022.

---

[15] Clark, C.. Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Ex- ploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.1000 10044, 2019.
[16] Coleman, C., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P., Liang, P., Leskovec, J., and Zaharia, M. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations, 2020. URL Hd-HJg260VYDr.
net/forum?
[17] Dahl, G. E., Schneider, F., Nado, Z., Agarwal, N., Sastry, C. S., Hennig, P., Medapati, S., Eschenhagen, R., Kasimbeg, P. Suo, D., et al. Benchmarking neural network training algorithms. arXiv preprint w3X1::2306.07199 2023.
[18] Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 16344-16359, 2022.
[19] Dasgupta, I., Lampinen, A. K., Chan, S. C., Creswell, A., Kumaran, D., McClelland, J. L.., and Hill, F. Language models show human-like content effects on reasoning. arXiv preprint MKKv:2207.0705// 2022.
[20] Dehghani, M., Tay, Y., Arnab, A., Beyer, L.,, and Vaswani, A. The efficiency misnomer. In International Conference on Learning Representations, 2022. URL tttt:p:///penrevieee http/forrmmiiiiin YMLR...R.
[21] Dettmers, T. and Zettlemoyer, L. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arxiv: 1907.04840, 2019.
[22] Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), pp. 4171-4186. Association for Computational Linguistics, 2019.
[23] Dozat, Incorporating nesterov momentum into adam. ICLR Workshop track, 2016.
[24] Evei, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-299222 PMLR, 2020.
[25] Fahlman. S. and Lebiere, C. The cascade-correlation learning architecture, Advances in neural information processing systems, 2, 1989.
[26] Fan, A., Grave, E., and Joulin, A. Reducing transformer depth on demand with structured dropout, 2019.
[27] Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01442, 2020.
[28] Gao, L.,, Biderman, S., Black, S., Golding, I.., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint 10Xiv:2101.00000
2020.
[29] Geiping, J. and Goldstein, T. Cramming: Training a Language Model on a Single GPU in One Day, December 2022. URL
org. /22p///221222222 14034. arXiv:2212.1400 14034 [cs].
[30] Gong, L., He, D., Li, Z., Qin, T., Wang, L., and Liu, T. Efficient Training of BERT by Progressively Stacking. In Proceedings of the 36th International Conference on Machine Learning, pp. 2337-2346. PMLR, May 2019. URL sttps://proceeednng. mlr, .press/ v97/gong19a html. ISSN: 2640-3498.
[31] Guo, C., Zhao, B., and Bai, Y. Deepcore: A comprehensive library for coreset selection in deep learning. In International Conference on Database and Expert Systems Applications, pp. 181-195. Springer, 2022.
[32] Guu, K., Lee, K., Tung. 7.,, Pasupat, P., and Chang, M.-W. Realm: Retrieval-augmented language model pre- pre-training 2020.
12

---

[33] He, H., Huang, G., and Yuan, Y. Asymmetric valleys: Beyond sharp and flat local minima. Advances in neural information processing systems, 32, 2019.
[34] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arxivv2203.
2022.
[35] Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q. Deep networks with stochas- tic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Purt IV 14, pp. 646-661. Springer, 2016.
[36] Irandoust, S., Durand, T., Rakhmangulova, Y., Zi, W., and Hajimirsadeghi, H. Training a Vision Transformer from scratch in less than 24 hours with GPU, November 2022. URL http:///arxi...ooraa//2/2////// 221pg/////////...../.g// 05187. arXiv:221 1102211.111877 [es].
[37] Izsak, P.,, Berchansky, M., and Levy, O. How to train bert with an academic budget. arXiv preprint arXiv:2104.07705, 2021.
[38] Izsak, P., Berchansky, M., and Levy, O. How to Train BERT with an Academic Budget,
September 2021. URL http::tttt////iee orgooogg/aa. httpg/bbs2104... 07705. arXiv:2104.07705 [es].
[39] Jiang, A. H., Wong, D. L.-K., Zhou, G., Andersen, D. G., Dean, J., Ganger, G. R., Joshi, G.. Kaminksy, M.. Kozuch, M., Lipton, Z. C., and Pillai, P. Accelerating Deep Learning by Focusing on the Biggest Losers, October 2019. http: /tappiv///xii...//// org/abs/////eeaaooeog.g.oeee /910. 1970.00662..... urXiv:191.007662 [cs, stat].
[40] Kaddour, J. Stop wasting my time! saving days of imagenet and bert training with latest weight averaging. arXiv preprint arXiv:2209. 14981, 2022.
[41] Kaddour, J. The minipile challenge for data-efficient language models. arXiv preprint arXiv:2304.08442, 2023.
[42] Kaddour, J., Liu, L.., Silva, R., and Kusner, M. J. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35:16577-16595, 2022.
[43] Kaddour, J.. Harris, J.. Mozes, M., Bradley, H., Raileanu, R., and McHardy, R. Challenges and Applications of Large Language Models, July 2023. URL http://axa// http///rxivv..rr/a///2...7 abs33770 10169.
[44] Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprins Setiv:2001.08366 2020.
[45] Katharopoulos, A. and Fleuret, F. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525-2534. PMLR, 2018.
[46] Kawaguchi, K. and Lu, H. Ordered sgd: new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics, pp. 669 679, PMLR, 2020.
[47] Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXiv:2104.14333 2021.
[48] Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
[49] Kleinberg, B., Li. Y., and Yuan. Y. An alternative view: When does sgd escape local minima? In International conference on machine learning, pp. 2698-2707. PMLR, 2018.
[50] Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big Transfer (BiT): General Visual Representation Learning, May 2020. URL http://arxiv. org/ abaaaass/111 1912. 11370. arXiv:1912.11370 [cs].
13

---

[51] Komatsuzaki, A. One epoch is all you need. arXiv preprint (eee::996.00600 2019.
[52] Kudo, T. and Richardson, J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint /vXiv:/80.006226 2018.
[53] Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burche C., and Carlini, N. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499) 2021.
[54] Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating Training Data Makes Language Models Better, March 2022. URLhttp: //arxiv.org..oo.ee.......ooooo www.g//////////oeoooeeee /2107. 06499. arXiv:2107. 06499 [es].
[55] Lengellé, R. and Denoeux, T. Training mips layer by layer using an objective function for internal representations. Neural Networks, 91)):83 97, 1996.
[56] Li, C., Zhuang, B., Wang, G., Liang, X., Chang, X., and Yang, Y. Automated Progressive Learning for Efficient Training of Vision Transformers. In 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 12476-12488 12486, New Orleans, LA, USA, June 2022. IEBE, ISBN 978-1-66888 16681-6-656-946-3. https://ieeexp.one ieee. org httgg//ocummet//8794222/
doi: 10. 100100999CPP3322220020222222
URL
[57] Liu, H. Does this work with 16-mixed precision. Sophia/issues/ 16, 2023. GitHub repository issue.
[58] Liu, H., Li, Z., Hall, D., Liang, P., and Ma, Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training, May 2023. URL http://arxiv, org org/abs/ 2305 14342. MxXi:2205511444 [cs, math].
[59] Liu, S. and Wang, Z. Ten lessons we have learned in the new" sparseland": A short handbook for sparse neural network researchers. arXiv preprint arXiv:2302.02596, 2023.
[60] Liu, Y., Agarwal, S., and Venkataraman, S. Autofreeze: Automatically freezing model blocks accelerate fine-tuning. CORR, abs/2102.01386, 2021.
[61] Longpre, S., Yauney, G., Reif, E., Lee, K., Roberts, A., Zoph, B., Zhou, D., Wei, J., Robinson, K., Mimno, D.. and Ippolito, D. A Pretrainer's Guide to Training Data: Mcasuring the Effects Data Age, Domain Coverage, Quality, Toxicity, May 2023. URL http: // //tp///ax/vvv/// org/ abs/2305. 13169. MtXiv:2305.1316 [cs].
[62] Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[63] Loshchilov, and Hutter, F. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL htt.s: //ppr//veee.....e..// bet/foorunidd=Skqgaaa
[64] Menghani, Efficient deep learning: survey on making deep learning models smaller, faster, and better. ACM Computing Surveys, 55(12):1-37, 2023.
[65] Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Hous- ton, M., Kuchaiev, O., Venkatesh, G., and Wu, H. Mixed precision training. In International Conference on Learning Representations, 2018. URL https: https://oopereviiet net/forum? id-rigs9JgRZ.
[66] Mindermann, S., Brauner, J. M., Razzak, M. T., Sharma, M., Kirsch, A., Xu, W., Höltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pp. 15630-15649, PMLR, 2022.
[67] Mocanu, D. C., Mocanu, E.,, Stone, P., Nguyen, P. H., Gibescu, M., and Liotta, A. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 91):2338, 2018.
14

---

[68] Narang, S., Chung, H. W., Tay, Y., Fedus, W., Fevry, T., Matena, M., Malkan, K., Fiedel, N., Shazeer, N., Lan, Z., al. Do transformer modifications transfer across implementations and applications? arXiv preprint arXiv: 1972, 2021.
[69] Nawrot, P. nanot5: A pytorch framework for pre-training and fine-tuning 15-style models with limited resources. ArXiv, abs/2309.02373. 2023.
[70] Nawrot, P., Tworkowski, S., Tyrolski, M., Kaiser, L., Wu, Y., Szegedy, C., and Michalewski, H. Hierarchical transformers are more efficient language models. ArXiv, abs/21 2021.
[71] Nawrot, P., Chorowski, J., La ncucki, A., and Ponti, E. Efficient transformers with dynamic token pooling, ArXiv, abs/221 11.0661,, 2022.
[72] Park, D., Papailiopoulos, D., and Lee, K. Active learning is strong baseline for data subset selection. In Has it Trained Yer? NeurIPS 2022 Workshop, 2022.
[73] Patterson, D., Gonzalez, J., Hölzle, U., Le, Q., Liang. C., Munguia, L.-M., Rothchild, D., So, D. R., Texier, M., and Dean, J. The carbon footprint of machine learning training will plateau, then shrink. Computer, 55(7):18-28 28,8 2022. doi: 10. 109/M..0022..3114.3
[74] Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning on data diet: Finding important examples early in training. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P. and Vaughan, J. W. (eds.), Advances in neural information processing systems, volume 34, pp. 20596 20607. Curran Associates, Inc., 2021. URL https://prooceedingg neureeeereinnnnnnnneesoeee neurips cc/ pdf.
paper pnper_fllee www.ffillsppppppe http:///0......o........23333332
[75] Portes, J., Blalock, D., Stephenson, C.,, and Frankle, Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates, November 2022. URL http://arxiv org/abs/ 2206. 00832. arXiv:2206.000 00832 [es].
[76] Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.114444 2021.
[77] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits transfer learning with a unified text-to-text transformer. J. Much. Learn. Res., 21:1400 21:140:1-1100:6 2020.
[78] Sandler, M., Zhmoginov, A., Vladymyrov, M., and Miller, N. Training trajectories, mini-batch losses and the curious role of the learning rate. arXiv preprint arXivv2301.02311. 2023.
[79] Sanyal, S., Kaddour, J., Kumar, A., and Sanghavi, S. Understanding the effectiveness of early weight averaging for training large language models. arXiv preprint 2rXiv:230660322/, 2023.
[80] Schmidt, R. M., Schneider, F., and Hennig, P. Descending through crowded valley- benchmarking deep learning optimizers. In Meila, M. and Zhang, T. (eds.), Proceedings the 38th International Conference Machine Learning, volume 139 of Proceedings
of Machine Learning Research, pp. 9367-9376. PMLR, 18-24 Jul 2021. URL https: /pppooeedings...../ mlr. www.....s................ wwwpppessssssse h1t9///chmit2221111 aetmiidtaaa.... html.
[81] Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O. Green ai, 2019.
[82] Shazeer, N. Glu variants improve transformer, 2020.
[83] Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost, 2018.
[84] Shazeer, N. M.. Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. E., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ArXiv, abs/1701.06533 2017.
[85] Shen, L.,, Sun, Y., Yu, 7., Ding, I.., Tian, X., and Tao, D. efficient training of large-scale deep learning models: A literature review. arXiv preprint arXiv:2304.0358 2023.
15

---

[86] Shen, S., Walsh. P., Keutzer, K., Dodge, J., Peters. M., and Beltagy. I. Staged Training for Transformer Language Models. Proceedings of the 39th International Conference on Machine Learning, pp. 19893-19908, PMLR. June 2022. URt.https:/proceeeee
mmlr.
press/v162/ shen22f. html. ISSN: 2640-3498.
[87] Smith, L. N. and Topin, N. Super-convergence: Very fast training of neural networks using large learning rates, 2018.
[88] Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, S. Beyond neural scaling laws: beating power law scaling via data pruning- arXiv preprint arXiv:2206. 14486, 2022.
[89] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, (5(1):192--1958 2014.
[90] Tay, Y. Dehghani, M., Abnar, S., Chung. H. W., Fedus, W., Rao, J., Narang, S., Tran, Q.. Yogatama, D., and Metzler, D. Scaling laws model architectures: How does inductive bias influence scaling?, 2022.
[91] Tay, Y., Dehghani, M., Bahri, D., and Metzler, D. Efficient transformers: survey. ACM Computing Surveys, 55(6):1-28, 2022.
[92] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023.
[93] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, t.,, and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[94] Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi- task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804 004444.7711.1.. 2018.
[95] Wang, A., Pruksachatkun, Y.. Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Superglue: A stickier benchmark for general-purpor purpose language understanding systems. Advances neural information processing systems, 32, 2019.
[96] Wang, P., Panda, R., Hennigen, L. T., Greengard, P., Karlinsky, L., Feris, R., Cox, D. D., Wang, Z., and Kim, Y. Learning to grow pretrained models for efficient transformer training. The Eleventh International Conference on Learning Representations, 2023. URL https: // //openreview
[97] Wang, X., Chen, Y., and Zhu, W. A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555-4576 2021.
[98] Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Arunkumar, A., Ashok, A., Dhanasekaran, A. S., Naík, A., Stap, D., Pathak, E., Karamanolakis, G., Lai, H. G., Purohit, I., Mondal, I., Anderson, J., Kuznia, K., Doshi, K., Patel, M., Pal, K. K., Moradshahi, M., Parmar, M., Purohit, M., Varshney, N., Kaza, R., Verma, P., Puri, R. S., Karia, R., Sampat, K., Doshi, S., Mishra, S., Reddy, S., Patro, S., Dixit, T., Shen, X., Baral, C., Choi, Y., Smith, N. A., Generalization via declarative Hajishirzi, H., and Khashabi, D. Super-naturalinstretion instructions 1600+ nlp tasks, 2022.
[99] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E.,, Le, Q., and Zhou, Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
[100] Wu, X., Dyer, E., and Neyshabur, B. When do curricula work?
arXiv preprint
arXiv:20/2.03 2020.
[101] Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, W. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining, May 2023. URL http:/ http//axxvvv.rrr org/abs/23055 10429. arXiv:2305.1044 [cs].
16

---

[102] Xie, S. M., Santurkar, S., Ma, T., and Liang, P. Data selection for language models via importance resampling, 2023, URL https://arxiv. http://aaxivv.ogaaaa////aa/ wtg.abs//302..oocm.o 03169.
[103] Xing, C., Arpit, D., Tsirigotis, C., and Bengio, A walk with sgd. arXiv preprint arXiv:1802.08777 2018.
[104] Xue, F., Fu, Y., Zhou, W., Zheng, Z.,, and You, Y. To Repeat Not To Repeat: Insights from Scaling LLM under Token-Crisis, May 2023. URLhttpi: /tpp://///xiv org/abs/////eoooooooo.o.ooe 2205. 13230. arXiv:2305.13230 [cs].
[105] Yao, X., Zheng, Y., Yang, X., and Yang. Z. NIp from scratch without large-scale pretraining: A simple and efficient framework. In International Conference on Machine Learning, pp. 25438-25451, PMLR, 2022.
[106] Zhang, C., Bengio, S., and Singer, Are all layers created equal? arXiv preprint aXXiv:190.001996, 2019.
[107] Zhang. M. and He, Y. Accelerating training of transformer-basee language models with progressive layer dropping. Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. Advancee in neural information processing systems. volume 33, pp. 14011-14023, Curran Associates, Inc., 2020. URL
ctpppperccppprrrrppppppppppp fpape_filess/ files/ pdf..
[108) Zhang, M. R., Lucas, J., Hinton, G., and Ba, Lookaheadd Optimizer: k steps forward, step back, December 2019. URL http: // arxia.iiivv...a. www.aaiwwv....aaaaaaaa stat].
08610. arXiv:1907.08610 [cs,
[109] Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, OPT: Open Pre-trained Transformer Language Models, June 2022. URL http:://axaii..oto// /ar//aaxvvv.rro/ org/ abs 2205. 01068. arXiv:2205.01066 [es].
[110] Zhu, C., Ni, R., Xu, Z., Kong, K., Huang, W. R., and Goldstein, T. Gradinit: Learning to initialize neural networks for stable and efficient training. Advances in Neural Information Processing Systems, 14:16410-16422 2021.
[111] Zhu, Y., Kiros, R., Zemel, S., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pp. 19-27. IEEE Computer Society, 2015.
[112] Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., and Shen, C. A survey on efficient training of transformers. arXiv preprint 23 107, 2023.
[113] Zhuang, J., Tang, T., Ding, Y., Tatikonda, S. C., Dvorek, N., Papademetris, X., and Dun- can, J. AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients. In Advances in Neural Information Processing Systems, volume 33, pp. 18795-18806. Curran Associates, Inc., 2020. URL https:// neurips cate.cccppaee httpps.ccppppp///ppppepppppppeee pc/ppaprr/020//aaaee
aad11995877522075111146611111111 htt.aaaaww.bbttttttaaattttttt.. html.
[114] zhuofan xia, Pan, X., Jin, X., He, Y., Xue', H., Song. S., and Huang, G. Budgeted training for vision transformer. In The Eleventh International Conference on Learning Representations, 2023. URL https: /ttp::////peeeiieeeee wttp:riiii.eeetftrroeeeeee nwtttfeuummmeee (at.rrummnid-VVBB--111111
17

---

preserved A Limitations and Future Work
Firstly, while we ran extensive ablations within the 6-, 12-, and 24-hour training regimes, is possible that results do not generalize to much longer ones. We justify this choice for two reasons. Firstly, all downstream task performances observed are only a few percentage points worse than state-of- the-art performances; e-g.. our 24-hour BERT model reaches -79% / 58.5% on GLUE//uppeCIL.EE while fine-tuning the original BERT-base model' reaches 80.9% / 60.8%, respectively. Similarly, 24h-T5 model reaches 39.5% SNI, while using the original checkpoint, reaches 41.0% [69]. Secondly, we argue that an efficient training method should work well specifically in settings with limited budgets.
Next, as illustrated in Appendix B, there is an abundance efficient training algorithms, and rigorously evaluating all of them is prohibitively expensive. Hence, one limitation of this work remains that we did not consider other efficiency-promotiie algorithms, and we hope to explore more in future work.
Another limitation is our sole focus on language model pre-training. Investigating approaches for (i) efficient fine-tuning of (large) language models or (ii) pre-training on other data-intensive modalities such as images and video remains promising too. We expect that our experimental protocol utilizing RSTs will benefit future works doing so.
B
Related Work
B.1
Efficient Training Algorithms
There is an abundance of proposals for efficient training: surveys of these can be found in [8, 85, 43]. We roughly categorize them into architecture-cemrric,, data-centric, optimization-centric, and others.
Architecture-centrie strategies. These decide how to avoid forward/backward passes of specific weights in the network. The idea of gradually growing network to accelerate training, in layer stacking, dates back to the 90s [25, 55]. There are a number follow-ups to the layer stacking paper [30], including automated stacking using elastic supemets [56] loss- and
stacking [86] and stacking small pre-trained models to train larger models [96].
Methods akin to layer dropping include FreezeOut [9], LayerDrop[26] (which focuses on network sparsity), and AutoFreeze [60]. In these methods, forward/hackward passes for specific layers are skipped based on either pre-determined schedule [9, 26] or based on statistics from prior forward/backward passes [60].
While not the focus of this work, similar motivation can also be found in dynamic sparse training methods, which aim identify relevant sub-networks during training and promote the prunability of the network [67, 21, 24]. However, these approaches do typically not lead to training speedups since unstructured sparsity is not GPU-friendly, which is why we do not consider them here [59].
Data-centric strategies. Besides batch selection methods, which subsample relevant data from mega-batch throughout training as discussed in Section 5, there are a number of other approaches aiming to either order or subsample training data. We did not consider them in our experiments since they either have a different motivation than training efficiency or are not directly suitable for drop-in modification of the training procedure. However, we briefly summarize three classes of such work. Firstly, one of the oldest classes of data-sclection strategies for training is curriculum learning (for a recent survey, see Wang et al. [97]). We do not consider curriculum learning here as it has already been extensively evaluated [100], and because it was initially motivated to improve generalization rather than achieve efficiency gains. Secondly, losk-specific retrieval [32, 105] use task specific data to retrieve similar data from large unsupervised corpus. Different from the task-agnostic batch selection methods we consider here, these methods are specifically designed to improve downstream task performance. Lastly, another line of work tries to subsample the entire dataset decoupled from the training procedure by various scoring heuristics [16, 31, 74, 54, 88, 1, 102, 101, 61], and we believe that further investigating such can be promising direction for future work.
"nttps://hugginngaaee www.erttttteeeeeee www-bbr--base-eee uncased
18

---

3.5
3.5
baseline
baseline
stacking (2.4, 7.2)
stacking (2.4, 7.2)
MLM loss
3.0
stacking (3, 7.2)
Span loss
3.0
stacking (3, 7.2)
stacking (7.2, 9.6)
stacking (7.2, 9.6)
2.5
2.5
2.0
2.0
0
12 RST (hours)
24
12
24
RST (hours)
(a) BERT
(b)
Figure 7: Layer stacking grid search: tune the intervals which the model is doubled. The notation "stacking (a, b)" signifies that the model's size was doubled once at a hours and then again at b hours, measured using RST.
Optimization-centric strategies. Lots of optimizers have been proposed with the goal of speeding up convergence [23, 108, 113]; yet, Schmidt al. [80] report that none of them consistently outper- forms Adam(W) [48, 62] when put to a rigorous test. Instead of modifying the training procedure, another line of work observed intermediate performance speedups by averaging weights along the training trajectory post-hoc training [40]: however, such gains arise primarily through effectively lowering the learning rates without directly intervening in the training process [78, 79], which is different the in this work considered methods which do intervene.
Others. These include ways improve the faster computation of Transformer building blocks [91, 18, 70], allocate compute conditioned on the input [84, 71, 2], network initialization [7, 110).
B.2 Efficient Training Meta-Studies
Budget-centric recipes. A different line of work investigates budget-centric training recipes, for example, for academic settings with multiple GPUs [38. 114], or hard time constraints such as training on single GPU for one day [36, 29, 69]. Our work adopts some of the recipes proposed by Geiping & Goldstein [29], Nawrot [69] and aims to complement them by investigate (other) speedup techniques.
Empirical Meta-Studies of Training Transformer-bastt models. Narang et [68] study Trans- former modifications across implementations and applications, finding that most do not meaningfully improve performances. Similarly, Tay et al. [90] study the scaling properties of various Transformer- based architectures (some of which are designed for efficiency), concluding that the original Trans- former proposed by Vaswani et al. [93] has the best scaling behavior. Dehghani et al. [20] discuss how common cost indicators of machine learning models have different pros and cons and how they can contradict each other. Further, they show how training time can be gamed. Our work is heavily inspired by Dehghani et al. [20] and aims complement it in two ways: (1) by proposing to normalize WCT across different systems (using RST) and (2) by a thorough experimental study in the case of pre- mre-trrining Transformer-basee language models.
Closest to our work the concurrent work by Dahl et al. [17], who exhaustively discuss various pitfalls of benchmarking neural network optimizers. Among other considerations, they propose to benchmark algorithms within a fixed runtime budget, a practice we agree with and use our work too. We believe work additionally complements theirs by other methods beyond optimizers.
19

---

3.5
10.0
baseline
40
baseline
40
Y 20
7/ 160
Yf 20
7/ 100
3.0
7.5
20
100*
2.5
5.0
2.0
2.5
12
8
24
RST (hours)
RST (hours)
(a) BERT
(b)
Figure 8: Grid search performed on the hyperparameter 7/ for layer dropping in a 24-hour budget
setting. For the T5 model with layer dropping we also test smaller learning rate (1e 2), because observe instabilities with the original one (2e 2). We mark runs with learning rate 2e
with *. The upper plots depict the training loss for each method, while the lower plots showcase the average number of active layers.
C Hyper-Parameter Search
C.1 Layer stacking: When To Stack
Figure illustrates that layer stacking has relatively low sensitivity to different stacking RST hour times, namely ((2.4, 7.2), (3, 7.2), 9.6)). with [(2.4, 7.2), , (3, 7.2)} yielding similar perfor- mance and (7.2, 9.6) slightly underperforming. For our experiments in Section 4.3, for both BERT stacking step ratio proposed by Gong et al. [30]. training steps
and T5, we choose (3, 7.2), as it maintains the same
C.2 Layer dropping: How to Drop
In Figure 8, we observe that different choices of yy yield comparable performance for both the BERT and T5 models. However, the layer dropping training curves for the T5 model notably spikier than those the baseline. suggesting that the layer dropping method demonstrates less stability during training with this architecture. As result, also tested smaller learning rate (le 2), in contrast to the original training rate (2e 2), which ultimately yielded better results. The parameters selected for our experiments in Section 4.3 were Yf 20, Ir le 2 for T5 and Yf 100 for BERT.
C.3 Selective backprop: Selectivity Scale
We tune the selectivity scale 3, where 3 E {1. 2, Jiang et al. [39] use 33% and 50% selectivity in their experiments, which approximately corresponds 3 {1, 2). respectively. We find that the larger the 3 value, the worse the pre-training performance. Note that the higher the / value, the more forward passes selective backprop needs to perform in order to collect enough samples for backward pass, which decreases the total number of parameter update steps within the RST budget. For the experiments in Section 5.3, we chose 1, as it consistently achieves best performance.
C.4 RHO loss: Mega-batch Size
RHO loss requires one additional hyper-parameter the size of the mega-batch, from which the mini-batch then gets subsampled from. We tune this hyper-parameter in Figure 10a and similar to Appendix C.3, we find that the larger this size gets, the worse the validation loss. We started tuning based on BCWK, and given the clear hierarchy we observed; we decided not to tune it on other datasets and simply set it 2x (3072).
Another implicit set of follow suggestions by Mindermann et al. [66] and choose the same architecture as the target model.
is how to pre-train the proxy/irreuuibbl loss models. Here,
20

---

4.0
4.0
4.0
baseline
0 = 1
3.5
3.5
3.5
0 2
NLL
3.0
TIN
3.0
I.LL.
3.0
B = 3
2.5
2.5
2.5
2.0
2.0
2.0
9 RST (hours)
6
12
9 RST (hours)
12
3
12
RST (hours)
(a) Bookeorpus Wikipedia
(b)
(c) MiniPile
Figure 9: Selective backprop grid search: we tune the 3 hyperparameter. Each plot shows the validation loss over time during training for the given dataset.
4.0
3072
4608
3.5
BCWK
6144
3.00
C4
15360
3.0
2.75
MP
2.5
2.50
2.25
2.0
3
9
12
2.00
RST (hours)
3
6
12
RST (hours)
(a) RHO loss grid search: we tune the mega batch size
BCWK multiples of the mini-batch size (1536): (2x (3072), 3x (4608),
Validation losses for training the RHO loss proxy/holdout model across three datasets.
(6144), (15360)).
We split all datasets into 20% proxy model pre-training, 1% proxy model validation, and -99 target model pre-training (later, during target model pre-training, further split the remaining 79% into train and validation set). For BCWK and C4, we train for 12 hours; for MP, we train for hours only since it is much smaller and want to avoid over-fitting Other than varying the dataset splits and training budgets, we use the same pprrmmprraeeters from Section 3.
Figure 10b shows the validation loss during proxy model pre-training; none of the models over-fit, and all of them reach reasonable losses. For reference, one may compare their values with the baseline in Figure 9, which shows the validation losses using data from the same dataset sources.
C.5
Lion: Learning Rate (LR) And Weight Decay (WD)
The authors of Lion provide the following guidelines [13, page 14]:
"The default values for Br and Bg in AdamW are set as 0.9 and 0.999, respectively, with an of le 8, while in Lion, the default values for 31 and 32 are discovered through the pro- gram search process and set as 0.9 and 0.99, respectively."
We adopt these default 31, hyper- parametrrss Next, we look LR and WD (also from [13)).
21

---

Architecture
LR
WD
BERT-Base-like
{ [ee-4, 3c-4, 5c-4, 7c-4)
{0.03, 0.05, 0.07, 0.1}
T5-Base
(5e-4, 7.5c-4, le-3, 2.5e-3, 5c-3, 2e-2]
{0.0)
Table 5: Grid Search Space for Lion.
3.0
baseline
wd=0.08
baseline
Ir=1e-4
wd=0.05
1r=2e-2
1r=3e-4
wd=0.07
1r Je-3
MLM loss
Ir= 5e-4
wd=0.1
MLM loss
2.5e-
7e-4
le-3
Ir 7.50-4
Ir 50-4
3
2
1.9
5
10
15
20
12 RST (hours)
24
hours (RST)
(a) BERT
(b) T5
Figure 11: Lion grid search for both the BERT and T5 model. Training loss over time during pre-training.
"Based on our experience, suitable learning rate for Lion typically 3-10x smaller than that for AdamW. Note that the initial value, peak value, and end value of the learning rate should be changed simul- taneously with the same ratio compared to AdamW. We do not mod- ify other training settings such as the learning rate schedule, gra- dient and update clipping. Since the effective weight decay is *x; update +=w X; update *= lr, the value of X used for Lion is 3-10x larger than that for AdamW W in order to maintain a similar strength."
To recap (details in Section 3), for AdamW, W. BERT, we use a WD of 0.01, while we disable it for T5 respectively.
base LRs of le-3 and 0.02, respectively. For
Hence, following the above guidelines, we define the grid search space as described in Table 5.
For BERT, determine L.R of 7e-4 and weight decay of 0.1 be best, illustrated in Figure a For T5, we do not use any weight decay (following Raffel et al. [77], Shazeer [82]) and find an LR of 7.5e-4 yield the best performance, as shown in Figure llb. For all optimizers, we follow Nawrot [69] to integrate RMS-LR scaling to facilitate convergence.
C.6 Sophia: Learning Rate (LR), Weight Decay (WD) and
The official code repository" suggests the following:
22

---

3.5
1900
baseline
000.01
Weights 1.2 Norm
0
MLM loss
0=0.5
p=0.1
2.5
200
5.0
7.5
10.0
5.0
10.0
RST (hours)
RST (hours)
Figure 12: Inserting Sophia as Drop-in Replacement (FP16) for BERT resulted in NaN Losses.
4.5
1050
baseline
Weights Norm
p=0.01
MLM loss
0.03
000.5
p=0.1
2.0
250
10
20
10
20
RST (hours)
RST (hours)
Figure 13: Lowering LR to le-4 Sophia for BERT Slows Down Convergence (FP16).
"Choose Ir to be about the same the learning rate that you would use for AdamW. W. Some partial ongoing results indicate that Ir can be made even larger, possibly leading
faster convergence. Consider choosing in [0.01, 0.1]. p seems transferable across different model sizes. We choose rho 0.03 in 125M SophiaG. The (Ir, rho) for 355M, Sophia- G is chosen to be (5e 4, 0.05) (more aggressive and therefore, even faster!) Slightly increasing weight decay to 0.2 seems also helpful."
First, we followed the above guidelines, replaced AdamW with Sophia, and simply tuned p (0.01, 0.03. 0.05, 0.13. All of these runs led to NaN losses, which we illustrate in Figure 12. Next, we lowered the learning rate le-4, which mitigated the instabilities but resulted much slower convergence, shown in Figure 13.
We then decided to switch from FP16 to BF16, which improved training stability. Further, manually tuned the L.R, WD, and p values, as shown in Figure 14. Since we noticed a strong negative correlation between P and the training loss, we decided stick p 0.01. The best performance was achieved with an LR of 4e-4 and a WD of 0.015. We follow Nawrot [69] to integrate RMS-LR scaling facilitate convergence, for all optimizers.
For T5, we vary the learning rate within the range of {2e- 2, 5e 3, 2.5c 3, 1,l1,,,.55e -,5 5e -4}, and in {5e - 2, 1e 2}. Figure 15 show the results; similar to BERT, observe better performance with a smaller The best performance comes with p = 1e 2 and LR 1e-3.
We acknowledge that there are additional typer-parameters like and A that we did not tune because we followed the author's suggestions. Future work may investigate their effects on the training speed-ups too.
23

---

3
baseline
baseline
rho:le-2 Ir:4e-4
rho:le-2 Ir:le-4
rho:le-2 Ir:5e-4
rho:le-2 Ir:2e-4
MLMM loss
rho:1e-2 Ir:1e-3
MLM loss
rho:1e-2 Ir:3-4
rho:3e-2 Ir:5e-4
rho:1e-2 Ir:de-4
rho:3e-2 Ir:lo-3
rho:le-2 Ir:50-4
2
2
10
20
10
20
hours (RST)
hours (RST)
(a) Weight decay
(b) Weight decay 0.15
Figure 14: Sophia grid search for BERT (BF16). Training loss over time during pre-training.
baseline
baseline
1=22e-2
12=2
Ir=5e-3
1r=5e-3
MLM loss
5
1--3.
MLM loss
1r=2.5e-3
Ir= le-3
Ir= le-3
Ir= 7.5e-4
Ir= 7.5e-4
5e-4
Ir 50-4
2
12
24
RST (hours)
RST (hours)
(a) Rho = le-2
(b) Rho Se-2
Figure 15: Sophia grid search for the T5 model. Training loss over time during pre-training.
D Additional Results
D.1 Training BERT, Evaluating on GLUE
We notice that on specific GLUE datasets, the efficiency methods have little effect performance (MNLI, SST-2, QNLI, QQP. MRPC). Across all datasets, CoLA appears to have the largest gains from efficient training. Averaged across all datasets, efficient training methods have little effect. We report the exact numbers in Table (left) for layer stacking and layer dropping. In these experiments, layer dropping barely outperforms the baseline given 6-hour RST budget. For 12-hour budget, both stacking and layer dropping inch above the baseline, and stacking only produces more accurate results than the baseline and layer dropping for 24-hour budget. In study, we also compare the efficient methods on T5-base model evaluated on the SNI benchmark and report the results in Table 2. From our observations, layer stacking is particularly noteworthy, demonstrating superior performance a 6-hour training period, significantly outperforming the other methods. However, the training budget is increased to 12 hours, the gap between bascline and layer stacking starts to diminish. In a 24-hour training scenario, baseline exhibits a marginally better performance than layer stacking, highlighting the efficacy of the baseline method given sufficient training time. Notably, both baseline and layer stacking consistently outperform layer dropping across all the time budgets.
the performance
D.2 Layer dropping small datasets
The layer dropping paper [107] trains on the Bookcorpus+ Wikkpedia dataset epochs, while the original BERT paper trains for only 40 epochs [22]. They do not report results for the baseline trained

---

baseline
slacking
layer drop
selective backprop
RHOLoss
100
öhrs
77
40
70
100
80
12hrs
78
40
70
100
85
24hrs
78
40
75
MNLI SST-2 STSB
RTE
QNLI
QQP MRPC
CoLA
avg
Figure 16: BERT models evaluated on GLUE. The black vertical eπror bars indicate the standard deviation over three seeds. The black horizontal line shows the average performance of the baseline. For clarity, the individual tasks are plotted against the left-hand axis, while the average accuracy is plotted against the right-hand axis.
3.0
Forwards (training)
0.20
Forwards (selecting)
Wall-clock Time (hours)
Backwards
2.5
0.15
0.10
2.0
0.05
1.5
train
val
0.00
Baseline
SB
5
20
(b) Motivation behind selective backprop (SB) [39].
RST (hours)
(a) Ablation for layer dropping, investigating its lack of performance. prevents overfitting when perform- ing multiple epochs over the datasct.
with a schedule based on fewer epochs. Given the possibility of overfitting due to the high number of epochs layer dropping's similarity dropout [89], and dropout's known efficacy for language model pre-training with repeated data [104], we raise the question of whether layer dropping acts as regularizer.
Note that given the abundance of pre- traiiraining data for language models, even the largest and most- expensively trained models [34, 14, 92] are typically trained within a one-epoch regime, which has been shown to improve performance over training for multiple epochs on smaller dataset [51, 77]. Hence, we exhaust the compute budget before passing through all available training data more than once. In contrast, Zhang He [107] use smaller dataset and thus complete 186 epochs during training. Thus, their training setup likely corresponds to overfitting regime.
25

---

To investigate this, we repeat the experiment with a truncated training dataset, resulting in the experiment performing roughly 180 epochs. The result is shown in Appendix D.1. The plot confirms our suspicion: when the baseline training procedure overfits, layer dropping can help mitigate this, preventing the validation loss from increasing as training time increases.
26


Reviews:
[{"review_id": "4ZwojmqhCZ", "reviewer": "Reviewer_b1c1", "summary": "The paper presents an analysis of 3 algorithms for training transformer models with a focus on efficiency. The authors present a way to measure wall clock time irrespective of the underlying hardware and make a principled comparison of the 3 algorithms by predefining a compute budget and adapting each algorithm and training recipe for the available time. The results of the paper show that none of the methods is universally an improvement over standard training.", "questions": "My main reservation for the paper is whether the experiments are enough to prove useful to the community.", "limitations": "The authors properly present and analyze the limitations of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Meta-analyses are very common in other fields but are sorely lacking in ML/Deep learning so the goal of this work is very welcome and in my humble opinion useful to the community.\n- Setting a compute budget and trying to optimize each algorithm accordingly is not as common as it should be in similar works.", "weaknesses": "- The main weakness of the work is possibly the choice of algorithms to be analyzed. There are only 3 while there could be a plethora of others as mentioned in the paper's related work. Moreover, the 3 methods evaluated are not particularly widespread (possibly because they don't work as well as the paper shows).\n- Another possible weakness is the fact that there is separate hyper parameter tuning per method irrespective of the compute budget. This means that the results could be widely different were one to apply the above methods to a new problem where the optimal or near-optimal hyper parameters are not known a priori.\n- RST although fair, it is not a transferable metric which means that a different paper will not be able to compare with the RST results of this paper and will have to re-run the experiments to compute the relative speed of the methods on their hardware. This severely diminishes its usefulness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n> _The main weakness of the work is possibly the choice of algorithms to be analyzed. There are only 3 while there could be a plethora of others as mentioned in the paper's related work. Moreover, the 3 methods evaluated are not particularly widespread (possibly because they don't work as well as the paper shows)._\n\nThank you for pointing this out. Based on this feedback, we have added three more recent and popular efficient training algorithms from this plethora of other methods mentioned in the related work section. To better organize all methods we have separated them into three different types: (1) **dynamic architectures** (i.e., the existing [layer stacking](http://proceedings.mlr.press/v97/gong19a/gong19a.pdf)/[dropping](https://proceedings.neurips.cc/paper_files/paper/2020/file/a1140a3d0df1c81e24ae954d935e8926-Paper.pdf) methods), (2) **batch selection**, where we add RHO-Loss [(Mindermann et al., 2022)](https://proceedings.mlr.press/v162/mindermann22a/mindermann22a.pdf) (and the existing [selective backprop](https://arxiv.org/pdf/1910.00762.pdf) method) , and (3) **efficient optimizers**, where we add recent optimizers Lion [(Chen et al., 2023)](https://arxiv.org/pdf/2302.06675.pdf) and Sophia [(Liu et al., 2023)](https://arxiv.org/pdf/2305.14342.pdf). The three new methods are very popular, e.g., their GitHub repositories have been highly starred: [1600 for Lion](https://github.com/lucidrains/lion-pytorch), [700 for Sophia](https://github.com/Liuhong99/Sophia), and [161 for Rho-Loss](https://github.com/OATML/RHO-Loss), and have been released recently (Sophia was released on May 25th, 2023; Lion on February 15th, 2023; RHO-Loss on June 16th, 2022).\n\nPlease let us know if there are other methods you would like to see tested.\n\n> _Another possible weakness is the fact that there is separate hyper parameter tuning per method irrespective of the compute budget. This means that the results could be widely different were one to apply the above methods to a new problem where the optimal or near-optimal hyper parameters are not known a priori._\n\nThanks for this. We agree that placing a budget on hyperparameter tuning is important. However, if we also placed a budget here, one could argue that the reason the baseline outperforms efficient training algorithms is because the baseline has fewer hyperparameters to tune, and so better hyperparameters can be found compared to other methods under this fixed budget. To avoid this critique, we fully tuned all efficient training algorithms, making them as strong as possible. We expect that in new problems with a fixed budget on training and hyperparameter tuning, the performance of these efficient training algorithms will degrade even further w.r.t. the baseline. Thank you again for bringing this up, we will clarify this point in the final version.\n> _RST although fair, it is not a transferable metric which means that a different paper will not be able to compare with the RST results of this paper and will have to re-run the experiments to compute the relative speed of the methods on their hardware._\n \nWe believe there is a slight confusion here. We saved all timings on the reference system for the model architectures, batches, sequence lengths, and other configurations used in our experiments. We will release these timings, allowing other researchers to run experiments with the same RST budget as computed against this reference system. This will prevent them from having to rerun any experiments in our paper. Sorry that this was not clearer, we will clarify this in the final version. "}, {"review_id": "3SkVWjndCw", "reviewer": "Reviewer_DsDW", "summary": "This paper studies three efficient pertaining techniques for Transformer models (layer stacking, layer dropping, and selective backpropagation). At variance with previous works on the subject, the authors adequately control for the learning rate schedule by evaluating performance at fixed compute budgets (as defined through a scaled wall-clock time). In doing so, they find that most of these methods provide unclear gain over a vanilla baseline -- and this finding holds across T5/BERT-like models. ", "questions": "Although exploring an interesting and valuable premise, this paper is insufficiently impactful and held back by its somewhat unclear and unprincipled methodology in the results section. I would recommend for the authors to: (1) unify their methodology across layer stacking/dropping/selective backpropagation and all models/datasets, to deliver a clearer results section; (2) improve the presentation of their results, with clearer figures (potentially plotting Pareto fronts as proposed in the main text); (3) get rid of the RST idea, which adds confusion for no reason. In its current form, I would rate this paper as a **Reject (3)**.\n\n**EDIT: following rebuttal I have updated my score to a Weak Accept (6)**.\n\n**Q1.** Based on the comments in W2 above on RST, could the authors clarify the value they see in RST? Especially regarding smoothing out fluctuations.\n\n**Q2.** Could the authors provide additional results for selective backpropagation at different compute budgets, and expand on the experiments combining SBP with different datasets?", "limitations": "The authors have included a limitations section. ", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* **S1.** At variance with previous works, the authors adequately control for the effect of the learning rate schedule. This is a common mistake in countless papers, which should be more discussed as it has previously impacted prominent work around scaling laws (see Hoffmann et al., 2022 as a response to Kaplan et al., 2020). \n\n* **S2.** This paper provides negative results on the timely topic of efficient pretraining of Transformers, allowing the community to take a step back and retrospectively analyse the validity of previous findings. Under that light, negative results can be as valuable as positive ones.\n\n* **S3.** (minor) The color scheme (which is consistent across text & plots) is a good idea to visually help readers. ", "weaknesses": "* **W1. Limited significance due to limited adoption of the practices evaluated.** The impact and significance of this work is limited, as it evaluates three methods which have not been widely adopted by the community. For a negative result paper to be truly valuable, it is better if it addresses a common practice rather than a result that has not gotten traction anyway. \n\n\n* **W2. It is unclear how the proposed Reference System Time approach improves upon other methods.** The authors bring-up RST as one of the significant contribution of the paper, and base their analysis on its usage. However, it's unclear how it differs in practice from simply using wall-clock time -- especially since the authors don't actually perform cross-hardware comparisons, which would be one of the benefit of formalising RST. \n   * **W2.1** The authors claim \"Unfortunately, WCT can fluctuate even on the same hardware, for instance, due to the usage of non-deterministic operations, hidden background processes, or inconsequential configurations, such as the clock rate\". It is not clear how RST improves upon this -- the first recording used to scale RST may be noisy as well, and so could the recordings being scaled. \n   * **W2.2** It is unclear whether Figure 1 is based on real measurements or is simply there for illustrative purpose. \n\n\n* **W3. The results are disparate, incomplete, and lack clarity.** The results are middling, lacking a clear finding (overall it seems layer dropping is worth it across all budgets for BERT, layer stacking may bring gains for longer training budgets on BERT but only on shorter budgets for T5, and that selective back propagation is never worth it). For a paper with negative results, clarity is key to improve upon past work.\n    * **W3.1.** Selective back propagation is compared in a completely different setup, on a single budget, and downstream performance is never measured for SBP+T5 -- only validation loss. Furthermore, while SBP+BERT are trained on 3 different datasets (a rather interesting study given the nature of SBP), only validation losses are provided for comparisons (no downstream task performance) and no conclusion is proposed on the influence of data source on SBP. The lack of a unified framework for comparisons between layer stacking/dropping and SBP make the paper feel more like a pot pourri of ideas rather than a principled & systematic comparison.\n    * **W3.2.** Reporting training loss instead of validation loss in Figure 2 is questionable: as the authors later discuss in their ablations in Figure 5.b), layer dropping acts similarly to drop out -- so it makes sense it is behind on training loss in Figure 2. Validation loss here would be far more valuable. \n    * **W3.3.** The figures are not clear: the x-axis in Figure 3 is not specified, the legends lack an analysis/mention of the key results for each figure, it's impossible to see differences in Figure 4, etc. Specifically to Figure 4/Table 1, the authors mention the concept of Pareto front multiple times in the main text; maybe plotting it would be far clearer than the figures/table proposed for downstream evaluation. \n\n\n* **W4. Insufficient clarity and lack of added educational value.** Negative results papers can prove themselves especially valuable if they help correct a bad practice in the community; here, a general lack of clarity and of adequate framing make this paper fall short of being an educational read.\n   * **W4.1.** The introduction of RST is confusing and frankly not necessary, as discussed in W2.\n   * **W4.2.** The issue around learning rate schedules could benefit from additional discussion & illustration. In particular, it would be interesting to mention that this is the crux of the difference between the scaling laws of Kaplan et al., 2020 and Hoffman et al., 2022. Not accounting for the influence of the LR schedule caused the Kaplan work to inappropriately recommend scaling model size primarily, instead of scaling model and data jointly. \n   * **W4.3.** The paper contains numerous imprecisions and overall lacks clarity (see W3.3. as well). The authors sometime abuse the citations by citing far too many work at once instead of the most relevant ones (l14, l32, l78, etc.). The default citation style of NeurIPS doesn't help here, making it difficult to identify the works at a glance... l144 it's unclear for instance what the sentence of curriculum and deduplication brings to the work -- since this never further discussed. Note also that Figure 6 is never referenced in the main text, and that \"Besides the dizziness due to conflicting ideas\" on l30 is not adequate language for a paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> _W1. Limited significance_\n\nThank you for pointing this out. Based on this feedback, we have added three more recent efficient training algorithms, see the global rebuttal response. \n\n> _W2. unclear how the proposed Reference System Time approach improves_\n\nWe believe there is a small confusion here. The difference between WCT and RST is that WCT does not account for changes in hardware and software configurations. RST does this by mapping all timings on an arbitrary system back to the reference system. We show an example of this in Figure 1, where we performed the same baseline training run across different hardware (a 3090 and an A100) and different software (different CUDA drivers on a 3090). And in fact, we ran all of our experiments across different machines (3090s and A100s), which enabled us to run such a large number of experiments (including hyer-parameter tuning, etc.). Sorry this was not clearer, we will make this clear in the final version. \n\n> _not clear how RST improves upon this_\n\nThanks for this. To ensure our RST measurement is not prone to fluctuation, we averaged these timings over 1000 iterations (for each model architecture, batch, sequence length, system configuration, etc.). We put this in a footnote, but we will move this to the main text to highlight this.\n\n> _W2.2 It is unclear whether Figure 1 is based on real measurements_\n\nFigure 1 is based on real measurements (the time of a baseline training run). We will make this explicit in the final version, thanks! \n\n> _W3. The results are disparate, incomplete, and lack clarity._ \n\nThank you for pointing that out. To improve clarity and incomplete results, we ran additional experiments and summarized our main findings; both can be found in the global response. \n\n> _W3.1. Selective back propagation is compared in a completely different setup_\n\nThanks. Based on this suggestion, we added validation and downstream performances for both SBP and Rho-Loss for all three considered budgets and a new downstream benchmark (SuperGLUE) to the new rebuttal PDF. We’ve also investigated the influence of different training data sources on both SBP and Rho-Loss in (Table 5 and Figure 4, rebuttal PDF). We opted to spend compute on this instead of on additional T5 experiments, as the effects here are large for batch selection methods. Across three training datasets, none of the batch selection methods outperform the validation loss or downstream performance of the baseline.\n\n> _The lack of a unified framework for comparisons between layer stacking/dropping and SBP make the paper feel more like a pot pourri of ideas._\n\nWe have now included downstream comparisons for all methods in the rebuttal PDF (in Tables 1 and 2, Figures 1 and 2). We compared SBP and Rho-Loss with the baseline using validation loss instead of training loss as these methods intentionally select high-loss training batches to improve generalization. Across three training datasets, neither SBP or Rho-Loss outperformed the validation loss or downstream performance of the baseline (Tables 3, 5; Fig. 4 in rebuttal PDF). If we don’t count the time required to select batches, RHO-Loss can slightly improve the validation loss (Table 4, rebuttal PDF).\n\n> _W3.2. Reporting training loss instead of validation loss in Figure 2 is questionable._\n\nThank you for bringing this up. We report training loss because we sample without replacement from our training dataset (C4), which is so large that we never see the same data-point twice throughout training. This means that the training loss is always computed on inputs that have not been seen before and so is an unbiased estimate of the generalization error, just like the validation loss. \n\nAlso, previously this figure reported the training loss computed while the layers were dropped. In the rebuttal PDF, we have included a new version of the figure where we compute the loss with all layers enabled. This does not qualitatively change the interpretation of the figure.\n\n > _W3.3. The figures are not clear._\n\nThank you! This should be the training budget. We have added this to the rebuttal PDF (Figure 4). This figure shows that the efficient batch selection algorithms do not outperform the baseline regardless of the training dataset.\n\n> Specifically to Figure 4/Table 1, the authors mention the concept of Pareto front multiple times in the main text; maybe plotting it would be far clearer than the figures/table proposed for downstream evaluation.\n\nWe believe there is a slight confusion here, these are in fact points on the Pareto curve of downstream performance vs. time (RST for 6, 12, 24 hours). The reason we opted for bar plots and tables is because all of the methods are essentially overlapping each other. \n\n> W4. Insufficient clarity and lack of added educational value.\n  \nBased on your prior comments, we have revised the manuscript to improve clarity (unfortunately, we cannot upload revised manuscripts in the rebuttal phase this year) and scope (additional experiments in rebuttal PDF). If you have any other recommendations, we would be happy to implement them.    \n\n> _W4.1. The introduction of RST is confusing_\n\nSee our response to W2 above. \n\n > _W4.2. The issue around learning rate schedules could benefit from additional discussion._\n\nThis is a very nice suggestion, we will investigate this further and add a discussion on this to the final version. \n\n> _W4.3. The paper contains numerous imprecisions_\n\nWe will revise our manuscript and (1) compress citations to the most relevant ones, (2) explain the background of sentence curriculum and deduplication in the pretraining dataset, (3) cite Figure 6 in the main text, and (4) we will rewrite the paragraph starting with \"Besides the dizziness due to conflicting ideas\". We hope this addresses your concerns, and we welcome any more suggestions on the writing.\n\n> Q1.\n\nYes, please see the response to W2 above.\n\n> Q2.\n\nYes, we have included these additional results in the rebuttal PDF. "}, {"review_id": "TgmiDpTXBc", "reviewer": "Reviewer_iTEM", "summary": "The paper reevaluates several training algorithms aimed at enhancing the efficiency of Transformer-based models, such as layer stacking, layer dropping, and selective backpropagation. The authors effectively manage the training resources by employing a metric called reference system time. However, the key result of the study indicates that these three efficient training algorithms yield only modest improvements compared to standard training methods.", "questions": "Do the results of this study depend on the size of the models? Can we anticipate that the effectiveness of these efficient training techniques would be more diminished or enhanced when training larger models with a greater compute budget?", "limitations": "In Section 6, the authors acknowledge the limitations of their work and highlight that it is not feasible to evaluate all types of training efficient algorithms due to the potentially exorbitant costs involved.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors conduct a thorough evaluation of efficient training algorithms, providing valuable insights into their effectiveness and highlighting the marginal gains such methods achieve. This revisit contributes to a deeper understanding of existing methods and offers a practical perspective on their applicability.\n- The authors point out the shortcomings of using wall-clock time for reference, and instead propose reference system time (RST) for a better estimate, which is valuable rule to highlight.", "weaknesses": "- While revisiting these methods undoubtedly holds value for the research community, it is important to note that the obtained results may be somewhat trivial and lack significant insights. The paper might be better suited for more specialized venues. \n- While the primary focus of the study lies in evaluating the original approaches, providing additional discussions on methods and evaluations that build upon these evaluated techniques would offer a more comprehensive understanding of how these methods have been adopted and evolved since their initial proposal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> While revisiting these methods undoubtedly holds value for the research community, it is important to note that the obtained results may be somewhat trivial and lack significant insights. The paper might be better suited for more specialized venues.\n\nThanks for this. We have added three more recent efficient training algorithms. Please see the global rebuttal response.\n\n> While the primary focus of the study lies in evaluating the original approaches, providing additional discussions on methods and evaluations that build upon these evaluated techniques would offer a more comprehensive understanding of how these methods have been adopted and evolved since their initial proposal.\n\nThanks for this, we will add a discussion on methods and evaluations that build on the efficient training algorithms we compare. On this note, the new methods we have added are motivated as building on the prior work we have compared here. Specifically, Sophia [(Liu et al., 2023)](https://arxiv.org/pdf/2305.14342.pdf) builds upon Lion [(Chen et al., 2023)](https://arxiv.org/pdf/2302.06675.pdf), as [Liu et al., 2023](https://arxiv.org/pdf/2305.14342.pdf) claim that Lion “only achieves limited speed-up on LLMs“. RHO-Loss [(Mindermann et al., 2022)](https://proceedings.mlr.press/v162/mindermann22a/mindermann22a.pdf) builds on [selective backprop](https://arxiv.org/pdf/1910.00762.pdf), as [Mindermann et al., 2022](https://proceedings.mlr.press/v162/mindermann22a/mindermann22a.pdf) argue that solely prioritizing high training loss results in two types of examples that are unwanted: (i) mislabeled and ambiguous data, as commonly found in noisy, web-crawled data; and (ii) outliers, which are less likely to appear at test time. Thank you for bringing this up, we will add these things in the final version.\n\n> Do the results of this study depend on the size of the models? Can we anticipate that the effectiveness of these efficient training techniques would be more diminished or enhanced when training larger models with a greater compute budget?\n\nGreat question! Our BERT model has 120M parameters and T5 has 247M, and across these model sizes the results appear to be consistent. We suspect the effect is similar for other LM sizes. That said, if there is another model or model size you would like to see, let us know, and we would be happy to include it. \n\n"}, {"review_id": "xnCjuQ74Tv", "reviewer": "Reviewer_VTeA", "summary": "Many algorithms have been proposed to make the training of ever larger models more efficient.\nThe authors present a critical empirical study of three selected training algorithms (layer stacking, layer dropping, and selective backpropagation) with fixed training budgets and find that these algorithms often do not make BERT and T5 pre-training significantly more efficient.", "questions": "I agree with the authors that \"As illustrated in Section 5, there is an abundance of efficient training algorithms, and rigorously evaluating all of them is prohibitively expensive.\" (Section 6). What would be an alternative approach to avoid such explosion of algorithms and experiments?", "limitations": "Yes, see the section \"Limitations and Future Work\".", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper re-visits evaluation standards, and offers a careful, rigorous, and clean re-evaluation of three efficient training algorithms. \nIt is an excellent example for the importance of also publishing \"negative\" results (no gains in metrics, no new algorithm).\nSpecifically, this finding can save development time in the implementation and maintenance of these algorithms and reduce the complexity of the pre-training algorithms.\n", "weaknesses": "I agree on the limitations pointed out by the authors in the section \"Limitations and Future Work\", including evaluation of small subset of efficient training algorithms only, language model pre-training only.\nOverall, the paper might be felt to be too simple and straight forward, the more as it addresses a well-known issue in a noisy field. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> I agree on the limitations pointed out by the authors in the section \"Limitations and Future Work\", including evaluation of small subset of efficient training algorithms only, language model pre-training only. Overall, the paper might be felt to be too simple and straight forward, the more as it addresses a well-known issue in a noisy field.\n\nThank you for this. Our motivation for evaluating a subset of efficient training algorithms on language models only was to allow us to spend more careful detail on an in-depth analysis of these methods. We believe the field of efficient training methods for language models will grow rapidly in the coming years, as the most successful models can currently only be trained by a handful of entities. \n\nAs far as we know, we haven’t seen any paper comparing the main approaches in this field. Our goal was to fill this gap to help researchers and practitioners understand which approaches deserve more attention and to spur development in these directions. We were surprised to discover that, in large part, current efficient training algorithms do not surpass learning-rate-decayed baseline training. We believe these observations could act as a turning-point for the efficient training field and prompt investigation into new directions entirely. Thank you for bringing this up; we will add more discussion on this in the final version.\n\n> I agree with the authors that \"As illustrated in Section 5, there is an abundance of efficient training algorithms, and rigorously evaluating all of them is prohibitively expensive.\" (Section 6). What would be an alternative approach to avoid such explosion of algorithms and experiments?\n\nThis is an interesting open question. The approach we took is to evaluate representative algorithms on popular architectures, using well-studied tasks. We believe our approach is particularly useful as the field continues to grow because, by using our published RST timings, it will allow future algorithms, architectures, and tasks to be evaluated against our results, regardless of the hardware setup.\n"}, {"review_id": "UAb3B2GH3Q", "reviewer": "Reviewer_BpVn", "summary": "This paper comprehensively evaluates three efficient training algorithms for transformer language models: layer stacking, layer dropping, and selective backpropagation. Such an evaluation is crucial due to the extensive computational resources needed for training transformer-based models. \n\nThe authors highlight the importance of specifying a training budget in performance comparisons. They introduce a measurement called \"reference system time\" (RST) to facilitate comparisons across different hardware configurations. The study examines these algorithms under various training budgets, model architectures, and datasets. The main finding is that these efficient training methods do not always significantly improve over the standard training baseline. Layer stacking tends to outperform the baseline and other methods in most instances. However, layer dropping sometimes falls short, while selective backpropagation often diminishes performance. \n\nThe paper also explores how these algorithms perform in a downstream task setting, showing that pre-training performance does not necessarily correlate with the ability to generalize in these tasks. It concludes by stressing the importance of caution when using these efficient training methods due to their potential overheads.  Further implications of this study can guide the choice of efficient training methods in transformer-based language models by considering the trade-off between improved performance and computational resources.", "questions": "1. It would be beneficial to understand how the proposed Reference System Time (RST) handle variations in software optimization levels for different algorithms, which might influence the perceived efficiency of other methods.\n\n2. For the RST measure, how much does the reference system time vary across different systems and configurations?\n\n3. The paper asserts that the algorithms provided 'little effect' or 'marginal improvements'. Can you clarify what you consider to be substantial or significant improvements in efficiency and why the results obtained didn't meet that criteria? \n\n4. In some comparison studies with baseline training, it's unclear whether the hyperparameters for the compared methods and the baselines were tuned independently. It would be beneficial if the authors clarified this aspect. Moreover, further details regarding the methodology behind establishing the \"budget-adjusted baseline\" would be appreciated.", "limitations": "The authors have clearly demonstrated some limitations in their paper, specifically highlighting that their study focuses on a select few efficient training algorithms and acknowledging that further exploration of additional algorithms is required. Furthermore, the authors admit that their review is primarily centered on language model pre-training and that the findings may not be necessarily applicable to fine-tuning or other data-intensive modalities. \n\nHowever, the authors could have expanded on potential negative societal impacts. For instance, if these inefficient training algorithms are utilized for large-scale projects, it could lead to unnecessary usage of computing resources, subsequently affecting energy consumption and carbon emissions. Additionally, the misrepresentation of algorithm efficiency could potentially mislead the academic and industrial communities in their pursuit of efficient models. \n\nTo improve the paper, it would be beneficial for the authors to provide more context on possible wider implications, particularly focusing on the ecological impact and potential misdirection of research efforts. They might also consider suggesting some directions for future work in improving the efficiency of training algorithms, both in terms of reducing computational costs and minimizing environmental impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Originality: The paper's focus on evaluating efficient training algorithms for transformer language models under a defined training budget is a novel perspective. The introduction of \"reference system time\" to compare training times across different hardware configurations shows ingenuity.\n\n2. Quality: The paper exhibits high quality in its methodology, experimental setup, and analysis. The experiments are thorough and well-conducted, involving different parameters like training budgets, model architectures, and datasets. The findings are meticulously analyzed, providing an honest assessment of the capabilities and limitations of the methods.\n\n3. Clarity: The paper is well-written and logically organized. The authors clearly explain the motivations behind their work, the reason for choosing specific efficient training algorithms, and their experimental methodology. They discuss their findings in an understandable and straightforward manner.\n\n4. Significance: The paper notably contributes to training language models. Its focus on efficient training methods and the introduction of RST offer practical solutions to the computational challenges researchers face. This research opens the door for more studies on efficient training of such models. The paper's findings and suggestions are invaluable for researchers and will aid in making informed decisions when implementing efficient training strategies.", "weaknesses": "1. Limited Scope: The evaluation in the paper is limited to three specific efficient training algorithms (layer stacking, layer dropping, and selective backpropagation).\n\n2. Limited Novelty: This paper mainly focuses on revisiting methods in a specific context. There are not many novel ideas or groundbreaking discoveries introduced in the paper. Studies expected to be published in top-tier conferences like NeurIPS should ideally contain significant novel contributions.\n\n3. Lack of Rigor in Experimental Design: While the authors propose using Reference System Time (RST), they do not sufficiently justify its superiority over other potential measures of computational effort. \n\n4. Overemphasis on Computational Efficiency: The authors focused on computational efficiency, sidelining the quality or performance of the models. This imbalance in focus can give a skewed perspective.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> _1. Limited Scope_\n\nThanks. Based on this feedback we have added three more recent efficient training algorithms. Please see the global response. \n\n> _2. Limited Novelty_\n\nThank you for bringing this up. We argue that the observation that none of the popular recently-proposed efficient training algorithms improve upon a learning-rate-decayed baseline is actually ground-breaking for the efficient training field. Papers that make observations like this have been hugely impactful, for instance \n\n* [Are GANs Created Equal? A Large-Scale Study](https://proceedings.neurips.cc/paper/2018/file/e46de7e1bcaaced9a54f1e9d0d2f800d-Paper.pdf) discovered that most GAN architectures reach similar scores with enough hyperparameter optimization and random restarts\n* [In Search of Lost Domain Generalization](https://openreview.net/pdf?id=lQdXeXDoWtI) finds that the trivial baseline (ERM) outperformed most DG algorithms when evaluated under the same experimental conditions \n* [You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings](https://openreview.net/forum?id=BkxSmlBFvr) discovered that old techniques are competitive and often even outperform recent model architectures when the hyper-parameters are tuned correctly \n* [Rethinking the Value of Network Pruning](https://arxiv.org/abs/1810.05270), found that fine-tuning pruned models obtained by state-of-the-art structured pruning algorithms only give comparable or worse performance than training that model with randomly initialized weights\n* [Pitfalls of Graph Neural Network Evaluation](https://arxiv.org/abs/1811.05868), uncovered serious shortcomings in existing evaluation strategies for GNNs\n\nWe actually started out trying to develop a new efficient training algorithm but then found that when we went to compare existing methods using a unified timing measure, reference system time (RST), none of the methods outperformed the simple baseline. Our shock at this discovery is what motivated us to publish this work. Thank you again for mentioning this, we will make this clearer in the final version.\n\n> _2. Lack of Rigor in Experimental Design_\n\nThanks. Let us clarify the benefits of RST over the other most popular measures of compute: WCT and FLOPs. WCT can vary widely across different hardware, and can even fluctuate on the same hardware, for instance, due to the usage of non-deterministic operations, hidden background processes, or inconsequential configurations, such as the clock rate. We show an example of this in Figure 1, where we performed the same baseline training run across different hardware (a 3090 and an A100) and the same hardware but different software (different CUDA drivers on a 3090). FLOPs do not account for parallelism (e.g., RNNs vs Transformers) or hardware-related details that can affect runtime. For example, FLOP counting ignores memory access times (e.g., due to different layouts of the data in memory) and communication overheads, among other things. \n\nTo address these things, RST ties all computation to a specific reference hardware. We average RST measurements over 1000 iterations (for each model architecture, batch, sequence length, system configuration, etc.), to ensure they are not influenced by fluctuations. In our code release, we will provide all RSTs measured by our hardware. This will enable researchers in the future to benchmark new algorithms using exactly our setup without owning the same hardware. Sorry this was not clearer, we will clarify this in the final version.\n\n> _3. Overemphasis on Computational Efficiency_\n\nWe added training, validation, and downstream performance results in the rebuttal PDF. If there are any additional quality or performance metrics you would like to see we would be happy to add them.\n\n> Q1. \n\nThanks for this question. We were curious about such fluctuations as well, to prevent them from influencing our results, we average RST measurements over 1000 iterations (for each algorithm, model architecture, batch, sequence length, system configuration, etc.). Thank you for bringing this up, we will clarify this in the final version.\n\n> Q2\n\nTo see how much the timing on a reference system would change if we selected different systems as the reference, we can look at the WCT of individual systems (imagining each was the reference system). This is shown in the top set of WCT bars in Figure 1. In theory, any of the three systems could be the reference system, the bottom set of RST bars show what happens when the last system (the 3090, shown in dark red) is chosen as the reference system, and all other systems are mapped to the timing of this system. This ensures that the same baseline run is timed identically, regardless of the system it was run on. \n\n> Q3\n\nThanks for this. By a `significant improvement’ we mean that, for a fixed RST budget, the average (training, validation, downstream) performance of an efficient training method must be greater than or equal to the average performance of the baseline plus a standard deviation (i.e., in Tables 1 and 2 we bolded the best method, and all methods whose average plus a standard deviation is greater than or equal to the best method). We will add these details to the final version.\n\n> Q4\n\nThanks for this. In all experiments, the hyperparameters for each method were tuned independently. We added this to the supplement, but we can move it to the main paper if you think this would help. We will also add more details on our budget-adjusted baseline: the idea is that we train the full model (unlike layer stacking/dropping) using standard data loaders (unlike selective backprop/Rho-loss) using Adam (instead of Lion or Sophia). Further, we use a one-cycle learning rate schedule and adjust the learning rate schedule based on the elapsed time conditional on a time budget (measured in RST). The implementation for this is simple and will be included in our code release. \n"}]

Author rebuttal:
We would like to thank all reviewers for their insightful and encouraging reviews: 
* **b1c1** - _“the goal of this work is very welcome and in my humble opinion useful to the community_”; 
* **DsDW** - “_allowing the community to take a step back and retrospectively analyse the validity of previous findings_”’; 
* **iTEM** - “_[it] contributes to a deeper understanding of existing methods and offers a practical perspective on their applicability_”; 
* **VTeA** - “_It is an excellent example for the importance of also publishing "negative" results (no gains in metrics, no new algorithm)._”; 
* **BpVn** - “_The paper's findings and suggestions are invaluable for researchers and will aid in making informed decisions when implementing efficient training strategies._”. 

In response to requests for comparing more methods (**b1c1, DsDW**), we added three more recent efficient training algorithms and one more benchmark ([SuperGLUE](https://arxiv.org/abs/1905.00537)). The results can be found in the attached rebuttal PDF.

To better categorize all approaches, we have separated the methods into three different types: (1) **dynamic architectures** (i.e., the existing [layer stacking](http://proceedings.mlr.press/v97/gong19a/gong19a.pdf)/[dropping](https://proceedings.neurips.cc/paper_files/paper/2020/file/a1140a3d0df1c81e24ae954d935e8926-Paper.pdf) methods), (2) **batch selection**, where we add RHO-Loss [(Mindermann et al., 2022)](https://proceedings.mlr.press/v162/mindermann22a/mindermann22a.pdf) (and the existing [selective backprop](https://arxiv.org/pdf/1910.00762.pdf) method), and (3) **efficient optimizers**, where we add recent optimizers Lion [(Chen et al., 2023)](https://arxiv.org/pdf/2302.06675.pdf) and Sophia [(Liu et al., 2023)](https://arxiv.org/pdf/2305.14342.pdf). The three new methods are very popular, e.g., their GitHub repositories have been highly starred: [1600 for Lion](https://github.com/lucidrains/lion-pytorch), [700 for Sophia](https://github.com/Liuhong99/Sophia), and [161 for Rho-Loss](https://github.com/OATML/RHO-Loss), and have been released recently (Sophia was released on May 25th, 2023; Lion on February 15th, 2023; RHO-Loss on June 16th, 2022). We have included a one-page PDF with new experimental results for all methods.

For the new optimizers, we noticed numerical instabilities when adding them as drop-in replacements into our training pipeline. Changing the mixed precision helped, which is why we report their results in BF16 instead of FP16. We also re-ran the baseline in that new precision to ensure direct comparability. 

Our main findings are:



* **Training loss** (comparing: Layer stacking, Layer dropping, Lion, Sophia (for **batch selection** approaches: selective backprop and RHO-Loss, we instead compare the validation loss, as they intentionally select training batches with high loss)): The only approach to consistently outperform the training loss of the fully-decayed learning rate baseline across budgets and models is Layer stacking (see Figure 1 in the rebuttal PDF). This improvement reduces as the budget increases to 24 hours.
* **Validation loss** (selective backprop, RHO-Loss): Across three training datasets, none of the **batch selection** methods outperform the validation loss of the baseline (Table 3 and Figure 4 in rebuttal PDF). If we don’t count the time required to select batches, RHO-Loss can slightly improve the validation loss (Table 4, rebuttal PDF). 
* **Downstream tasks**: For a 24-hour budget, none of the efficient training algorithms we evaluate improves the downstream performance of the baseline (Tables 1 and 2, Figures 2 and 3, rebuttal PDF).
* Methods with lower per-iteration costs than the baseline (i.e., **dynamic architecture** methods: Layer stacking, Layer dropping) can slightly improve downstream performance for lower budgets (6 hours, 12 hours), but the improvement disappears with longer training.
* Methods with higher per-iteration costs (i.e., **batch selection** methods: selective backprop and RHO-Loss, and some **efficient optimizer** methods: Sophia) are significantly worse than the baseline in some downstream tasks (GLUE, SNI), for all budgets.
* If we ignore the additional per-iteration computations of the three above methods, the downstream performance is still matched by the baseline.

Thank you for your time spent on reviewing our work. 


Meta-review: The submission compares the performance of recently proposed improvements to transformers. It aims to provide a fairer comparison than previous work by introducing the concept of 'reference system time, which attempts to allow results to transfer to different experimental hardware. Ultimately the paper finds that none of the chosen methods give large improvements over baselines when controlling for compute budget, and the reviewers agree that this is a useful negative result. The paper received borderline scores. The major concern raised in the original reviews, which I agree with, is that the experiments used techniques that are not widely used, which limits the impact of the findings. However, the paper has significantly expanded this in the revision, for example comparing with well publicised recent work like the Sophia optimizer. While the reviewers agree that the revision has significantly improved the paper, some chose to focus their scores on the original submission. Overall, while I am not surprised by the negative results, I think they are a useful reminded to the community to focus on fair comparison, and I recommend acceptance.
Decision: Accept (poster)