SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jean Kaddour $^{1*}$  Oscar Key $^{1*}$  Piotr Nawrot $^{2}$  Pasquale Minervini $^{2}$  Matt J. Kusner $^{1}$
<sup>1</sup>Centre for Artificial Intelligence, University College London
{jean.kaddour.20, oscar.key.20, m.kusner}@ucl.ac.uk
{piotr.nawrot, p.minervini}@ed.ac.uk
The computation necessary for training Transformer-based language models has skyrocketed in recent years. This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, layer dropping), batch selection (selective backprop, RHO loss), and efficient optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed computation budget using such methods, we find that their training, validation, and downstream gains vanish compared to a baseline with a fully-decayed learning rate. We define an evaluation protocol that enables computation to be done on arbitrary machines by mapping all computation time to a reference machine which we call reference system time. We discuss the limitations of our proposed protocol and release our code to encourage rigorous research in efficient training procedures: https://github.com/JeanKaddour/NoTrainNoGain.
Language models are advancing rapidly, surpassing human-level performances in some experimental setups [11, 47, 99, 19, 12]. These improvements are primarily due to model, data, and training budget scaling [44, 34, 3]. Training a single state-of-the-art language model requires hundreds of thousands of GPU hours, costs millions of dollars, and consumes as much energy as multiple average US family households per year [81, 73, 14, 92, 43].
To remedy this, there is a rapidly growing line of work on efficient training algorithms, which modify the training procedure to save computation [81, 8, 112, 85, 64, 43]. Specifically, three distinct classes of such algorithms are dynamic architectures (layer stacking [30] and layer dropping [107]) which ignore some of the weights during training, batch selection (selective backprop [39] and RHO loss [66]) which skip irrelevant data, and efficient optimizers (Lion [13] and Sophia [58]) which claim to convergence faster than Adam(W) [48, 62].
However, we find that the evaluation methodology is not standardized, and there are inconsistencies in the quantification of a "speed-up". Two general trends are: comparisons of (1) intermediate performances throughout training instead of final ones within a pre-defined training budget, and (2) incomplete speedup metrics, e.g., training progress as a function of the number of iterations (or epochs) instead of wall-clock computation times despite unequal per-iteration costs.
As pointed out by Dehghani et al. [20], Dahl et al. [17], this can be unfair to baselines, which were not tuned for efficiency within the same compute budget.
*Equal contribution, alphabetical order.
GLUE	SuperGLUE
6h	12h	24h	6h	12h	24h
Baseline	77.1 ± 0.2	77.8 ± 0.2	78.3 ± 1.1	57.8 ± 0.9	58.5 ± 1.1	58.6 ± 1.2
Layer stacking	76.8 ± 0.8	78.4 ± 0.4	79.4 ± 0.2	58.6 ± 1.0	57.9 ± 0.7	58.7 ± 2.0
Layer dropping	76.8 ± 0.3	78.1 ± 0.2	78.6 ± 0.1	58.6 ± 0.8	58.5 ± 0.7	58.4 ± 0.8
Selective backprop	75.3 ± 0.6	76.6 ± 0.4	77.9 ± 0.3	57.4 ± 0.4	59.1 ± 0.9	58.5 ± 0.4
RHO loss	75.7 ± 0.1	76.5 ± 1.3	77.8 ± 0.2	57.6 ± 1.1	57.8 ± 0.6	58.7 ± 0.7
Baseline (BF16)	77.0 ± 0.3	77.8 ± 0.2	77.9 ± 0.3	57.6 ± 0.5	57.9 ± 0.5	57.9 ± 0.6
Lion (BF16)	62.0 ± 13.7	72.0 ± 0.5	71.4 ± 0.8	56.1 ± 2.5	57.5 ± 0.2	57.2 ± 2.3
Sophia (BF16)	73.9 ± 1.3	71.1 ± 4.2	72.3 ± 3.8	58.0 ± 0.6	57.8 ± 0.7	57.5 ± 0.7
An example for (1) includes learning rate schedules, which can be arbitrarily stretched to improve the final performance while "sacrificing" the quality of intermediate checkpoints [103, 49, 87]. This can make comparisons of intermediate performances unfair to baselines. For (2), additional regularization techniques can improve the per-iteration convergence at higher per-iteration costs [5, 27, 42, 17], rendering a wall-clock time comparison more appropriate.
Table 1: Downstream performance, BERT. Results for efficient training methods on the GLUE and SuperGLUE dev sets for three budgets (6 hours, 12 hours, and 24 hours) after pre-training a crammed BERT model [22, 29]. We report average validation of GLUE and SuperGLUE scores across all tasks (standard deviations for three seeds). We use mixed precision training with BF16 for the optimizer comparison as we found FP16 precision lead to numerical instabilities (Section 6).
SNI
6h	12h	24h
Baseline	34.2 ± 0.2	38.3 ± 0.4	39.5 ± 0.1
Layer stacking	38.9 ± 0.2	39.2 ± 0.1	38.9 ± 0.2
Layer dropping	31.5 ± 0.3	34.4 ± 0.8	38.0 ± 0.5
Lion	20.8 ± 1.1	30.7 ± 0.4	33.7 ± 0.0
Sophia	26.7 ± 0.8	31.5 ± 0.8	34.1 ± 1.1
Table 2: Downstream performance, T5-Base [77, 69]. SNI [98] test ROUGE-L results (three seeds).
In this paper, we propose a simple evaluation
protocol for comparing speedups of efficient training algorithms. We use this protocol to evaluate these algorithms for pre-training Transformer-based language models from scratch. We compare different training budgets (6, 12, and 24 hours), model architectures (BERT-Base [22] and T5-Base [77]), and (for batch selection algorithms) datasets (C4 [77], Wikipedia and BookCorpus [111], and MiniPile [41]). To account for variability in measured wall-clock time on different hardware and software configurations, we propose a simple measure that we call reference system time (RST).
Our key findings are as follows:
- Training loss (layer stacking, layer dropping, Lion, Sophia): The only approach to consistently outperform the training loss of the fully-decayed learning rate baseline across budgets and models is layer stacking (Lion matches this performance for certain BERT training budgets). This improvement reduces as the budget increases to 24 hours.
- Validation loss (selective backprop, RHO loss): Across three training datasets, none of the batch selection methods outperform the validation loss of the baseline.
- **Downstream tasks:** For a 24-hour budget, none of the efficient training algorithms we evaluate improves the downstream performance of the baseline.
- Methods with lower per-iteration costs than the baseline (i.e., dynamic architecture methods: layer stacking, layer dropping) can slightly improve downstream performance for lower budgets (6 hours, 12 hours), but the improvement disappears with longer training.
- Methods with higher per-iteration costs (i.e., batch selection methods: selective backprop, RHO loss, and some efficient optimizer methods: Sophia) are significantly worse than the baseline in some downstream tasks (GLUE, SNI), for all budgets.
- If we ignore the additional per-iteration computations of the three above methods, the downstream performance is still matched by the baseline.
2 Comparing Efficient Training Algorithms
What is the fairest way to compare efficient training algorithms? Ideally, each method should be given an identical compute budget, and methods that converge to better solutions than a baseline within this budget achieve a speed-up. Below, we discuss the issues of commonly used metrics to specify the compute budget, and propose an improved metric.
The Pitfalls of Iterations Specifying a budget in number of training iterations is usually not suitable because the iteration time can vary between methods.
The Pitfalls of FLOPs While FLOPs count the number of elementary arithmetic operations, they do not account for parallelizability (e.g., RNNs vs Transformers) or hardware-related details that can affect runtime. For example, FLOP counting ignores memory access times (e.g., due to different layouts of the data in memory) and communication overheads, among other things [20, 8].
The Pitfalls of Wall-Clock Time WCT can fluctuate even on the same hardware, for instance, due to the usage of non-deterministic operations<sup>2</sup>, hidden background processes, or inconsequential configurations, such as the clock rate. Further, we would like a metric that allows researchers to run on shared compute clusters, where hardware configurations will vary.
Our Proposal: Reference System Time (RST) We propose a simple time measure that will allow us to standardize any timing result w.r.t. a reference hardware system (e.g., NVIDIA RTX 3090, CUDA runtime 11.8, PyTorch 2.0, etc.). We define the time elapsed on a reference system as reference system time (RST). To convert the training run of an arbitrary device to RST, we first record the time per training iteration on the reference training system. Then, we compute the RST by multiplying the number of iterations run on the arbitrary device by the time per iteration on the reference system. This way, the time is grounded in practical time units but can be applied to any system.
2.1 The Pitfall of Comparing Intermediate Performances
One difficulty with comparing efficient training methods stems from the importance of the learning rate schedule to model performance. Various works have demonstrated that deep learning models tend to reach their maximum performance only once the learning rate has been fully decayed [109, 76]. If the learning rate schedule is stretched, this can delay the number of iterations required for the model to reach a given performance threshold since the optimizer keeps "exploring" the local loss basin, wasting iterations oscillating [33, 42].
This leads to unfair comparisons when the learning rate schedule is not set fairly between methods; a non-obvious pitfall that has been discussed previously in the literature [50, 20], but which we still observed in several of the works we evaluated [30, 107, 39, 58]. Consider comparing a baseline B with an efficient training method X, where both methods are trained for the same number of iterations, and the learning rate is decayed in terms of the number of iterations. B and X reach a similar performance at the end of training, but because X completes the iterations in less WCT, a speed-up is declared.
However, this is not a fair comparison because the training budget was measured in iterations, but the performance of the models was evaluated at a point specified in WCT, specifically the time it took X to complete the budget. Thus, B is evaluated at an intermediate point during training when its learning rate schedule has not been fully decayed. If the learning rate schedule of B had been shortened so it finished at the same WCT as X, the two methods may have reached similar performances in the same WCT time. The same problem arises when the budget is specified using different units to the point of measurement, including the cases where FLOPs are used for the budget.
To avoid this issue, each method should be trained to a fixed budget with the performance measured at the end of this budget. Importantly, the learning rate and other schedules must be decayed as a function of the same unit used to measure the budget.
<sup>2</sup>https://pytorch.org/docs/stable/notes/randomness.html
<sup>3</sup>We average the time required per iteration across 1000 iterations given some model architecture, batch, sequence length, system configuration, etc.
Given this computation measure, we can now evaluate efficient training algorithms under fixed computation budgets. We conduct experiments using two established and widely used Transformer language models: BERT [22] and T5 [77]. We choose a single-GPU setting following recent works [29, 36, 69] to facilitate reproducibility and access for compute-restricted researchers. However, in principle, our protocol can be run on any hardware and straightforwardly used in a distributed setup.
BERT: Encoder-only. We follow the setup and hyperparameters of Geiping & Goldstein [29] with minor modifications. We pre-train a BERT-base-like model with 16 layers instead of 12, which consists of 120M parameters, using a masked language modeling (MLM) objective. We use the AdamW optimizer [62] with hyperparameters  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.98$ ,  $\epsilon = 10^{-12}$ , and weight decay of  $10^{-2}$  [62]. Further, we use a one-cycle learning rate schedule [87] with a peak learning rate of  $10^{-3}$  and gradient clipping of 0.5. The batch size is 1536 sequences, and the sequence length is 128. We fine-tune and evaluate the pre-trained BERT models using the GLUE [94] and SuperGLUE [95] benchmarks. For fine-tuning on GLUE, we use the hyper-parameters from Geiping & Goldstein [29]. On SuperGLUE, we tune them using the validation accuracy of BoolQ [15]. We found inconsistencies in the literature on how to aggregate the SuperGLUE scores; here, we average the following scores: for CB, we use the F1 score; for the rest (BoolQ, COPA, RTE, WiC, WSC), we use the accuracy.
T5: Encoder-Decoder. We pre-train a T5v1.1-Base [77, 82] model using the original span-corrupting MLM objective and SentencePiece [52] tokenizer. We follow Nawrot [69] and use the AdamW optimizer [62] with tensor-wise LR scaling by its root mean square (RMS), base learning rate 0.02, no weight decay, cosine schedule with final of  $10^{-5}$  [63], gradient clipping of 1.0, and  $10^{4}$  warm up steps. We use a batch size of 144 examples, where each example consists of an input of 512 tokens and an output of 114 tokens. We evaluate our pre-trained T5 models on the Super-Natural-Instructions [SNI, 98] benchmark. To fine-tune the model on SNI, we strictly
Algorithm 1 Layer stacking [30]
Input: number of layers  $L$ , number of stacking operations  $k$
Initialize model  $f_0'$  with  $L / 2^k$  layers  $f_0 \gets \text{Train}(f_0')$
for  $i\gets 0,\dots ,k - 1$  do
$$
\begin{array}{l} f _ {i} ^ {\prime} \leftarrow \left(f _ {i}, f _ {i}\right) \{S t a c k l a y e r s. \} \\ f _ {i} \gets \text {T r a i n} \left(f _ {i} ^ {\prime}\right) \\ \end{array}
$$
follow the hyperparameters of [69, 98]. For both pre-training and fine-tuning, we use TF32 precision.
Dataset. Unless specified otherwise, we use the C4 [77] dataset for less than one epoch (i.e., without data repetitions) without sentence curriculum and de-duplication [53, 29]. In Section 5, we additionally use two other datasets.
Learning-rate schedule. We adjust the learning rate schedule based on the elapsed time conditional on a time budget (measured in RST), similar to [37, 29], who measure raw WCT.
Hyper-parameter search. For all considered methods, we tune their hyper-parameters based on the pre-training loss. We list all details about each method's hyper-parameters, our considered grid search ranges, and the best values we found in Appendix C.
Reference System. We record the RST on two separate systems for BERT and T5. For BERT, we choose a single NVIDIA RTX 3090, CUDA 11.7, PyTorch 1.13. For T5, we choose an NVIDIA A100, CUDA runtime 11.8, PyTorch 2.0.
4 Case Study 1: Dynamic Architectures
4.1 Layer stacking
Layer stacking [30], as summarized in Algorithm 1, replicates a  $L$ -layer model into a  $2L$ -layer model by copying its parameters, effectively warm-starting the stacked model with parameters transferred from the smaller model. Thereby, it benefits from faster per-iteration times in the early training phases when using fewer layers. Gong et al. [30] attribute the success of this method to attention
(b) T5
Figure 1: Training losses, dynamic architecture methods (layer stacking, and layer dropping). Results are shown for RST budgets of 6 hours ( $\cdots$ ), 12 hours ( $\cdots$ ), and 24 hours ( $\cdots$ ), on C4.
distributions of bottom layers being very similar to attention distributions of top layers, indicating that their functionalities are similar.
4.2 Layer dropping
Layer dropping exploits the following observation: the layers of a network do not contribute equally to the loss reduction throughout training [35, 6, 106]. It does so by randomly choosing parts of the model to be skipped during each training step. Specifically, it replaces a subset of Transformer blocks with the identity function. As a result, it reduces the overall computational cost of the model for each training step because it skips these layers during the forward and backward passes.
To minimize the impact of layer dropping on the pretraining loss of the model, Zhang & He [107] employ a time and depth schedule that determines the probability of dropping each block. The time schedule begins with a zero probability of dropping each block. Then it increases this probability throughout the training process until it reaches a maximum of  $(1 - \bar{\alpha})$ , where the hyperparameter  $\bar{\alpha} = 0.5$  as chosen by Zhang & He [107].
The depth schedule ensures that blocks located earlier in the model are dropped with a lower probability than those located later/deeper. An important hyperparameter of the
Algorithm 2 Layer dropping [107]
1: Input: iterations  $T$ , layer keep probability  $\bar{\alpha}$ , temperature budget  $\gamma_f > 0$ , layers  $L$ , functions (self-attention, layer-norm, feed-forward)  $f_{ATTN}, f_{LN}, f_{FFN}$ , loss function  $\mathcal{L}$ , data  $(\mathbf{x}_0, \mathbf{y})$ , output layer  $f_O$   
2:  $\gamma \leftarrow \frac{\gamma_f}{T}$   
3: for  $t \gets 1$  to  $T$  do  
4:  $p \gets 1$  {Keep probability.}  
5:  $\alpha_t \gets (1 - \bar{\alpha}) \exp(-\gamma \cdot t) + \bar{\alpha}$   
6:  $p_d \gets \frac{1 - \alpha_t}{L}$  {Layer decay.}  
7: for  $i \gets 0$  to  $L - 1$  do  
8:  $s \sim \text{Bernoulli}(p)$  {Keep or drop.}  
9: if  $s == 0$  then  
10:  $\mathbf{x}_{i+1} \gets \mathbf{x}_i$  {Drop.}  
11: else  
12:  $\mathbf{x}_i' \gets \mathbf{x}_i + \frac{f_{ATTN}(f_{LN}(\mathbf{x}_i))}{p}$   
13:  $\mathbf{x}_{i+1} \gets \mathbf{x}_i' + \frac{f_{FFN}(f_{LN}(\mathbf{x}_i'))}{p}$   
14:  $p \gets p - p_d$  {Decay prob.}  
15:  $\ell \gets \mathcal{L}(f_O(\mathbf{x}_L), \mathbf{y})$   
16:  $f_{ATTN}, f_{LN}, f_{FFN}, f_O \gets \text{Update}(\ell)$
depth schedule in layer dropping is  $\gamma_{f}$ , which controls the rate at which the probability of dropping layers increases. A higher value of  $\gamma_{f}$  leads to a quicker increase in the probability. Zhang & He [107] set  $\gamma_{f}$  to 100 in their experiments. We refer the reader to the original work by Zhang & He [107] for more details.
Training losses. Figure 1 shows the pre-training losses for the baseline, layer stacking, and layer dropping on BERT and T5 when given a budget of 24 hours in RST. In both settings, layer stacking achieves the lowest loss within this budget. The gap between layer stacking and the baseline closes almost completely as the budget is increased to 24 hours. However, layer dropping is consistently worse than the baseline throughout training. In both models, the gap between layer dropping and the baseline is larger than between the baseline and layer stacking at the end of training. Further, layer dropping introduces additional fluctuations in the loss when training T5, compared to the baseline.
Downstream performance. Figure 2 shows the SuperGLUE downstream performance after using the baseline and all dynamic architecture methods (layer stacking and layer dropping) to pre-train
Figure 2: BERT models evaluated on SuperGLUE. The black vertical error bars indicate the standard deviation over three seeds. The black horizontal line shows the baseline average performance. For clarity, the individual tasks are plotted against the left-hand axis, while the average accuracy is plotted against the right-hand axis.
BERT. We evaluate all methods using three different RST budgets (to get a sense of the Pareto-frontier of budget and performance [75]): 6 hours, 12 hours, and 24 hours. We notice that on specific SuperGLUE datasets, the efficiency methods have little or even detrimental effect on performance (CB, WiC, WSC, BoolQ). Across all datasets, COPA appears to have the largest gains from efficient training. Averaged across all datasets, efficient training methods have little effect. We report the exact numbers in Table 1 (right). On average, across all budgets, neither layer stacking and layer dropping significantly improve over the baseline.
We also compare the performance of dynamic architecture methods on the T5-base model evaluated on the SNI benchmark and report the results in Table 2. Here layer stacking significantly improves over the baseline for 6 and 12 hour budgets. However, given 24 hours for training, the final accuracy of layer stacking reduces back to the accuracy of the 6 hour model. Here the baseline significantly improves over all methods. Notably, for all budgets, layer dropping is consistently outperformed by the baseline.
5 Case Study 2: Batch Selection
Scaling up the size of web-crawled data for pre-training has been one of the major drivers to improve the capabilities of language models [44, 10]. A line of work has argued that training speed-ups emerge through the selection of informative examples. They argue that these examples can be identified from certain statistics collected throughout training [4, 45, 39, 46, 72]. Here, we focus on two such methods that directly alter the training procedure to steer gradient computations towards informative samples by subsampling examples from a mega-batch to curate the mini-batch, called batch selection.
5.1 Selective Backprop
Due to its simplicity, we first examine selective backprop [39], described in Algorithm 3. The high-level idea of selective backprop is to compute the backward pass only on the training examples with the highest loss. To construct such batches, it first computes the loss of every example in a
Algorithm 3 Selective backprop [39]
1: Input: iterations  $T$ , data  $\{(\mathbf{x}_i,\mathbf{y}_i)\}_{i = 1}^N$ , loss  $\mathcal{L}$ , mini-batch size  $B_{m}$ , mega-batch size  $B_{M}$ , number of inputs to compute loss CDF  $R$ , selectivity  $\beta >0$ , model  $\theta$ , loss history  $\mathcal{H} = ()$   
2: for  $t\gets 1$  to  $T$  do  
3:  $\mathcal{B}_m\gets \{\}$  {Initialize mini-batch.}  
4:  $\mathcal{B}_M\subset \{(\mathbf{x}_i,\mathbf{y}_i)\}_{i = 1}^N$  {Select mega-batch.}  
5:  $\{\ell_j\}_{j = 1}^{B_M}\gets \mathcal{L}(\mathcal{B}_M;\pmb {\theta})$  {Compute loss.}  
6: for  $j\gets 1$  to  $B_{M}$  do  
7:  $\mathcal{H}\gets (\ell_j,\mathcal{H})$  {Add to history}  
8:  $p\gets \mathrm{CDF}(\ell_j;\mathcal{H}_{1,\dots,R})^\beta$  {Selection prob.}  
9:  $s\sim \mathrm{Bernoulli}(p)$  {Select or not.}  
10: if  $s == 1$  then  
11:  $\mathcal{B}_m\gets \mathcal{B}_m\cup (\mathbf{x}_j,\mathbf{y}_j)$   
12: if  $|\mathcal{B}_m| == B_m$  then  
13:  $\pmb{\theta}\gets \mathrm{Update}(\pmb {\theta},\mathcal{B}_m)$  {Backwards pass.}  
14:  $\mathcal{B}_m\gets \{\}$
Algorithm 4 RHO loss [66]
1: Input: iterations  $T$ , train data  $\mathcal{D}_{\mathrm{train}}$ , validation data  $\mathcal{D}_{\mathrm{val}}$ , loss  $\mathcal{L}$ , small model  $\theta^S$  trained on  $\mathcal{D}_{\mathrm{val}}$ , mini-batch size  $B_{m}$ , mega-batch size  $B_{M}$ , learning rate  $\eta$ , model  $\theta$   
2: for  $(x_i, y_i) \in \mathcal{D}_{\mathrm{train}}$  do  
3:  $\ell_i^S \gets \mathcal{L}(x_i, y_i; \theta^S)$  {Small model loss on train}  
4: for  $t \gets 1$  to  $T$  do  
5:  $\mathcal{B}_M \subset \{(\mathbf{x}_i, \mathbf{y}_i)\}_{i=1}^N$  {Select mega-batch.}  
6:  $\{\ell_j\}_{j=1}^{B_M} \gets \mathcal{L}(\mathcal{B}_M; \theta)$  {Compute loss.}  
7:  $\mathcal{R} \gets \{\ell_j - \ell_j^S \mid j \in \mathcal{B}_M\}$   
8:  $\mathcal{R}_{\mathrm{top}} \gets \max_{B_m} (\mathcal{R})$  {top  $B_m$  of  $\mathcal{R}$ }  
9:  $\theta \gets \text{Update}(\theta, \mathcal{R}_{\mathrm{top}})$  {Backwards pass.}
6h	12h	24h
Baseline	2.42 ± 0.00	2.20 ± 0.00	2.10 ± 0.01
Selective backprop	2.73 ± 0.18	2.44 ± 0.06	2.18 ± 0.02
RHO loss	2.61 ± 0.00	2.37 ± 0.00	2.16 ± 0.01
Val. Loss	GLUE
Baseline	2.21	77.79 ± 0.2
Selective backprop	2.23	77.92 ± 0.1
RHO loss	2.19	77.16 ± 0.4
Table 3: Validation losses, batch selection methods (selective backprop, and RHO loss). Results are shown for RST budgets of 6, 12, and 24 hours, on C4.
Table 4: Batch Selection For Free. Results for a 12-hour RST budget on C4, removing all costs used to select batches.
uniformly-sampled mega-batch. It then samples a high-loss subset of the mega-batch by ranking points based on their loss percentiles w.r.t. historical losses among recently-seen inputs.
5.2 RHO Loss