Cramming: Training a language model on a single GPU in one day
Authors: Jonas Geiping
Tom Goldstein
Venue: Submitted to ICLR 2023 (2023)
Keywords: Transformers
Scaling
Pretraining
Language

Abstract: Recent trends in language modeling have focused on increasing performance through scaling, and have resulted in an environment where training language models is out of reach for most researchers and practitioners.  While most in the community are asking how to push the limits of extreme computation, we ask the opposite question:  
How far can we get with a single GPU in just one day? 
We investigate the downstream performance achievable with a transformer-based language model trained completely from scratch with masked language modeling for a single day on a single consumer GPU.
Aside from re-analyzing nearly all components of the pretraining pipeline for this scenario and providing a modified pipeline with performance close to BERT, we investigate why scaling down is hard, and which modifications actually improve performance in this scenario. We provide evidence that even in this constrained setting, performance closely follows scaling laws observed in large-compute settings. Through the lens of scaling laws, we categorize a range of recent improvements to training and architecture and discuss their merit and practical applicability (or lack thereof) for the limited compute setting.

Under review as a conference paper at ICLR 2023
CRAMMING: TRAINING A LANGUAGE MODEL ON A SINGLE GPU IN ONE DAY
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent trends in language modeling have focused on increasing performance through scaling, and have resulted in environment where training language models is out of reach for most researchers and practitioners. While most in the community are asking how to push the limits of extreme computation, we ask the opposite question: How far can we get with a single GPU in just one day?
We investigate the downstream performance achievable with transformer basedd language model trained completely from scratch with masked language modeling for single day components of the pretraining pipeline for this scenario and providing a modified pipeline with performance close to BERT, investigate why scaling down is hard, and which modifications actually improve performance in this scenario. We provide evidence that even in this constrained setting, performance closely follows scaling laws observed in large-compute settings. Through the lens of scaling laws, we categorize a range of recent improvements to training and architecture and discuss their merit and practical applicability (or lack thereof) for the limited compute setting.
single consumer GPU. Aside from re-analyzing nearly all
SCALING UP AND SCALING DOWN
Large-scale training of machine learning models with transformer architectures has lead to ground- breaking improvements in many sub-fields of natural language processing including language un- derstanding and natural language generation (Vaswani et al., 2017; Dosovitskiy et al., 2021; Radford al., 2019). The nowadays accepted (but historically surprising) key behavior of these systems is that they reliably scale they continuously improve in performance when the number of model pa- rameters and amount of data grow. These increases in performance are well-described by various power laws studied by Kaplan et al. (2020). This sets up dominant paradigm in which scaling the key to performance improvement (Sutton, 2019).
The power of scale has set off race to produce extremely large models, which in turn has created an environment where few researchers or practitioners feel that they are capable training a language model. The original BERT model Devlin et al. (2019), which became cornerstone transformer for many practical applications in natural language understanding, already required a significant amount of computation train. Yet, the reproduction and improvements in Liu al. (2019) further increased its performance by cranking up the level of computation by orders of magnitude. As these pre-trained checkpoints became popular for range of downstream applications (Wolf al., 2020), the competition for the largest language model became focal point for industrial labs. This led pretrained language models the expense of
training runs that improved the performance computation at the zettaF1.00 L.P scale (Raffel et al., 2020; Yang et al., 2020; Zaheer et al., 2021) and later the extremely large yottaFL.OP scale (Brown et al., 2020; Black al., 2022; Chowdhery et al., 2022; Rae et al., 2022).
Our goal is to turn this trend on its head and investigate how to best seale down language model training and what trade-off's emerge when doing so: What downstream performance can be achieved by a modest researcher when training from scratch with a single GPU for a single day? The ability train a language model to the performance level of BERT with such modest resources has several interesting implications. For one, if scaled-down model pretraining is a viable analogue of large-compute pretraining, then this opens up a host of further academic investi irestigations that

---

Under review as a conference paper at ICLR 2023
currently hard to realize for large-scale models. For example, research questions about the differences between existing and new pre-training tasks, tracing model predictions to data points (llyss et al., 2022), security questions such membership inference (Carlini et al., 2022) and data poisoning (Geiping et al., 2021), and a wide range of empirical investigations into topics such as stability or generalization that arise during training (Nagarajan & Kolter, 2019; Jiang et al., 2019). At the same time, we can imagine situations in which legal requirements make it unclear whether models trained on public data with uncertain origin are permissible, and where practitioner is interested in retraining their language models using specialized or trustworthy data source (Wilka al., 2017; Gold & Latonero, 2017).
In addition, we are motivated to benchmark the overall conceptual progress of research in this area over the last years, beyond simply turning the scaling knob. The goal of achieving BERT-like perfor- mance with modest training resources would have seemed unthinkable in 2018, and yet with modern advances and transformer training techniques this may now be possible.
To answer these questions, we consider a challenge we call "Cramming" learning whole language model the day before the test. Our studies begin by investigating many facets of the training pipeline to see which modifications actually improve performance in the scaled-down scenario. We provide evidence that even in this constrained setting, performance closely follows scaling laws observed in large-compute settings. An unsurprising consequence of these laws is that scaling down is hard; while smaller model architectures enable speeding up gradient computations, overall rates of model improvement changes to the training pipeline that exploit scaling laws to yield improvements by improving the effective rate of gradient computations without compromising model size. In the end, are able to train models that achieve respectable performance - often close to and sometimes exceeding BERT on GLUE tasks on shoestring budget.
time remain nearly constant. Nonetheless, we can find
TYING OUR HANDS BEHIND OUR BACK: A SETUP WITH LIMITED COMPUTE
Before we start this investigation, we want to outline the extent of limitations we are interested in. The rules for cramming are as follows:
A transformer- modeling, completely from scratch.
language model of arbitrary size is trained with masked-language
Existing pretrained models cannot be included in any part of the pipeline.
Any raw text (excluding downstream data) can be included for training. This means that one can achieve speedups by making judicious choices about how and when to sample data, provided the sampling mechanism does not require pre-trained model.
The downloading and pre- pre-processing of raw data is exempted from the total compute bud- get. Pre-processing may include CPU-based tokenizer construction, tokenization, and fil- tering, but cannot include representation learning (e.g. pre-training word embedding is not allowed, unless it is counted towards the final runtime).
Training proceeds on a single GPU for 24 hours.
Downstream performance is evaluated on GLUE (Wang et al., 2018). Downstream finetun- ing GLUE is limited to brief training with only the training data of the downstream task (we consider 5 epochs or less) and needs to work with hyperparameters set globally for all GLUE tasks. Downstream finetuning is excluded from the total compute budget.
In our implementation, we analyze both setup with classical rtx2080t: GPU (released September 2018) and a separate setup with a more modern rtxa rtxa6000 GPU (released October 2020). We pair each unit with 4 CPU cores and 32GB of RAM.
Why these limitations? We are principally interested in re-investigating the original BERT setup of Devlin et al. (2019) with limited compute. The optimal architecture of the transformer is not fixed, as the optimal size and shape depends on scaling laws (Kaplan et al., 2020). The limitations on usage of existing models rule out distillation from existing model (Ture al., 2019; Jiao et al., 2020; Sun et al., 2020; Wang et al., 2020b; Kaliamoorthi et al., 2021) and data filtering based on existing large models (Golchin al., 2022), both of which ultimately answer questions about compression and
2

---

Under review as a conference paper at ICLR 2023
Group
Target
Accelerator IPU
Time Limit
Total exaFLOP
(Devlin et al., 2019)
BERT
4 days
680
(Dettmers, 2018)
BERT
VV00
11 days
950
(Narasimhan, 2019)
BERT-large
1472 V100
47 min
519
(Raffel et al., 2020)
Ta-base
16 TPOV3
1 day
170
(landola al., 2020)
Squee7eBERT
Titan RTX
4 days
361
(Narang et a 2021)
'5 variations
TPUV3
1.75 days
298
(Tay et al., 2021)
T5-small-L.16
16 TPOV3
11.2 hours
82
(Izsak ct al., 2021)
BERT variation
8
I day
86
(Liu et al., 2019)
roBERTa-base
1024
I.25 day
13424
(Chowdhery et al., 2022)
PaLM
6144 TPUv4
50 days
7
Our Setup
BERT variation
1xtx0080t1
I day 1 day
5
Our Setup 2
BERT variation
rtxa6000
13
Table 1: Maximal Throughput available for select training runs of large language models. FLOP Counts for BERT reproductions and related models. Large-scale LMs included only for reference.
transfer of already processed information. Further, we do not want to limit data to the original dataset used to train BERT, wanting to allow for possible improvements through better data curation and quality. The xt xtx2280tt GPU is a natural candidate for this experiment, given that was released before Devlin al. (2019), but the more recent rt 000 is also interesting, being arguably the limit of a single-user workstation. At the finetuning stage we want to mimic the original BERT finetuning and evaluation setup, but provide additional limits to prevent gains based tuning of only the downstream procedure, for example via computationally extensive downstream training (Bahri et al., 2021a), use of multiple downstream datasets (for example continued pretraining with MNLI before finetuning other tasks (Izsak et al., 20) and extended hyperparameter optimization for each GIUE task (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2019).
RELATED WORK ON EFFICIENT TRANSFORMERS
How long does take train BERT? In general, this question is hard answer, due to wildly varying hardware and software setups and differing measures of efficiency (Dehghani et al., 202 An upper bound on the compute of training run can be established by finding the total number of (low-preed (lo-prrecision) floating point operations available over the wallclock budget of the run. This peak of total FLOPs in given time interval is generally not reached in actual compute, even for highly opti- mized models (Chowdhery et al., 2022), but represents the paid budget required to realize training run. We summarize budgets for a few select training runs Table 1. After the original training run for BERT TPUs, initial reactions estimated up to 11 days of compute for comparable results GPUs (Dettmers, 2018). However, sustained improvements, especially in software, have reduced the upper limit significantly (You et al., 2019 9;; Narasimhan, 2019). Yet, recipes and implementations generally require entire server nodes (for GPUs) or TPU slices and target larger BERT architectures.
Other work discussing improvements BERT targets compute settings closer to the original BERT, for example SqUeezeBERT (landola et al., 2020) employs 8 T Titan RTX cards for four days. Sel- lam et al. (2022) note that the original BERT training an outlier and doubling its training time more reliably reproduces the original results.
Our central point of comparison for BERT training with limited resources is the work of Izsak et al. (2021) who also attempt the goal of training BERT within 24 hours with overall similar limitations, but a full server node with 8 V100 0000 GPUs. Izsak al. (2021) choose a BERT LARGE architecture variant and train with sequence length of 128, including range of tweaks such as modified learning rates schedules, large batch sizes, sparse prediction and packed sequences. We re-evaluate this setup as baseline setting for our own compute budget (which is about 15x smaller).
Studies of Efficient Transformers Recent years have seen flurry of research working to improve and modify the transformer architecture proposed in Vaswani et al. (2017) and refer to Treviso et al. (2022) for a recent categorization and review of research in this area. Several meta-studies have investigated proposed improvements and modifications: Narang et al. (2021) evaluate large range of architectural modifications applied to the T5 model pipeline Raffel et al. (2020) on tasks in both language understanding and translation. The encoder-decoder structure of T5 is closer in spirit
3

---

Under review as a conference paper at ICLR 2023
to the original transformer setup, but is understood to behave similarly to BERT when using the encoder component (Liu et al., 2021a). Evaluating modifications with 1.75 days of compute on TPU slices they find that most improvements do not reliably materialize gains in final accuracy. Tay et al. (2021) work in the same setting and evaluate the optimal shape of T5 derived architectures and its relative effects downstream performance as models are scaled. Further exploration of the scaling behavior of various architectural improvements in Tay et al. (2022a) find that only few modifications outperform the original architecture of Vaswani et al. (2017) at all scales, especially when evaluating downstream accuracy. The meta-study investigating improvements in preparation for extreme-scale training Scao et al. (2022) focuses on minor modifications to layout, positional embeddings and data sources for autoregressive models, and other extremely-large scale training runs have so far been similarly conservative their settings (Brown et al., 2020; Black et al., 2022; Rae et al., 2022).
In general though, these evaluations target larger compute settings than we intend to use, and are concerned with whether improvements (often from academic sources and proposed with evaluations on small scales) translate to larger scales. In this work, we set aside the question of (up)scaling and focus only on the limited compute.
Scaling Laws The difficulty in finding tangible improvements is echoed in the scaling laws of Kaplan et al. (2020). Over wide range of transformer model shapes, Kaplan et al. (2020) find only model size (as number of parameters in non-embedding layers) strongly predicts performance. Further, for fixed compute budget, optimal model size can be derived, but performance is only mildly connected to model size- larger models processes less data per unit of compute, but improve faster by almost the same margin. While the precise coefficients and shape of these scaling laws continue to be iterated on (Hoffmann et al., 2022) and adapted for related settings (Bansal et al., 2022: Clark al., 2022; Bahri et al., 2021b), their overall logic appears hard to escape, even if power laws fit observations somewhat less well on small scales.
4 INVESTIGATIONS
For our experimental evaluation we implement and test considerable number of proposed modifi- cations to the setup of Devlin et al. (2019) for their merits in our limited compute setting as described in Section 2. We first clarify the common implementation and initial data setup, and then investigate architectural, training and dataset improvements.
4.1 IMPLEMENTATION DETAILS
We implement everything in PyTorch (Paszke et al., 2017) and to limit our gains from the "soft- ware lottery" (Hooker, 2021) we do not use specialized implementations (c.g. as proposed for attention mechanisms in Ivanov et al. (2021); Dao et al. (2022)), which would further bias results to- wards well-established components. We keep everything on the implementation level of the PyToch framework, allowing only automated operator fusion (Sarofeen et al., 2022) that can be applied to all components. We run all experiments and ablation studies with the same setup of automated mixed precision (Micikevicius et al., 2018) for standard 16- and 32-bit floating point precision (over full 32-bit float, scaled 16-bit (Rasley et al., 2020) and pure bfloar oaat1 6 (Wang & Kanwar, 2019). We find no benefit from offloading (Ren et al., 2021; Rasley et al., 2020) in our setting.).
Initial Data Setup We start our investigation with close analogue the original raw text sources of Devlin et al. (2019), using a recent dump of the English Wikipedia (20220301, (0)3e..en)))) and En- glish bookcorpus, noting the commentary of Tan (2019): Bandy & Vincent 1). We force all text into lower-case, strip accents and non-ascii characters and create an English tokenizer from scratch based only on this data. We choose WordPiece with a vocabulary size of 215 32768 (Wu et al., 2016). We found no significant change in performance with BPE (Sennrich et al., 2016) or SentencePiece with Unigrams (Kudo, 2018; Kudo & Richardson, 2019). Smaller vocabulary sizes 12 213 2 resulted in worse performance, while larger vocabulary sizes (216) we not reliably better. We pack tokenized data into randomized sequences of length 128 and separate unrelated fragments by <sep> The performance impact from dropping this separator was minimal. No im- pact was observed from including a <cls> token in pretraining. The shorter sequence length is sufficient for the downstream applications that are targeting and simplifies attention computa-
4

---

Under review as a conference paper at ICLR 2023
Lirenn
NEM
Mukge 15:0)
Black
Mead
7.5%
17.0
1900
Teloess Oegeard
Thot Thiers Iegrond
Figure 1: Various Transformer architectures and shapes, showing MLM loss versus number of tokens ingested. Left: Global view. Right: Zoom onto 10e8 that improvements through architectural reshaping are minimal; while there are some fluctuations loss early training. the rates of loss decay during most of training differ by a multiplicative constant (horizontal shift due to logarithmic horizontal axis) that depends strongly on the model size and not model type.
tokens. All models trained with the same budget. We
tions. Packing data into full sequences limits us to simpler sequence losses, but uses the available compute optimally Liu et al. (2019); Izsak al. (2021). For the targeted compute settings, this sequence length results in micro-batch sizes of 64 to 96 for most variations of the base BERT ar- chitecture on gtx2 stx2080ti, which we will accumulate into larger batch sizes. With our limited compute budget, this produces enough samples to run single-epoch training (Komatsuzaki, 2019; Hernandez al., 2022) where no data point is revisited.
4.2 MODIFYING THE ARCHITECTURE
The most obvious way to efficiently scale down training is by modifying the model architecture; in- tuitively, it seems likely that smaller/lower capacity models will be optimal in the cramming regime. In this section, we study the relationship between model type and training efficiency. We see that scaling laws create a strong barrier to scaling down. Per-token efficiency of training depends strongly model size, but not transformer type. Furthermore, smaller models learn less efficiently, and this largely mitigates any throughput gains. Fortunately, the fact that training efficiency nearly con- stant across models of the same size means that modifications that speed up gradient computation while keeping the parameter count nearly con- stant. This makes architecture selection fairly straightforward as we can make design choices based primarily on how they affect computation time for single gradient step.
boost performance by finding architecture
Scaling laws hold in the low-resource regime A large corpus of research in recent years has developed architectural improvements to speed up the original transformer. Many of these methods have not been found to improve training for the large-scale T5 architecture Narang et al. (2021); Tay et al. (2022a). But, in the low compute setting where data throughput is of utmost importance, maybe this is the way forward? Scaling laws have been observed by Kaplan et al. (2020) in the high- resource regime, and seem to hold strongly in the limit as resources grow. Surprisingly, these laws also hold in the limit extreme compute down-scaling, and they create a barrier to low-cost training.
We exemplify the effect of scaling laws for many transformer variants from the literature in Figure 1, where we train each architecture variant with optimized training hyperparameters described below in Section 4.3. We apply these architecture variants to shared baseline model that incorporates rre-Normalization and rotary embedding. Figure visualizes the progress of MLM loss versus the number of tokens ingested in total and all architectures run with the same time budget.
We observe that varying the transformer type and size has only minimal impact on the final loss after 24 hours. Models with more parameters learn more efficiently, as their MLM loss decreases faster on a per-gradient basis. However, smaller architectures make up for their slower learning efficiency by higher throughput, and thus process more tokens over the limited budget. Figure shows that different architectures are unpredictable throughout an initial stage of training (the first 1B tokens), after which the per-token efficiencies differ by only multiplicative constant (a horizontal shift due to the log axis). This constant depends almost entirely on the model size, not model type, so that all choices reach MLM loss around 1.9 at the end of training.
5

---

Under review as a conference paper at ICLR 2023
Exploiting the scaling law. The scaling laws seem to bar from making large gains via major changes to the transformer size and type, as per-token performance is tightly coupled to model size. coanelltransformen
As result, we find no improvements when using Nawrot et al., 2022), when dropping FFN layers (Sridhar et al., 2022), or when using recurrent layers (Lan et al., 2019), even when trained with BPTT as in Schwarzschild (2021). Rescaling architectures to be deep-narrow (Tay et al., 2021; Wies et al., 2021) provides no gains.
architecture (Dai et al., 2020;
While this principle closes one door for scaling down efficiently, it opens another; Because per- gradient efficiency remains nearly constant for all models of the same size, we can exploit scaling laws by quickly searching for architectural choices that speed up computation while keeping model size roughly constant. A number of obvious optimizations fall into this category. and we describe them below, in addition to several other tweaks that provide marginal but worthwhile/free gains.
Attention Block: We disable all QKV biases (Dayma et al., 2021). This exploits the scaling law by removing a layer of computation, making the forward and backward pass somewhat faster, while keeping the model size nearly constant. We find that we can decrease gradient costs by reducing the number of attention heads (Merity, 2019; Araabi & Monz, 2020; Liu et al., 2021b; Javaheripi et al., 2022), as this parallelizes better on the GPU and provides a slight performance boost. We find no benefits from replacements the softmax operation (Richter & Wattenhofer, 2020). We further keep the original multi-head self-attention mechanism. A large amount of work has been focused on efficient attention (Sukhbaatar et al., 2019; Beltagy et al., 2020; Wang et al., 2020a; Liu et al., 2021c) and studies of efficient attention (Tay et al., 2020a;b). But, because we set the maximal sequence length to 128, attention complexity less of concern in our setting. To verify this, we implement the recently proposed FLASH mechanism (Hua et al., 2022), but find no benefits. We further experiment with Fourier attention as proposed Lee-Thorp et al. (2021), 1), but find no improvements. We supplement the attention with rotary embeddings (Su et al., 2021; Black et al., 2022), which we find to provide small benefits.
We find empirical gains from disabling all linear layer biases (Dayma et al,, 2021). Just for the attention layers, this leverages the scaling law by accelerating gradient com- putation without noticeable impacts on model size. As a result, we get higher throughput without compromising the rate at which the model improves. We keep the original feedforward block largely unchanged, finding no benefits from changing to another activation than GELU. We do see small improvements from re-ordering the block into gated linear unit (Dauphin al., 2017). In contrast to other work, e.g. (Black et al., 2022), we do not increase the number of parameters in the FFN block to compensate for the halving of the hidden dimensionality due to gating.
Feedforward Block:
Embedding: We implement scaled sinusoidal positional embeddings as described in Hua et al. (2022), finding incremental benefits over learned or unscaled sinusoidal embeddings. We see no im- provements from decoupling the input and output embeddings (Chung et al., 2020). The suggestion from Lan et al. (2019) to factorize the input embedding provides no gains in our setting. We include a layer normalization at the end of the embedding block.
Layer Structure: As observed in many studies, we find that pre-normalization with Layer Norms beneficial over post Layer Norms (Baevski & Auli, 2018; Xiong et al., 2020). We see no ad- ditional benefit from other variants of this modification, such as (Liu et al., 2020b; Shleifer et al., 2021). Further, replacing Layer Normalization with RMS Normalization provides no gains (Zhang & Sennrich, 2019). We note that the key effect of pre-normalization is to stabilize training and enable larger learning rates and reduced warmup, and we see limited benefits from including it by itself. We see no benefits from stochastic dropping of entire layers as described in (Zhang & He, 2020).
Head Block: We find that we can remove the nonlinear head without ill effect. We can further drop the decoder bias (Radford et al., 2019) and gain in memory using sparse token prediction (Liu et al., 2019; Izsak et al., 2021). We add final Layer Norm stabilize training further.

---

Under review as a conference paper at ICLR 2023
3.000
Cru-Cade
22
Liner
Corsdast
MIM 1oss
Ramp
MIM Loos
2:3
Codine Decay
2,09
Step
0,0004
1.35
00002
1000
100%
2400
2500
3006
Microbarch Soaps
Microbatch Steps
Microbatch Steps
Figure Leamning Rate Schedules. Although globally many schedule result similar behavior, we see in the zoom in the middle, that differences do exist. The right side shows the corresponding learning rate schedules.
4.3 MODIFYING THE TRAINING SETUP
We study the impact of training ppprm-prrseeterss on the BERT-base architecture. The original BERT training recipe understandably results is poor model performance in the cramming setting, and so we revisit number of standard choices.
Objective: We train with only masked language modeling on fully packed blocks of tokens with a masking rate of 15% and the original setup of Devlin et al. (2019) where 10% of all masks are filled with random words and 10% unchanged. We see no improvement from masking larger rates, e.g. at 40% as proposed in (Wettig et al., 2022). We see no difference enabling or disabling the mentioned 20% rule. We evaluate other functions for the masked- masked-language objective, such as mean-squared error (Hui & Belkin, 2021) or L1 loss, but find no benefits.
Choice of Optimizer: We keep Adam (Kingma & Ba, 2015) as the optimizer of choice, with weight decay of 0.01 described in (Loshchilov & Hutter, 2017), 81 0.9, B2 0.98 and 10-12, We find no noticeable change in varying these parameters in reasonable amounts, e.g. E 10-6, 3, 0.9, 82 0.999. We test other first-order adaptive optimizers (Shazeer & Stern, 2018; Liu et al., 2020a) but find no advantages in our setting. We further find no advantages using higher- order optimizers (Yadav, 2020; Anil al., 2021), but note that especially for higher-order optimizers there is greater amount of variability in implementation.
Following the advice of Izsak et al. (2021), we re-scale the learning rate schedule so that it is tied to our budget and the learning rate decays as the budget reduces to zero. Interestingly, we observe in Figure 2 that while globally a large number of learning rate shapes lead to similar reductions in loss, find that we can make some gains through the choice of schedule. We find that a simple one-cycle learning rate (Smith & Topin, 2018) with a peak learning rate of 10-3 leads to minimal pretraining loss within our budget.
Learning Rate Schedule and Peak:
Batch Size Schedule: A particularity of our setting is that, due to being limited to a single GPU, the micro-batch size that finds its way onto this GPU (96 for most experiments) is several times smaller than the optimal batch size. We find that the optimal batch size in this setting is around 1536 for minimal pretraining loss and 4032 for maximal downstream performance, i.e. we accumulate gradients and only perform update every 16 and 42 forward/backward passes, respectively.
Fortunately, we can find small speedups by using an aggressive batch size schedule; we increase the number of averaged micro-batches linearly over the course of training. This results in more progress earlier in training, and leads to a small benefit to performance. We also experiment with automatic and adaptive batching rules (De et al., 2017; Bollapragada et al., 201 Sa;b), but find that the best results from these adaptive schedules resemble the fixed linear schedule. For simplicity we just stick the simpler linear schedule.
Dropping Dropout The original BERT model of Devlin al. (2019) includes dropout as in Vaswani et al. (2017), which prevents overfitting when training data is small relative to total compute budget. While it can be helpful as regularizer, dropout effectively reduces the number of gradient
7

---

Under review as a conference paper at ICLR 2023
Dataset
Batch Size
MNLI (m)
Bookcorpus- Wikipedia
1536
79.8
The Pile
1536
80.5
The Pile (natural data subset)
1536
80.8
C4-Subset
1536
79.1
Bookcorpus- Wikipedia, Deduduplication > 100
I536
79.9
Bookeorpus-Wikipediaa, Deduduplication >> 50
1536
79.5
Bookcorpus- Wikipeedia,, filtered with / = 0.3, sorted
1536
80.8
Bookeorpus-Wkiipddii.. sorted
1536
81.0
C4-Subset, Deduduplication 100
1536
79.2
C4-Subset, filtered with = 0.3
1536
79.9
C4-Subset, filtered with = 0.3, sorted
1536
81.4
C4-Subset, filtered with = 0.3, larger, sorted
1536
81.9
Bookcorpus-
4032
80.5
C4-Subset, filtered with = 0.3
4032
82.2
C4-Subset, filtered with = 0.3, sorted
4032
82.5
C4-Subset, filtered with = 0.3
8064
80.9
Table 2: Dataset Variations for the optimal model from Section 4.2 and optimal training routine from Sec-
tion 4.3, modifying final batch size in conjunction with dataset format.
updates seen by each parameter, as updates do not occur when the associated feature is dropped. At the same time, update runtime is not strongly effected by the presence of dropout, and so dropout results in a net reduction in updates per second.
In the cramming setting, training data is large compared to compute. Overfitting is not possible due to the single epoch schedule, and we disable dropout during pretraining (Brown et al., 2020) to maximize the number of parameter updates. We re-enable dropout during downstream fine-tuning with a dropout value of 0.1. Further, we experiment with length curricula (Li et al., 2022) and token dropping (Hou al., 2022), but find gains.
4.4 OPTIMIZING THE DATASET
We found above that scaling laws create barrier to making major gains (beyond computational ef- ficiencies) with architectural modifications. However, scaling laws not preclude us from training on better data. Once we have exhausted our ability to train more tokens per second, we should seek to train on better tokens.
We consider two data based pathways to better down-scaling, First, we filter, process, sort the existing data various ways. Second, we can swap our data source. To this end, we experiment with several subsets of The Pile (Gao et al., 2020), containing and Wikipedlaa (en). From these Pile datasets we tokenize the first 4 x 10 entries to generate enough tokens for single pass. Another popular source of data is C4, the colossal, cleaned version of Common Crawl (Raffel et al., 2020), from which we stream the first 20 10 entries. For each data source we regenerate its own WordPiece tokenizer described in Section 4.
text from only Gutenberg, Books3
Of these four sources, we find the Pile to perform best in terms of downstream MNLI performance. However, turns out we can further improve especially the C4 datset through additional processing. We first evaluate deduplication as described in Lee et al. (2022) via exact substring deduplication, but find this not help in downstream performance in our case. We then test filtering for un- compressible data. We use the tokenizer itself to remove all training sequences from C4 set that cannot compressed well; we simply set a threshold f, e.g. dataset where the number of tokens in the entry is larger than t times the number of raw characters. This removes, for example, sequences consisting of to-oppesssssssss HTML or markdown code. Surprisingly, this results in measurable improvement on C4, summarized in Table 2.
0.3, and drop all entries from the
We then see some further improvements from two directions. First, sorting all tokenized se- quences by some metric, and second, increasing the final batch size. For filtering sort all tokenized sequences by their average (unigram) token prevalence, so that likely sequences occur first. This has some positive effect, and can be strengthened slightly by drawing from larger
8

---

Under review as a conference paper at ICLR 2023
MNLI (m/mm)
SST-2
STSB
RTE
QNLI
QQP
MRPC
Avg.
BERT-Base
83.5/83.6
92.1
86.7
58.3
90.3
87.6
88.7
83.9
BERT- B2se(2080t)
54,2/54,1
80.9
14.3
52.0
59.8
65.4
78.1
57.4
Crammed LM (2080ti)
82.0/82.5
90.4
84,9
56.5
88.2
87.0
85.0
82.1
Crammed LM (A6000)
83.3/83.8
92.1
84.5
56.1
88.5
87.3
87.5
82.9
Table 3: Comparison in GLUUB-evv performance of baseline BERT to crammed model. Avg. Score is all scores excluding CoLA,
pus, as the unlikely sequences never get reached. Finally, increasing the batch size to 4032 at the end of training (as mentioned in Section 4.3) is disproportionally effective C4, but less bookcorpus wwikiwwikiiiii We believe that both modifications ultimately reduce the likelihood of training being hindered by fluctuations in the data distribution.
FINETUNING PERFORMANCE ON GLUE
Finally, we evaluate performance the GLUE benchmark of Wang et al. (2,18), minus WNLI as in Devlin et al. (2019). We note that we only use MNLI (m) during the previous sec- tions and not tune hyperparameters based on the full GLUE scores. We finetune both the pre- trained BERT-base checkpoint and our models under the same constraints laid out in Section 2. For BERT-base, we finetune all datasets for epochs with batch size of 32 and learning rate 2 x 10 10-5 For the crammed models, we find that this is not optimal and minor improvements can be gained from a batch size of 16 and learning rate of 4 x 10-5 with cosine decay (this setup does not improve the pretrained BERT checkpoint).
Table 4: Comparison in GLUE-dev performance of baseline BERT to crammed model. Avg. Score is all scores excluding CoLA, GLUE is the full average over the same tasks as in Devlin al. (2019).
CoLA
Avg.
GLUE
BERT-Base
$6.0
83.9
80.8
BERT- (B0T-))ss(208Ori
2.0
57.4
51.2
Crammed LM (2080ti)
41.9
82.1
77.6
Crammed LM (A6000)
37.0
82.9
77.8
Table 3 and Table 4 describe the performance of this setup on the GLUE downstream tasks (as me- dian over 5 trials). There we compare the original BERT-base checkpoint, a reproduction of the BERT pretraining settings stopped after our budget is reached, and the modified recipe, evaluated with the single day rtx2080ti setup and the single day A6000 setup. Overall, performance is surprisingly decent, especially for the larger datasets of MNLI, QQP, QNLI and SST-2, where down- stream finetuning can smooth remaining differences. However, even the smaller datasets mostly work. The average is brought down however by massive drop on CoI.A (corpus of linguistic ac- ceptability) (Warstadt et al., 2019). This behavior is intriguing and we offer two hypotheses. First, it is conceivable that the chosen global hyperparameters for finetuning are bad fit for CoLA in particular. CoLa performance can be brittle with respect to hyperparameter, with Jiao et al. (2020) training longer only on CoLA or Joshi et al. (2020) training less only on CoLA. Nevertheless, for BERT, a set of global hyperparameters exists, pointing at a deficiency in the crammed model. As second hypothesis, is conceivable that these models need process more text before they memo- rize enough data to do well on CoLA. This would be in contrast to Liu et al. (2021d) who find that CoLA is learned relatively quickly compared other downstream tasks when probing intermediate BERT checkpoints. On the other hand. deficiencies on CoLA in particular are also common in ap- proaches that distill BERT into smaller architectures (Sun et al., 2019; Ture et al., 2019; Mukherjee et al., 1), which might come with limited capacity for linguistic acceptability.
6 CONCLUSIONS
We discuss how much performance a transformer-based language model can achieve when crammed into a setting with very limited compute, finding that several strands of modification, such as espe- cially training recipe and data setup lead to decent downstream performance though, cramming language models appears hard, as we empirically find many implications of Ka- plan al. (2020) to still hold in this regime. We hope that this work can provide a baseline for explorations of the question of cramming we formalize in Section 2 and cast a new light on a num- ber of improvements and tricks proposed for transformer architectures in recent years.
GLUE. Overall

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We provide code to reproduce all experiments.
REFERENCES
Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable Second Order Optimization for Deep Learning. arXiv:2002.0901 {cs, math, stat], March 2021. URL http: / //aaxiiv.orgg/////aa...aaa.a...a http://rggga//aaaa/e///eo/eee/ /tw.bb/222000000....0
Ali Araabi and Christof Monz. Optimizing Transformer for Low-Resource Neural Machine Trans- lation. In Proceedings of the 28th International Conference on Computational Linguistics, pp- 3429-3435, Barcelona, Spain (Online). December 2020. International Committee on Computa- tional Linguisties. doi: 10.18863/v1/2020.ccoo wtt.://00...oonnmmmooeee main. URL https: // /ttps////aaaaatagaaaa.ooooaa org/2020. co www.iniggminn...
Mikel Artetxe, Jingfei Du, Naman Goyal, Luke Zettlemoyer, and Ves Stoyanov. On the Role of Bidirectionallity in Language Model Pre-Training. drxiv:2005.117266ccmm May 2022. doi: 10.
48550/arXiv.2205.1 11726. URL http::t/////// /ttp://aaxvvvv/. wwtprxii.orgg.s/////p//o// http/bb//20000 11726.
Alexei Baevski and Michael Auli. Adaptive Input Representations for Neural Language Modeling. In International Conference on Learning Representations, September 2018. URL https://
openreview nnt/ forum? Ld-Byx2X20qFQ.
Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness-Aware Minimization Improves Language Model Generalization. 1xXiv:210000855 [cs], October 2021a. URL httt::://axxvv.rrggg
abs/2110 08529.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining Neural stut], February 2021b. doi: 10.48550/arXiv.2102.
Scaling Laws. arxi::21022.677/(ccodndd 06701. URL http::tttt//tt// org/ absaabss1111 22t.2.22... 06701.
Payal Bajaj. Chenyan Xiong, Guolin Ke, Xiaodong Liu, Di He, Saurabh Tiwary. Tie- Yan Liu, Paul Bennett, Xia Song, and Jianfeng Gao. METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals. wrXiv:2204.06644 [cs], April 2022. URL http: //tp:///aaiiiia orq/ abs 2204. 06644.
Jack Bandy and Nicholas Vincent. Addressing "Documentation Debt" in Machine Learning: A Ret- rospective Datasheet for BookCorpus. NeurIPS 2021 Track Datasets and Benchmarks, November 2021. URL https: wwwp//vww..eeee.eeeeeee....o..we net www.//ee///ommneeeeooeenttmm di-immmmdddiddddieee wvJeu.
Yamini Bansal, Behrooz Ghorbani. Ankush Garg, Biao Zhang, Maxim Krikun, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data Scaling Laws in NMT: The Effect of Noise and Ar- chitecture. arXiv:2202.0199. [cs], February 2022. URL https: //axxiv org/ abs/2202. 9944v..
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang. Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training. arXiv:2002.1 12804 [cs], February 2020. URL
/ttt:://rriss v.w.vv..o org/ / abs/2002. 12804.
Iz. Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The Long-Document Transformer. arXiv:2004.05150 [cs], December 2020. URL http: //axxivv////i....//. org/ aaa/a/bs//200 05150.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivan- shu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach, GPT-NcoX-200 An Open-Source Autoregressive Language Model. 27X75:2204.06745 [cs]. April 2022. URL http:// org/ abs 2204 06745.
Raghu Bollapragada, Richard Byrd, and Jorge Nocedal. Adaptive Sampling Strategies for Stochas- tic Optimization. SIAM Journal on Optimization, 28(4):33123333, January 2018a. ISSN 1052-6234. doi: 10. 10.1137/17M1154677 URL https: 1137/17M115460
epubs. s siam. org/doi/ abs 10.
10

---

Under review as a conference paper at ICLR 2023
Raghu Bollapragada, Dheevatsa Mudigere, Jorge Nocedal, Hao-Jun Michael Shi, and Ping Tak Peter Tang. A Progressive Batching L-BFGS Method for Machine Learning. arxiv: 1xxi::8802.05374/cs math, stat), May 2018b. URL httt:://aaxi..orrttt org. wttp://rrgg//////ao////o
abs/1802.05374
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vosse Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess. Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. 34th Conference on Neural Information Processing Systems (NeurIPS 2020), December 2020. URI. http://arxiza org/
14165.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership Inference Attacks From First Principles. arxiv:2112.03357ffesss April 2022. doi: 10.48550/arXiv221122.0 12.0050 URL htt.:ht::/aaaai...gggaaass////// 03570.
Ivan Chelombiev, Daniel Justus, Douglas Orr, Anastasia Dietrich, Frithjof Gressmann, Alexandros Koliousis, and Carlo Luschi. GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures. arxiv:2106.0588) [es], June 2021. URL
2106. 05822v1.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov. Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellster Douglas Hck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaL. Scaling Language Modeling with Pathways. arXiv:2204.022 023 /1 [cs], April 2022. URL http:
2204 02311.
Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. Re- thinking Embedding Coupling in Pre-trained Language Models. In International Conference Learning Representations, September 2020. URL https: http::::////pppvvveeeeee wtpp//ppeeeeeeeeeeee/ee/ee net forum?
id= idd=xFFI_Nttger
Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoff- mann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew Johnson, Katie Millican, Albin Cas- sirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Jack Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. Unified Scaling Laws for Routed Language Models. arXiv:2202.01669 [cs], February 2022. URL http: //arxiv. wwwpaaa.///aaaao.caa....aaaaaaea abs /202 01169.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre- training Text Encoders as Discriminators Rather Than Generators. In International Conference on Learning Representations, September 2019. URL https hhttts::/opeereiiiieete http//pppweeew.eeeeeeeeee net forum? 1x==1xMHBBBBB
Kevin Clark, Minh-Thang Luong, Quoc V. Le. and Christopher D. Manning. Pre-Training Trans- formers as Energy-Based Cloze Models. arXiv:20/2.0856/ [cs], December 2020. URL. http:
//arxiv... wwwoorr/abbbo 2212.0222200 08561.
Zihang Dai, Guokun Lai, Yiming Yang. and Quoc Le. Tranf--Transfommerr: Filter- ing out Sequential Redundancy for Efficient Language Processing. In Advances in Neural Information Processing Systems, volume 33, pp. 4271-4282. Curran As- wttc//appe//22///aaaaeee sociates, Inc., 2020. URL att html.
2cd291

---

Under review as a conference paper at ICLR 2023
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with 1O-Awareness. 2tth::2205.141155//ee May 2022. doi: 10.485500/2X1v.2225... 14135. URL http::ttt/aae http::/axxiv..ogg///.../o abs / 2205.
Yann Dauphin, Angela Fan, Michael Auli, and David Grangier. Language Modeling with Gated Convolutional Networks. In Proceedings of the 34th International Conference Machine Learn- ing. pp. 933-941. PMLR, July 2017. URL https: http:////occeedeeeeeeeoe wwt.emmrr.ppp... wttp:///ww......../ee./...ooo press/v7C/ dauphin176 7a html.
Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phúc Lå Khác, Luke Melas, and Ritobrata Ghosh. DALL-E Mini, July 2021. URL https://gtt thub. com/ borisdaymal /a dalle-mini..
Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein, Big Batch SGD: Automated Inference using Adaptive Batch Sizes. arxiv:1610.05792/css math, stat], April 2017. URL http:// arxiv.org/abl... / 610 05792.
Mostafa Dehghani, Yi Tay, Anurag Arnab, Lucas Beyer, and Ashish Vaswani. The Efficiency Misnomer. In International Conference on Learning Representations, September 2021. URL.
https://cpenreviev net/ httpr//wwnnee//oooiiieeeeee
Tim Dettmers. TPUs vs GPUs for Transformers (BERT), October 2018.
URL https://
timdettmera www.c0mmm0000 112/01//1//111111111111111111111 100001//1111////////////
gpus--pps---- for-troree fo foreers-berbbbr///////o
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], May 2019. URL http:: org/ abs / 1810. 04805.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.
[cs], June 2021. URL http: http::::/wxxwv........ea 11991.1199.. 11929.
Francesco Fusco, Damian Pascual. and Peter Staar. PNLP-Mixer: An Efficient MlL- Architecture for Language. arxiv:2202.04355 [cs], February 2022. URL https:///rxxi....// wttp:/wwwww..o...//a/o///o/oa abs/ 2202. 04350v1.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027 [cs], December 2020. URL http: / arxiv. org /ttp://211..000
Jonas Geiping, Liam H. Fowl, W. Ronny Huang. Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' Brew: Industrial Scale Data Poisoning via Gradient Match- ing. In International Conference on Learning Representations, April 2021. URL https: / owwerreewiee.ntttffoormmee-ed-od
Shahriar Golchin, Mihai Surdeanu, Nazgol Tavabi, and Ata Kiapour. A Compact Pretraining Ap- proach for Neural Language Models. arxiv:2208. /2367[cs], August 2022. doi: 10.48550/arXiv.
URL http://ttaaaee http://aaavvv/..///e
wwtp://///abbaaaaeeeeeae 12367.
Zachary Gold and Mark Latonero. Robots Welcome: Ethical and Legal Considerations for Web Crawling and Scraping. Washington Journal of Law, Technology & Arts, 13 (3):275-312, 2017. URL https:/ attps://heinon::: http//heennnnnnneeeeeeeeeeeeeeee org/ HHHHHHLLPhhhae journals/ washjolta136i-285
Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. On the Trans- former Growth for Progressive BERT Training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 5174-5180, Online. June 2021. Association for Computational Linguis- tics. doi: 10. 10.1333/vv1/201 maacl-main.405 URL https: / /aaaaaaanntllogoooeee or2/2021. naacl -eacl-maii 406.
12

---

Under review as a conference paper at ICLR 2023
Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving De- BERTa using ELECTRA-StyLe Pre-Trec Training with GradienttDiseenaapne Embedding Sharing. arXiv:2111.09543 [cs], December 2021. URL http::/arxiv...to wttp://////...//////////
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nel- son Elhage, Zac Haffield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling Laws and Interpretability of Learning from Repeated Data. arxiv:2205. 1et.:2225..0477ccs 10487[cs), May 2022. URL http: / http///axvvvv/// org/ abs 22225.00455
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Compute-Optimal Large Language Models. arXiv:2203. [es], March 2022. URL http:://rxixa org/ abs 2203 15556.
Sara Hooker. The hardware lottery. Communications of the ACM, 64(12):58-66 65, November 2021. 11.....411////11.77 ISSN 0001-0782. doi: 10. 10.1145/3467017. URL http::tt////eett hopp:://do..oo /10/ /ttp:///0oo.ooo..1111111///11110
Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou, Token Dropping for Efficient BERT Pretraining. arXiv:2203.13244 [cs], March 2022. URL httt::///txiii
org/ abs /2003/222.222 220.213240...
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer Quality in Linear Time. In Proceedings of the 39th International Conference on Machine Learming, pp. 9099-9117, PMLR, June 2022. URL httpt::t//cc.. htttpps://proceedingg.. . mmlr. www.ppresssvll... h2/aa/22aaaaaaaa.aaaaaaaaaaaaaaa html.
Like Hui and Mikhail Belkin. Evaluation of Neural Architectures Trained with Square Loss vs Cross Pross-Entrroyy in Classification Tasks. arXiv:2006.07332 [cs, stat), October 2021. URL http:
//arxiv. abs:///b//2/ 2006 07322.
Forrest N. landola, Albert E. Shaw, Ravi Krishna, and Kurt W. Keutzer. BqueeZeBERT: What can computer vision teach NLP about efficient neural networks? 13Xiv:2006.11316 [cs], June 2020. URL htLLtt////axii org/ aass/2000 11316.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Data- models: Predicting Predictions from Training Data. arxiv:2202.006222cr stat), February 2022.
doi: 00.48550//rxi..222..000
URL http:://aaaaee wwtp://...g.aba..o 02622
Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. Data Movement Is All You Need: A Case Study on Optimizing Transformers. arXiv:2007.00072 [cs, stat), November 2021. URL http: http//aaxivvv//a woorg/abaa 20007000772
Peter Izsak, Moshe Berehansky, and Omer Levy. How Train BERT with an Academic Bud- get. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 10644-10652. Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 00.18653/v1/022 http:////221.mmppm..eeo wewnlmmmain.83... URL https: hatp:://accaattttho wwwphhhllll..o...goooeooe/e// wttp///.......ooo emppl-.mmllmmmmm inainn in. 831.
Mojan Javaheripi, Shital Shah, Subhabrata Mukherjee, Tomasz L. Religa, Caio C. T. Mendes, Gustavo H. de Rosa, Sebastien Bubeck, Farinaz Koushanfar, and Debadeepta Dey. LiteTrans- formerSearch: Training-free On-device Search for Efficient Autoregressive Language Models. 22XV:22003.0209 [cs], March 2022. URL http: aawiiiiiiii.////ii org/ abs 2203. 02094.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic Generalization Measures and Where Find Them. arXiv:19/2.02/788 (cs, stat], December 2019.
URL aatp:://arxin org/ abs///bs//// 19112 02178.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for Natural Language Understanding. arXiv:1909.100 10351 [cs), October 2020. URL http: /arxiaassss org/abs 909. 10351.
13

---

Under review as a conference paper at ICLR 2023
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Span- BERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the As- sociation for Computational Linguistics, 8:64-77, 2020. doi: 10.1162/tacl a 00300. URL
htttps://aclanthho wwwphhhllll..ooooo/.ooooo hrgg tacl-1.5.
Prabhu Kaliamoorthi, Aditya Siddhant, Edward Li, and Melvin Johnson. Distilling Large Language
Models into Tiny and Effective Students using pQRNN. arxiv: 2101.08890 [cs], January 2021.
URL httph:tt///// /aaxxvivivv.. org/ abs 08890v1.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. 0rexi::0011.08361 /cs, stat), January 2020. doi: 10.485500/rXiv.20001 2006666....... URL.
http://arxiv..tt org/ abs 20000000.0000 08361.
Guolin Ke, Di He, and Tin Liu. Rethinking Positional Encoding in Language Pre-training. In International Conference on Learning Representations, September 2020. URL https:// openreview.. n forum? (add==0--8y22-3
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR), San Diego, May 2015. URL htth:://taxxxi org/abs/ 1/1//411121 6980.
Aran Komatsuzaki. One Epoch Is All You Need. arXiv:1966.06669 [cs, stat), June 2019. URL http://arxiv.orgg: aawpg////////aaa 190..06669..
Taku Kudo. Subword Regularization: Improving Neural Network Translation Models with Mul- tiple Subword Candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 66-75, Melbourne, Australia. July 2018. Association for Computational Linguistics. doi: 10. 1111118/1//11--1111 1007. URL https: // //aclanthology... org P080P101
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In EMNLP (Demonstration), July 2019.
https:: httpstts:///peneeee
/ttpp///////peeiiieeee/e .nt. www.wrrutee/eeeeeeemee 11y bH.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In International Conference on Learning Representations, September 2019. URL https: 7ABtvv..
/ //openreview.....m/m////oooooove wwwppenwweieeeeeettorrmeeeeeeeee
Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating Training Data Makes Language Models Better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424-8445, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1/553//1/022..alllooo 2022 ac1-..ng.11oo5500.. 577.
URL https: / laclanthologyy...oooooooooooolla org/
James Lee-Thorp Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. FNet: Mixing Tokens with Fourier Transforms. arxiv:2105.0m 03824 [cs), May 2021. URL https: /ttps/////vvvcoii 01.55.3824v..
Tao Lei, Ran Tian, Jasmijn Bastings, and Ankur P. Parikh. Simple Recurrence Improves Masked Language Models. arxiv:205.115888c55 May 2022. URL http://arxiv.orz /bs//
11588.
Conglong Li, Minjia Zhang, and Yuxiong He. Curriculum Learning: A Regularization Method for Efficient and Stable Billion-Scale GPT Model Pre-Tnaining. arXiv:2108.0660 [cs], February 2022. URL http: /tt:///axxivv org/ abs 2200//11.... 06084.
Frederick Liu, Siamak Shakeri, Hongkum Yu, and Jing Li. EncT5: Fine-tuning T5 Encoder for Non- autoregressive Tasks. axix:2110008426(css), October 2021a. doi: 10.48550/arxi.21110.00222 URL
14

---

Under review as a conference paper at ICLR 2023
Liyuan Liu, Haoming Jiang, Pengeheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the Variance the Adaptive Learning Rate and Beyond. In International Conference on Learning Representations, March 2020a. URL https: https::/apeereviiieee/ http///ppeeeeee.eeee/eeeeeeee/ep net e wwwtffounm forum? d rkgz2aEKDD
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the Dif- ficulty of Training Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5747-5763, Online, November 2020b. Associ- ation for Computational Linguistics. doi: 18.1 1ttp://////2eeeee hwtp:0000.mmmmmiiinm..ee.. URL https: //aclanthoology org/ emnlp 446.
Liyuan Liu, Jialu Liu, and Jiawei Han. Multi-head or Single-head? An Empirical Comparison for Transformer Training. arxiv:2/066.9655(cc1 June 2021b. doi: 10:4850WarXxx22266.6666.
URL http:///rxix orgg http:/bb//21.... 29650.066000.....
Xiaoyu Liu, Jiahao Su, and Furong Huang. Tuformer: Data-driven Design of Transformers for
Improved Generalization or Efficiency. In International Conference on Learning Representations, September 2021c. URL https: wwtp//ppnreeeeeeeeeeeeeee net/ (tte://foouuue.iooooosgggggg
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. arXiv: 1692 [cs], July 2019. URL 1907.11692.
abs/
Zeyu Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. Probing Across Time: What Does RoBERTa Know and When? In Findings of the Association for Computa- fional Linguistics: EMNLP 2021, pp. 820-842, Punta Cana, Dominican Republic, November 2021d. Association for Computational Linguistics. doi: 1021 10116655///2221111nnissmmmm// URL https: http:///aaaaatttotaooeo org/ 2021 www.iiiinnnniiiiin.n
71.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, arXiv: [cs, math], November 2017. URL /ttp:://aaaaae org/ /atps///1..... 05101.
Stephen Merity, Single Headed Attention RNN: Stop Thinking With Your Head. arXiv:991.11123 [cs], November 2019, URL
/91/11911..1 11423.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. In International Conference Learning Representations, 2018. URL
net/ forum?id=r1gssseee
Sören Mindermann, Jan Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Win- nie Xu, Benedikt Höltgen, Aidan N. Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. arxiv:2206.07137/cc/) June 2022. doi: 10448550/2XX..2206077133 URL http:://rrr org/abs/ 220dab///22... 07137.
Subhabrata Mukherjee, Ahmed Hassan Awadallah, and Jianfeng Gao. XuremeDiiti/Tranffomoi Task Transfer for Task-agnostic Distillation. arXiv:2106.0450 [cs], June 2021. URL http:
//aaxiivorrg///.e...e.. /2..6 04563.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smooth- ing help? In Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL wws.aattttttasssattaaaaaaaaaaa..
nips wwt.pppccppppp///..a.e cc/ppape/2211 9/has// Abstract. html.
Vaishnavh Nagarajan and J. Zico Kolter. Generalization in Deep Networks: The Role of Distance from Initialization. arXiv:1901.01622 [cs, stat], January 2019. URL http: / /wwr/vivvvco org/ abs/1901. 67.
15

---

Under review as a conference paper at ICLR 2023
Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding. Jake Mar- cus, Adam Roberts, and Colin Raffel. Do Transformer Modifications Transfer Across Implemen- tations and Applications? In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pp. 5758-5 5773, Online and Punta Cana, Dominican Republic, Novem- ber 2021. Association for Computational Linguistics. doi: 10. 10..8865/v1/221111 2tt555//12222...mmmmmaaa....... Lemnlp-main.465. URL https: /aaaaaaannoooooooo.aaaaaaaaooala org/202/ emnlp main. 465.
Shar Narasimhan. NVIDIA Clocks World's Fastest BERT Training Time and Largest Transformer Based Model, Paving Path For Advanced Conversational AI, August 2019. URL https://
developer: wwt.oper..viiii..... com blog/ wetooggtrriiingggg.. ceetniingg-er----- bert-with gpus/.
Piotr Nawrot, Szymon Tworkowski, Michat Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical Transformers Are More Efficient Language Models. arxiv:2110.137111cc// April 2022. URL http: org/ bttg/abs/// wwtp//sc...............aaaataata 13711.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentia- tion in PyTorch. In NIPS 2017 Autodiff Workshop, Long Beach, CA, 2017. URL https: / //openreview. www.ee//ffeeee
Bo Peng. RWKV-LM. Zenodo, August 2021. URL 5196577.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAl, pp. 24, 2019.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAlcese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kun- coro, Áida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Men- sch, rean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yu- jia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Au- relia Guy, Chris Jones, James Bradbury. Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling Language Models: Methods, Analysis & Insights from Training Go- pher. 1rXiv:2112.111 11446 [cs], January 2022. URL http: //taxxiivv/////////////////aaava org/ abs / 2212222222 112..... 11446.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to- Text Transformer. arXiv: [cs, stat], July 2020. URL 1910.10683,
erg/abs/
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase and Yuxiong He. DeepSpeed: System Op- timizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference Data Mining, KDD '20, pp. 3505-3506, New York, NY, USA, August 2020. Association for Computing Machinery. ISBN 978- 7999--
Knowledge Discovery &
doi: N011445/3944466.44670. URL 111////......//.../.. /1//333444888 3406703.
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. (ZeRO-Offload): Democratizing (Billion-Scale) Model Training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pp. 551- 564, 2021. ISBN 978-1-939133-23-4 URL https: www.wwwsenx.ccommmm wwt.co/fw///eeee////oooo//o
ren-
16

---

Under review as a conference paper at ICLR 2023
Oliver Richter and Roger Wattenhofer. Normalized Attention Without Probability Cage. arXiv:2005.09561 [es, stat], May 2020. URL htttp://axxiv org/ abaabb//2/ abs/2005 09561.
Aurko Roy, Rohan Anil, Guangda Lai, Benjamin Lee, Jeffrey Zhao, Shuyuan Zhang, Shibo Wang, Ye Zhang, Shen Wu, Rigel Swavely, Tao, Yu, Phuong Dao, Christopher Fifty, Zhifeng Chen, and Yonghui Wu. N-Grammer: Augmenting Transformers with latent n-grams. Nadi::2207.0636660 org/abs/ 2207. 06366.
July 2022. doi: 10448550/aXxi..2277.66666 URL http://arxas
Christian Sarofeen, Piotr Bialecki, Jie Jiang, Kevin Stephano, Masaki Kozuki, Neal Vaidya, and Stas Bekman. Introducing nvFuser, deep learning com- piler for PyTorch, August 2022.
URL. introducing- wwwpggg--uuur-a--eee dedep-------------eee---e Learning for-arpo-o------oootttoo-
Teven Le Scao, Thomas Wang. Daniel Hesslow, Lucile Saulnier, Stas Bekman, M. Saiful Bari, Stella Biderman, Hady Elsahar, Jason Phang. Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong. Julien Launay, and Iz Beltagy. What Language Model to Train if You Have One Million GPU Hours? In Challenges {', April 2022. URL https: https:// htpps///ppeeeeieww net wtw.leewwwee/foummiieeeeeee forum?id=aa -d=mmddd.--/rIa
Avi Schwarzschild. Easy-To-Hard, October 2021. URL easy-to-hard.
com/ aks2203/
Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D A'mour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Turc, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick. The MultiERTs:: BERT Reproductions for Robustness Analysis. In International Conference on Learning Representations, March 2022. URL httpt: /ttp///eeeeeeeeeeeeeeeee/.. net / forum?id= [OE_FOgFDGA.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics (Volume Long Pupers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. N0.18653/v1/P16. 1162. URL https: / //aclanthology. wwt.:oo........1oo...e.. orgpp/6-11222
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. 1mxiv:/804.04235/cc stat], April 2018. doi: 1445500/axXi..18000422... URL http: wlaaxii..rggaass///.e
04235.
Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy. Staged Training for Transformer Language Models. arXiv:2203.06211 [cs], March 2022. URL http:
// //arxiv.org/aan wtt.:oog//ab///...././/// 06211.
Sam Shleifer, Jason Weston, and Myle Ott. NormFormer: Improved Transformer Pretraining with Extra Normalization. arXiv:2110.09455 [cs), November 2021. URL http://arxiv.org.
abs/2110. 09456.
Leslie N. Smith and Nicholay Topin. Coppr--onveergncee: Very Fast Training of Neural Networks Using Large Learning Rates. arXiv: 1en::00000002 [cs, stat], May 2018. URL htttp://arxiv..
org/abs//70n 1008. 07120.
David So, Wojeiech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Le. Searching for Efficient Transformers for Language Modeling. Advances in Neural Information Processing Systems, May 2021. URL https: http:////eeeeeeeeeeeeeeeeeeeeeee /ooe
_VaII.
Sharath Nittur Sridhar, Anthony Sarah, and Sairam Sundaresan. TrimBERT: Tailoring BERT for Trade-offs. arXiv:2202. 12411 [cs], February 2022. URL htRp::ttp///rriw.orrr 12411.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding. arxiv:2104.098. [cs], April 2021. URL https: : arxiv.. org/abs 24 04824v2.
17

---

Under review as a conference paper at ICLR 2023
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive Attention Span in Transformers. arXiv:1905.07799 [cs, stat), August 2019. URL CRLattt://aaxx.. hattp://axxiv.org/ abs/1905 07799.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient Knowledge Distillation for BERT Model Compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4323-4332, Hong Kong, China, November 2019, Association for Com- putational Linguistics. doi: 10.1 10.11863/1/DD19.. 1441. URL /tttt:::s/acaanttoolooooo org D19-1441.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobile- BERT: A Compact Task-Agnostic BERT for Pessorcce-Limiter Devices. arXiv:2004.02984 [cs], April 2020. URL
/ 2004. 02984.
Richard Sutton. The Bitter Lesson. Incomplete Ideas (blog), pp. 1, March 2019. URL http:
//www. www.inccompleteidesss... net
Liling Tan. What the bookcorpus?, December 2019. URL allaatoons/o222766655$ce22221212
com/
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long Range Arena: A Benchmark for Efficient Transformers. arXiv:2011.04006 [cs], November 2020a. URL http::/aaxiv..ot abs/ 2011. 04006.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient Transformers: A Survey. arXiv:2009.067:3 [cs], September 2020b. URL http: //arxi////xivvee hwtproorg/aaaaa 2209 06732.
Yi Tay, Mostafa Dehghami, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale Efficiently: Insights from Pretraining and Finetuning Transformers. In International Conference on Learning Representa- fions, September 2021. httpttttt/p/eeeeeeeet
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling Laws Model Archi- tectures: How does Inductive Bias Influence Scaling? axxi::2207.10551(css July 2022a. doi: (0.48550/rXxi22207.055) URL wwtph:ttta.i...///////tt/
abs /200.2200..02 222021011555 10551.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying Language Learning Paradigms. mmxi::2005.0513//c May 2022b. URL http: wwwaaiiii.r....e.eeeoooooo abs/22055 05131.
Marcos Treviso, Tianchu Ji, Ji-Ung Lee, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Pedro H. Martins, André F. T. Martins, Pe- ter Milder, Colin Raffel, Edwin Simpson, Noam Slonim, Niranjan Balasubramanian, Leon Der- czynski, and Roy Schwartz. Efficient Methods for Natural Language Processing: A Survey. arxiv:2209.0000 0009990s))0 August 2022. URL http:: http::///wwwww//.//e/eee abs wttp:o/ggab//.....//o/ 2209.00099
Julia Ture, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. 108992290.00866 [cs], September 2019.
URL / /ttp:p///axxvv.ooo org/ abs/ 19008 08962.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv:1706.03762 [cs], De- cember 2017. URL http://///....
wwt.iiivorrg/eoo abs//bb///11 /006/ 03762.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In International Conference on Learning Representations, September 2018. URL https:// openreview.nen for/fo 2et.ormmmm.drrrr4222255....
18

---

Under review as a conference paper at ICLR 2023
Shibo Wang and Pankaj performance on Cloud TPUs,
Kanwar.
BFloat16:
The secret to high
August 2019.
URL https://
cloud. gwwgle com bfloat1 6-the-tthheeee weteee--seeettt-tttee hww.-eee--eeeeeett--oooiirrrrrra
-n-clleee--o-------e------ oud-tpus/.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-Attention with Linear Complexity. (tt::2006004768V/ [cs], June 2020a. URL htttt://aaxii...o
abs/2006. 04768v3.
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? arXiv:2204.05832 [cs, stat), April 2022. URL http:
//aaxiiv.rrg/////a....oooaao...a abs 2204 05832.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression Pre-Trained Transformers. 109997 [cs], April 2020b. http:hhttariiiii .rgg absaabss///// 2oob/2202..o 10957.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural Network Acceptability Judg-
ments. arxi::/805. October 2019, doi: 10.48550/arXiv.1800 180. URL http: / ///axxii.rrg/abb////......
Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should You Mask 15% in Masked Language Modeling? arxiv: 2202.08005 [cs], February 2022. URL https: /
/2002. 08005v1.
Noam Wies, Yoav Levine, Daniel Jannai, and Amnon Shashua. Which transformer architec- ture fits my data? A vocabulary bottleneck self-attention. In Proceedings of 38th ternational Conference Machine Learning, pp. 11170-11181. PMLR. July 2021. URL attps://proceeede
ww..m.r.pp... mal..press/// wttpe/ss/vv.//weweeeee... wwwpwwweweeaa... html.
Rachel Wilka, Rachel Landy, and Scott A. McKinney. How Machines Lear: Where Do Companies
Get Data for Machine Learning and What Licenses Do They Need. Washington Journal of Law, Technology & Arts, 13(3):117-244, 2017. URL https: htt.hhiinnnnnnn..eeeemm orooorgg//ooooo Hogg/HOL/// ph
hein. ba
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace' Transformers: State- of-the-art Natural Language Processing. 111.0:1910.03771 [cs], July 2020. URL http:// arxiv. org/ abs 1tt//999000000 03771.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey. Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's 's Neu- ral Machine Translation System: Bridging the Gap between Human and Machine Translation. arxiv:1609 (1466,08144(cs),, October 2016. URL http: /ttp//aavvvv//e.//////eee/.awa.. org/ abs / 1169..6... 08144.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing. Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie- Yan Liu. On Layer Normalization in the Transformer Archi- tecture. 2Xi2:2002.04745 [cs, stat], June 2020. URL 04745.
Abhay Yadav. Making L-BFGS Work with Industrial-Stret Strennth Nets. In BMVC 2020, pp. 13, 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. 1rXiv:1906.082 [es], January 2020. URL http:///eee //arxiv/// org/ abs 6066 08237.
19

---

Under review as a conference paper at ICLR 2023
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel. Kurt Keutzer, and Cho-Jui Hsieh. Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. In International Conference Learning Representa- fions, September 2019. URL https: / revirrevice ew, net/ www.ee/oorrrrt//////
forum? Stt.ummmiid-SSSxnttetee.e
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, and Amr Ahmed. Big Bird: Transformers for Longer Sequences. 0rXiv:2007.14055 [cs, stat], January 2021. URL http: arxiv.. org/
abs arg/abs/// 2007. 14062.
Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. arXiv:1910.07607 [cs, stat], October 2019. URL http: htt//axivv..o////// wttp:////t/g..ooo.///o// 07467.
Minjia Zhang and Yuxiong He. Accelerating Training of Transformer- Language Models with Progressive Layer Dropping. In Advances in Neural Information Processing Systems, vol-
33, pp. 14012-3 2020. URL httpss::/ppoceeecimnssss neurips cttppps.cccppaaee cc/paper/
LVAR22o0n1eM222SSKKKRS
Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W. Ronny Huang, and Tom Goldstein. GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training. arxiv:2100.080098/c1 November 2021. doi: 10.48550aaxxi221122.09999 URL http:// httppp///axxw.ooo/a//a/////ssss/ 08098.
A APPENDIX
B LIMITATIONS
In this work, we limited our investigation to transformer-basee architectures trained with MLM objectives. However, we do think that the general task of cramming posed in Section 2 is interesting even when relaxing these constraints. There have been number modifications proposed the objective in particular (Joshi et al., 2020; Bao et al., 2020; Bajaj et al., 2022; Tay et al., 2022b). While Artetxe et (2022) and Wang et al. (2022) find MLM still to hold up well as pretraining objective, other suggestions such as ELECTRA (Clark et al., 2019; 2020; He et al., 2021) could be employed which might beneficial for crammed models. Also, the optimal architecture might not be transformr-bbace (Merity, 2019; Fusco et al., 2022; Peng, 2021)
C OTHER MODIFICATIONS
A few recent developments not included in this study Roy al. (2022), Shen et al. (2022), and Mindermann et al. (2022). Modifications further not included in this study are more involved initial- ization (Zhu et al., 2021), additional objective modifications (Müller et al., 2019), progressive growth (Gu et al., 2021; Shen et al., 2022), convolutional variants (landola et al., 2020; Chelombiev et al., 2021; So al., 2021), sequence recurrence (Lei al., 2022) and TUPE embeddings (Ke et al., 2020).
D ADDITIONAL INFORMATION
20

---

Under review as a conference paper at ICLR 2023
Name
MLM Loss
MNLI-m
MNLI-mm
Tokens/Second
Modified Transformer
1.89
81.02
81.35
50946
DeepNarrow (12 Layers)
1.94
80.90
80.97
78396
DeepNarrow (24 Layers)
1.98
80.78
81.14
41289
E = 128
2.14
76.68
77.62
53267
FFN every 2 blocks
1.93
80.43
80.97
64774
FFN every 3 blocks
1.97
80.44
80.93
71634
FFN every 4 blocks
2.00
80.03
79.67
73319
H 512
1.93
80.61
80.93
83718
H 1024
1.95
80.07
80.68
32004
4 Layers
2.00
78.45
79.00
137127
Layers
1.93
79.49
79.82
96156
Layers
1.89
81.11
81.08
74248
10 Layers
1.89
81.02
81.21
61431
16 Layers
1.92
81.39
82.10
39406
24 Layers
2.01
80.64
80.97
26927
Recurrent (1-12)
2.40
77.46
77.81
52405
Recurrent (2-6)
2.04
80.45
80.73
53148
Recurrent (3-4)
2.00
80.78
81.33
51634
Recurrent (4-3)
1.98
80.95
81.26
51952
BERT-tiny
3.30
56.71
57.21
914694
BERT-mini
2.49
72.22
73.21
429593
BERT-Large (Izsak variant)
2.38
76.93
77.47
13448
Original BERT
7.54
35.45
35.22
41978
With decoder bias
1,89
80.97
81.20
51155
E = 6 in Layer Norm
1.90
80.49
81.35
51728
Learned Embedding
1.88
80.51
81.03
52601
No Norm after Embedding
1.94
79.65
80.34
52175
No Final Norm
1.89
80.40
80.89
51207
No Skip of Head Transform
1.88
80.49
81.19
51728
No Rotational Embedding
1.88
80.91
81.52
53526
Post-LN
7.54
31.82
31.82
52270
With QKV bias
1.89
80.70
80.88
51112
With bias in Linear Layers
1.89
80.64
81.49
50584
Heads
1.88
81.75
81.99
47967
Table 5: Additional raw results for experiments considered in the main body. First two blocks: Architectural
variants discussed in Section 4.2. Third block: Ablation study of finally adopted model. All experiments with the training setup described in Section 4.3 for a day on single GPU with mixed precision. Batch size 4032 and dataset bookcor pus pwpppuss wwssswwilt wwwkwwwikiiieee a. Downstream evaluation as described Scction 5.
21


Reviews:
[{"review_id": "VpEkt6zLidI", "reviewer": "Reviewer_xMcb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the paper investigate language model pipeline to see which modifications improve performance in the scaled-down scenario ( a single GPU for 24 hours).  ", "summary_of_the_review": "This paper does a lot of Interesting investigations. ", "strength_and_weaknesses": "Strength:\n1. In this paper, several modifications (architecture, training setup and datasets) are explored to check whether there is any improvement. All of these aspects are important and interesting. These can give good insights for the community. \n\n2. Some interesting conclusion are got, for example, training recipe and data setup lead to decent downstream performance on GLUE.\n\nWeaknesses:\n\n\n1. The investigation of modifications lack convincing experiments. For example, only one task performance (MNLI) is reported for when studying the impact of training hyper-parameters. Other tasks can have a different trend.  And when exploring the effect of the architecture, only MLM loss is report. The performance of downstream tasks can be also important. \n\n2. The technical novelty of this paper is a little limit. The total contributions are also limit. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O8d9Km8oLKg", "reviewer": "Reviewer_4joz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "While research on enormous language models have dominated empirical NLP lately, most researchers and practitioners do not have the resources and access to work with these models. This work seeks to answer the empirical question: what is the best performance one can achieve on the GLUE benchmark by training a BERT-like model from scratch on a consumer GPU for one day? The numerous Transformer variants proposed in recent years present another challenge to answering this question – which of these variants are beneficial when one’s compute is extremely constrained?\n\nThe authors investigate a wide range of design choices in terms of model architecture, training recipe, and data curation. They also note that the final MLM loss is mostly correlated with the FLOPs spent, not the particular Transformer type and size. This motivates them to choose architectures that parallelize well on GPUs. The final result demonstrates that some combination of the variants proposed in the three years after BERT yields a model that is almost as performant as the original BERT base while using 1/50 of the FLOPs.", "summary_of_the_review": "Neat empirical investigation with conclusions that might interest many. However, it is not clear if ICLR is the best venue. Moreover, one might argue that MLM loss is not the best criterion to study model and training design choices.", "strength_and_weaknesses": "This work presents a thorough empirical investigation on a topic that is of interest to many researchers who do not have access to large compute clusters. The overall methodology appears to be sound and the final result promising.\n\nHowever, I am not sure if ICLR is the best venue for this work. There are no new theoretical or algorithmic contributions nor any new insight into representation learning. This does not change my belief that this is an informative paper to many in the community, but it might find a more suitable audience if submitted to venues such as EMNLP. (Disclaimer: I do not regularly publish in this area. I am happy to defer to more experienced reviewers and the AC on this point.)\n\nAn additional concern is that it is not clear if the MLM loss is the best metric to track when comparing different model/training design choices. This paper (https://openreview.net/pdf?id=F5uYcwABMu), for example, clearly demonstrates that models with near-identical pretraining loss can perform very differently on downstream tasks due to implicit biases. This should be taken into consideration since many conclusions in this work are based on the MLM loss alone. I encourage the authors to also observe downstream performance and note if they agree or disagree with the MLM loss.\n\nThe connection between “cramming” and scaling laws can be clarified. The scaling laws mentioned in this work are empirical observations that a model’s performance strongly correlates its size but not necessarily its shape. The empirical results from this work show that this holds for the low-compute regime, which is somewhat surprising. However, these “laws” are merely empirical observations. It is not clear what the authors mean by “we discuss scaling laws in the low compute regime and find that cramming is hard because the conclusions of Kaplan et al. (2020) so easy to reproduce” (emphasis mine) in the conclusion. It would be better to simply state that the empirical observation from Kaplan et al. holds in the setups investigated, which motivated using architectures that parallelize well. A related concern is that since the dominating factor for performance is the number of FLOPs we can squeeze out of a GPU within a given timeframe, this makes the conclusions of this work somewhat hardware-specific, e.g., they might not hold on TPUs or newer/older GPUs.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LcmGfQxOYO3", "reviewer": "Reviewer_x7E8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigated pretraining a mask language model in a resource-constrained setting, i.e. a single GPU for one day. The authors empirically tested various architectural and data changes in order to maximize performance. There are some interesting findings, such as per-gradient efficiency only depends on model size, several strategies to filter and sort the training data brought improvement, etc. As a results, the authors were able to push the performance close to BERT base if excluding some outlier tasks.", "summary_of_the_review": "Overall, this paper tackles an important problem, the experiments design are sound and the empirical findings are informative for language models pretraining. If the authors could add more clarity to generalizability and robustness of the findings, e.g. experiments with sequence lengths longer than 128, further ablate on the causes of drop in CoLA, etc. then the results would be more valuable to language models pretraining. ", "strength_and_weaknesses": "Strengths:\n- This paper adds more insights on scaling-down, which is less understood as most concurrent efforts are around scaling-up. \n- The experiments have a good coverage in terms of testing various architectural changes from recent literature.\n- The final performance on downstream tasks (GLUE) are impressive given the limited compute budget. \n  \nWeaknesses:\n- The biggest missing piece is an ablation study on the improvement from various architecture changes. In the current version, the authors provided a comprehensive list of things they tried, what helped and what didn't. But it's unknown which change(s) brought the bigger improvement.  \n- Related to this is the poor performance on CoLA. Although the authors provided several hypotheses, some of them should be tested to verify whether they're actually related to any of the architecture or data changes, or mostly due to reduced model size. For example, one possible cause provided by the authors is that reasonable performance CoLA would need more training data. But are models in these experiment trained on less data compared to BERT-base? Is it possible to see whether the performance gap can actually be closed if the models were trained longer than one day (i.e. seeing more data)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JJy5zDi_UNr", "reviewer": "Reviewer_Fi67", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study the performance of transformer models on downstream tasks as the total computational budget is decreased.  This process, known as cramming in the paper, turns the problem of training these enormous language models in a new direction from the typical scenario used in industrial labs that train models on a seemingly endless supply of resources. The author's place and exception small limit on the total computation that is allowed to train a transformer model from scratch to the total FLOPs available on a single GPU in 24 hours. By considering the scaling laws of large model transformers the authors mainly investigate training setups that keep the total number of parameters in the model constant but reduce the cost of performing a gradient update. By enumerating a small number of interesting features of the transformer training design space the authors demonstrate that cramming can achieve interesting and sometimes comparable results with larger models using more computation in particular settings and for particular datasets.", "summary_of_the_review": "Overall I find the motivation for the work and claims made by the authors to be an interesting departure from the traditional language training papers that use exorbitant computational resources. It seems more practical to answer questions about how researchers can do more with less when it comes to allocating resources for training transformer models.\n\nMy remarks should be taken with a grain of salt as I am not an expert in this particular area but I would feel more inclined to experiment with transformer models if I felt I could train them to a reasonable level of ability on my modest desktop setup. I believe this sentiment represents the spirit of the paper and the results should be of interest to other members of the research community that are hesitant to participate in this research area because of the perceived computational overheads.", "strength_and_weaknesses": "Strengths:\n- The motivation for the study proposed in the paper is interesting for a number of reasons. The volume of computation required by many modern transformer models has been prohibitively expensive and therefore out of reach for most researchers for quite a while. By studying the implications of constraining the computational resources on the ability of the model to perform well on certain tasks the authors could provide a way for researchers with limited budgets to participate and utilize these models in fundamentally new ways.\n- The trend in the paper to consider modifications that mainly reduce the gradient update cost without significantly impacting the total number of parameters in the model, based on the scaling laws, provides an interesting and unifying theme throughout. The persistence of the scaling laws to influence the performance of the model on tasks is reinforced through empirical evidence throughout and yields interesting insights.\n- Performance evaluation on a shoe-string budget of FLOPs compare to other prominent models is impressive.\n\nWeaknesses:\n- Similar studies were conducted on a single node with 8 GPUs as noted by the authors. Though that setup had considerably more computational resources the total volume of computation was still a fraction of the amount used by many large research institutions. In light of that work, the scenario presented in this paper may seem somewhat derivative and only marginally interesting.\n- It is not clear if or how the observations made in the cramming regime may be used to make more informed decisions regarding the training process in the normal training setting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject