Scaling Laws for BERT in Low-Resource Settings
Year: 2023

Abstract:
Large language models are very resource intensive, both financially and environmentally, and require an amount of training data which is simply unobtainable for the majority of NLP practitioners. Previous work has researched the scaling laws of such models, but optimal ratios of model parameters, dataset size, and computation costs focused on the large scale. In contrast, we analyze the effect those variables have on the performance of language models in constrained settings, by building three lightweight BERT models (16M/51M/124M parameters) trained over a set of small corpora (5M/25M/125M words). We experiment on four languages of different linguistic characteristics (Basque, Spanish, Swahili and Finnish), and evaluate the models on MLM and several NLU tasks. We conclude that the power laws for parameters, data and compute for lowresource settings differ from the optimal scaling laws previously inferred, and data requirements should be higher. Our insights are consistent across all the languages we study, as well as across the MLM and downstream tasks. Furthermore, we experimentally establish when the cost of using a Transformer-based approach is worth taking, instead of favouring other computationally lighter solutions.

heading: 1 Introduction
text: Pre-trained neural language models based on the Transformer architecture have shown impressive results on many NLP tasks to the point that their use has become standard practice. The capabilities of these models improve as the complexity (in terms of parameters) of their architecture (Wei
et al., 2022a) and the size of the corpora on which the pre-training is performed increase (Zhang et al., 2021). For this reason, there is now a tendency to build ever-larger models trained on ever-growing corpora. This trend has resulted in a never-ending increase of the computational requirements to perform model pre-training, but also for the subsequent fine-tuning and inference processes at production time. Moreover, building very large models require huge training corpora, which is only available for a handful of rich-resource languages.
Kaplan et al. (2020) and Hoffmann et al. (2022) propose power-law formulas that relate model size, corpora size and computation power, and help find the optimal settings in advance given a fixed budget. However, their analysis is focused on autoregressive models of relatively big sizes, that require large corpora to train. In this paper, we analyze whether the conclusions drawn in these works also apply to discriminative (encoder-only) models in low-resource settings, where both the data size and budget are constrained. We analyze the performance of several combinations of model and data sizes using a simulated low-resource scenario in four linguistically diverse languages from different families (Basque, Spanish, Swahili and Finnish).
Our study reveals that the data size and model size power law values provided by Kaplan et al. (2020) and Hoffmann et al. (2022) are not optimal in these scenarios. Instead, our experiments show that data size should be relatively bigger than what those scaling laws estimated when training small models (16-124M parameters) for optimal performance. Furthermore, given a fixed computational budget, it is better to train big models instead of
7771
computing more model updates in smaller models. Additionally, we establish the minimally required combinations of compute, model and dataset sizes of Transformer-based approaches that outperform other lighter neural baselines, taking CO2 emissions into consideration.

heading: 2 Related Work
text: Since the emergence of the attention-based Transformer (Vaswani et al., 2017) architecture and the masking pre-training strategies introduced since BERT (Devlin et al., 2019), different pre-training strategies have been published. But aside from the improvements to the architecture or training procedures, the qualitative improvement in results is mainly the result of increasing the model size, alongside the amount of text corpora used to train them: Chinchilla (70B parameters) (Hoffmann et al., 2022), LaMDA (137B) (Thoppilan et al., 2022), GPT-3 (175B) (Brown et al., 2020), Gopher (280B) (Rae et al., 2021) and PaLM (540B) (Chowdhery et al., 2022). This fast-growing increase in model sizes and data is proven to surface new abilities in larger models, but not present in smaller models (Wei et al., 2022a).
The relationship between the size of the pretraining corpus and the performance of the language model in NLU tasks has been addressed in the literature before. The performance improves when the amount of data is increased (Zhang et al., 2021; Hu et al., 2020; Micheli et al., 2020; Raffel et al., 2020), although, at a certain point, the increase in performance slows down when the model size is kept fixed (Inoue et al., 2021; Martin et al., 2020; Micheli et al., 2020; Raffel et al., 2020; Liu et al., 2021). Furthermore, it is more convenient to improve the diversity of training datasets, than to add more text from the same domains (Inoue et al., 2021; Martin et al., 2020; Liu et al., 2021).
The correlation between model size and model performance on NLU tasks has also been analyzed. The performance of the model improves when scaling the model size (and FLOPs) (Turc et al., 2019; Raffel et al., 2020; Xia et al., 2022; Clark et al., 2022). However, all these works used very large pre-training datasets. They do not analyze if the increase in performance slows down bottlenecked by the pre-training dataset size and thus, the conclusion of scaling being always beneficial cannot be extended to low-data scenarios.
The works of Kaplan et al. (2020) and Hoffmann
et al. (2022), whose aim is aligned with this work, empirically study the optimal ratios of the training tokens, model parameters, and computation to train dense language models and infer scaling laws.
Kaplan et al. (2020) train models of a size ranging from 768 to 1.5 billion parameters with datasets ranging from 22 million to 23 billion tokens and conclude that LM performance improves smoothly as we increase the model size, dataset size, and amount of computation. They show that all three factors must be scaled up in tandem, to avoid bottleneck issues. Furthermore, they note that larger models are more sample-efficient, and that convergence is inefficient, suggesting that it’s better to under-train a bigger model than converge a smaller one on the same computing budget.
Hoffmann et al. (2022) find the optimal model size and the number of training tokens for given a fixed FLOPs budget. For this purpose, they draw their own scaling laws, based on the losses of over 400 models, ranging from 70M to 16B parameters, and trained on 5B to 400B tokens. They state that model size and the number of training tokens should scale equally, based on three alternative approaches, while Kaplan et al. (2020) extrapolates that every time the model size is increased by 8, the data only needs to be increased by 5. Thus, after concluding that the performance of most of the current large language models is bottlenecked by the undersized corpora, they train Chinchilla.
However, the scaling laws of Kaplan et al. (2020) and Hoffmann et al. (2022) are not useful for the low-resource settings we want to focus on. According to Kaplan et al. (2020) we need very small training corpora (e.g. 744K tokens for a BERTBASE , which is clearly not enough or optimal).
Hoffmann et al. (2022) infers significantly bigger training corpora: e.g. 3M tokens for a BERTMINI or 86M tokens for a BERTBASE . Current models for low-resource languages are trained with corpora around that range (Joshi et al., 2020): 161M for Irish (Barry et al., 2021), 130M for Luxembourgish (Lothritz et al., 2022), 45M for Galician (Vilares et al., 2021), 16M for Swahili (Martin et al., 2022b) and 4.4M for Quechua (Zevallos et al., 2022). However, increasing pre-training data several orders of magnitude has been proved beneficial for base size models (Liu et al., 2019).
Finally, those optimal scaling laws have been deduced from models trained over one epoch, while in low/medium-resource settings models are often
trained over several epochs (Martin et al., 2020; Lothritz et al., 2022; Zevallos et al., 2022).
Nonetheless, for certain NLU tasks (e.g. NeQA and Quote Repetition) scaling language models is detrimental (Perez and McKenzie, 2022), creating inverse scaling laws. However, Wei et al. (2022b) implies U-shaped scaling laws where even larger models might be able to solve those tasks that comprise a true and a distractor task.

heading: 3 Experimental Setup
text: We aim to find the optimal combination of modelsize, dataset-size and computing in low-resource environments and assess whether they follow the scaling laws established in the literature. In addition, we seek to find the minimum requirements to overcome computationally lighter neural baselines. Therefore, we carry out experiments for 3 corpus sizes and 3 model sizes, in 4 languages, training a total of 36 different models.

heading: 3.1 Language Selection
text: We conduct the experiments in four languages from different language families, selected among those that have enough monolingual data to train LMs, as well as enough available evaluation datasets for NLU tasks. Hence, the low-resource setting has been simulated in some cases. Among other languages that fulfil those criteria, we opted for Basque (eu), Spanish (es), Swahili (sw) and Finnish (fi). In addition to being part of disjoint language families, these languages are linguistically diverse with different complexities across morphology, syntax, verb system and vocabulary (Coloma, 2015) (see Appendix A).

heading: 3.2 Corpora
text: For each language, we created three corpora comprising 125M, 25M and 5M words, respectively. We limited the number of corpora sizes to three in order to control the number of experiments, and thus the computational resources needed.
Preliminary experiments showed a big fall in the results when reducing pre-training data to just 1M words, in consistency with Zhang et al. (2021). Since obtaining corpora of about 5M words is achievable by most languages that have annotated datasets (Joshi et al., 2020), we set the lower bound at 5M words. Zhang et al. (2021) shows that 10M to 100M words of pretraining data are enough for a language model to acquire the linguistic capacities
of syntax and semantics. Thus, we set the other two corpora sizes at 25M and 125M words, keeping a constant increase rate among them.
Regarding the nature of the texts, corpora for Basque and Spanish are a mix of 75% news and 25% text from Wikipedia. We selected the newspaper Berria1 for Basque, and El Pais2 for Spanish. Corpora for Swahili and Finnish were built by randomly selecting documents (longer than 10 sentences) from the web-crawled cc100 corpus (Conneau et al., 2020; Wenzek et al., 2020).

heading: 3.3 Models
text: In a similar fashion to (Turc et al., 2019), we employ three different variants of the BERT model, dubbed BERT124M 3, BERT51M and BERT16M . These models have 12, 8 and 4 layers respectively, also shrinking other parameters proportionally (hidden dimension, number of attention heads, etc.), since model shape does not affect performance significantly (Kaplan et al., 2020). Table 1 shows a detailed view of the parameters in each model. We also increased the vocabulary sizes from the original 30K subword tokens to 50K because it is beneficial for agglutinative languages (Agerri et al., 2020). We trained each model up to 500K steps with a batch of 256 and a sequence length of 512. For more pre-training details see Appendix B.

heading: 4 Evaluation Settings
text: We evaluate all models intrinsically and extrinsically. For the intrinsic evaluation, we tested the models on masked language modeling; for the extrinsic evaluation, we selected four NLU downstream tasks with available datasets in all the selected languages: Name Entity Recognition and Classification (NERC), Topic Classification (Topic), Sentiment Analysis (SA) and Questionanswering NLI (QNLI). Our selection of tasks in-
1https://www.berria.eus 2https://elpais.com 3This corresponds to BERTbase in Devlin et al. (2019)
cludes one sequence labeling task and three sequence classification tasks including sentiment analysis and QNLI, which are tasks that require a deeper NLU than the shallow linguistic tasks of NERC and Topic Classification (Zhang et al., 2021). Table 2 shows the details of each dataset.

heading: 4.1 MLM
text: Masked Language Modeling (MLM) is one of the default pre-training objective functions of BERT. We report both the loss and accuracy of MLM.
For this purpose, we created test datasets, from news sources not used for pre-training the models. For Basque we gathered texts from Argia4 news magazine. For Spanish, we opted for texts from the newspaper El Mundo 5. For Swahili, as a data source not used in the pre-training, we randomly selected a sub-corpus from the pre-train data for SwahBERT model (Martin et al., 2022a), which is mostly made up of news (%80). For Finnish we opted for a subset of cc100 not used in the pretraining, due to the lack of document-level news corpora available with an open license.

heading: 4.2 NERC
text: Named Entity Recognition and Classification (NERC) is a token classification task. For Basque, we used the in-domain NERC dataset from the BasqueGLUE benchmark (Urbizu et al., 2022). For Spanish, we opted for the Conll2002 dataset (Sang, 2002). For Swahili, we selected Masakhaner (Adelani et al., 2021). And lastly, for Finnish, we used FiNER (Ruokolainen et al., 2019). Each dataset has 4 categories, and we use the F-score as the performance metric.

heading: 4.3 Topic Classification
text: Topic classification is a sequence classification multi-class task. For Basque, we chose the
4www.argia.eus 5www.elmundo.es
BHTCv2 dataset including 12 thematic classes (Urbizu et al., 2022). The Spanish counterpart is MLdoc (Schwenk and Li, 2018) which has 4 classes. For Swahili, we employed Swahili: News Classification Dataset (David, 2020) which has 4 thematic classes. Since a development dataset split was missing, we randomly selected the 20% of the training split to create it. Furthermore, since the fine-tuning dataset is bigger than the smallest of the pre-training dataset (5M), we downsampled this training to 10K examples. And for Finnish, we selected the 10% version of the Yle corpus6, which contains 10 thematic classes. Performance is measured with the F-score score.

heading: 4.4 SA
text: Sentiment Analysis (SA) is a sequence classification task. For Basque, we employed the dataset BEC2016eu (Urbizu et al., 2022), which has positive, negative and neutral classes. InterTass2020 (Cumbreras et al., 2016) is the Spanish dataset selected for SA, which also has positive, negative and neutral classes. For Swahili, we utilized the dataset presented by Martin et al. (2022b). The dataset was mapped to polarity annotation following guidelines from the article: joy ([1]) = positive, disgust ([4]) = negative, neutral ([0]) and surprise ([5]) = neutral. Only examples with a single label were mapped. Original train/dev/test splits were maintained. And lastly, for Finnish, we chose Finnish sentiment7 which only contains positive and negative labels. We use F-score as the performance metric.

heading: 4.5 QNLI
text: Question-answering NLI (QNLI) is a sequence classification task. For Basque, we employed QNLIeu (Urbizu et al., 2022). And for Spanish, Swahili and Finnish, we adapted already available
6www.github.com/spyysalo/yle-corpus 7www.huggingface.co/datasets/sepidmnorozy/
Finnish_sentiment
conversational Question Answering (QA) datasets, into a sequence-pair binary classification task following the design of QNLI for English (Wang et al., 2019). The QA dataset selected were SQADes (Carrino et al., 2020), Tydiqasw and Tydiqafi (Clark et al., 2020). Each QNLI dataset has a QuestionSequence pair for entailment8. Tydiqa only provides splits for train and development. Thus, we used that development split as our test split, and randomly select some examples from the training set to create our development set. We follow the English QNLI design and use accuracy as the evaluation metric.

heading: 4.6 Systems and Baselines
text: For the extrinsic evaluation, we fine-tuned each of the 36 BERT models making use of the Transformers library (Wolf et al., 2020), with a lr of 3e−5, an effective batch size of 32 and training up to 10
8In the case of Basque and Spanish, those sequences are a single sentence, while the sequences for Finnish and Swahili are short paragraphs, because the QA datasets were annotated following different methodologies.
epochs9, which are considered default values (Devlin et al., 2019; Mosbach et al., 2020). For each task and language, we report an average of 5 runs.
In order to compare the performance of our models to other lighter approaches, we implement a competitive neural baseline based on contextual embeddings using Flair (Akbik et al., 2019). For sequence labeling tasks, embeddings are passed into a BiLSTM-CRF system based on the architecture proposed by (Huang et al., 2015). For text classification tasks, the computed Flair embeddings are fed into a BILSTM to produce a document-level embedding which is then used in a linear layer to make the class prediction.
We pre-train our own contextual Flair embeddings using the 125M corpora for each language with the following hyperparameters: Hidden size of 2048, sequence length of 250, a mini-batch size of 100 and 10 epochs. The rest of the training parameters are left in their default setting10.
In addition to LSTM-based neural baselines, we also include in the comparison a multilingual BERT
9Selecting the best-performing epoch on the development set.
10Each model took 80h to train on an Nvidia TitanV GPU.
model, namely mBERTbase (Devlin et al., 2019). We assume that this kind of multilingual language models will always be available and that they can somehow be an alternative in some low-resource settings, namely when limited resources refer to the computational capacity for pre-training and to the availability of enough text in the target language. In that line, we perform this comparison only for the Basque and Swahili languages which approximately include a training corpus size in mBERT no larger than those of the corpora used in our experiments (roughly 35M for eu and 11M for sw).

heading: 5 Results

heading: 5.1 Down-Scaling Laws
text: Figures 1-411 show the relation between the FLOPs and the MLM loss in the development dataset for the models in the four languages. Different colours stand for different model sizes (16M/51M/125M), and different symbols represent the pre-training data (5M/25M/125M). Each line is formed by 5 checkpoints (every 100K steps).
We can appreciate how the lowest loss is achieved by the biggest model with the most pre-training data, as expected from scaling laws, which dictate an improvement in performance when model-size, dataset-size and compute budget are increased simultaneously. Furthermore, the plot shows that increasing pre-training data is more beneficial than increasing the model size or amount of compute in this low-resource scenario; the gap between the loss obtained from increasing pre-training data is much bigger than the improvements obtained when increasing model-size or training steps.
The figures also show that models trained with small datasets yield larger MLM losses in development with further training (× curves), which we attribute to overfitting, as the training MLM loss does shrink as training advances. Big models with medium datasets (red and black ⋄ curves) also show the same tendency, although to a lesser extent. According to the figures, unless we use a dataset of at least 125M in the case of BERT51M and BERT124M , or a dataset of at least 25M in the case of the smallest BERT16, we should consider applying early stopping to our models to avoid overfitting, which corresponds with the first checkpoint of 100K steps we plotted in most cases.
11Zoomed in for BERT16M in the Appendix G.
Nevertheless, over-fitting issues during pretraining, do not have a direct impact neither on MLM accuracy nor on downstream tasks (see Appendix D). Thus, for the evaluation of the models regarding MLM accuracy (analysis available in Appendix C) and NLU downstream tasks (Section 5.2), we employed the final checkpoints at 500K steps.
Regarding languages, a comparison of the four figures (Figures 1-4) shows that the correlation between MLM loss and combinations of model-size, dataset-size and FLOPs is consistent across languages. MLM accuracies are also consistent across languages (Appendix C).
Hoffmann et al. (2022) estimates that 3M, 25M and 86M tokens are optimal to train BERT16M BERT51 and BERT124 respectively, while Kaplan et al. (2020) estimates much lower values. Our results show that the amount of data needed to train an LM optimally is no less than 25M words for BERT16M and 125M for BERT51 and BERT124. We carry out an in-depth comparison of these results with additional data in Appendix H.

heading: 5.2 Evaluation on NLU tasks
text: This section analyses the performance of our models on the NLU tasks listed in section 4, to measure the effect of model-size and pre-training data-size once finetuned on the downstream tasks.
The results for the NERC task are shown in Table 3. As expected, there is a clear positive correlation between the evaluation metric and the model and corpora size, but corpora size has slightly more impact on the performance. The results for topic classification at Table 4 follow the same trends, albeit with smoother differences. Table 5 presents the results for SA, again repeating the trends, but with a few outliers. Lastly, for QNLI (see Table 6), we observe there is a general trend of improving results while increasing dataset and model sizes. However, many results present large standard deviations, leading to several outliers12 that stand out from the general trend.
The models trained obtain competitive results, as shown by the results for NERC, topic classification and SA for Swahili, which are new SotA for those datasets to the best of our knowledge. Besides, some of the results obtained with the BERT124M and 125M words are comparable with SotA models trained over huge datasets (See appendix F).
12outliers marked in red

heading: 5.2.1 Evaluation vs Baseline Systems
text: Table 7 contains the results for the models trained with the corpora of 125M words, compared to the BiLSTM-CRF Flair baseline (trained with the same 125M corpora) and mBERT (for the languages with a comparable target-language pre-training corpus size). BERT models outperform the Flair neural baseline, but, depending on the evaluation dataset (task and language), the baseline is outperformed only by the BASE124M model or by all three model sizes. Furthermore, for some datasets, even the
BERT16M models trained with the smallest corpus (5M), not included in this table, outperform the Flair baseline (trained over a corpus of 125M words). Finally, computational costs (analysed in Section 5.3) ought to be a factor to consider and decide if the gain in performance is worth the increase in computational requirements.
The boost in performance when increasing the model size is larger in downstream tasks than in the MLM intrinsic task, particularly when shifting from the smallest BERT16M to the intermediate
BERT51M . This indicates that a larger model is better suited for fine-tuning, as the number of trainable parameters is also higher.
The results and scaling trends across languages are very consistent. The results and the trends we obtained are also consistent across different tasks, with the exception of QNLI, where results are volatile13 and have many outliers.
5.3 FLOPs and CO2 Emissions Table 8 shows the computational costs and CO2 emissions for each system for training, finetuning14 and inference. We calculated the FLOPs following the same method as Hoffmann et al. (2022). For non-transformer baselines, FLOPs were computed following (Zhang et al., 2018). CO2 emissions were estimated with Machine-Learning Impact calculator15 (Lacoste et al., 2019). The neural baseline based on Bi-LSTMs is lighter FLOP-wise, on pre-training, fine-tuning and inference time, even against the smallest BERT16M model. Still, the Flair baseline has higher CO2 emissions for finetuning, due to its inability to parallelize from the recurrent nature of the LSTMs.
If we revisit the results on MLM and NLU tasks (Sections 5.1 and 5.2) with computational costs in mind, we can say that if we only have a tiny corpus
13with an average standard deviation of 1.8 14Finetuning values are computed for a single run at Spanish
topic classification. 15https://mlco2.github.io/impact#compute
(5M token) available, the results obtained with a small model (BERT16M ) are on par with its bigger siblings at MLM, Topic, SA and QNLI, but not in NERC, where increasing the model size (up to BERT51M ) is needed to get competent results. In a scenario with a small dataset (25M), BERT16M would only obtain comparable results at topic classification and SA, but a BERT51M model obtains results as good as, or even better than BERT124M . Thus, we can opt for the BERT51M and use only half of the compute. However, if we are working with a pre-training dataset bigger than 125M, BERT124M obtains the best results by far, indicating that it is worth investing the compute needed to train such a model.
However, here we are comparing models of different sizes, trained for the same amount of steps. What would happen if we want the best model for a fixed computational budget? We answer that in Appendix E, where we compare the BERT51M and BERT124M , pre-trained on a comparable amount of computation. In line with (Kaplan et al., 2020), we conclude that it is better to undertrain a BERT124M than overtraining a BERT51M with the same amount of computation.

heading: 6 Conclusions
text: We present a study of the performance of language models in constrained settings, to analyze if the same scaling laws studied for large-language mod-
els apply to low-resource scenarios. We find out that the estimated values for optimal balance of model size and corpora size do not hold in these scenarios, and that pre-training tokens should be higher than the amount of model parameters.
From our experiments, we conclude that it is preferable to train big models on as much data as possible rather than using the computational power to further train smaller models. We see a clear trend where bigger models tend to quickly overfit when pre-trained for many epochs with small corpora. Still, even when they overfit in the pretraining stage, bigger models consistently outperform smaller models in downstream applications which require fine-tuning.
The experimental results are consistent among languages. Additionally, we empirically establish when the computational cost of using a Transformer-based approach is worth taking.
All the pre-training corpora, models and datasets created in this work are publicly available16.
Limitations
First of all, our study is limited to languages that use the Latin script. Still, the 4 languages are from different language families and are typologically diverse.
Secondly, the low-resource scenario is simulated. As mentioned in 3, in order to carry out the experiments the languages involved were required to have enough monolingual data to train LMs, as well as available evaluation datasets for NLU tasks.
The source of the pre-training corpora for Swahili and Finnish (cc100) is not completely comparable with the corpora used for Basque and Spanish (75% news, 25% Wikipedia), due to the unavailability of a large curated corpus for Swahili, and the lack of big news corpora for Finnish with an open license that allowed us to share freely the pre-training data.
16https://github.com/orai-nlp/low-scaling-laws
Our study is limited to 3 language model sizes and 3 pre-training corpora sizes. Including other model sizes like a BERT-Large or a model between 51M and 16M (where there is a big gap in results), and adding more pre-training corpora sizes (let’s say 625M and 1M words) were out of the scope of this work.
In addition, we use the default hyperparameters that are commonly used for BERT-base (BERT124M ) for the pre-training and fine-tuning of the BERT51M and BERT16M models without any hyperparameter tuning.

heading: Acknowledgements
text: This work has been partially funded by the Basque Government (ICL4LANG project, grant no. KK-2023/00094). It has also received funding from the following MCIN/AEI/10.13039/501100011033 projects: (i) DeepKnowledge (PID2021-127777OB-C21) and ERDF A way of making Europe and, (ii) DeepR3 (TED2021-130295B-C31) and European Union NextGeneration EU/PRTR. Rodrigo Agerri currently holds the RYC-2017-23647 fellowship (MCIN/AEI/10.13039/501100011033 and 63.34ESF Investing in your future). We also acknowledge the support of Google’s TFRC program.

heading: A Linguistic Characteristics of Selected Languages.
text: Table 9 shows the linguistic characteristics of the languages we selected for our experiments, which are Basque (eu), Spanish (es), Swahili (sw) and Finnish (fi). On one hand, we have the language families they belong to, and on the other hand, their complexity in morphology, syntax, verb system and vocabulary according to Coloma (2015).

heading: B Pre-Training Details
text: We use a cased sub-word vocabulary containing 50K tokens trained with the unigram language model based sub-word segmentation algorithm proposed by Kudo (2018). The vocabularies are learned from each training corpus with a character coverage of 99.95%, to ignore rare characters. Thus, we obtain 3 vocabularies for each language, one for each size of the pre-training corpora (5M, 25M, 125M), which are shared among LMs of different sizes throughout our experiments.
We apply several Maskings to the same sentences, to create different examples from the same text17, which is a common practice during the preprocessing of the pre-training data. We applied 10 different random maskings to each text and we employed whole-word masking, where whole words are masked instead of the sub-word units. All models were trained on TPUv3-8 machines using the same set of default hyperparameters (Devlin et al., 2019) in all model sizes: a learning rate 1e−4, β1 = 0.9, β2 = 0.999, L2 weight decay of 0.01, a learning rate warmup of 10K steps, and training the models for a total of 500K steps with a batch size of 256 and a sequence length of 512. This means that we will be doing many epochs (500/2500/12500K) over the same corpus, a common practice, when there is not an enormous pre-training corpus available, for instance, in the original publication of BERT (Devlin et al., 2019), the model is pre-trained for 40 epochs.
Although the models are trained for the same amount of steps and batch size, the time needed for training each of them is different, with larger models taking more time. We trained all our models using TPUv3-8 machines; in which we trained BERT124M models to 500K steps in 76 hours, BERT51M models in 32 hours and BERT16M models in 10 hours.
17I love cats: I love [MASK]; I [MASK] cats; [MASK] love cats

heading: C MLM Evaluation
text: Table 10 shows the accuracies obtained in the MLM task for each language (Basque, Spanish, Swahili and Finnish), for each model size and corpus size combination. As expected, larger models trained with the biggest corpora yield the best results, and a positive correlation exists between model/corpora size and accuracy in every language we compare. Moreover, results show that in overall, in these low-resource settings, it is preferable to increase the pre-training data over model size. Increasing pre-training data improve results on MLM for all languages, with the exception of the BERT16M model trained with the 125M token dataset. The gain obtained with the smallest BERT16M models as we keep adding training data diminishes, which suggests that performance is reaching a plateau in these models.
On the other hand, increasing the model size only helps once we reach a certain amount of pre-training data. Increasing model size from BERT16M to BERT51M does not improve MLM accuracy for a 5M corpus, suggesting that 3M nonembedding parameters are enough to absorb the knowledge of such a small dataset. However, increasing model size from BERT51M to BERT124M for the same 5M corpus does improve the overall performance for all languages except for Finnish, This might be due to larger language models being more sample-efficient (Kaplan et al., 2020).
Surprisingly, BERT51M outperforms BERT124M consistently across all languages when pre-trained with a 25M corpus; this goes against the intuition of larger models being more sample-efficient. Furthermore, the table shows that a slightly smaller model with more data can outperform a larger model with smaller corpora; every BERT51M model trained with 125M token corpora outperforms BERT124M model trained with 25M tokens.

heading: D Does Overfitting at Pre-training Propagate to Finetuning at Downstream Tasks?
text: The loss curves in Section 5.1 suggested that some model-dataset size ratios, which have the least data and more model parameters, have been trained for too long, to a degree in which the loss starts to increase significantly.
To analyze if those overfitting issues from when we keep pre-training over and over again on the same training data propagates to the downstream
tasks once finetuned, here we compare the checkpoints of the models at 100K steps, with the last checkpoint of our models at 500K steps. We did this comparison with 2 models, BERT16 and BERT51, both of them trained on the smallest corpora (5M words), which are among those with the most pronounced increasing loss curves.
The results for each checkpoint for both models after finetuning on the tasks are shown in Table 12. All in all, the 500K step checkpoints are on a par with the 100K step counterparts, without a clear winner, but definitely equalizing the gap that there is at MLM loss.
Thus, since the decline in loss when kept pretraining does not spread to the downstream tasks, we decide to employ the last checkpoints (500K steps), to evaluate and compare the models at Appendix C and Section 5.2, to avoid adding another variable to the evaluation.

heading: E Optimizing for a Fixed Budget
text: We have shown that increasing the amount of pretraining data and model size improves their performance. Thus, the conclusion regarding data in low-resourced settings is to use all the data there is available, independently of the model size.
With respect to the model size, however, even if the available corpus size suggests that increasing it improves the performance, there is usually a limited computational budget constraining this. Thus, we need to choose the best model size within our budget. Kaplan et al. (2020) concludes that convergence is inefficient, which means that we obtain optimal performance by training larger models and stopping significantly short of convergence when working with a fixed compute budget.
For this purpose, we compare the BERT51 and BERT124, pre-trained on a comparable amount of compute. We employed the pre-training corpus of 125M words, and pre-trained BERT51 for 500K steps, and BERT124 for 200K steps.
The results obtained are shown in Table 11. BERT124M , the model with the most parameters outperforms BERT51M in most of the tasks: 4/4 for Spanish, 3/4 for Swahili, 2/4 for Basque and 2/4 for Finnish. These results agree with the claim of Kaplan et al. (2020) that convergence being inefficient. However, since there is not a big gap in the results, other factors might be also considered. For example, an undertrained BERT124M model has more room for improvement with further pretraining, while BERT51M is cheaper and faster to finetune and deploy.

heading: F Comparison with SotA on Downstream Tasks
text: In Table 13 we compare the results of our BERT124M trained over the corpora of 125M words, the baselines of Flair, and mBERT with the current state-of-the-art results on each language and task. We improve SotA results for NERC, topic
classification18 and sentiment analysis for Swahili, and obtain similar results for topic classification for Finnish.

heading: G MLM Loss Plots Zoomed in for
text: BERT16M
Since the loss curve lines for BERT16M for the corpora of 25M and 125M tokens are hard to see in Figures 1, 2, 3 and 4, we zoomed in on them in the figures 5, 6, 7 and 8 respectively.
18Our model is finetuned with a subset of the dataset

heading: H Takeaways from Scaling Laws for Low-Resource Settings
text: In Tables 14-17 we compare our results to the predictions of previous scaling laws from Kaplan et al. (2020)19 and Hoffmann et al. (2022)20.
Tables 14-17 show the estimates of Kaplan et al. (2020) do not hold in this low-resource setting, by several magnitudes of order. Table 15 shows how
19a = 0.73 and b = 0.27 20a = 0.5 and b = 0.5
Hoffmann estimations for optimal model size are not that far from our results, but Table 17 suggests that the optimal dataset size required to train small language models is around an order of magnitude higger than what the scaling laws of Hoffmann et al. predict. Furthermore, Tables 14 and 16 show that the optimal FLOPs needed for those models are a few orders of magnitude higger than predicted by Hoffmann et al., where models are trained for a single epoch, which is clearly not optimal in lowresource settings.
All in all, we can underline the following takeaways for NLP practitioners working on LMs in low-resource settings:
• Use as much text as available.
• Pretraining for several (100s) epochs is clearly beneficial.
• Given a fixed computational budget, it is better to train big models instead of using the computational power to compute more model updates in smaller models.
• For a dataset of 125M words: BERT124M , trained for at least 3.37E+19 FLOPs21 is recommended.
• For a 25M dataset: BERT124M , trained for 5.39E+18 FLOPs22 is recommended, but a BERT51M model trained for 3.01E+18 FLOPs23, obtains similar results, and it is lighter for finetuning and inference.
• For a 5M dataset: BERT124M , trained for 1.35E+18 FLOPs24 or less is recommended, but a BERT51M model trained for 7.01E+17 FLOPs25, obtains similar results, which is lighter for finetuning and inference.
21500K steps with batch=256 and sequence-length=512 2280K steps with batch=256 and sequence-length=512 23150K steps with batch=256 and sequence-length=512 2420K steps with batch=256 and sequence-length=512 2535K steps with batch=256 and sequence-length=512
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations (after conclusions)
7 A2. Did you discuss any potential risks of your work? We are not aware of any potential risks of our work.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and introduction (1).
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: 3 Experimental Setup
text: B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3 Experimental Setup and 4 Evaluation Settings
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments?

heading: 5 Results and Appendix

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 3 Experimental Setup, 5 Results and Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 3 Experimental Setup and Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 3 Experimental Setup and 5 Results

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3 Experimental Setup, 4.6 Systems and Baselines and 5 Results
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: 34ESF Investing in your future). We also acknowledge the support of Google’s TFRC program