SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fuzhao Xue $^{1}$  Yao Fu $^{2}$  Wangchunshu Zhou $^{3}$  Zangwei Zheng $^{1}$  Yang You $^{1\dagger}$ $^{1}$ National University of Singapore  
 $^{2}$ University of Edinburgh  
 $^{3}$ ETH Zurich
Recent research has highlighted the importance of dataset size in scaling language models. However, large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is likely to be approaching its scaling limit for LLMs. To further enhance LLMs, a straightforward approach is to repeat the pre-training data for additional epochs. In this study, we empirically investigate three key aspects under this approach. First, we explore the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting, leading to multi-epoch degradation. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives, while less influential factors consist of dataset quality and model FLOPs. Finally, we explore whether widely used regularization can alleviate multi-epoch degradation. Most regularization techniques do not yield significant improvements, except for dropout, which demonstrates remarkable effectiveness but requires careful tuning when scaling up the model size. Additionally, we discover that leveraging mixture-of-experts (MoE) enables cost-effective and efficient hyper-parameter tuning for computationally intensive dense LLMs with comparable trainable parameters, potentially impacting efficient LLM development on a broader scale.
Large Language Models (LLMs) have demonstrated remarkable performance on various NLP tasks [14, 21], and have even become a part of our daily lives through applications such as ChatGPT and Bard. This success has been largely attributed to scaling up transformer-based language models, as evidenced by recent work [10, 20, 28]. In the early stages of transformer scaling, researchers observed that larger models could achieve comparable performance with smaller models using fewer training steps and less pre-training data [10], leading to early views that model size might be one of the most critical factors in achieving better performance.
Dataset size is more important than we thought in LLM scaling. Recent work [8] found that the pre-training dataset size plays a more significant role than previously thought and proposed compute-optimal scaling (i.e., Chinchilla scaling law), where model size and training dataset size should be scaled equally for optimal performance given a fixed computation budget. For instance, an under-trained larger model like Gopher-280B [20] can be outperformed by a well-trained smaller model like Chinchilla-70B if not enough data is used in larger model training. The intuition here is that the decreased model size can be compensated by the increased size of data. The effectiveness of the Chinchilla scaling law is further validated by the recent success of LLaMA-65B [31] and PaLM-2 [1].
†Correspondence to youy@comp.nus.edu.sg
Figure 1: Modeling number of tokens in stock and number of tokens required by training compute-optimal LLM.
Insufficient tokens hinder LLM scaling. State-of-the-art LLMs require vast amounts of internet-scale text data for pre-training, such as the 780 billion tokens used for PaLM-540B [3] and the 1.4 trillion tokens used for Chinchilla-70B [8]. However, this raises two critical questions: (1) how many tokens are needed to fully train SoTA LLMs, and (2) how many tokens are available for pre-training? To answer these questions, we modeled token requirements using Chinchilla scaling law and estimated the scale of potential high-quality pre-training data based on recent research [32]. Unfortunately, as shown in Figure 1, the growth rate of high-quality text data on the internet is much slower than the growth rate of data required by LLMs. For instance, to fully pre-train PaLM-540B, 10.8 trillion tokens would be needed, but the total stock of high-quality text data is only around 9 trillion tokens. Moreover, the high-quality text data is growing at a rate of  $4 - 5\%$  per year, in line with the world economy, which is much slower than the pace of LLM growth and hardware improvement. According to recent study [32], high-quality text data may not suffice the requirements of scaling LLMs and in a pessimistic scenario, and we may run out of new data between 2023 and 2027. In light of the compute-optimal scaling study, this may already have occurred. Therefore, data may be becoming a more significant bottleneck for scaling transformers than hardware. In this paper, we refer to this problem as the "token-crisis".
In addition, it is important to note that the pre-training dataset size prediction discussed above is based on the compute-optimal training proposed in [8], which only considered the training cost of LLMs and ignored the inference cost. However, given that LLMs are often used as a service, such as in the case of Bard and Bing, and perform a significant amount of inference every day, it is crucial to consider the inference cost in compute-optimal modeling as well. Therefore, in order to achieve the best performance with the least computation cost per sample, it is likely that LLMs will require even more data than we estimated in Figure 1. This further emphasizes the importance of making full use of the off-the-shelf high-quality text data for LLM pre-training.
The token-crisis is even more severe when it comes to non-English data. According to the Web Technology Surveys<sup>3</sup>, English content makes up over  $56\%$  of the web, with non-English data from over 100 languages comprising only  $44\%$  of the total. This long-tailed distribution of data makes it much harder for LLMs to perform well on non-English tasks. Despite PaLM's impressive 540B parameters and training on 780B tokens [3], including  $22\%$  non-English data, it still lags behind models such as mT5 [37] and ByT5 [36] on non-English tasks like Multilingual QA. With native English speakers making up just  $5\%$  of the world's population, achieving comparable performance on non-English tasks is highly desirable from fair access and democratizing LLMs perspectives.
Using pre-training data repeatedly. To alleviate the token-crisis, one straightforward approach is training LLM for multiple epochs. The practice of multi-epoch training varies in subfields of machine learning. Although there exist models like Vision Transformers [6] that are typically trained for many epochs (e.g., 300 epochs on ImageNet [24]), LLMs are often trained for only one or a few epochs [3, 8, 31]. Currently, it is unclear what multiple epochs mean for language model pretraining: while Hoffmann et al. [8] suggest that training LLM with repeated tokens can be harmful to performance, Taylor et al. [30] observed improvements when training a 120B model for 4 epochs. Therefore, although training with repeated data may seem superficially simple, it has a nontrivial influence on practitioners and therefore further investigation is needed to determine its effects.
Contributions and Insights This paper presents a systematic empirical study of repeating pretraining data for token-crisis problem, making it the first work of its kind. We summarize 11 insights
<sup>3</sup>https://w3techs.com/technologies/overview/content_language
Figure 2: We observe a linear relationship between the number of tokens required to fully train the model and the number of model trainable parameters, which validates that the Chinchilla scaling law still holds when training T5 on C4 dataset.
Figure 3: We train models at three different scales (i.e., T5-Base, T5-Large, T5-XL) with different dataset sizes but the same amount of total computation budget (i.e., batch size 128, sequence length 512, training steps  $2^{19}$  (around 524K).
in three aspects. First, we investigate what would happen when training with repeated pre-training data in Section 2 and found (1) Encoder-Decoder $^4$  model on C4 dataset is comparably data-hungry as stated in Chinchilla scaling law; (2) Larger models are more prone to overfitting when training with repeated data.
We study three components (i.e., data, model, objectives) in Section 3 to explore the key factors contributing to multi-epoch degradation. We found (3) Training LLM with a larger dataset for multiple epochs can alleviate the multi-epoch degradation; (4) The use of high-quality data<sup>5</sup> does not mitigate multi-epoch degradation. (5) The number of parameters plays a crucial role in multi-epoch degradation, even when the computation budget is fixed. The effect of FLOPs on this issue is negligible; (6) The Mixture-of-Experts transformer can even be employed to predict the behavior of dense models that have comparable parameters but require much more computation; (7) Utilizing a mixture of training objectives, such as UL2 [29], can accelerate LLM learning, but it also leads to faster memorization, resulting in worse multi-epoch degradation.
We then investigate whether off-the-shelf regularization technologies can alleviate multi-epoch degradation in Section 4. We found (8) While most existing transformer regularization techniques struggle, dropout proves to be highly effective, despite its infrequent usage in LLM pre-training; (9) Dropout can be introduced only at the later stage of pre-training (after a few epochs) to ensure faster learning at the early stage of pre-training; (11) When scaling to very large models, dropout requires additional tuning.
We finally make use of insight (6) to alleviate the challenge introduced by insight (10). That is our final insight (11): The MoE model can serve as a more affordable and faster alternative to fine-tune hyper-parameters (e.g., dropout rate) of large dense models with comparable parameters but significantly more FLOPs. We think this approach has great potential to have a broader impact on efficient LLM development.
2 What Are the Consequences of Repeating Pre-training Data?
In this study, we adopt T5 1.1 as our default pre-training configuration. This means that, unless specified otherwise, we utilize the C4 dataset [21] along with the identical model architecture, training objectives, and hyper-parameters as described [21]. Detailed hyper-parameters can be found in Appendix J.
Insight (1): Training T5 on C4 is Data-Hungry In this section, our focus is to investigate the effects of training a LLM for multiple epochs under token-crisis conditions. Before delving into that, it is essential to examine whether we have a similar data-hungry observation like Chinchilla scaling
4We verified Encoder-Decoder is actually not that different from Decoder-only model in Appendix F
5The data quality here is relative. For instance, we think the quality of C4 is low when comparing with Wikipedia but C4 is good when comparing with C4 unclean. In addition, we limit the quality discussion here to web-scale pre-training data. High-quality instruction tuning data is not in this scope.
Table 1: We finetune pre-trained T5 checkpints on SQuAQ[22] dataset. The C4 Top-1 Acc denotes the masked token prediction accuracy on C4 validation set before fine-tuning. On SQuAD, we report both Exact Match (EM) and F1 score.
Model	C4	SQuAD
Val Acc	EM	F1
T5 Base (235 tokens, 1 epoch)	64.6	82.4	90.0
T5 Base repeat 28 (227 tokens, 28 epochs)	61.7 (-2.9)	79.9 (-2.5)	88.1 (-1.9)
Figure 4: We repeatedly use  $2^{27}$  and  $2^{29}$  tokens for  $2^{8}$  times by using batch size 128 and 512 for  $2^{18}$  steps.
Table 2: We pre-train two T5 models on a subset of C4 and a subset of Wikipedia with the same computation budget. Both of these two subsets have around i.e.,  $2^{27}$  tokens. We then finetune these two checkpoints on SQuAD and report the Exact Match (EM) and F1 score.
Dataset	SQuAD
Acc	EM	F1
C4 (235 tokens)	82.4	90.0
C4 (227 tokens)	79.9 (-2.5)	88.1 (-1.9)
Wikipedia (235 tokens)	82.4	89.9
Wikipedia (227 tokens)	79.4 (-3.0)	87.6 (-2.3)
law in a widely-used open-sourced setting, specifically training an encoder-decoder transformer on the C4 dataset. To assess this, we follow [8] and train models with six different configurations. For detailed configurations, please refer to Appendix K. We then compare the validation accuracy for masked token prediction at various computation budgets. When a larger model outperforms a smaller model, it indicates that the smaller model has received sufficient tokens. The number of tokens used to train the smaller model can then be considered as the token requirement for full training. Figure 2 illustrates our findings, showing a linear relationship between the number of tokens required and the model size. Overall, our results indicate that the Chinchilla scaling law holds true when training T5 with the C4 dataset.
Insight (2): Multi-epoch Degradation As suggested in Figure 1, we anticipate encountering token scarcity issues as we continue to scale. Consequently, our next investigation revolves around training LLMs with repeated data. To explore this, we randomly select several subsets of the C4 dataset containing approximately  $2^{35}$ ,  $2^{29}$ , and  $2^{27}$  tokens, resulting in each token being repeated 1,  $2^{6}$ , and  $2^{8}$  times, respectively. The results, as presented in Figure 3, demonstrate the expected performance degradation when training LLMs with repeated tokens. Furthermore, we observe that larger models are more susceptible to overfitting under token-crisis conditions. Specifically, when trained without a sufficiently large dataset, T5-XL, despite consuming more computational resources, performs worse than T5-Large having access to  $4 \times$  data ( $2^{29}$  vs  $2^{27}$  tokens).
Downstream performance check Given that fine-tuning LLMs allows us to unlock additional capabilities, such as following instructions and aligning with human behavior [19], it is crucial to assess whether the pre-training degradation also adversely affects downstream tasks. Considering that the fine-tuning dataset is typically smaller in scale, we perform fine-tuning on SQuAD [22] dataset using pre-trained checkpoints. The results, presented in Table 5, indicate that the token-crisis experienced during pre-training indeed has a detrimental impact on downstream tasks. For instance, the model trained with  $2^{27}$  tokens, despite achieving a validation set score of 2.9 points in pre-training, experiences a drop of 1.9 points in F1 score for the downstream task.
3 What Are the Key Factors Contributing to Multi-EPOCH Degradation?
Insight (3): Dataset Size In Figure 3, we investigated the impact of dataset size and the number of token repeats while keeping the total computation budget fixed. To further explore the importance of dataset size, we conducted another experiment where we fixed the number of token repeats and varied the dataset size. Specifically, we repeatedly used  $2^{27}$  and  $2^{29}$  tokens for  $2^{8}$  times, corresponding to training the model with a batch size of 128 and 512 for  $2^{18}$  steps. As depicted in Figure 4, we observed a significant overfitting phenomenon when training with  $2^{27}$  tokens for  $2^{8}$  epochs. However, when using  $2^{29}$  tokens for the same number of training steps, the model did not experience degradation. Since we changed both batch size and number of tokens, to ensure a more fair comparison, we conduct another set of ablation study by fixing the batch size in Appendix G. These results indicate that employing a larger dataset can alleviate the issue of multi-epoch degradation.
Insight (4): Dataset Quality Taylor et al. [30] successfully trained a 120B model on 106B tokens for 4 epochs. They suggest that dataset quality may be a key factor to avoid overfitting, although they did not conduct experiments to validate this hypothesis. As suggested in [3, 31], we assume Wikipedia dataset [5] is our high-quality dataset. To perform a fair comparison with the C4 dataset, we sampled approximately  $2^{27}$  tokens from Wikipedia and trained the model for  $2^{19}$  steps. Due to the differences in pre-training data usage, we directly compare the performance on a downstream task. As presented in Table 2, the model trained on a subset of Wikipedia exhibits a similar level of degradation to the model trained on a subset of C4. This indicates that the high-quality pre-training data from Wikipedia does not alleviate the issue of multi-epoch degradation. Certainly, the low quality here is relative. For extremely low-quality data like C4 with cleaning, the data quality will probably harm the performance.
Insight (4) & (5): Decoupling Parameters and FLOPs Scaling the foundation model is a crucial aspect, but it is unclear which factor plays a more significant role in multi-epoch degradation. During the scaling process, both the number of parameters and the computational cost increase. To disentangle the effects of these two factors, we introduce Mixture-of-Experts (MoE) [13] and parameter sharing (ParamShare) [4] to increase or decrease parameters with comparable computation cost. MoE allows for a substantial increase in the number of parameters while maintaining comparable FLOPs per sample. We implement a T5 MoE model based on ST-MoE [38], where an MoE layer is added every 4th transformer block, with each MoE layer consisting of 16 experts. On the other hand, ParamShare reduces the number of parameters while keeping the FLOPs fixed. Following ALBERT [12], we create a T5 ParamShare model with 6 layers of trainable parameters in both the encoder and decoder. We reuse each trainable transformer block twice, resulting in the same FLOPs as the vanilla T5 model but with only around  $0.5 \times$  the number of parameters.
Figure 6: Comparing the overfitting trend of dense model and MoE model with comparable parameters but different FLOPs.
To analyze the effects of parameter variation while maintaining comparable FLOPs, we use ParamShare with  $0.5 \times$  parameters, vanilla model with  $1 \times$  parameters and MoE with  $2 \times$  parameters. In Figure 5a, we observe that the model with less parameters is less influenced by the usage of repeated tokens, indicating a reduced impact of multi-epoch degradation. On the other hand, the MoE model with more trainable parameters is highly prone to overfitting when trained on limited data. To further investigate the behavior of MoE models, we conduct ablation study on the number of experts in Appendix E. We found MoE models are less data-efficient and more data-hungry [11, 17, 34, 35]. While MoE can be a beneficial inductive bias when sufficient data is available, caution should be exercised when using MoE models under token-crisis or in low-resource language scenarios.
To investigate the influence of the computation budget, we fix the number of trainable parameters and vary the FLOPs by conducting experiments with three different configurations: (1) None of the 12 base-level transformer layers in both the encoder and decoder are shared. (2) We still use 12 layers of trainable parameters but reuse each layer twice, resulting in a model with  $2 \times$  the FLOPs compared to the baseline. (3) We further increase the computation by using 6 wider trainable layers (referred to as large-level transformer layers) and reusing each layer four times. This configuration results in a model with  $4 \times$  the FLOPs compared to the baseline but with comparable parameters. The results, as shown in Figure 5b, indicate that although the model with more computation achieves slightly better performance when scaling the FLOPs alone, we do not observe a clear increase in degradation.
Insight (6): Using MoE to Predict the Behavior of Larger Dense Models The previous findings regarding MoE models are relatively negative. However, we made an interesting observation when comparing a large-level dense model and a base-level MoE model. Despite having a comparable number of trainable parameters (784M vs. 701M), the MoE model has only around  $0.39 \times$  the FLOPs and  $2.1 \times$  the throughput. Surprisingly, these two models exhibit almost the same overfitting trend, as shown in Figure 6a. To further investigate this observation on a larger scale, we compare an XL-level dense model with a large-level MoE model. Similar to the previous finding, these models, with comparable parameters (2.4B vs. 2.8B), exhibit a similar overfitting trend, despite the large-level MoE model having only  $0.32 \times$  the FLOPs and  $3.8 \times$  the throughput.
We believe that this finding is significant. In the era of large models, training a model at the final large scale is extremely expensive. Therefore, being able to predict the behavior of larger models using smaller and more cost-effective models, as stated in GPT-4 [18], is highly desirable. With our finding, we can accurately predict the behavior of larger models using sparse MoE models with significantly lower carbon emissions. We provide an example of how this finding can be utilized in Section 5.
3.3 Training Objective
Insight (7): Can Diverse Training Objectives Alleviate Multi-EPOCH Degradation? We investigate whether diverse training objectives can improve models from different aspects and alleviate the token-crisis. Specifically, we study the UL2 training objective proposed by Tay et al. [29], which is a mixture of widely used pre-training objectives (e.g., PaLM 2), including next token prediction and masked language modeling. UL2 covers the two most important training objectives in LLM
Figure 7: We compare T5 and UL2 with the same training hyper-parameters under two different settings, i.e., using enough pre-training data once and repeating limited data for multiple epochs.
Table 3: We pre-train T5 and UL2 with limited data for multiple epochs. We then fine-tune these models on SQuAD to check the larger gap of UL2 pre-training performance indeed has a more negative impact on downstream tasks.
Dataset	SQuAD
Acc	EM	F1
MLM w/o repeat	82.4	90.0
MLM w/ repeat	79.9 (-2.5)	88.1 (-1.9)
UL2 w/o repeat	82.5	90.1
UL2 w/ repeat	79.6 (-2.9)	87.6 (-2.5)
Table 4: We conduct an ablation study on how widely used tricks, i.e., dropout [26], droppath [9], label-smoothing [27], and weight decay [16] alleviate token-crisis.
Limited Data	Dropout	DropPath	Label-Smoothing	Weight Decay	Val Acc
X	X	X	X	X	64.5
X	✓	X	X	X	63.6
✓	X	X	X	X	61.7
✓	✓	X	X	X	62.9
✓	✓	✓	X	X	63.0
✓	✓	X	✓	X	62.6
✓	✓	X	X	✓	Nan
pre-training. Similar to the previous experiments, we use  $2^{27}$  tokens for  $2^{8}$  epochs. However, since UL2 pre-training objective is more challenging than the objective used in vanilla T5, it is unfair to directly compare the pre-training validation masked token prediction accuracy. Therefore, we report downstream results on the SQuAD dataset for reference. It is important to note that we use the same hyper-parameters as T5 for a fair comparison.
The results, summarized in Figure 7 and Table 3, compare the vanilla masked language modeling (MLM) objective in T5 with the UL2 training objective using the same training hyper-parameters. We examine both scenarios of using enough pre-training data and using limited data for multiple epochs. Although we cannot directly compare the validation accuracy of T5 and UL2, it is evident that UL2 is more prone to overfitting and exhibits a more pronounced multi-epoch degradation. For instance, when using the vanilla masked language modeling objective on the base-level model, the performance does not drop during training. However, with the UL2 objective, the validation accuracy starts to decline early on in the pre-training phase. Moreover, in the downstream evaluation (Table 3), the performance drop of UL2 is larger than that of vanilla T5. However, it is important to highlight that, although UL2 shows negative results in our token-crisis setting, it actually verifies the effectiveness of the UL2 training objective. The findings in this section indicate that UL2 can accelerate the model's learning process, which is precisely the key aspect of a well-designed LLM pre-training objective.
4 Can Regularization Alleviate Multi-EPOCH Degradation?