SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Exploring Widely Used Regularization Technologies We explore widely used regularization technologies, including dropout [26], droppath [9], label-smoothing [27], and weight decay [16], to alleviate the multi-epoch degradation observed under token-crisis. We present the results of the ablation study in Table 4. Our findings indicate that using dropout alone is highly effective in alleviating the multi-epoch degradation. Adding additional regularizations on top of dropout does not lead to further performance improvements. In fact, introducing weight decay can even make the training process unstable. This finding is actually good news for practitioners because dropout is easy to implement with model parallelism. Although we can see a slight improvement when using label-smoothing, we do not consider it as default because label-smoothing may introduce unforeseen issues for beam search. Regarding DropPath, its effectiveness may be limited in our case due to the
Figure 8: We train three Large-level models, including one model without dropout, one model using dropout from start, and one model using dropout after  $2^{15} \approx 32K$  steps (i.e., w/ dropout stage).
(a) Base-level models
(b) Large-level models
Figure 9: We set dropout as 0.0 or 0.1 when training at different scales with limited data.
(c) XL-level models
use of tensor parallelism [25] during pre-training. Introducing more communication across GPU or TPU cores to support DropPath may slow down the training process and reduce hardware utilization.
Insight (8): Dropout as an Effective yet Underutilized Regularization Technique in LLM An interesting observation is that most existing LLMs with over 10 billion parameters, such as GPT-3, PaLM, LLaMA, Chinchilla, and Gopher, do not utilize dropout as a regularization technique. We hypothesize that the reason is that using dropout can potentially slow down the model's learning process when an ample amount of training data is available, as demonstrated in Table 4. However, it is worth noting that Galactica [30] is an exception as they incorporate dropout in their training process. This could explain why they were able to successfully train a 120 billion-parameter model without experiencing overfitting, despite emphasizing the importance of data quality as a key factor in their work. Please note We have investigated the influence of data quality in Section 3.1 and found it to be less significant than we thought.
Insight (9): Gradual Integration of Dropout during Training In order to ensure that the model performs well throughout the entire training process, we explore an alternative approach where dropout is introduced only at a later stage of the training process. For the early stage of training, we do not use dropout. In Figure 8, we conducted an experiment using  $2^{27}$  tokens for  $2^{8}$  epochs, where a Large-level model was pretrained for a total of  $2^{19}$  steps. In the first  $2^{15}$  steps, we did not employ dropout, and for the remaining  $2^{19} - 2^{15}$  steps, we applied dropout with a rate of 0.1. The results show that the model with dropout introduced at a later stage performs comparably to the model with dropout from the beginning. Upon closer examination, we also observed that the model using dropout later outperforms the model with dropout from the start during the early stages of pre-training. These findings suggest that gradually integrating dropout during training can achieve comparable performance to using dropout from the beginning, while potentially offering some advantages in the initial phases of training. This approach allows for flexibility in the application of dropout, ensuring the model's performance is not compromised during the early training stage.
Insight (10): Dropout Performance at Different Model Scales While dropout has shown promising results at the Base-level, we aimed to investigate its effectiveness when scaling up the models. We conducted experiments using dropout with a rate of 0.1 and trained models with limited data for multiple epochs, as shown in Figure 9. The results clearly demonstrate that dropout can have a significant positive impact on performance across different scales of models. However, it is important to note that dropout is not a perfect solution, particularly when dealing with XL-scale model. Even with dropout applied, there is still a slight drop in validation accuracy at the later stages of training for the XL-level model. This suggests that while dropout can effectively mitigate overfitting and improve performance, it may face additional challenges when scaling up to larger models.
Figure 10: We first sweep dropout from 0.1 to 0.5 for Large-scale MoE and then sweep the same range for XL-scale Dense model to validate.
Figure 11: We compare the model trained with enough data and the models trained with limited data with and without dropout.
5 MoE Hyper-Parameter Tuning
We discovered that as we scale up, dropout may necessitate additional hyper-parameter tuning. However, conducting hyper-parameter tuning at a large scale can be extremely costly. For instance, in our specific scenario, training T5-XL five times would require approximately $37,000 USD for renting Google Cloud TPUs. Considering even larger models like PaLM and GPT-4, trained on even larger datasets, this cost becomes unmanageable. To address this issue and minimize the expense associated with hyper-parameter tuning, we leverage the insight that a Sparse MoE model can approximate the optimal hyper-parameters by predicting the behavior of a larger dense model.
Insight (11): Determining Optimal Hyper-parameters for Dense Models through MoE Sweeping We first validate the aforementioned insight regarding MoE behavior prediction still holds after incorporating dropout in Appendix I. Then, to identify the optimal hyper-parameters for dense models, we employed a two-step process. First, we conducted a sweeping analysis of the dropout rate in the Large-scale MoE model across the range of  $\{0.1, 0.2, 0.3, 0.4, 0.5\}$ . Subsequently, we performed the same sweeping procedure for the XL-scale Dense model to validate the accuracy of the dropout rate identified by the MoE model. As illustrated in Figure 10, both the Dense XL and MoE Large models exhibit nearly identical curves. Notably, they indicate that setting the dropout rate to 0.2 or 0.3 yields the optimal performance.
This discovery holds significant practical value for practitioners. The advantage of leveraging the MoE model for hyper-parameter tuning is evident. It requires considerably fewer computational resources compared to the dense model, despite possessing comparable parameters. This translates into substantial savings in computation resources for debugging and hyper-parameter tuning. For example, in this specific set of experiments, sweeping the MoE Large model incurred an expenditure of approximately 10.6K USD on the Google Cloud Platform. Conversely, training the Dense XL model only once required 7.4K USD. Consequently, the entire development process, including sweeping, amounted to a total cost of 18K USD, which is only 0.48 times the expense of directly tuning the Dense XL model. As we scale up to larger models and conduct more experiments, the potential of MoE Hyper-Parameter Tuning to conserve computational resources and reduce carbon emissions becomes increasingly promising for future endeavors.
Final Performance after Dropout Sweep Having determined the appropriate dropout rate based on the MoE model, we proceed to scale up the models accordingly. Figure 11 illustrates the outcomes of this scaling process. Notably, we only used around  $2^{27}$  tokens, which should be only able to train a 16M T5 model according to Chinchilla scaling law. However, we can see the model can still improve when scaling to 2.8B parameters, i.e., over  $1700 \times$  larger than the 16M model. We believe this is a significant result for such a simple method, i.e., introducing an appropriate dropout rate via MoE Hyper-Parameter Tuning.
In this study, we investigated the token-crisis problem and thoroughly explored various approaches to training LLMs with repeated tokens. Our investigation covered what is token-crisis, what would happen if we use repeated data under token crisis, why there is multi-epoch degradation when repeating data, and how can we alleviate this issue with off-the-shelf approaches. We also demonstrated the effectiveness of using MoE to predict the behavior of more computationally expensive dense models, offering a valuable means of accelerating LLM development more broadly.
Yang You's research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ByteDance grant, ARCTIC grant, SMI grant and Alibaba grant. We also thank the TPU computing resources from Google TRC (TPU Research Cloud) grant.
A Frequent Asked Questions
We list the potential frequent asked questions and the point-to-point answers as follows:
A.1 Selection of Encoder-Decoder T5 Model and C4 Dataset
Firstly, the C4 dataset is a widely studied large-scale pre-training dataset that has been open-sourced. It is relatively easier to prepare and maintain compared to the dataset used in LLaMA. T5 was originally proposed and developed in conjunction with the C4 dataset. Therefore, using the T5 model with the C4 dataset aligns with established practices and facilitates comparability with prior research.
Furthermore, while decoder-only architectures have been predominant in existing Language Models (LLMs), it is still unclear whether decoder-only models consistently outperform encoder-decoder architectures. Recent research by Tay et al. [29] demonstrated that encoder-decoder architectures exhibit superior performance at the 20B scale. Additionally, the specific architecture employed by OpenAI's GPT-4 remains unknown to external researchers. Consequently, there is ongoing debate and uncertainty regarding the superiority of decoder-only models.
In reality, the difference between decoder-only and encoder-decoder architectures may not be as significant as initially perceived. Both architectures utilize an autoregressive decoder, and the main distinction lies in determining which tokens are fed into the encoder and which ones are fed into the decoder. Another difference arises in multi-turn dialogue systems, where encoder-decoder architectures may require recomputation of certain activations.
A.2 Lack of Significant Improvement of UL2 in Table 3
The lack of significant improvement of UL2 over the vanilla T5 model in Table 3 can be attributed to a specific difference in our implementation. Unlike the original UL2 implementation by Tay et al. [29], we did not utilize dropout in our experiments. By removing dropout from the UL2 training process, it is possible that we experienced a performance drop, leading to the comparable results observed between UL2 and the vanilla T5 model in our experiments.
A.3 Reason for Not Training a Larger Scale Model (10B+)
While training a larger-scale model like  $10\mathrm{B}+$  parameters would indeed provide valuable insights, the primary reason for not doing so is the limitation of computational resources. Conducting a comprehensive set of experiments covering various aspects of the token-crisis issue requires substantial computational power, which includes significant financial costs, time requirements, and carbon emissions. Therefore, in order to balance these factors and optimize our research efforts, we decided to scale up to approximately 3B parameters to explore the token-crisis problem within the available resource constraints.
A.4 Performance Drop Without Enough Data Even If We Are Using Dropout
Even when employing a suitable dropout rate, there is still a substantial gap between models trained with full tokens (without dropout) and those trained with a limited number of tokens through data repetition. This discrepancy arises because, for the purpose of clarity, we repeated only a small number of tokens (i.e.,  $2^{27}$ ) across multiple epochs ( $2^{8}$  epochs). In more typical scenarios, such as repeating 1T tokens for 10 epochs, the observed gap would probably be smaller.
B.1 Multi-Epoch LLM Pre-training
The topic of training LLMs for multiple epochs only received little attention in existing literature. Hoffmann et al. [8] suggested that training LLMs for multiple epochs may have detrimental effects. On the other hand, Tay et al. [29] trained a 120B model for 4 epochs without observing multi-epoch degradation, although they attribute their success primarily to the high-quality data used. biderman et al. [2] found that deduplicating pre-training data had no clear benefit on language modeling performance. Additionally, Hernandez et al. [7] discovered that repeating a small fraction of data during LLM pre-training can significantly harm model performance.
In contrast to these works, our study focuses specifically on the token-crisis problem and investigates the consequences of further scaling LLMs by repeating a fixed amount of data multiple times. To the best of our knowledge, ours is the first paper to explore the token-crisis and train LLMs for multiple epochs. Many of the insights we present, such as using Mixture-of-Experts (MoE) models to predict the behavior of more computationally expensive dense models, are novel and valuable contributions to our research community.
B.2 Pre-training with Synthetic Data
Existing research has examined the concept of pre-training LLMs with synthetic data as a means of mitigating data scarcity. For instance, Ri et al. [23] designed artificial languages with structural properties that mimic natural language, while Wu et al. [33] successfully pre-trained LLMs through simpler synthetic tasks. Other works, such as TAPEX [15], address the challenge of data scarcity in specific domains.
In contrast, our paper focuses on training LLMs in the context of the token-crisis problem with multiple epochs, which is distinct from the perspective of using synthetic data. However, we believe this line of research is relevant because it has the potential to alleviate the token-crisis in LLMs, thereby addressing the associated challenges and limitations.
Scalability to State-of-the-Art Models: Although we conducted experiments on models with 3B parameters, we acknowledge that we did not explore the performance of our approach on state-of-the-art scale models, such as the 175B-parameter GPT-3. The primary reason for this limitation is the lack of computational resources required for running experiments on very large models multiple times. Given the token-crisis scenario and the need for extensive experimentation, conducting experiments on larger models remains prohibitively expensive.
Dataset Quality Assumptions: Our Insight (4) relies on using C4 as low-quality data and Wikipedia as high-quality data. While we acknowledge that the data quality of C4 is acceptable, we adopted this setting because many existing LLMs have utilized Wikipedia for pre-training for at least one epoch, and this choice is supported by claims of better data quality. It is worth noting that other datasets with different characteristics may yield different results when applied to our approach.
Limitations of Insight (6) in Handling FLOPs Disparities: Our Insight (6) may encounter challenges when dealing with significant gaps in FLOPs between models. For example, we found that the Base-level MoE model with 64 experts in every MoE layer cannot perfectly predict the behavior of an XL-level dense model, although it still exhibits a higher susceptibility to overfitting compared to the Base-level MoE model with 16 experts. This observation aligns with the notion that very sparse models may have under-trained parameters, leading to imperfect predictions. Further investigation is necessary to address this limitation and enhance the accuracy of Insight (6) when confronted with substantial FLOPs disparities.
D Ablation Study on Batch Size of Larger Dataset Size
Figure 12: We fix the batch size and number of repeats, and then train the model with 4 times more steps to go through 4 times more tokens.
We can see when using smaller batch size than what we did in Figure 4, model still has little multi-epoch degradation. This further verifies that larger dataset can alleviate the multi-epoch degradation.
E Ablation Study on Number of Experts for MoE
Figure 13: we use relatively limited tokens to train three T5-MoE models with 4, 16, 64 experts at each MoE layer. We can observe MoE with more experts, which have more trainable parameters have a more serious multi-epoch degradation.
To further investigate the behavior of MoE models, we train three T5-MoE models with 4, 16, and 64 experts using a relatively limited number of tokens<sup>6</sup>. In Figure 13, we observe that as the number of trainable parameters increases with more experts, the MoE models experience more severe multi-epoch degradation. These findings indicate that the additional parameters in MoE models are indeed leading to faster memorization of the training data.
F Encoder-Decoder vs Decoder-Only
We can see encoder-decoder is clearly better than decoder-only but the MoE based decoder-only model having comparable trainable parameters with encoder-decoder performs almost the same as
$^6$ Following the convention in [38], the number of experts represents the number of expert FFNs in each MoE layer. In our case, we use one MoE layer every 4 layers in both the encoder and decoder, resulting in a total of 6 MoE layers in the T5 MoE Base model. For more details about the MoE hyperparameters, please refer to Appendix J.
Figure 14: We train three models (i.e., encoder-decoder, decoder-only, MoE-based decoder-only) with the same data (C4) and training objective (Span-Corruption).
encoder-decoder model. Therefore, as suggested by UL2 paper, the different behaviours of encoder-decoder and decoder-only are more from the training objective instead of model architecture. That is the reason why we explore UL2 training objective in our Section 3.3.
G Ablation Study on Batch Size of Larger Dataset Size
Figure 15: We fix the batch size and number of repeats, and then train the model with 4 times more steps to go through 4 times more tokens.
We can see when using smaller batch size than what we did in Figure 15, model still has little multi-epoch degradation. This further verifies that larger dataset can alleviate the multi-epoch degradation.
H Downsteam Evaluation
Table 5: The updated fine-tuning results. We include the standard deviation of 5 runs.
Model	BoolQ	RTE	SQuAD
Acc	Acc	EM	F1
C4-SpanCorr-Repeat1	77.0±0.7	68.9±0.8	82.3±0.9	90.0±0.9
C4-SpanCorr-Repeat28	73.8±1.0	66.4±0.3	80.0±0.6	88.1±0.5
C4-UL2-Repeat1	78.8±0.9	72.2±1.2	82.3±0.8	90.2±0.7
C4-UL2-Repeat28	74.5±0.6	68.4±0.7	79.3±1.1	87.3±1.2
C4-Wiki-Repeat1	74.2±0.7	69.3±1.1	82.6±0.8	90.1±0.6
C4-Wiki-Repeat28	71.4±0.4	64.9±0.5	79.6±1.0	87.9±1.3
I Verification of MoE Behavior Prediction with Dropout
To validate the aforementioned insight regarding MoE behavior prediction, we conducted an examination incorporating dropout, as depicted in Figure 16. Remarkably, our findings reveal that the MoE model maintains a remarkably similar training behavior to that of the dense model, even when utilizing additional computation resources and possessing comparable parameters. This observation
Figure 16: We add dropout to train two models with comparable trainable parameters, i.e., Large-level Dense model and Base-level MoE.
further reinforces the effectiveness of the MoE model in approximating the behavior of larger, denser models.
J Default Hyper-parameters
Table 6: Default pre-training hyper-parameters.
Name	Value
Learning Rate	0.01
Learning Rate Decay	Square Root Decay (0.8)
Optimizer	Adafactor
Batch Size	128
Training Steps	524288 (219)
Dropout	0.0
DropPath	0.0
Label Smoothing	0.0
Weight Decay	0.0
Table 7: Default MoE hyper-parameters.
Name	Value
Num of Experts	16
MoE Layer Layout	Every fourth
Train Capacity Factor	1.25
Test Capacity Factor	2.0
Num Selected Experts	2
Router Weight	1e-2
Z-loss Weight	1e-4
We follow T5 1.1 implementation in T5x and ST-MoE implementation in Flaxformer. The default hyper-parameters are shown in Table 6 and Table 7. The pre-training parameters in Table 6 are used in both dense and MoE model training.
We conduct experiments on Google Cloud TPU. For Base-level and Large-level models, we use 32 TPU v3 cores, and 128 TPU v3 cores are employed to train XL-level models.
K Model Configurations
Table 8: Default MoE hyper-parameters.
Scale	# Enc Layers	# Dec Layers	# Heads	Hidden Dim	MLP Dim	#Params
Small	8	8	6	512	1024	78M
Mid	10	10	10	640	1280	140M
Base	12	12	12	768	2048	247M
Base Plus	16	16	14	896	2560	432M
Large	24	24	16	1024	2816	783M
Large Plus	24	24	22	1408	4096	1.4B
XL	24	24	32	2048	5120	2.8B
For Base, Large, XL scale, we follow the default setting in T5x. To obtain a more smooth scaling curve, we add more configurations like Small, Mid, Base Plus and Large Plus. Please note the XL scale was not used to draw the scaling curve in Figure 2 because of the expensive training cost.
L More Results
In this section, we share more other approaches we explored.
L.1 Training with Larger or Smaller Batch
Empirically, larger batch size is easier to overfit, and smaller batch size can usually reduce overfitting. We found this widely-used commonsense still holds in LLM token-crisis. However, we think this trick is not that useful because training LLM with smaller batch size is inefficient.
L.2 Training with Longer or Shorter Sequence
We train the same amount of tokens with longer or shorter sequences with the same total token usage. That is, for longer sequence, we train for fewer steps and more steps are used to train with shorter sequences. We found using longer sequences can achieve better performance.
L.3 Randomly Activating One Embedding Layer from Multiple Embedding Layers
We suggested the over-fitting can be alleviated when adding more randomness into input embeddings. We therefore init multiple embedding layers and randomly activate one of these embedding layers for each input sequence. However, we did not observe this design can alleviate multi-epoch degradation even if we regularize different embedding layers have different embeddings.
L.4 Adding Dropout Before and After Linear Layers
Considering dropout is very effective, we tried to add dropout before and after each linear layer. Adding dropout after one layer means a column-wise weight masking. Adding dropout before and after one layer denotes element-wise weight masking. We found such regularization is too strong and makes LLM achieve inferior performance even if we set a small dropout rate on the XL-level model.