To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis
Authors: Fuzhao Xue
Yao Fu
Wangchunshu Zhou
Zangwei Zheng
Yang You
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language Model
Transformer Scaling
Foundation Model Pre-training
TL;DR: We empirically investigate repeating pre-training data for additional epochs to enhance LLMs, when high-quality text data on the web is approaching its scaling limit for LLMs.

Abstract: Recent research has highlighted the importance of dataset size in scaling language models. However, large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is likely to be approaching its scaling limit for LLMs. To further enhance LLMs, a straightforward approach is to repeat the pre-training data for additional epochs. In this study, we empirically investigate three key aspects under this approach. First, we explore the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting, leading to multi-epoch degradation. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives, while less influential factors consist of dataset quality and model FLOPs. Finally, we explore whether widely used regularization can alleviate multi-epoch degradation. Most regularization techniques do not yield significant improvements, except for dropout, which demonstrates remarkable effectiveness but requires careful tuning when scaling up the model size. Additionally, we discover that leveraging mixture-of-experts (MoE) enables cost-effective and efficient hyper-parameter tuning for computationally intensive dense LLMs with comparable trainable parameters, potentially impacting efficient LLM development on a broader scale.

To Repeat or Not To Repeat:
Insights from Scaling LLM under Token-Crisis
Fuzhao Xue
Yao Fu²
Wangchunshu Zhou²
Zangwei Zheng'
Yang You If
National University of Singapore
University of Edinburgh
3ETH Zurich
Abstract
Recent research has highlighted the importance of dataset size in scaling language models. However, large language models (LLMs) are notoriously coken-hungry during pre-training, and high-quality text data the web is likely to be approaching scaling limit for LLMs. To further enhance LLMs, a straightforward approach is to repeat the pre-training data for additional epochs. In this study, we empirically in- vestigate three key aspects under this approach. First, we explore the consequences of repeating pre pre-training data, revealing that the model is susceptible to overfitting, leading to multi-epoch degradation. Second, we examine the key factors contribut- ing multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives, while less influential factors consist dataset quality and model FLOPs. Finally, explore whether widely used regu- larization can alleviate multi-epoch degradation. Most regularization techniques do yield significant improvements, except for dropout, which demonstrates remarkable effectiveness but requires careful tuning when scaling up the model size. Additionally, discover that leveraging mffexereoof-experts (MoE) enables cost-effective and efficient pper-parameter tuning for computationally intensive dense LLMs with comparable trainable parameters, potentially impacting efficient LLM development on broader scale.
1 Introduction
Large Language Models (LLMs) have demonstrated remarkable performance on various NLP tasks 14 and have even become part of our daily lives through applications such as CharGPT and Bard. This success has been largely attributed scaling up transformer-based language models, as evidenced by recent work [10) 20 28). In the early stages of transformer scaling, researchers observed that larger models could achieve comparable performance with smaller models using fewer training steps and less pre- pre-training data [10), leading to early views that model size might be one of the most critical factors in achieving better performance.
Dataset size is more important than we thought in LLM scaling, Recent work [8] found that the pre-training dataset size plays more significant role than previously thought and proposed compute-optimal scaling (i.e. Chinchilla scaling law), where model size and training dataset size should be scaled equally for optimal performance given fixed computation budget. For instance, an under-trained larger model like Gopher-280B (20) can be outperformed by well-trained smaller model like Chinchilla-70B if not enough data is used in larger model training. The intuition here that the decreased model size can be compensated by the increased size of data. The effectiveness
of the Chinchilla scaling law is further validated by the recent success LLaMA-65B (3 and PaLM-2 (].
'Correspondence to youy @ comp.nus.cdee edu.cd
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

10t3
PaLM Gooper
GPT-3
Number Tokens
10
Turing NLG
10
GPT-2
1010
_BETT
Tokens in Stock
GPT-1
#Tokens Required
10
2018
2019
2020
2021
2022
2023
Year
Figure 1: Modeling number of tokens in stock and number of tokens required by training compute- optimal LLM.
Insufficient tokens hinder LLM scaling, State-of-the-art LLMs require vast amounts of internet- scale text data for pre-training. such as the 780 billion tokens used for PaLM-540B (3) and the 1.4 trillion tokens used for Chinchilla-70B [8]. However, this raises two critical questions: (1) how many tokens are needed to fully train SoTA LLMs, and (2) how many tokens are available for pre- pre-training? To answer these questions, we modeled token requirements using Chinchilla scaling law and estimated the scale of potential high-quality pre-training data based on recent research [32] Unfortunately, as shown in Figure the growth rate of high-quality text data the internet is much slower than the growth rate of data required by LLMs. For instance, to fully pre-train PaL.M-500B, 10.8 trillion tokens would be needed, but the total stock of high-quality text data is only around trillion tokens. Moreover, the high-quality text data is growing at a rate of 4-5% per year, in line with the world economy, which is much slower than the pace of LLM growth and hardware improvement. According to recent study (32). high-quality text data may not suffice the requirements of scaling LLMs and in pessimistic scenario, and we may run out of new data between 2023 and 2027. In light of the compute-optima scaling study, this may already have occurred. Therefore, data may be becoming a more significant bottleneck for scaling transformers than hardware. In this paper, we refer to this problem as the "token-crisis".
In addition, it is important note that the pre-training dataset size prediction discussed above is based on the compute-optimal training proposed in [8]. which only considered the training cost of LLMs and ignored the inference cost. However, given that LLMs are often used as a service, such as in the case of Bard and Bing, and perform a significant amount of inference every day, it is crucial to consider the inference cost in compute-optines best performance with the least computation cost per sample, it is likely that LLMS will require even more data than we estimated in Figure II This further emphasizes the importance of making full use of the off-the-shelf high-quality text data for LLM pre-training.
modeling as well. Therefore, in order to achieve the
The token-crisis is even more severe when it comes to non-English data. According to the Web Technology S English content makes up over 56% of the web, with non-English data from over 100 languages comprising only 44% of the total. This long-tailed distribution of data makes it much harder for LLMs to perform well on non-English tasks. Despite PaLM's impressive 540B parameters and training on 780B tokens (3). including 22% non-English data, it still lags behind models such as mT5 [37] and ByT5 (36] on non-English tasks like Multilingual QA. With native English speakers making up just 5% of the world's population, achieving comparable performance on non-English tasks is highly desirable from fair access and democratizing LLMs perspectives.
Using pre-training data repeatedly. To alleviate the token-crisis, straightforward approach is training LLM for multiple epochs. The practice of multi-epoch training varies in subfields of machine learning. Although there exist models like Vision Transformers (6) that are typically trained for many epochs (e.g., 300 epochs on ImageNet (24)). LLMs are often trained for only one or a few epochs [3] (1). Currently, it is unclear what multiple epochs mean for language model pretraining: while Hoffmann el al. (8] suggest that training LLM with repeated tokens can be harmful to performance, Taylor el al. [30] observed improvements when training a 120B model for 4 epochs. Therefore, although training with repeated data may seem superficially simple, it has a nontrivial influence on practitioners and therefore further investigation is needed to determine its effects.
Contributions and Insights This paper presents a systematic empirical study of repeating pre- training data for token-crisis problem, making it the first work of its kind. We summarize 11 insights

---

Num of Tokens Required (Billion)
Base
80
Large
MLM Accuracy (%)
XL
64
40
62
20
60
200 400
600
800
50
100 150
200
250
Num Parameters (Million
Num Data Repeat
Figure 2: We observe a linear relationship between the number of tokens required to fully train the model and the number of model trainable parameters, which validates that the Chinchilla scaling law still holds when train- ing T5 on C4 dataset.
Figure 3: We train models at three different scales (i.e., T5-Base, T5-Large, T5-XL) with different dataset sizes but the same amount of total computation budget (i.e., batch size 128, sequence length 512, training steps 2 (around 524K).
in three aspects. First, we investigate what would happen when training with repeated pre-training data in Section 2 and found (1) Shccoder-Decoded model on C4 dataset is comparably data-hungry as stated in Chinchilla scaling law; (2) Larger models are more prone to overfitting when training with repeated data.
We study three components (i.e., data, model, objectives) in Section to explore the key factors contributing to multi-epoch degradation. We found (3) Training LLM with a larger dataset for multiple epochs alleviate the multi-epoch degradation; (4) The of high-quality data does does mitigate multi-epoch degradation. (5) The number of parameters plays a crucial role in multi-epoch degradation, even when the computation budget is fixed. The effect of FLOPs on this issue is negligible: (6) The cexture-of-Experee transformer can even be employed to predict the behavior of dense models that have comparable parameters but require much more computation; (7) Utilizing mixture of training objectives, such as UL2 (29], can accelerate LLM learning, but also leads to faster memorization, resulting in worse multi epoch degradation.
We then investigate whether off-the-shelf regularization technologies can alleviate multi-epoch degradation in Section 4 We found (8) While most existing transformer regularization techniques struggle, dropout proves to be highly effective, despite its infrequent usage in LLM pre-training: (9) Dropout can be introduced only at the later stage of pre-training (after few epochs) to ensure faster learning at the carly stage of pre-training; (11) When scaling to very large models, dropout requires additional tuning.
We finally make use of insight (6) to alleviate the challenge introduced by insight (10). That is final insight (11): The MoE model serve more affordable and faster alternative to fine-tune parameters (e.g., dropout rate) of large dense models with comparable parameters but significantly more FLOPs. We think this approach has great potential have a broader impact on efficient LLM development.
2 What Are the Consequences of Repeating Pre-training Data?
In this study, adopt T5 1.1 default pre-training configuration. This means that, unless specified otherwise, we utilize the C4 dataset (21) along with the identical model architecture, training objectives, and typer-parameters as described (21) Detailed hyper-parameted can be found in Appendix[]
Insight (1): Training T5 on C4 is Data-Hungry In this section, our focus is to investigate the effects of training a LLM for multiple epochs under token-crisis conditions. Before delving into that, essential to examine whether we have similar data-hungry observation like Chinchilla scaling
"We verified Encoder-Decoder is actually not that different from Decoder-only model in Appendix F "The data quality here is relative. For instance, we think the quality of C4 is low when comparing with Wikipedia but C4 is good when comparing with C4 unclean. In addition, we limit the quality discussion here to web-scale pre-training data. High-quality instruction tuning data is not in this scope.
3

---

Table 1: We finetune pre-trained T5 checkpints on SQuAQ 22] dataset. The C4 Top-1 Ace denotes the masked token prediction accuracy on C4 validation set before fine-tuning. On SQuAD, we report both Exact Match (EM) and Fl score.
Model
C4
SQuAD
Val Acc
F1
T3 Base (2 tokens, 1 epoch)
64,6
82.4
90.0
T5 Basc repeat 2 (2? tokens, 2 epochs)
61.7 (-2.9)
79.9 (-2.5)
88.1 (-1.9)
Table 2: We pre-train two T5 models on a subset of C4 and subset of Wikipedia with the same computation budget. Both of these two subsets have around i.e. 2 tokens. We then finetune these two checkpoints on SQuAD and report the Exact Match (EM) and Fl score.
MMM Accuracy (%0
50
40
Dataset
SQuAD
1x bsz, 1x tokens
30
4x bsz, 4x tokens
Acc
EM
F1
3 Num Steps
2
C4 (235 tokens)
82.4
90.0
tM
C4 (227 tokens)
79.9 (-2.5)
88.1 (-1.9)
Figure 4: We repeatly use 2 and 2 to- kens for 2 times by using batch size 128 and 512 for 21 steps.
Wikipedia (235 tokens)
82.4
89.9
Wikipedia (2 tokens)
79.4 (-3.0)
87.6 (-2.3)
law in a widely-used open-soureed setting, specifically training an encoder-decoder transformer on the C4 dataset. To assess this, we follow [8] and train models with six different configurations. For detailed configurations, please refer to Appendix K We then compare the validation accuracy for masked token prediction at various computation budgets. When larger model outperforms smaller model, it indicates that the smaller model has received sufficient tokens. The number of tokens used to train the smaller model can then be considered as the token requirement for full training. Figure 2 illustrates our findings, showing a linear relationship between the number tokens required and the model size. Overall, results indicate that the Chinchilla scaling law holds true when training T5 with the C4 dataset.
Insight (2): Multi-epoch Degradation As suggested in Figure | we anticipate encountering token scarcity issues as we continue to scale. Consequently, our next investigation revolves around training LLMs with repeated data. To explore this, we randomly select several subsets of the C4 dataset containing approximately 235 229 and 227 tokens, resulting in each token being repeated 1, 28, and 2 times, respectively. The results, as presented in Figure demonstrate the expected performance degradation when training LLMSs with repeated tokens. Furthermore, we observe that larger models are more susceptible overfitting under token-crisis conditions. Specifically, when trained without sufficiently large dataset, T5-XL, despite consuming more computational resources, performs worse than T5- having access to 4x data (2 vs 2 tokens).
Downstream performance check Given that fine-tuning LLMs allows us to unlock additional capabilities, such as following instructions and aligning with human behavior [19), it is crucial to assess whether the pre- pre-traimin training degradation also adversely affects downstream tasks. Considering that the fine-tuning dataset is typically smaller in scale, we perform fine-tuning on SQuAD (2) dataset using pre-trained checkpoints. The results, presented Table indicate that the token-crisis experienced during pre-training indeed has a detrimental impact on downstream tasks. For instance, the model trained with 227 tokens, despite achieving validation set score of 2.9 points in pre-training, experiences a drop of 1.9 points in Fl score for the downstream task.

---

60
MLM Accuracy (%)
MLM Accuracy (%)
50-
0,5x params w/o repeat
50-
1x FLOPs w/o repeat
0.5x params w/ repeat
1x FLOPs w/ repeat
40
1x params w/o repeat
2x FLOPS w/o repeat
1x params w/ repeat
2x FLOPs w/ repeat
2x params w/o repeat
4x FLOPs w/o repeat
30
2x params w/ repeat
4x FLOPs w/ repeat
Num Steps
Num of Steps
(a) We use comparable FLOPs and change the num- ber of trainable parameters by using PhramShare with 0.5x parameters, vanilla model with 1x pa- rameters and MoE with 2x parameters.
(b) We fix the number of trainable parameters and change FLOPs by increasing the parameter sharing and re-using of each trainable transformer block.
The dash lines mean this model is trained with enough tokens for epoch (i.e., repeated data usage). The solid lines mean that these models are trained with limited data for multiple epochs.
Figure 5:
What Are the Key Factors Contributing to Multi-Epoch Degradation?
3.1 Data
Insight (3): Dataset Size In Figure we investigated the impact of dataset size and the number of token repeats while keeping the total computation budget fixed. To further explore the importance of dataset size, we conducted another experiment where we fixed the number of token repeats and varied the dataset size. Specifically, we repeatedly used 227 and 229 tokens for 2 times, corresponding to training the model with a batch size of 128 and 512 for 2 steps. As depicted in Figure we observed significant overfitting phenomenon when training with 22 tokens for 2 epochs. However, when using 22 tokens for the same number of training steps, the model did not experience degradation. Since we changed both batch size and number of tokens, to ensure a more fair comparison, we conduct another set of ablation study by fixing the batch size in Appendix G These results indicate that employing a larger dataset can alleviate the issue of multi-epoch degradation.
Insight (4): Dataset Quality Taylor et al. (30) successfully trained 120B model on 106B tokens for 4 epochs. They suggest that dataset quality may be key factor to avoid overfitting, although they did not conduct experiments to validate this hypothesis. As suggested in [3] 31), we assume Wikipedia dataset (5) is our high-quality dataset. To perform fair comparison with the C4 dataset, we sampled approximately 2 tokens from Wikipedia and trained the model for 2 steps. Due to differences in pre-training data usage, we directly compare the performance on a downstream task. As presented in Table the model trained on subset of Wikipedia exhibits similar level of degradation to the model trained on a subset of C4. This indicates that the high-quality pre-training data from Wikipedia does not alleviate the issue of multi-epoch degradation. Certainly, the low quality here is relative. For extremely low-quality data like C4 with cleaning, the data quality will probably harm the performance.
3.2 Model
Insight (4) & (5): Decoupling Parameters and FLOPs Scaling the foundation model is crucial aspect, but it is unclear which factor plays a more significant role in multi-epoch degradation. During the scaling process, both the number of parameters and the computational cost increase. To disentangle the effects of these two factors, we introduce Soxtuureoff-Exprrr (MoE) [13) and parameter sharing (ParamShare) [4] increase or decrease parameters with comparable computation cost. MoE allows for a substantial increase in the number of parameters while maintaining comparable FLOPs per sample. We implement T5 MoE model based on ST-MoE (38). where an MoE layer is added every 4th transformer block, with each MoE layer consisting of 16 experts. On the other hand, ParamShare reduces the number of parameters while keeping the FLOPs fixed. Following ALBERT (12), we create T5 ParamShare model with 6 layers of trainable parameters in both the encoder and decoder. We reuse each trainable transformer block twice, resulting the same FL.OPs as the vanilla T5 model but with only around 0.5 the number of parameters.
5

---

MUM Acouracy (%)
MLM Accuracy
50
50
05
45
40-
Derise Large
Dense XL
30
MoE
McE
3
Steps
Steps
(a) We compare Large-level dense model and Base-level MoE model. Both models have sim- ilar trainable parameters (784M vs 701M), but the MoE model has significantly reduced FLOPs (0.39x) and improved throughput x2.1x).
(b) We compare XL-level dense model and Large-level MoE model. Both models have similar trainable parameters (2.8B vs 2.4B). but the MoE model has significantly reduced FLOPs (0.32x) and improved throughput (3.8x).
Figure 6: Comparing the overfitting trend of dense model and MoE model with comparable parameters but different FLOPs.
To analyze the effects of parameter variation while maintaining comparable FLOPs, ParamShare with 0.5x parameters, vanilla model with 1x parameters and MoE with 2x parameters. In Figure 5a| observe that the model with less parameters is less influenced by the usage of repeated tokens, indicating a reduced impact of multi-epoch degradation. On the other hand, the MoE model with more trainable parameters is highly prone to overfitting when trained on limited data. To further investigate the behavior of MoE models, we conduct ablation study on the number of experts in Appendix E We found MoE models are less data-efficient and more data-hungry [I] 17 34 35
While MoE can be a beneficial inductive bias when sufficient data is available, caution should be exercised when using MoE models under token-crisis in low-resource language scenarios.
To investigate the influence of the computation budget, we fix the number of trainable parameters and vary the FLOPs by conducting experiments with three different configurations: (1) None of the 12 base-level transformer layers in both the encoder and decoder are shared. (2) We still use 12 layers of trainable parameters but reuse each layer twice, resulting in model with 2x the FLOPs compared to the baseline. (3) We further increase the computation by using 6 wider trainable layers (referred to as levee-level transformer layers) and reusing each layer four times. This configuration results in model with x the FLOPs compared to the baseline but with comparable parameters. The results, as shown in Figure [b] indicate that although the model with more computation achieves slightly better performance when scaling the FLOPPs alone, we do not observe clear increase in degradation.
Insight (6): Using MoE to Predict the Behavior of Larger Dense Models The previous findings regarding MoE models are relatively negative. However, we made an interesting observation when comparing large-level dense model and base-level MoE model. Despite having comparable number of trainable parameters (784M vs. 701M), the MoE model has only around 0.39x the FLOPs and 2.1x the throughput. Surprisingly, these two models exhibit almost the same overfitting trend, as shown in Figure 6a To further investigate this observation on a larger scale, we compare an XL-level dense model with large-level MoE model. Similar the previous finding, these models, with comparable parameters (2.4B vs. 2.8B), exhibit a similar overfitting trend, despite the large-level MoE model having only 0.32x the FL.PPs and 3.8x the throughput.
We believe that this finding is significant. the era of large models, training a model at the final large scale is extremely expensive. Therefore, being able to predict the behavior of larger models using smaller and more cost-effective models, as stated in GPT-4 [18], is highly desirable. With our finding, accurately predict the behavior of larger models using sparse MoE models with significantly lower carbon emissions. We provide an example of how this finding can be utilized in Section 5
3.3 Training Objective
Insight (7): Can Diverse Training Objectives Alleviate Multi-Epoch Degradation? We investigate whether diverse training objectives can improve models from different aspects and alleviate the token- crisis. Specifically, we study the UL.2 training objective proposed by Tay et al. 229 which is a mixture of widely used pre-training objectives (e.g., PaL. 2), including next token prediction and masked language modeling. UL2 covers the two most important training objectives in LLM

---

60
Table 3: We pre-train T5 and UL2 with limited data for multiple epochs. We then fine-tune these models on SQuAD to check the larger gap of UL2 pre-training performance indeed has a more nega- tive impact on downstream tasks.
MLM Accuracy (%)
40
MLM w/o repeat
MLM repeat
UL2 w/o repeat
Datasct
SQuAD
UL2 w/ repeat
Acc
EM
F1
Num of Stops
MLM w/o repeat
82.4
90.0
Figure 7: We compare TS and UL2 with the same training syper-parameters under two different settings, i.e., using enough pre- pre-training data once and repeating limited data for multiple epochs.
MLM w/ repeat
79.9 (-2.5)
88.1 (-1.9)
UL2 w/o repeat
82.5
90.1
UL2 w/ repeat
79.6 79.6(-2.9)
87.6 (-2.5)
Table 4: We conduct an ablation study on how widely used tricks, i.e. dropout (26). droppath ( label-smoothing (27). and weight decay (16) alleviate token-crisis.
Limited Data
Dropout
DropPath
Smoothing
Weight Decay
Val
x
64.5
/
X
X
X
63.6
X
X
AXX
X
X
61.7
AYXIK
X
62.9
X
EXX
63.0
X
62.6
Nan
pre-training. Similar to the previous experiments, we use 227 tokens for 2 epochs. However, since UL2 pre-training objective is more challenging than the objective used in vanilla T5, it is unfair to directly compare the pre-training validation masked token prediction accuracy. Therefore, we report downstream results the SQuAD dataset for reference. is important note that srper-parameters as T5 for a fair comparison.
the same
The results, summarized in Figure 7 and Table compare the vanilla masked language modeling (MLM) objective in T5 with the UL2 training objective using the same training ayperpparametrs. We examine both scenarios of using enough pre-training data and using limited data for multiple epochs. Although we cannot directly compare the validation accuracy of T5 and U1.2, it is evident that UL.2 pronounced multi-epoch degradation. For instance,
more prone overfitting and exhibits when using the vanilla masked language modeling objective on the base-level model, the performance does drop during training. However, with the UL2 objective, the validation accuracy starts to decline early on in the pre-training phase. Moreover, in the downstream evaluation (Table 3), the performance drop of UL2 is larger than that of vanilla T5. However, is important to highlight that, although UL2 shows negative results in our token crisis setting, it actually verifies the effectiveness of the UL2 training objective. The findings in this section indicate that UL2 can accelerate the model's learning process, which is precisely the key aspect of well-designed LLM pre-training objective.
4 Can Regularization Alleviate Multi-Epoch Degradation?
Exploring Widely Used Regularization Technologies We explore widely used regularization technologies, including dropout [26). droppath (9). label-smoothing (27). and weight decay [16]. alleviate the multi-epoch degradation observed under token-crisis. We present the results of the ablation study in Table 4 Our findings indicate that using dropout alone is highly effective in alleviating the multi-epoch degradation. Adding additional regularizations on top of dropout does not lead to further performance improvements. In fact, introducing weight decay can even make the training process unstable. This finding actually good news for practitioners because dropout easy to implement with model parallelism. Although we can see slight improvement when using label-smoothing, we do not consider as default because label-smoothing may introduce unforeseen issues for beam search. Regarding DropPath, its effectiveness may be limited in case due to the
7

---

60
MLM Accuracy (%)
50
40
w/o dropout
dropout
30
w/ di dropstage
Num of Steps
Figure 8: We train three Large-level models, including one model without dropout, model using dropout from start, and one model using dropout after 2 ze 32K steps (i.e., w/ dropstage).
2.3
HUM Accurac (96)
2.50
50
P50
MLM
Base wio drapout Bise wi dropout
10
Large wib dropout
Large dropout
30-
XL WiO dRODOLr
XL WI drspout
Num of Steps
Num of Steps
Num Steps
(a) Base-level models
(b) Large-level models
(c) XL-level models
Figure 9: We set dropout as 0.0 or 0.1 when training at different scales with limited data.
use of tensor parallelism 25 during pre-training. Introducing more communication across GPU or TPU cores to support DropPath may slow down the training process and reduce hardware utilization.
Insight (8): Dropout as an Effective yet Underutilized Regularization Technique in LLM An interesting observation is that most existing LLMs with over 10 billion parameters, such as GPT-3, PaLM, LLaMA, Chinchilla, and Gopher, do not utilize dropout a regularization technique. We hypothesize that the reason is that using dropout can potentially slow down the model's 's learning process when an ample amount of training data is available, as demonstrated in Table However, it is worth noting that Galactica (30] is an exception as they incorporate dropout in their training process. This could explain why they were able to successfully train a 120 bilioo--arammete model without experiencing overfitting, despite emphasizing the importance of data quality as a key factor their work. Please note We have investigated the influence of data quality Section 3.1 and found it to be less significant than we thought.
Insight (9): Gradual Integration of Dropout during Training In order to ensure that the model performs well throughout the entire training process, we explore an alternative approach where dropout is introduced only a later stage of the training process. For the early stage of training, we do not use dropout. In Figure 8 we conducted an experiment using 227 tokens for 2 epochs, where Large-level model was pretrained for total of 219 steps. In the first 215 steps, we did not employ dropout, and for the remaining 29 - 215 steps, we applied dropout with a rate of 0.1. The results show that the model with dropout introduced at a later stage performs comparably to the model with dropout from the beginning. Upon closer examination, we also observed that the model using dropout later outperforms the model with dropout from the start during the early stages of pre-training. These findings suggest that gradually integrating dropout during training can achieve comparable performance to using dropout from the beginning, while potentially offering some advantages the initial phases of training. This approach allows for flexibility in the application of dropout, ensuring the model's performance is not compromised during the early training stage.
Insight (10): Dropout Performance at Different Model Scales While dropout has shown promising results at the Base-level, we aimed investigate its effectiveness when scaling up the models. We conducted experiments using dropout with rate of 0.1 and trained models with limited data for multiple epochs, as shown in Figure [ The results clearly demonstrate that dropout can have significant positive impact on performance across different scales of models. However, it is important to note that dropout is not a perfect solution, particularly when dealing with XL-scale model. Even with dropout applied, there is still slight drop in validation accuracy at the later stages of training for the XI-leeeel model. This suggests that while dropout can effectively mitigate overfitting and improve performance, it may face additional challenges when scaling up larger models.
8

---

70.0
Dense MoE Large
65
67.5
65.0
Dropout Rate
64-
MLM Accuracy
60.0
63
57.5
55.0
2 tokens dropout
2d tokens wio dropout
62
52.5
tokens wi dropout
0.3
0.5
50.0
2
MLM Accuracy (%)
Params
Figure 10: We first sweep dropout from 0.1 to 0.5 for Large-scale MoE and then sweep the same range for XC-se Dense model to validate.
Figure 11: We compare the model trained with enough data and the models trained with limited data with and with- out dropout.
5 MoE Slyper-Parameter Tuning
We discovered that we scale up, dropout may necessitate additional ppprr-parrmeter tuning. However, conducting in our specific scenario, training T5-XL five times would require approximately $37,000 USD for renting Google Cloud TPUs. Considering even larger models like PaLM and GPT-4, trained on even larger datasets, this cost becomes unmanageable. To address this issue and minimize the expense associated with hyper-parameter tuning, we leverage the insight that a Sparse MoE model can approximate the optimal
tuning at a large scale can be extremely costly. For instance,
by predicting the behavior of a larger dense model.
Insight (11): Determining Optimal Hy paraaeeesss for Dense Models through MoE Sweeping We first validate the aforementioned insight regarding MoE behavior prediction still holds after incorporating dropout in Appendix D Then, to identify the optimal hyper-parametts models, employed two-step process. First, conducted sweeping analysis of the dropout rate in the Large-scale MoE model across the range of {0.1, 1, 0.2, 0.3, 0.4, 0.5). Subsequently, we performed the same sweeping procedure for the XL-Scale Dense model validate the accuracy of the dropout rate identified by the MoE model. As illustrated in Figure 10 both the Dense XL and MoE Large models exhibit nearly identical curves. Notably, they indicate that setting the dropout rate to 0.2 or 0.3 yields the optimal performance.
for dense
This discovery holds significant practical value for practitioners. The advantage of leveraging the MoE model for hyper-parameter tuning is evident. It requires considerably fewer computational resources compared the dense model, despite possessing comparable parameters. This translates into substantial savings in computation resources for debugging and papper-prramter tuning. For example, in this specific set of experiments, sweeping the MoE Large model incurred an expenditure of approximately 10.6K USD on the Google Cloud Platform. Conversely, training the Dense XL model only once required 7.4K USD. Consequently, the entire development process, including sweeping, amounted to total cost of 18K USD, which is only 0.48 times the expense of directly tuning the Dense XI. model. As we scale up to larger models and conduct more experiments, the potential of MoE Hlyee--Paaamett Tuning to conserve computational resources and reduce carbon emissions becomes increasingly promising for future endeavors.
Final Performance after Dropout Sweep Having determined the appropriate dropout rate based on the MoE model, proceed to scale up the models accordingly. Figure T1 illustrates the outcomes of this scaling process. Notably, we only used around 2?7 tokens, which should be only able to train a 16M TS model according Chinchilla scaling law. However, we can see the model can still improve when scaling to 2.8B parameters, i.e., over 1700 larger than the 16M model. We believe this is a significant result for such simple method, i.e., introducing appropriate dropout rate via MoE Hyper- Parammer Tuning.
Conclusion
In this study, we investigated the token-crisis problem and thoroughly explored various approaches to training LLMs with repeated tokens. Our investigation covered what is token-crisis, what would hap- pen if we use repeated data under token crisis, why there is multi-epoch degradation when repeating data, and how can we alleviate this issue with off-the-shell approaches. We also demonstrated the effectiveness of using MoE to predict the behavior of more computationally expensive dense models, offering valuable means of accelerating LLM development more broadly.

---

Acknowledge
Yang You's research group is being sponsored by NUS startup grant (Presidential Young Professor- ship), Singapore MOE Tier-1 grant, ByteDance grant, ARCTIC grant, SMI grant and Alibaba grant. We also thank the TPU computing resources from Google TRC (TPU Research Cloud) grant.
References
[1] Anil al., "Palm 2 technical report," arXiv preprint arXiv:2305. 100::2005.0403, 2023.
[2] Biderman et al., Pythia: suite for analyzing large language models across training and scaling, 2023. arXiv: 2ww.:2344.00 01373 [cs.Cl. CL]
[3] A. Chowdhery et al., "Palm: Scaling language modeling with pathways," arXiv preprint
arXiv:2204.022 02.00,4...11...1 2022.
[4] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser, "Universal transformers," arXiv preprint arXiv:1807.0388: 100..0319, 2018.
[5] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional
transformers for language understanding." arXiv preprint
2018.
[6] A. Dosovitskiy et al., "An image is worth 16x16 words: Transformers for image recognition scale," arXiv preprint arXiv:2010.1 1929, 2020.
[7] D. Hermandez et al., "Scaling laws and interpretability of learning from repeated data," arXiv preprint arXiv:2205. 10487, 2022.
[8] J. Hoffmann et al., "Training compute-optimal large language models," arXiv preprint
arXiv:2203.1 15556, 2022.
[9] Huang. Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger, "Deep networks with stochas- tic depth," in Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, Springer, 2016, pp. 646-661. [10] J. Kaplan et al., "Scaling laws for neural language models," arXiv preprint arXiv:2001.0836/ 2020.
[11] J. Kim et al., "Scalable and efficient moe training for multitask multilingual models," arXiv preprint arXiv:2109. 10465, 2021.
[12] Z. Lan, M. Chen. S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "Albert: A lite bert for self-supervised learning of language representations," arXiv preprint arXiv:/909, 1942, 2019. [13] D. Lepikhin et al., "Gshard: Scaling giant models with conditional computation and automatic sharding," arXiv preprint arXiv:20066 1666,,, 2020.
[14] J. Li, Z. Zhang, and H. Zhao, "Selftprompting large language models for open-domain qa." arXiv preprint arXiv:2212. 22.0::22 2022.
[15] Q. Liu et al., "Tapex: Table pre-training via learning a neural sql executor," arXiv preprint 10fi:2007.00653 2021.
[16] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," arXiv preprins
arXiv:1711.0511 2017.
[17] Y. Lou, Xue, Zheng, and Y. You, "Cross-token modeling with conditional computation." arXiv preprint arXiv:2109.02006 2021.
[18] OpenAI, "Gpt-4 technical report," ArXiv, vol. abs/2303.08774 2023.
[19] L. Ouyang al. "Training language models follow instructions with human feedback," in
Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, Cho, and A. Oh, Eds., vol. 35, Curran Associates, Inc., 2022, pp. 27 1773--2774 744, hww.lallbbbbbbbttttte//////////e [20] J. Saee////eeebbbsa4711///550000000
[Online]. Available: http::: htw.::sssiiiiiirccrieeeeeese
fil.fieess. silesspapeer pappp//pppr22111 pdf
Rae el al., "Scaling language models: Methods, analysis insights from training gopher,"
arXiv preprint arXiv:2 2. /446, 2021.
[21] Raffel et al., "Exploring the limits of transfer learning with a unified text-to-text transformer." Journal of Machine Learning Research, vol. 21, no. 140, pp. 1-67, 2020. [Online]. Available: http:// /tmp: hmwr../mmr.rao.... wttg/paperrs//1/220000 html
10

---

[22] P. Rajpurkar. J. Zhang, K. Lopyrev, and P. Liang, "SQuAD: 100,000+ questions for machine comprehension of text," in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 2383-2392 DOT: 10. (8653/v1/DIs 1264 [Online]. Available: https
//aclanthology org/D16-1264
[23]
R. Ri and Y. Tsuruoka, "Pretraining with artificial language: Studying transferable knowledge in language models," in Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 7302-7315. DOL: 10. 11653/11/20222 acl monl-long... 504
[Online].
Available: "railable:httts:::: /attpp://accl //tp://aaaaaanoooooooooooo.o org 2222. acl- 1ong--on55555 -504
[24] O. Russakovsky et al., "Imagenet large scale visual recognition challenge," International journal of computer vision, vol. 115, pp. 21 212-2 2015.
[25] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, "Megatron-Im: Training multi-billion parameter language models using model parallelism," arXiv preprint arXiv:/909.08D0
2019.
[26] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," Journal of Machine Learning Research, vol. 15, no. 56. pp. 1929-1958. 2014. [Online]. Available: http: /ttt:://jml.. org/ papers/v15/ htt.://rriatattaaaaemm
html
[27] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna, "Rethinking the inception architecture for computer vision," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818-2826.
[28] Y. Tay et al., "Scale efficiently: Insights from pre-training and fine-tuning transformers," arXiv preprint arXiv:2109. 10686, 2021.
[29] Y. Tay et al., "Unifying language learning paradigms," arXiv preprint arXiv:2205.05132 2022.
[30] R. Taylor et al., "Galactica: A large language model for science," arXiv preprint
arXiv:2211.0903: 2022.
[31] H. Touvron et al., "Llama: Open and efficient foundation language models," arXiv preprint
arXiv:2302. 139721,...3971, 2023.
[32] P. Villalobos, J. Sevilla, L. Heim, T. Besiroglu, M. Hobbhahn, and A. Ho, "Will we run out of data? analysis of the limits of scaling datasets in machine learning," arXiv preprint
1XXi:2221.043222 2022.
[33] Y. Wu, F. Li, and P. S. Liang, "Insights into pre-training via simpler synthetic tasks," in Advances in Neural Information Processing Systems, Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35, Curran Associates, Inc., 2022, pp. 21 844-21 857. [Online]. Available: httpss:
neurips http:ecccpppppaeaeeeeeee _flee httpppprrrflleeppppeereeeeee ppppp//022//////////////////2221 921e/89879d5f 38818888666331111888800000086660
Conf erence. pdff
[34] F. Xue, X. He, X. Ren, Y. Lou, and Y. You, "One student knows all experts know: From sparse to dense," " arXiv preprint arXiv:2201. 10890, 2022.
[35] F. Xue, 7. Shi, F. Wei, Y. Lou, Y. Liu, and Y. You, "Go wider instead of deeper;" in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, 2022, pp. 8779-8787.
[36] L. Xue et al., "Byt5: Towards a token-free future with pre-trained byte-to-byte models," Transactions of the Association for Computational Linguistics, vol. 10, pp. 291-306, 2022.
[37] L. Xue et al., "Mt5: A massively multilingual pre-trained text-to-text transformer," arXiv
preprint arXiv:2010. 1934, 2020.
[38] B. Zoph et al., "St-moe: Designing stable and transferable sparse expert models." URL
https://arxiv. org/abs/2202.08800 080002002..000........22222202 2022.
11

---

Appendix
A Frequent Asked Questions
list the potential frequent asked questions and the point-to-point answers as follows:
A.1 Selection of Encoder-Decoder T5 Model and C4 Dataset
Firstly, the C4 dataset is widely studied large-scale pre-training dataset that has been open-sourced. relatively easier to prepare and maintain compared the dataset used LLaMA. T5 was originally proposed and developed in conjunction with the C4 dataset. Therefore, using the T5 model with the C4 dataset aligns with established practices and facilitates comparability with prior research.
Furthermore, while decoder-only architectures have been predominant in existing Language Models (LLMs), it is still unclear whether decoder-only models consistently outperform encoder-decoder architectures. Recent research by Tay et al. (29) demonstrated that encoder-decodee exhibit superior performance the 20B scale. Additionally, the specific architecture employed by OpenAl's GPT-4 remains unknown to external researchers. Consequently, there is ongoing debate and uncertainty regarding the superiority of decoder-only models.
architectures
In reality, the difference between decoder-only and encoder-decoder decoder architectures may not be as significant as initially perceived. Both architectures utilize autoregressive decoder, and the main distinction lies in determining which tokens are fed into the encoder and which ones are fed into the decoder. Another difference arises in multi-turn dialogue systems, where encoder-decoder architectures may require recomputation of certain activations.
A.2 Lack of Significant Improvement of UL2 in Table 3
The lack of significant improvement of UL2 over the vanilla T5 model in Table 3 can be attributed to a specific difference in our implementation. Unlike the original UL2 implementation by Tay er al. [29], did not utilize dropout in experiments. By removing dropout from the UL2 training process, it is possible that we experienced performance drop, leading to the comparable results observed between UT.2 and the vanilla T5 model in experiments.
A.3 Reason for Not Training Larger Scale Model (10B+)
While training a larger-scale model like 10B+ parameters would indeed provide valuable insights, the primary reason for not doing so is the limitation of computational resources. Conducting a comprehensive set of experiments covering various aspects of the token-crisis issue requires substantial computational power, which includes significant financial costs, time requirements, and carhon emissions. Therefore, in order to balance these factors and optimize our research efforts, we decided scale up approximately 3B parameters explore the token-crisis problem within the available resource constraints.
A.4 Performance Drop Without Enough Data Even If We Are Using Dropout
Even when employing suitable dropout rate, there is still substantial gap between models trained with full tokens (without dropout) and those trained with limited number of tokens through data repetition. This discrepancy arises because, for the purpose of clarity, repeated only small number of tokens (i.e., 227) across multiple epochs (2 epochs). In more typical scenarios, such as repeating 1T tokens for 10 epochs, the observed gap would probably be smaller.
12

---

B Related Work
B.1 Multi-Epoch LLM Pre-training
The topic of training LLMs for multiple epochs only received little attention in existing literature. Hoffmann et al. [8] suggested that training LLMs for multiple epochs may have detrimental effects. On the other hand, Tay et al. [29] trained 120B model for 4 epochs without observing multi-epoch degradation, although they attribute their success primarily to the high-quality data used. biderman er al. (2) found that deduplicating pre-training data had no clear benefit on language modeling performance. Additionally, Hernandez et al. 7 discovered that repeating a small fraction of data during LLM pre-training can significantly harm model performance.
In contrast to these works, our study focuses specifically the token-crisis problem and investigates the consequences of further scaling LLMs by repeating a fixed amount of data multiple times. To the best of our knowledge, ours is the first paper explore the token-crisis and train LLMs for multiple epochs. Many of the insights we present, such as using Mixture-of- fixture-of-Expertss (MoE) models to predict the behavior of more computationally expensive dense models, are novel and valuable contributions to our research community.
B.2 Pre-training with Synthetic Data
Existing research has examined the concept of pre-training LLMs with synthetic data means of mitigating data scarcity. For instance, Ri et al. (23) designed artificial languages with structural properties that mimic natural language, while Wu et al [33] successfully pre-trained LLMs through
simpler synthetic tasks, Other works, such as TAPEX [5] address the challenge of data scarcity in specific domains.
In contrast, our paper focuses on training LLMs in the context of the token-crisis problem with multiple epochs, which is distinct from the perspective of using synthetic data. However, believe this line research is relevant because it has the potential to alleviate the token-crisis in LLMs, therehy addressing the associated challenges and limitations.
C Limitations
Scalability to State-of-the-Art Models: Although we conducted experiments on models with 3B parameters, we acknowledge that we did not explore the performance of our approach on state-of-the- art scale models, such as the 175B- 75B-parameter GPT-3. The primary reason for this limitation is the lack of computational resources required for running experiments on very large models multiple times. Given the token-crisis scenario and the need for extensive experimentation, conducting experiments on larger models remains prohibitively expensive.
Dataset Quality Assumptions: Our Insight (4) relies on using C4 as low-quality data and Wikipedia as high-quality data. While acknowledge that the data quality of C4 is acceptable, adopted this setting because many existing LLMs have utilized Wikipedia for pre- pre-training for at least one epoch, and this choice is supported by claims of better data quality. It is worth noting that other datasets with different characteristics may yield different results when applied to our approach.
Limitations of Insight (6) in Handling FLOPS Disparities: Our Insight (6) may encounter chal- lenges when dealing with significant gaps in FLOPs between models. For example, we found that the Base-level MoE model with 64 experts in every MoE layer cannot perfectly predict the behavior of an XL-level dense model, although it still exhibits a higher susceptibility to overfitting compared to the Base-level MoE model with 16 experts. This observation aligns with the notion that very sparse models may have under-trained parameters, leading to imperfect predictions. Further investigation is necessary address this limitation and enhance the accuracy of Insight (6) when confronted with substantial FLOPs disparities.
13

---

D Ablation Study on Batch Size of Larger Dataset Size
60
MLM Accuracy
50-
bsz, tokens
30
0.5
1.0
1.5
Num of Steps
Figure 12:
We fix the batch size and number of repeats, and then train the model with 4 times more steps to go through times more tokens.
We can see when using smaller batch size than what we did in Figure 4 model still has little multi- epoch degradation. This further verifies that larger dataset can alleviate the multi-epoch degradation.
E
Ablation Study on Number of Experts for MoE
62
MLM (96)
588
56
54
52
4 Experts
16 Experts
48
64 Experts
2
Steps
1e5
Figure 13: we use relatively limited tokens to train three T5-MoE models with 4, 16, 64 experts at each MoE layer. We can observe MoE with more experts, which have more trainable parameters have a more serious multi-epoch degradation.
To further investigate the behavior of MoE models, train three T5-MoE models with 4, 16, and 64 experts using relatively limited number of tokens In Figure 13 we observe that as the number of trainable parameters increases with more experts, the MoE models experience multi-epoch degradation. These findings indicate that the additional parameters in MoE models are indeed leading faster memorization of the training data.
F Encoder-Deecode vs Decoder-Only
We can see encoder-decoder is clearly better than decoder-only but the MoE based decoder-only model having comparable trainable parameters with encoder-decoder performs almost the same as
FFllowwig the convention (38), the number experts represents the number of expert FFNs in each MoE layer. In our case, we use one MoE layer every 4 layers in both the encoder and decoder. resulting in total of 6 MoE layers in the T5 MoE Base model. For more details about the MoE hyperparameters, please refer to Appendix
14

---

65
60
MLM Accuracy (N0
55
45
40
Decoder-MoE
Decader
Num Stops
Figure 14: We train three models (i.e. encoder-decoder, decoder-only, MoE-based decoder-only) with the same data (C4) and training objective SpannCorruptiion..
eecoder model. Therefore, as suggested by UL.2 paper, the different behaviours of encoder- decoder and decoder-only are more from the training objective instead of model architecture. That the reason why explore UL2 training objective in our Section 3.3.
G Ablation Study on Batch Size of Larger Dataset Size
MLM Accuracy
50
tokens
30
1x bsz, 4x takens
0.0
0.5
1.0
1.5
Num of Steps
Figure 15: We fix the batch size and number of repeats, and then train the model with 4 times more steps to go through times more tokens.
We can when using smaller batch size than what did in Figure 15 model still has little multi-epoch degradation. This further verifies that larger dataset can alleviate the multi-epoch degradation.
H Downsteam Evaluation
Table The updated fine-tuning results. We include the standard deviation runs.
Model
BoolQ
RTE
SQuAD
Acc
Acc
EM
F1
C4-SpanCorr- Spp-eeet
77.0.10.7
68.9.10.8
82.3-0.9
90.0.10.9
C4- C4-SpanCor-Repea
73.8111.0
66.4.10.3
80.0
88.1100.5
C4-UL2-Res
78.8+0.9
72.2+1.2
82.3=0.8
90.2+00.7
74.5+00.6
68.4+0077
79.3=1.1
87.3+1.2
C4- Wiki-Repeatl
74.2+10.7
69.3+1.11
82.6=0.8
90.1+0.6
H4-Wiii-RepeeaZ"
71.410.4
64.9.00.5
7.06-1.0
87.911.
I Verification of MoE Behavior Prediction with Dropout
To validate aforementioned insight regarding MoE behavior prediction, we conducted an exam- ination incorporating dropout, depicted in Figure T6 Remarkably, our findings reveal that the MoE model maintains remarkably similar training behavior to that of the dense model, even when utilizing additional computation resources and possessing comparable parameters. This observation
15

---

60
MUM Accuracy (%)
50
MoE Base w/ dropout
30
Dense Large dropout
Num of Steps
Figure 16: We add dropout to train two models with comparable trainable parameters, i.e, Large-level Dense model and Base-level MoE.
further reinforces the effectiveness of the MoE model in approximating the behavior of larger, denser models.
16

---

Default Hyper-parameters
Table 6: Default pre-training cypee-parameter..
Name
Value
Learning Rate
0.01
Learning Rate Decay
Square Root Decay (0.8)
Optimizer
Adafactor
Batch Size
128
Training Steps
524288 (219)
Dropout
0.0
DropPath
0.0
Label Smoothing
0,0
Weight Decay
0.0
Table 7: Default MoE parammtaraestrs.
Name
Value
Num of Experts
16
MoE Layer Layout
Every fourth
Train Capacity Factor
1.25
Test Capacity Factor
2.0
Num Selected Experts
2
Router Weight
le-2
Z-loss Weight
le-4
We follow T5 1.1 implementation in T5x and ST-MoE implementation in Flaxformer. The default prper-parameters are shown Table 6 and Table 7 The pre-training parameters in Table o are used in both dense and MoE model training.
We conduct experiments on Google Cloud TPU. For Base-level and Large-level models, we use 32 TPU v3 cores, and 128 TPU v3 cores are employed to train XL-level models.
17

---

K Model Configurations
Table 8: Default MoE hayper-paameeee
Scale
Enc Layers
Dec Layers
Heads
Hidden Dim
MLP Dim
#Params
Small
8
512
1024
78M
Mid
640
1280
140M
Base
12
12
12
768
2048
247M
Base Plus
896
2560
432M
Large
24
24
16
1024
2816
783M
Large Plus
24
22
1408
4096
1.4B
XL
24
24
32
2048
5120
2.8B
For Base, Large, XL scale, we follow the default setting in T5x. To obtain more smooth scaling curve, add more configurations like Small, Mid, Base Plus and Large Plus. Please note the XL scale was not used to draw the scaling curve Figure because the expensive training cost.
18

---

L
More Results
In this section, we share more other approaches we explored.
L.1 Training with Larger Smaller Batch
Empirically, larger batch size is easier to overfit, and smaller batch size can usually reduce overfitting. We found this widely-used commonsense still holds in LLM token-crisis. However, think this trick is not that useful because training L.M with smaller batch size is inefficient.
L.2 Training with Longer or Shorter Sequence
We train the same amount of tokens with longer shorter sequences with the same total token usage. That is, for longer sequence, we train for fewer steps and more steps are used to train with shorter sequences. We found using longer sequences achieve better performance.
L.3 Randomly Activating One Embedding Layer from Multiple Embedding Layers
suggested the over-fitting can alleviated when adding more randomness into input embeddings. We therefore init multiple embedding layers and randomly activate one of these embedding layers for each input sequence. However, we not observe this design can alleviate multi-epoch degradation even if we regularize different embedding layers have different embeddings.
L.4 Adding Dropout Before and After Linear Layers
Considering dropout is very effective, we tried to add dropout before and after each linear layer. Adding dropout after one layer means column-wise weight masking. Adding dropout before and after one layer denotes element- wise weight masking. found such regularization too strong and makes LLM achieve inferior performance even if we set small dropout rate on the XL-level model.
19


Reviews:
[{"review_id": "meoLnCDv6v", "reviewer": "Reviewer_SEVe", "summary": "Data is one of the most important factors for training a well-performed Large Language Model(LLM). There are not enough studies that are deep into the effect of data. This paper addresses this key problem and analyzes the effect from the important aspects including the effect of pre-training, the effect of downstream tasks, and so on. ", "questions": "1. Please update the experiment results with more data, for example, 400B.\n2. If possible, show the performance of GPT-3.", "limitations": "Please see the weakness.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Data is very important for LLM. This work analyzes the impact of data from various aspects. \n2. The experiments are rich and detailed. \n3. The paper is well-written except some typos, for example missing ‘.’ in Line 74.\n", "weaknesses": "1. The backbone of this paper is T5 1.1. And we know that GPT is one of the most important generative models. Hence, it is necessary to add GPT as the backbone.\n2. Data is the key factor for LLM. The data volume used in the experiment could affect the conclusion seriously. In this work, the maximum amount of data is 2**35, about 34B data. It is so small data for training LLM that the experiment results may not be reliable enough. \nThe figures of training loss over train tokens from LLAMA 1 and LLAMA 2 show that 300~400B data could display a stable trend (at least 200B). In other words, 2**27(0.13B) tokens repeated many times are different with 1TB tokens repeated the same times.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Use GPT instead of T5 as backbone. \n\nThank you so much for your valuable insights. As we discussed in Appendix A.1, the differences between the encoder-decoder and decoder-only models might not be as big as our community thought. There are two main distinctions to consider: (1) The encoder-decoder architecture actually has approximately twice the number of \"trainable parameters\" compared to the decoder-only model. This is one important reason why encoder-deocder is sometimes better. (2) The encoder-decoder setup requires clearly separated input-output pairs to work effectively, enabling bidirectional attention only on the input tokens. Interestingly, this mechanism is also employed in decoder-only prefix-LMs. An example of such a model is UPaLM [1]. They fine-tuned PaLM (a model trained by casual LM objective) to create UPaLM with just a few training steps. This suggests that the difference between prefix-LMs and casual LMs is relatively minor. Moreover, since both prefix-LM and encoder-decoder use input-only bidirectional attention, we can view the prefix-LM as an encoder-decoder model that shares the trainable weights across the encoder and decoder. \n\nTo show the encoder-decoder and decoder-only are not that different from the model architecture perspective, we conduct another set of experiments in Figure 1 in our rebuttal pdf. We can see the encoder-decoder is clearly better than vanilla decoder-only model due to more parameters. If we use MoE-based decoder-only model with comparable trainable parameters with encoder-decoder model, the gap can be patched almost perfectly.\n\nIn addition, to be honest, this project was initiated before ChatGPT gained widespread popularity. As evident in our paper, we explored various factors related to the \"token-crisis\" phenomenon, which necessitated a wide array of implementation and timely training efforts. As a result, we opted for the well-established T5 framework, which was widely studied at the time. More importantly, once again, we believe the T5 architecture is quite similar to the widely used decoder-only design.\n\n\n> Q2: If possible, please update the experiment results with more data, for example, 400B.\n\nThank you so much for your suggestion. We totally agree that super-scale experiments like training a 175B model with 400B tokens for multiple epochs would make our conclusion more solid. This was also mentioned in our Appendix A.3. While we acknowledge the potential benefits of super-scale experiments, resource limitations (especially during the rebuttal phase) prevented us from pursuing this scale. If we have enough computation resource, we promise that we will have a run on this level. However, considering the resource we have so far, to make our conclusion as sound as possible, as depicted in Figure 3 and Figure 4, we conducted experiments to verify larger model is usually more data-hungry and easier to overfit within fewer epochs. This supports that our insights based on the smaller model with more epochs can also apply to larger models with fewer epochs.\n\n[1] Tay, Yi, et al. \"Transcending scaling laws with 0.1% extra compute.\" arXiv preprint arXiv:2210.11399 (2022).\n"}, {"review_id": "eAdOGnBGFN", "reviewer": "Reviewer_m3sH", "summary": "This paper delves into the token-crisis issue in language models, a situation characterized by performance decline when the same pre-training data is used across multiple epochs. The authors scrutinize several methods of training language models with recurring tokens, encompassing regularization strategies and the use of mixture-of-experts for refined hyper-parameter tuning. Furthermore, they look into the possibilities of creating supplementary data utilizing existing language models and formulating more data-conservative model structures. The paper provides a comprehensive analysis of the token-crisis issue and its roots, as well as the efficacy of various strategies aimed at alleviating this problem.", "questions": "Refer to the weakness", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has several strengths across different dimensions:\n\n- The paper focuses an important problem in language modeling, the token-crisis problem, which has not been extensively studied before and has big potential. The authors provide a thorough investigation of the problem and explore various approaches to mitigating it, including regularization techniques and mixture-of-experts for efficient hyper-parameter tuning. \n- The presented insights have the potential to improve the performance and efficiency of language model learning. The paper's contributions, including a thorough investigation of the token-crisis problem and its causes, as well as the effectiveness of various approaches to mitigating this issue, are significant for the field of natural language processing.\n- The paper is well-written and well-organized, with clear explanations of the problem and the proposed solutions. The authors provide a detailed analysis of the factors contributing to multi-epoch degradation and the effectiveness of various approaches to mitigating this issue. The experiments are well-designed and the results are presented clearly and comprehensively.", "weaknesses": "The first weakness of this work is that all the experiments were conducted on the T5-style masked language modeling, which is opposite to the recently surging GPT-style (i.e., causal) language modeling. And it's unclear whether all the sights/findings are applicable or transferable to the CLMs. \n\nMeantime, the experiments were conducted on models with up to 3B parameters and did not explore the performance of the proposed approaches on larger models, such as the GPT3 or its equivalents. This limits the generalizability of the results to larger-scale models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: All the experiments were conducted on the T5-style masked language modeling, which is opposite to the recently surging GPT-style (i.e., causal) language modeling. And it's unclear whether all the sights/findings are applicable or transferable to the CLMs.\n\nThank you for your insights. We totally agree that the training objective is highly important in the token-crisis problem. That is the reason why we investigate UL2[2] training objective (which is used in PaLM-2[3]) in Section 3.3. The UL2 training objective not only includes the T5-style masked language modeling but also covers causal language modeling and a more challenging masked token prediction (longer masked span or higher mask ratio), which is similar to fill-in-the-middle objective[4], an advanced training objective proposed by OpenAI and used in open-sourced models like StarCoder [5].\n\n[2] Tay, Yi, et al. \"Ul2: Unifying language learning paradigms.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[3] Anil, Rohan, et al. \"Palm 2 technical report.\" arXiv preprint arXiv:2305.10403 (2023).\n\n[4] Bavarian, Mohammad, et al. \"Efficient training of language models to fill in the middle.\" arXiv preprint arXiv:2207.14255 (2022).\n\n[5] Li, Raymond, et al. \"StarCoder: may the source be with you!.\" arXiv preprint arXiv:2305.06161 (2023).\n\n\n> Q2: The experiments were conducted on models with up to 3B parameters and did not explore the performance of the proposed approaches on larger models, such as the GPT3 or its equivalents.\n\nThank you for your valuable suggestion. We fully agree that it would have potential benefits of extending our experiments to models like GPT-3 or its counterparts. However, as mentioned in our response to Reviewer SEVe's Question 1, the primary challenge lies in resource availability. Training a 175B model on a substantial dataset for multiple epochs requires significant computational resources, which only a handful of institutions possess. \n\nIt's worth emphasizing that our interest extends to conducting ablation studies that delve into the various components of multi-epoch training. These studies necessitate even more computational resources than training LLaMA-2 70B from scratch if we consider larger models. In the current paper, even focusing solely on Figure 10 comes with a considerable cost, approximately 47K USD for Google Cloud TPU usage. This cost is approximately seven times higher than training BERT-Large from scratch.\n\nTo contribute to our research community, we have to scale down to align with a reasonable budget. At the same time,  to make our conclusion as sound as possible, as depicted in Figure 3 and Figure 4, we conducted experiments to verify larger model is usually more data-hungry and easier to overfit within fewer epochs.\n"}, {"review_id": "GSzjvrp47P", "reviewer": "Reviewer_kBHt", "summary": "The authors propose a concept called token-crisis, which means the growth rate of high-quality text data available is much slower than the growth rate of data required by LLMs. This paper is the first empirical study of the repeating pre-training data for the token-crisis problem. Some major findings are: larger models are more prone to overfitting and would affect downstream tasks; using dropout is an effective way to alleviate the multi-epoch degradation, and setting the dropout rate to 0.2 or 0.3 yields the optimal performance.", "questions": "1. See Weaknesses.\n\n2. Since most of the experiments are done on the C4 dataset, which is known to be low quality, I'm wondering if the conclusions would still hold when using high-quality data. I think there currently is a trend to utilize higher quality data (whether generated from GP4 or hand-crafted) to fine-tune or pre-train smaller LLMs, which seems to obtain very good results.", "limitations": "The limitation section is pretty honest and actually echoes some of my concerns.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experiments are very comprehensive, and most of the conclusions are well-supported.\n\n2. Many finds in the paper provide valuable insights to train better open-source LLMs in academia with limited resources.\n\n3. The dropout mitigate token-crisis findings seem to be pretty useful.", "weaknesses": "1. Why there are no experiments testing for a smaller number of data repetitions? The smallest number of repetitions in the paper is 2^8, which is very large.\n\n2. The paper seems to not answer the question in the title: to repeat or not to repeat. If we have limited data, should we repeat or not in pre-training? There seems to be no experiment showing how many repetitions are optimal with the same set of data.\n\n3. The dataset-quality-does-not-matter-much conclusion seems to be a bit not-well-supported. If the Wiki data is of higher quality than C4 data, why the downstream performance of Wiki pre-trained without repeating model does not outperform the C4 pre-trained model? Considering there is a recent paper [1] that claims that true high-quality data helps a lot in code generation, I'm a bit unsure about this conclusion.\n\n4. Does model architecture matters? All experiments use T5, an encoder-decoder model. Do all conclusions hold when using a decoder-only model?\n\n[1] Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C.C.T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O. and Salim, A., 2023. Textbooks Are All You Need.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Why there are no experiments testing for a smaller number of data repetitions? The smallest number of repetitions in the paper is 2^8, which is very large.\n\nThank you for your question. In our plots, we actually have the experiments testing for a small number of data repetitions. For instance, in Figure 3, we trained Base, Large, and XL models with a larger dataset (four times larger) and a reduced number of epochs (e.g., 2^6). Furthermore, in several of our figures (e.g., Fig 4, 5, 6, 7), the plotted validation performance spans the entire training process, encompassing datapoints from the initial stages, which actually involve only a few epochs of data.\n\n\n> Q2: The paper seems to not answer the question in the title: to repeat or not to repeat. If we have limited data, should we repeat or not in pre-training? There seems to be no experiment showing how many repetitions are optimal with the same set of data.\n\nThanks for the great question again! We actually make our conclusion through the investigation but did not clearly highlight this in the conclusion section. To address this, we revised the conclusion section of our paper to provide a more explicit summary of our findings. Specifically, if we do not conduct any tricks, according to our experiments in Figure 4,5,6,7, it is okay to repeat for a few epochs. A concurrent work [6] also figured out the specific scaling law of training with repeated data. In this paper, we think studying the factors of scaling LLM under token crisis is also highly important because we find the scaling law (or the optimal repetitions) is very sensitive to many factors like dataset size, model size, training objective, and regularizations. Based on our findings, in Figure 9, we can see simply setting dropout as 0.1 can greatly change the mult-epoch degradation so that the scaling law would also be very different after adding dropout. Therefore, if we add an appropriate dropout for multiple epoch LLM training, it would be okay to repeat. \n\n[6] Muennighoff, Niklas, et al. \"Scaling Data-Constrained Language Models.\" arXiv preprint arXiv:2305.16264 (2023).\n\n\n\n> Q3: The dataset-quality-does-not-matter-much conclusion seems to be a bit not-well-supported. If the Wiki data is of higher quality than C4 data, why the downstream performance of Wiki pre-trained without repeating model does not outperform the C4 pre-trained model? Considering there is a recent paper (Textbook is all you need) that claims that true high-quality data helps a lot in code generation, I'm a bit unsure about this conclusion.\n\nThis is a great question. We also discussed the dataset quality assumption in Appendix C. Since Wikipedia dataset is actually smaller than C4, even if we use the full wikipedia dataset, the model is actually trained for a few epochs when training with 500K steps. However, the C4 is large enough so that there is no repeat at all. Therefore, it is reasonable that wikipedia is slightly worse than C4 when training with enough data. \n\nFor the Textbook is all you need paper, the data quality is extremely high, which is also very similar to the downstream tasks like HumanEval (See Section 6 of the Textbook paper). So using such high-quailty instruction following data without web-scale pre-training data to achieve good performance on some benchmarks is possible. However, in this paper, we are discussing the quality of web-scale pre-training data. Note the high quality here is relative. Compared with C4, wikipeida is good. But if we compare wikipedia with Phi-1 or Vicuna’s ShareGPT instruction following data, wikipedia is relatively not that ideal. \n\n\n> Q4: Does model architecture matter? All experiments use T5, an encoder-decoder model. Do all conclusions hold when using a decoder-only model?\n\nPlease see the response for Reviewer SEVe Q1 and our Appendix A.1.\n\n> Q5: Since most of the experiments are done on the C4 dataset, which is known to be low-quality, I'm wondering if the conclusions would still hold when using high-quality data. I think there currently is a trend to utilize higher quality data (whether generated from GP4 or hand-crafted) to fine-tune or pre-train smaller LLMs, which seems to obtain very good results.\n\nWe argue that, firstly, C4’s quality is relatively low when compared with wikipedia but it doesn’t mean C4 is a terrible dataset. Note that LLaMA also used C4 as part of its pretraining data. As we mentioned above, The textbook data works on HumanEval because its data is similar to HumanEval benchmark to some extent. If we want to get a model which is good at everything, it is still better to pretrain the model with web-scale dataset. The instruction finetuning is out of the scope of this paper. Due to the popularity of ChatGPT, users are writing instructions every day. We can even mine instructions in the daily conversation or movie transcript and then generate a huge amount of instructions. So the number of instructions would grow fast recently and it has not approached the upper bound. In other words, it is too early to discuss the token-cirsis of instruction following data. We think it is better to wait for what would happen in the instruction following data and see whether we will really run out of instructions before studying “instruction-crisis”.\n"}, {"review_id": "XeQ8YzRfxR", "reviewer": "Reviewer_vyo8", "summary": "The paper presents an empirical study on the effect of training on multiple epochs in the data limited regime. They show that Chinchilla's scaling laws holds for T5 style models. The authors show that repeated tokens result in degradation of accuracy. The also study some of the factors contributing to this degradation. Towards the end, they show that MoEs can used a proxy to tune the hyperparameters of the larger models.   ", "questions": "- Have the authors considered the effect of learning rate in overfitting?", "limitations": "No", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- a detailed empirical study\n- the results are carefully studied", "weaknesses": "- Some of the conclusions are trivial. For example, larger models are more susceptible to overfitting.\n- MoE is presented as a way to tune hyperparameters of larger models but it is not clear to what extent. For example, shoiuld MoE always be iso-parameter with the base dense model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Some of the conclusions are trivial. For example, larger models are more susceptible to overfitting.\n\nThanks for the suggestion. We argue that, although some of our conclusions are intuitively trivial, these conclusions play an important role in studying the token-crisis problem. For instance, to ensure that our investigation on relatively fewer tokens and moderate-size models can adapt better to train larger models with more data for fewer epochs, verified the insight that larger models are more susceptible to overfitting. Therefore, these conclusions not only provide insights to readers but also pave the way to more in-depth and insightful conclusions.\n\n> Q2: MoE is presented as a way to tune hyperparameters of larger models but it is not clear to what extent. For example, should MoE always be iso-parameter with the base dense model?\n\nWe cannot say MoE is always iso-parameter with the base dense model before studying all hyper-parameters. And as we know, it is prohibitively expensive and almost impossible to ablate all hyper-parameters even if for a moderate-size model in LLM training. However, in this paper, we studied MoE is iso-parameter with more computation-heavy dense model at three different scales (Base, Large, XL) and under six different dropout ratios (0, 0.1, 0.2, 0.3, 0.4, 0.5). This indicates a strong trend towards iso-parameter behavior with MoE, so this suggests MoE is a promising approach for efficient hyperparameter tuning.\n\n\n> Q3: Have the authors considered the effect of learning rate in overfitting?\n\nThank you for your insightful query. We fully agree that, as a general principle, training with an excessively small learning rate over an extended duration has the potential to exacerbate the overfitting issue. Therefore, we delved into the impact of learning rate on overfitting through an ablation study involving both smaller and larger learning rates. Our investigation revealed that while the learning rate does influence overall model performance, it only exerts a limited effect on altering the overfitting trend in our experiments.\n\n"}, {"review_id": "1DUGCONbKM", "reviewer": "Reviewer_2hBw", "summary": "This paper conducts an empirical study on the scaling of transformer models under limited training data, which they call a *token crisis*. They show that under token crisis, training T5 models for multiple epochs results in the degradation of pre-training and downstream task performance. They also show that the dataset quality is unimportant for this multi-epoch degradation. They reveal that using dropout can alleviate the multi-epoch degradation. They also observe that the behavior of Mixture-of-Expert (MoE) models can be used to predict the training behavior of dense models, and they use this observation to search for the best dropout rate for the dense model using MoE models.", "questions": "- Q1. Line 249: balance between regularization and model performance. Doesn't regularization mean less overfitting, which leads to better model performance? What does the balance mean here? Should it be the balance between regularization and training efficiency?\n- Q2. Does Chinchilla scaling law really hold for Enc-Dec models like T5? Chinchilla law is based on Decoder only models. More extensive experiments (more than the experiments shown in Insight (1)) are needed to verify such a claim. \n- Q3. The sentence on Line 111 (`When a larger model outperforms a smaller model, it indicates that the smaller model has received sufficient tokens`) is unclear to me. I don't see why the prior situation indicates the latter conclusion.\n- Q4. Line 153: Why is C4 with cleaning still extremely low-quality?\n\nPresentation \n===\nLine 244: \"We\" should not be capitalized.\n\n\nOther suggestion\n===\n**The following comment does not influence my rating of the paper, and I know that the authors are not required to compare their work with the following concurrent work**. I am just listing it since I believe that when the paper under review is published, readers will be wondering the following questions.\nAnother very recent work also studies token crisis, \"Scaling Data-Constrained Language Models [1]\". It will be beneficial to discuss the similarity and differences between that paper and the current paper under review. For example, why does the Chinchilla law need to be adjusted in [1] while the current paper claims that the Chinchilla law still holds? ", "limitations": "The limitations are well-addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written, and the empirical takeaways are clear and useful\n2. The *token crisis* is important and is expected to be more severe. It is good that someone studies this problem.", "weaknesses": "1. This paper only uses a single task as the downstream task, so it is unclear if the result will hold for other datasets.\n2. This paper only studies encoder-decoder models like T5, so it is unclear if the results will hold for other models. Specifically, decoder-only models (like GPT3, Chinchilla, and LLaMA) are more widely used and are the predominant architecture of current LLMs. Still, I think this paper has merits since for focusing on Enc-Dec models, and such a study is still valuable.\n3. Some conclusions are too assertive and are not fully supported by the experiment results.\n    - Section 3.1 Insight (3) attributes the performance difference between the two models to dataset size. However, the batch size is also different. I wonder if the key difference is the dataset size or the batch size.\n    - Section 3.3: UL2 degrades more. There should be some statistical significance comparison or variance for the downstream performance based on multiple runs shown in Table 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: This paper only uses a single task as the downstream task, so it is unclear if the result will hold for other datasets.\n\nThank you for your insightful suggestion. We conducted more downstream evaluation on BoolQ and RTE datasets, which are two widely used datasets in SuperGLUE benchmark. As shown in the rebuttal PDF, we observe a similar trend in both SuperGLUE and SQuAD results.\n\n> Q2: This paper only studies encoder-decoder models like T5, so it is unclear if the results will hold for other models. Still, I think this paper has merits because for focusing on Enc-Dec models, and such a study is still valuable.\n\nThank you so much for your positive feedback. Please see the response of SEVe Q1 and Appendix A.1.\n\n> Q3: Section 3.1 Insight (3) attributes the performance difference between the two models to dataset size. However, the batch size is also different. I wonder if the key difference is the dataset size or the batch size.\n\nThis is a great question! We conduct another set of experiments in rebuttal pdf file by fixing the batch size and let the model go through the 4 times larger dataset. We can see that model trained by fixed batch size has a similar trend to the model trained with a 4 times larger batch size.\n\n> Q4: Section 3.3: UL2 degrades more. There should be some statistical significance comparison or variance for the downstream performance based on multiple runs shown in Table 3.\n\nWe added the standard deviation in the PDF submission.\n\n> Q5: Line 249: balance between regularization and model performance. What does the balance mean here?\n\nDropout serves as a regularization technique to reduce overfitting in LLM training to improve model performance. However, it can slow down early learning. The \"balance\" refers to optimizing both training efficiency and model performance through the whole training process. To achieve this balance, we experimented with applying dropout mainly in later epochs to prevent overfitting, while avoiding its use in the initial training stages. We've clarified this point in the latest version of our paper and will update it accordingly.\n\n> Q6: Does Chinchilla scaling law really hold for Enc-Dec models like T5? Chinchilla law is based on Decoder only models. More extensive experiments (more than the experiments shown in Insight (1)) are needed to verify such a claim.\n\nFirst, we argue again that Decoder-only is not that different from Enc-Dec models. In addition, we check the Chinchilla scaling law in our paper because we want to see whether encoder-decoder architecture is also similarly data-hungry. Through our experiments depicted in Figure 2, we indeed observed that larger models outperform smaller ones given a fixed computation budget, with the requirement for an increased dataset size. This observation reinforces the necessity of investigating multi-epoch training. To ensure clarity and precision in our assertion, we have revised our claim as follows: \"Encoder-Decoder models trained on the C4 dataset exhibit comparable data-hungry behavior as described in the Chinchilla scaling law.\". This would be reflected in our forthcoming version of the paper.\n\n> Q7: The sentence on Line 111 (When a larger model outperforms a smaller model, it indicates that the smaller model has received sufficient tokens) is unclear to me. I don't see why the prior situation indicates the latter conclusion.\n\nLarger model is more data-hungry so the larger model is superior only after learning from enough tokens. > Q7: The sentence on Line 111 (When a larger model outperforms a smaller model, it indicates that the smaller model has received sufficient tokens) is unclear to me. I don't see why the prior situation indicates the latter conclusion. Larger model is more data-hungry, which means larger model is superior only after learning from enough tokens. Assume we get more computation budget and infinite training tokens, the question here is whether we should train a larger model or just train a smaller model for longer? Since smaller model is cheaper during inference, we only want a larger model when the larger model is better than the smaller one given the same training cost. Since smaller model has relatively limited capacity to consume more data, it would be outperformed when we have enough resource to train larger model for long enough. That is the case that the smaller model has received enough data and we should train a larger model instead.\n\n> Q8: Line 153: Why is C4 with cleaning still extremely low-quality?\n\nSorry for the confusing statements. As we discussed in Appendix C, the terms \"high\" and \"low\" quality are relative, and our classification of C4 as low-quality is made in comparison to Wikipedia. To prevent any misunderstanding, we will incorporate a footnote in our upcoming version.\n\n> Q9: Discuss the concurrent token crisis paper, \"Scaling Data-Constrained Language Models [1]\".\n\nThank you so much for pointing this out! We are very happy when seeing this concurrent work focusing on token-crisis and glad to discuss the difference.\n\nWhile both our paper and the referenced work address token-crisis concerns, they diverge in focus. The other work primarily delves into the specific scaling law associated with repeated data usage—identifying the optimal repetition times based on model and dataset sizes. In contrast, our primary objective is to scrutinize the multi-epoch scaling behavior. Notably, we explore influential factors like UL2 and MoE techniques, which exhibit heightened data dependency, and we discover the surprising effectiveness of dropout in mitigating multi-epoch degradation. \n\nThese papers offer orthogonal insights into the token crisis problem. The other work contends that the Chinchilla scaling law necessitates adjustment due to the use of repeated data—a distinct context from the original Chinchilla paper. In contrast, our experimentation with T5 does not involve data reuse, maintaining a setting more akin to the Chinchilla study.\n\n"}]

Author rebuttal:
Dear reviewers and AC:

We thank all the reviewers for their feedback. Two main concerns and concise arguments are summarized below. The full version can be found in the corresponding response of different reviewers.

* Use Decoder-only Model instead of Encoder-Decoder Model. =>  This project started before the popularity of ChatGPT. Due to the wide range of experiments, we did not finish this work until 2 months ago. As a result, we opted for the well-established T5 framework, which was widely studied at the time. More importantly, we argue that decoder-only architecture is not that different from the encoder-decoder architecture. As we discussed in Appendix A.1, there are two main distinctions to consider: (1) The encoder-decoder architecture has approximately twice the number of "trainable parameters" compared to the decoder-only model. (2) The encoder-decoder requires clearly separated input-output pairs to apply bidirectional attention only to the input tokens. To verify the viewpoint above, we compare encoder-decoder, decoder-only, and MoE-based decoder-only models in Figure 1 of our rebuttal pdf. We can see encoder-decoder is clearly better than decoder-only but the MoE based decoder-only model with comparable trainable parameters with encoder-decoder performs almost the same as encoder-decoder model. Therefore, as suggested by UL2[2] paper, the different behaviors of encoder-decoder and decoder-only are more from the training objective instead of model architecture. That is the reason why we explore UL2 training objective in our Section 3.3. 

* Larger Model and More Training Data. -> As we stated in Appendix A.3, we fully agree that a super-scale investigation would be helpful, but, since we want to investigate a number of different factors resulting in token-crisis and multi-epoch degradation, we have to implement and try a range of ablation experiments. Using a very large model and dataset is prohibitively expensive for our setting. To verify larger model is usually more data-hungry and easier to overfit within fewer epochs, in Figure 3 and Figure 4, we investigate the scaling-up behaviors in multi-epoch training. This supports that our insights based on the smaller model with more epochs can also apply to larger models with fewer epochs.



Best,

Authors


Meta-review: This paper studies the effect of repeating tokens while pretraining language models for more than one epoch, before finetuning on target tasks. Reviewers appreciated the relevance of the token-crisis problem this paper is focusing on, and the detailed analysis and ablations that are well-organized and clearly presented. Reviewers also acknowledge that the insights from this work can help researchers with limited resources train better language models. 

The main concerns were around the use of encoder-decoder models (T5) while the more common architecture for most recent general purpose LLMs is decoder-only, experiments with "only" up to 3B parameters models, and experiments on limited pretraining and downstream datasets. All these limit the strength of the author's claims and condition them on the specific examined setting. After the discussion and the additional results provided in the rebuttal, most reviewers agreed that despite these limitations the insights from this paper overall constitute a meaningful contribution. I strongly recommend including the additional results on more downstream tasks to (especially given the focus on encoder-decoder and their common use for finetuning unlike few-shot with decoder-only). I also strongly recommend modifying the abstract and introduction to properly state the scope of this work up ahead to properly set up the reader's expectations before reading the rest of the paper.
Decision: Accept (poster)