Simple and Scalable Strategies to Continually Pre-train Large Language Models
Authors: Adam Ibrahim
Benjamin Thérien
Kshitij Gupta
Mats Leon Richter
Quentin Gregory Anthony
Eugene Belilovsky
Timothée Lesort
Irina Rish
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models—saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English$\rightarrow$English) and a stronger distribution shift (English$\rightarrow$German) at the $405$M parameter model scale with large dataset sizes (hundreds of billions of tokens). Selecting the weak but realistic shift for larger-scale experiments, we also find that our continual learning strategies match the re-training baseline for a 10B parameter LLM. Our results demonstrate that autoregressive transformer-based LLMs can be successfully updated via simple and scalable continual learning strategies, matching the re-training baseline using only a fraction of the compute. Finally, inspired by previous work, we propose alternatives to the cosine learning rate schedule that help circumvent forgetting induced by LR re-warming and that are not bound to a fixed token budget.

Published in Transactions on Machine Learning Research (06/2024)
Simple and Scalable Strategies to Continually Pre-train Large Language Models
Adam Ibrahim e
research research@udamirraine @udamiraahim.nn benjamin. therien @emila. quebec kshitij, gupta @emila quebee mats. mats.richte richter Gemiila quebee qubitquentin & @grmail com l.soor @gmaiil com eugene. bewilobbblllossyy... connnooooniiii irina. rish @vrilla, quebec
Benjamin Thérien*te
Kshitij Gupta to
Mats L. Richter
Quentin Anthony 016
Timothée Lesort to
Eugene Belilovsky 18
Irina Rish T
Department of Computer Science and Operation Research, Université de Montréal, Montréal, Canada t
Department of Computer Science and Soflware Engineering, Concordia University, Montréal, Canada $ Mila, Montréal, Canada @ EleutherAT Q
Reviewed on OpenReview: https: openreview. net/ formm? id- DimPeeCKO
Abstract
Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models-saving significant compute compared to re-training. However, the distribution shift induced by data typically results in degraded performance on previous data poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decnging, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between commonly used LLM pre-training datasets (English-EEnglih) and a stronger distribution shift (English-GGerman at the 405M parameter model scale with large dataset sizes (hundreds of billions of tokens). Selecting the weak but realist shift for larger-scale experiments, we also find that our continual learning strategics match the re-training bascline for a 10B parameter LLM. Our results demonstrate that autoregressive haansoormrrbasee LLMs can be successfully updated via simple and scalable continual learning strategies, matching the re-training bascline using only fraction of the compute. Finally, inspired by previous work, we propose alternatives to the cosine learning rate schedule that help circumvent forgetting induced by LR wa-manning and that are not bound to fixed token budget.
1 Introduction
Over the past few years, large pre-trained models have enabled massive performance improvements in language modeling (Brown et al., 2020; Zhao al., 2023), visual understanding (Radford al., 2021; Alayrac et al., 2022; Kirillov et al., 2023), text-to-image generation (Rombach et al., 2022: Pernias al., 2024), and text-to-video generation (Brooks et al., 2024) name few. Large language models (LLMs) are at the

---

Published in Transactions on Machine Learning Research (06/2024)
center of all these improvements, providing an intuitive means for humans to interface with machine learning algorithms through language.
While LLMs are the cornerstone of current generative Al technology, they are expensive to train and keep up to date. However, as new and higher-quality datasets contimue to become available (Gao et al., 2020; Soboleva et al., 2023; Computer, 2023; Soldaini et al., 2024), organizations will need to update their models to stay abreast of the competition. Currently, LLMs are re-trained on a combination of old and newly collected data. Existing works aim to reduce these training costs by enabling low-cost hyperparamcter optimization (Yang et al., 2022) or providing guidelines for maximizing performance under given compute budget (Hoffmann et al., 2022). However, these works assume that models will be trained from random initialization, raising the following question: Should practitioners always combine existing datasets and troin from random imilialization to obtain the best performance? Doing so for every update of the models quickly becomes expensive.
To avoid complete re-training, we explore simple and scalable continual learning strategics for continuing to pre-train LLMs (up to 10B parameters) on large amounts of new data (200B+ tokens). We refer to our setting as "continual pre-training and highlight that it is distinct from existing settings in the literature (Gururangan et al., 2020; Ke et al., 2022; Scialom et al, 2022; Xie et al., 2023) due to the large amount of incoming data we consider. In this work, we do not intend to improve on the performance of models trained from a random initialization on all of the available data. Instead, we consider models trained on the union of existing datasets as baselines whose performance we seek to match using a combination of contimual learning strategies at scale.
Naively contimuing to train the model on new data, however, tends to lead to performance far below re-training all available data, often due to 1) poor adaptation (failure to optimize the new dataset) or 2) catastrophic forgetting (significant capability loss on the previous datasct). Firstly, the question of adaptation is central to our setting as training on large datasets is costly. One would presumably not choose to spend considerable computational resources training on a new dataset only to minimally adapt to it. However, most performant open-source LLMs (Touvron et al., 2023a;b; Jiang et al., 2023; Gemma Team et al., 2024) decay their learning rate to a small value by the end of training. We hypothesize, therefore, that the learning rate must be re-increased and re-decayed to improve adaptation per compute spent when training on a new dataset. We note that this has not been thoroughly studied in the continual learning literature. Secondly, catastrophic forgetting is key difficulty to overcome if one is to realize the full potential of continual pro-training. Adapting to hundreds of billions of new lokens is important, but it must not come at the cost of erasing most existing knowledge in the LLM. Recent work (Scialom et al., 2022) shows, in an LLM fine-tuning setting, that replaying previous data (as little as 1%) is sufficient to mitigate forgetting to a large extent. While continually pre-training on large amounts of new data will almost surely lead to more forgetting than fine-tuning, we hypothesize that an appropriate amount of replay could mitigate forgetting even in our setting. Moreover, recent works show that pre-training (Cossu al., 2022; Ramasesh al., 2022; Mehta et al., 2023) and increasing model size (Mirzadeh et al., 2022) both help to reduce the effects of forgetting. We, therefore, expect the trend of increasing language model capacity and pre-training dataset size in tandem (Kaplan et al., 2020; Hoffmann et al., 2022; Touvron et al., 2023b) will yield models increasingly capable of continual learning (Scialom et al., 2022), suggesting that our experimental results should only improve with models scale.
even
Given the great potential for continual learning to considcrably reduce costs associated with re-training models and the potential for LLMs to be strong continual learners, we ask ourselves the following question: when simple and scalable. contional learning techniques are applied, what is the performance difference between continually pre-trained LLMs relative to LLMs pre-trained from rendom initialization on the union of all data? To answer this question, we conduct a large-scale empirical study of continual lcarning techniques for LLM pre-training. Our empirical evaluation spans large (10B parameters) and small (405M parameters) decoder-only transformer models as well as weak (English - English) and stronger (English - German) distribution shifts. Our main contributions can be sunmarized as follows:
We establish the effect of learning rate re- warming and re-docaying for decoder-only transformer-basod LLMs pre-trained using a cosine schedule, showing that re-warming and re-decaying is necessary for adaptation during continual pre-training.
2

---

Published in Transactions on Machine Learning Research (06/2024)
Average English Eval Performancs
Compute Spent Units of
Tokens (.0=3008)
Average Final Validation Loss
2.5
2.3
50
47.3
Replay (PT File)
PT on Pile
2.0
4.0
Piin SP/Ger
1.5
1-73 273
26.3
322
1.5
30
1.0
20
0.5
0.5
10
0.0
0.0
10B SP
405M SP
405M Ger.
10B
SP
405M SP
405M Ger.
10B SP
405M SP
405M Ger,
(a)
Training compute
expended
(b) Average final validation
(c) Average evaluation performance
update/re-tret re-tran the model
Figure 1: Continual pro-training decreases computational costs of updating the model while maintaining similar final validation and average evaluation performance. We report results for the Pile U (almPajaaa(SP)) (SP)//Gmmaa Ger.) baseline model trained on the union of both datasets which we consider to be an upper bound on performance. We also report performance for two continually pre-trained models. "РT on Pile" starts from a pre-trained Pile checkpoint and only uses learning rate re-warming and re-decaying, while "Replay (PT on Pile)" re-warms the learning rate, re-decays it, and uses 5% replay for SlimPajama and 25% replay for German. We observe that the combination of LR re-warming, re-decaying, and replay allows our continually pro-trained model to attain similar average performance to the baseline model while requiring substantially less compute. We note that this setting assumes that pre-trained model is available (e.g., via HuggingFace hub an in-house model designed to be continually pre-trained).
We establish the effect of replaying previous data while keeping compute constant across two distribution shifts and many replay percentages. We find that, even when updating decoder-only transformer-basee LLMs on hundreds of billions of new tokens, it is possible significantly mitigate forgetting with an appropriate amount of replay.
We demonstrate, across two model sizes and distribution shifts, that a simple and scalable combination of LR re-warming, LR re-decaying, and compute-equivalent replay allows continually pre-trained decoder-only transformer-basse LLMs attain similar performance on average models re-trained the union of all data while using significantly less compute.
We propose infinite learning rate schedules (schedules allowing smooth transition across datasets) for the continual pre-training of LLMs as promising way to circumvent optimization difficulties associated with learning rate re- re-warming.
Our code is available at http:://giaat.ccc//eeeeeeelle// com//LLettherrAI/ttt-exx through pull requests 1194 and 1200. Model checkpoints throughout continual pre-training for most of our models are available https: //huggingface. co/coolectionnoooooooooooooooooo aeeeciinnncercaaaeeae
-aapiiaan---r--rrinnn----------- preliminary version of this work was made available ICML 2023 workshop paper in (Gupta et al., 2023).
2 Main Findings and Takeaways and Examples our Method's Practicality
Our experimental results assume that continually pre-trained LLMs undergo two more pre-training phases sequentially. That is, our results apply to situations where continually pre-trained LLM is randomly initialized and pro-trained on datasets Do, Dı... DN-1 in sequence where N 2 2 and tokens(D,) 2 100B. We note that this includes situations where the LLM in question is an open-source model (Touvron et al., 2023a;b; Jiang et al., 2023; Gemma Team et al., 2024) which has already been pre-trained on Do and situations where organizations may wish train an initial LLM with the intention of continually pre-training it on new data. The new data may be similar the previous data, corresponding a weak distribution shift (c.g., the latest web-scrape of different domains), or quite different. from previous data, corresponding to strong
3

---

Published in Transactions on Machine Learning Research (06/2024)
distribution shift (e.g., data from completely new language). Our experimental evaluation accounts for these difficulties, finding that appropriately applying LR re-warming, LR re-decaying, and replay is sulficient. to match the performance of re-training across weak and strong distribution shifts and two model sizes (see Fig. 1). To make our findings as accessible to the community as possible, we now provide Rules of thumb for applying our findings:
Rules of thumb for continual pre-training
Caveat 'he following guidelines are written the best of our current knowledge.
Learning rate schedule:
the learning rale was cosine-decayed from a large value Mmarr to a small value mmm during pre-training on the initial dataset, the following guidelines can help continually pre-train your model:
Re-warming and re-decaying the learning rate from (fmea)) to (TTmi)) improves adap- tation to a new dataset, e.g. compared to continuing from small learning rates O( Decreasing the schedule's maximum learning rate can help reduce forgetting, whereas increasing it can improve adaptation.
Infinite LR schedules are promising alternatives to cosine decay schedules. They transition into a high constant learning rate across tasks, helping prevent optimization--rlated forgetting by avoiding re- we-warning the LR between tasks. They also avoid committing specific budget of tokens as final exponential decay can be used train the model to convergence at any point during training.
Replay:
We find that even small amounts of replay are good at mitigating forgetting. We recommend experimenting with different replay fractions since relative differences between them appear very early during training. For example, one may experiment with different replay fractions for a limited token budget, using evaluations relevant to their use case, to find sweet spot between adapting to the new data and mitigating performance loss due the distribution shift.
Recent works employing our techniques Two notable recent works (Glorioso et al., 2024; DeepSeek-AI et al., 2024) have successfully applied combinations of the techniques proposed herein to continually pre-train LLMs at scale, providing further evidence of their efficacy. Glorioso et al. (2024) apply LR re-warming, LR re-decaying, and 60% replay in the context a decay phase over 50B tokens of high-quality data, applied after their initial pre-training phase. The authors observe improvements their model's performance without. suffering from catastrophic forgetting. DeepSeek-AI et al. (2024) select non-decayed checkpoint from the initial pre-training phase to ensure smooth LR transition into continual pre-training (e.g., suggested in Figure 9), V2 (DeepSock-AI, 2024) on 6T tokens. The resulting model significantly improves its code generation abilities, while retaining most of its natural language generation abilities. Together, these works highlight the generality the techniques we propose herein: applying the appropriate combinations work to continually pre-train LLMss on small and large continual pre-training datascts (o.g., 50B and 6000B, respectively) and for architectures beyond the dense transformer (e.g., hybrid SoM-traasformers and sparse Mixture of Experts models, respectively).
decay, and use 30% replay of pre-training data, to continually preitrain DeepSeek-
4

---

Published in Transactions on Machine Learning Research (06/2024)
3
Related Work
3.1
Continual learning
Contimual learning (CL) approaches aim to learn from an evolving data distribution, adapting to novel data while retaining knowledge gathered through prior training (French, 1999; Rolnick et al., 2019; Caccia et al., 2020; Lesort et al., 2021). The key challenge of continual learning is to avoid forgetting past information, while also adapting to novel information. This trade-off is known the rigidity-plasticity dilemma (Mermillod et al., 2013; Ostapenko et al., 2019; Riemer et al., 2019).
CL approaches are convenient even in small-scale settings to avoid re- re-training from scratch or to bridge the data availability issue (Smith et al., 2021). However, at scale, CL is more than a convenience; it may be necessary to process huge amounts of continually gathered data. The recent increase in training scale, most notably for LLMs (Scao et al., 2022; Brown et al., 2020; Zhao et al., 2023), offers new opportunities for CL to reduce the cost. of re-training and increase efficiency for memory, computing, and storage (Prabhu el. al., 2023; Aljundi et al., 2019; Harun et al., 2023a; Veniat et al., 2021; Harum et al., 2023b). Just as federated learning can enable the sharing of compute and data between different agents co-located in space (McMahan et al., 2017; Reddi et al., 2021; Douillard et al., 2023; Ryabinin et al., 2021), continual learning allows the sharing of compute and data progressively through time and could be a useful tool for large-scale training.
Recent work shows that optimizers such as SGD and Adam have interesting knowledge retention properties in DNNs that could be beneficial at scale for CL (Lesort et al., 2023) and that just a small amount of replay could be sufficient to boost knowledge accumulation (Scialom al., 2022). In this work, we want to benefit. from the efficiency of those approaches in the context of large language models pretraining and boost them with the right learning rate scheduling and replay policy.
3.2 Pre-training, Model Scale, and Continual Learning
Several existing works evaluate the impact of pre-training and model scale on continual learning. Cossu et al. (2022) investigale pre-training scenarios for language and vision. They find that unsupervised and self- supervised pre-training plays a fundamental role in mitigating forgetting, while supervision hurts performance. Similarly, Mchta et al. (2023) find that pro-trained models forget less than randomly initializod models, due to their weights lying in flatter regions of the loss landscape. They also find that larger models forget less which is connected to the findings of Ramasesh et al. (2022); Mirzadeh et al. (2022). The former finds that pre-trained models forget less as they are scaled up, suggesting that it may be due to the hidden reprosentations growing more orthogonal with scale. The latter finds that wider neural networks forget less compared to their parameter-equivalee for transfer: equations that can predict the performance of neural network on a new task as a function of its parameter count and pre-training dataset size. The authors find that this positive transfer consistently improves as the parameter count increases. Finally, Scialom et al. (2022) show that autoregressive LLMs have a strong ability to learn contimually which they hypothesize related to their pro-training objective.
deeper counterparts. Hernandez et al. (2021) establish scaling laws
3.3 Domain Adaptive Continual Pre-training (DACPT)
Existing work considers Domain Adaptive Continual Pre-training (DACPT), a setting where a series of unlabelled domains become available to the LM sequentially and practitioners wish to train on each domain in self-supervised fashion while retaining performance across each of them. While the objective is similar to our own, we consider general-purpose pre-training datasets that mix many domains opposed to domain-specific datasets. Ke et al. (2022) assume data from previous domains is not available when training new domains and develop new technique for this setting which involves an importance mask of parameters for all previous lasks to prevent. forgetting when pre-training with a masked language modeling (MLM) objective. Gururangan et al. (2020) investigated domain and task adaptive pre-training of RoBERTa (also MLM) and contributed a sample selection strategy for efficient continual pre-training. Similarly, Xie al. (2023) also propose a data selection strategy that reduces the computational cost of continual pre-training (shown for autoregressive LMs). Qin et al. (2023) investigate re-cycling fine-tuned adapter layers of previous base LMS
5

---

Published in Transactions on Machine Learning Research (06/2024)
the initialization of new adapters for adapting contimually updated versions of the base LM to specific tasks. Recently, Wu et al. (2024) proposed LLaMA Pro, method for the continual pre-training of L.MSs that enables learning new tasks without forgetting previous knowledge. However, unlike our work which considers adapting all existing weights, LLaMA Pro roquires growing the size of the model for each new update and only adjusting the new weights.
3.4 Continual Learning for LMs Applied to Specific Domains
Several related works apply continual pre-training to specific tasks and domains (Sun et al., 2020; Jang et al., 2022a;b; Gong et al., 2022; Zan et al., 2022; Yadav et al., 2023a; Ma et al., 2023; Yang et al., 2024). While these works also utilize contimual pre-training techniques, they differ from our work by focusing on particular domains instead of general pre-training techniques, on smaller-scale datasets < 10B tokens with smaller models. The only existing work that approaches our dataset scale is (Gogoulou et al., 2023), which explores continual autoregressive language modeling across English, Danish, Icelandic, and Norwegian datasets (73B each). While they do not use replay they do re-warm and re-decay the learning rate. The only existing work that approaches our model scale is (Yang et al., 2024). They continually pre-train and instruction tune LLaMA2 on small-scale academic plant science data. This concurrent work uses a very similar continual learning setup to the one we propose: replay, LR re-warming, and LR re-decaying. While, unlike our work, they do not build a controlled experimental framework to systematically evaluate the validity of these approaches for continual pre-iinining, it nice to see further experimental evidence validating our approach.
3.5 Learning Rate Schedules
Several studics have examined the impact of different learning rate (LR) schedules on the training stability and final performance of neural networks. Goyal et al. (2018) found that a gradual warm-up of LR early on in training can help overcome optimization challenges, particularly with large mini-batch sizes. Additionally, Popel & Bojar (2018) emphasized the importance of warm-up stage when training Post-LN Transformers. On the other hand, Xiong et al. (2020) discovered that Pre-LN Transformers are more stable and may not require warm-up stage. You et al. (2019) explored the role of the LR decay and found that a large initial LR prevents the network from memorizing noisy data, whercas a small LR helps learn complex patterns. Kaplan et al. (2020) explored LR schedules for pre-training Large Language Models (LLMs) and found that schedule choice did not significantly impact performance. Correcting this erroneous finding, Hoffmann et al. (2022) found that the LR schedule does play an important role. Hoffmann et al. (2022) and Rac et al. (2021) established best practices for using cosine schedule when pre-training LLMs, which have become widely adopted. In contrast, Raffel et al. (2023) and Zhai al. (2022) explore LR schedules that follow the inverse square root decay for large-scale pre-training. Raffel et al. (2023) utilized an inverse square root decay for training LLMs, allowing flexibility in adjusting the number of training steps. In Zhai et al. (2022), authors use these schedules referred to as "infinite learning rate schedules" to train vision transformers. These schedules enable indefinite training and the evaluation of multiple training durations in a single run. We note that our proposed infinite learning rate schedules for LLMs (Sec. 7.4) are inspired by this idea.
4 Background & Methodology
In this sect ion, we provide appropriate background and methodology as it relates to continual pre-training in the context of LLMs.
4.1 Linear Warmup and Cosine Decay Schedule
Hoffmann et al. (2022) and Rae et al. (2021) established best practices for using a cosine schedule when pre-training LLMs. Specifically, they recommend starting with a linear warmup phase and decaying the learning rate to 10x its maximum value such that the end of the cosine cycle set to match the number of tokens. While the linear warmup duration differs, most works have a duration between 0.1% and 0.5% of training steps (Zhao et al., 2023). Given that many popular open-source models (Touvron et al., 2023b;a; Almazrouei et al., 2023) follow this learning rate schedule recipe, it is critical to understand its nuances for
6

---

Published in Transactions on Machine Learning Research (06/2024)
Cosine Schedule
Linear Warmup Phase
Learning Rate
Cosine Annealing Phase
Maxx
Amin
0.0
Training Iterations
Figure 2: Linear warmup and cosine annealing schedule. For illustration purposes, the schedule uses linear warmup for 10% of training iterations. However, most works have duration between 0.1% and 0.5% of training steps (Zhao et al., 2023).
continually pre-training such models. The schedule first linearly increases the learning rate over T warrau timesteps, equivalently until some timestep tann warmuup:
Te Tmon T Twarmmup
(1)
where T7 is the value of the learning rate iteration t, and Tmas is the maximum learning rate. The schedule then transitions into cosine anncaling phase over Tann timesteps, equivalently until some timestep lend Tann + lawn:
7h Amin + (Niaa Mmnin)
lann Lend
+1)
cos
π
(2)
2
Lann
where Maan is the maximum learning rate and Mmnn the minimum learning rate, Fig. illustrates these two phases.
4.2 Compute-equivalent Replay
In many of our experiments, we compare models trained with replay to models trained without it. When making such comparisons, we keep the amount of compute constant for training both models. That is, we correspondingly reduce the number of tokens seen from the new dataset to accommodate the additional tokens seen from the replay buffer. We refer to this use of replay as compute-equivvaent replay. For instance, suppose datasets Do and DD, each contain 100B tokens. We wish to compare model (a) trained sequentially on Do and D, to model (b) trained sequentially on Do and D with 5% compute equivalent replay. Model (a) will see all tokens from both datasets for a total of 200B unique tokens. Model (b) will see 100B unique tokens of Do and 95B unique tokens of DD plus 5B replayed tokens from Dy for a total of 200B tokens. Tn this way, both compared models expend the same amount of compute.
For instance, in our settings that span only two datasets (Do, Di), use replay of data from Do when training Dj. We replay the data in the order it seen when pretraining Do, as did not observe noticcable differences when reshuffling the replay data in preliminary experiments. The use of methods for selecting replay samples is left as future work. We refer to models using replay as "D, x% Replay", where æ is the percentage of data each training batch that comes from Dg. Conversely, (100% x)% of the samples each training batch will be sampled from D,. When comparing models trained with replay other configurations, we ensure that the compute is equivalent by reducing the number of Dj tokens accommodate replay tokens from Do-
7

---

Published in Transactions on Machine Learning Research (06/2024)
5 Experimental Setup
To empirically evaluate the effectiveness of continually pre-training LLMs in comparison to training LLMs from random initialization, we select recent pre-training datasets from the literature, outline practical continual pre-training settings for investigation, and select several baselines to compare with our proposed tochniques. Our goal is to fairly compare our continual pre-training techniques to baselines in a controlled setting. We do not seek to obtain stale-ofthheart performance or compare with models out of the scope of this paper.
5.1 Datasets
We use three datasets for training and validation: SlimPajama (Soboleva et al., 2023), German Common- Crawl (Laippala et al., 2022), and Pile (Gao et al., 2020). For all datasets, use the same tokenizer as Black et al. (2022) trained specifically on the Pile. To create our training set for Slim Pajaaaa we randomly sub-sample the dataset (606B Total Tokens) to form a ~299 token subset (see Table 1') that is of comparable size Pile. We also further sub-sample this SlimPajama subset create three 100B token splits of the datasct (see Sec. 7.4 for details). For each of these datasets, we follow standard practice in LLM pre-training and select sampling percentages proportionally to the amount of data available in each domain such that one pass over the dataset does not repeat samples from any domain. To create the SlimPajama validation set we simply tokenize the default validation set that has been extensively deduplicated (Soboleva et al., 2023). To create the German training and validation sets, we split and tokenized the German Common Crawl scrape, available as part of the Oscar Datasot (Laippala al., 2022), into a 195.43B token training set and a 982.6M token validation set. The Pile dataset comes pre-shuffled and mixed, we simply used the default training and validation sets. The training set is 330B tokens total, though in our experiments we only train a 300B token subset.
Table 1: Domain sizes of the 300B token training set of Slim Pajamaa We sub-sampled the SlimPajama dataset (606B total tokens) into a 300B token split to make it of comparable size to Pile. We report the size of the subsampled domains that make up SlimPajama and the sampling percentage used at training time (e.g., the percentage of samples in each batch that come from certain domain).
Dataset
Size (Tokens)
Sampling (%)
Wikipedia
11.96B
4.00
Book
12.58B
4.20
C4
79.87B
26.69
Stack Exchange
10.09B
3.37
GitHub
15.63B
5.22
Common Crawl
155.89B
52.09
Arxiv
13.25B
4.43
Total
299.28B
100.00
5.2 Continual Learning Settings
We consider three realistic continual pre-training settings in the main body and provide results for third which we believe is less warranted in the appendix. Each setting was carefully selected to expose different challenges and strengths of continual pre-training Our setups assumne that continually pre-trained LLMs undergo two or more pre- pre-training phases sequentially. At the start of each phase, we reset the optimizer states, since optimizer states may not always be available, e.g. when using open-weight models from Hugging That is, our results apply situations where a continually pre-trained LLM is randomly initialized and pre-trained datasets Do; D Dı,..., Dx-1 in sequence where N > 2. For the realistic settings consider tokens(D)) (D,) > 100B. In each case, consider the following natural baselines:
We refer readers the Pile paper (Gao al., 2020) for its composition.
8

---

Published in Transactions on Machine Learning Research (06/2024)
1- D,, and A model trained from random initialization the union of all datasets i.c. _
model trained from random initialization on individual dataset Di, 0 OSiSN.
2 settings Here we assume a model available (e.g. via hugging face or pre-trained in-house) that has been pre-trained for autoregressive language modcling on a dataset (Do) using a lincar warmup and cosine decay LR schedule. We also assume that the schedule follows existing conventions in the literature (e.g. decaying to the token budget; see Sec. 4 for details) it is the case for most performant pre-trained LLMs (Rae et al., 2021; Hoffmann et al., 2022; Touvron et al., 2023a;b). Given model pre-trained on Do, we now assume that practitioner wants to update this model on a new dataset D2 using the same self-supervised objective. We consider the following concrete varialions of the two-dataset setting:
Two datasets, weak shift: this variation, we consider Do to be the Pile (Gao et al., 2020) and D to be pro-training on SlimPajama (Soboleva et al., 2023). SlimPajama is an extensively deduplicatod version of RedPajama (Computer, 2023) which is built based on the LLaMA dataset (Touvron et al., 2023a). We consider this to be weak but realistic distribution shift as both datasets are English-language and contain overlapping domains (CommonCrawl,, GitHub, Arxiv, Wikipedia, StackExchange, Book, and C4), but SlimPajama (2023) is newer dataset than Pile (2020) and is, therefore, likely to have newer data within these overlapping domains. Therefore, despite the potential for significant overlap, we believe this transition is realistic and likely to be of interest to practitioners wishing update an LLM on a similar distribution pro-training (c.g., newly collected data of the same sources with higher quality filtering).
Two datasets, stronger shift: In this variation, we consider Do to be pre-training on the Pile (Gao et al., 2020) and D1 to be pre-training on German Common Crawl. German Common Crawl is a ~ 200B token dataset taken from the Oscar dataset (Laippala et al., 2022). We note that this constitules a stronger shift given the change language. This setting is of particular interest for practitioners wishing to augment an LLM with a new natural language, programming language, or specific domain that is notably different in vocabulary from pre-training. We note, however, that the domain strays farther and farther away from the tokenizer's training corpus, the tokenizer may become key bottleneck to performance. We leave the treatment of the tokenizer to future work.
N >> 2 settings - We also consider the following settings with more dataset transitions to investigate how well the methods considered scale with more datasets:
Three datasets, no shift. We consider an = setting, where Do, D1, D2 are each district 100B token splits of SlimPajama. This setting is primarily used evaluate the ability of our techniques scale to many future updates and to assess the performance of our proposed infinite learning rate schedules.
Domain incremental continual pre-training: This setting considers consuming the tokens of SlimPajama sequentially ordered by domain. That is, we train on a sequence N future datasets (Do, Di....,, D each of is a distinct domain of SlimPajama 300B. We note that this is similar to DACPT (Ke et al., 2022), however, we consider much larger datasets for each domain. This set senting is particularly challenging due to the distribution shift experience at the transition between each domain. While it is certainly interesting, the SlimPajama data before training on it. The poor results in this setting (Sec. A.1 of the appendix) suggest that general-purpose LLMs should be continually pre-trained on a mixture of domains if possible, not updated per domain.
believe it is unnecessarily difficult compare to mixing
5.3 Training Setup
Using GPT-NeoX (Andonian et al., 2021) based on Seggatron-DepSpperr (Shoeybi al., 2019; Microsoft, 2020), train autoregressive decoder-only transformers with causal language modeling objective. The
9

---

Published in Transactions on Machine Learning Research (06/2024)
models use Pre-LN. Each model is trained using the same tokenizer as Black et al. (2022), which was trained exclusively on the Pile via the BPE algorithm (Semnrich et al., 2016). For all models, we train with the Ad optimizer (Loshchilov & Hutter, 2019) using a batch size of 1104 and a sequence length of 2048. An epoch of training approximately corresponds to 132, 366 total training steps. As mentioned in the previous section, we reset. the optimizer states between datasets. We consider two model sizes 405M and 9.6B parameters (referred to as 10B this work) including embeddings. We train the smaller models using data parallelism across 6-GPU nodes using micro-batch size of 4. The larger model is trained using tensor parallelism (Shoeybi et al., 2020) spanning six GPUs within node and pipeline parallelimm nodes; that is, each model replica spans 24 GPUs across four nodes. We train this model on 276 nodes using gradient accumulation of 4 steps. Each model uses optimizer sharding via ZeRO-1 (Rajbhandari et al., 2020), activation checkpointing (Chen et al., 2016), activation partitioning across lensor parallel ranks, and mixed precision FP16/FP32 to reduce GPU memory consumption and fully utilize NVIDIA tensor cores during training. We provided an extended description of all hyperparamcters in the appendix (Table. 13).
al., 2019) spanning four
5.4 German and English LM Evaluation Benchmark
We measure performance on wide variety of downstream tasks, which can be broadly categorized as follows.
English Benchmarks
Commonsense Reasoning (0-shot): HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2019), PIQA (Bisk et al., 2019), OpenBookQA (Mihaylov et al., 2018), ARC-Fasy, ARC-Challenge (Clark et al., 2018)
World Knowledge (5-shot): NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017)
Reading Comprehension (0-shot): BoolQ (Clark et al., 2019)
Math: MathQA (Amini al., 2019)
Popular Aggregated Results: MMLU (5-shot) (Hendrycks et al., 2021)
German Benchmarks from (Plüster, 2023), which translated their English counterparts using GPT 3.5 API
Commonsense Reasoning (0-shot): HellaSwag-DD (Zellers et al., 2019), ARC-Challenge-DF (Clark al., 2018)
World Knowledge (5-shot): TriviaQA-DE (Joshi et al., 2017)
Popular Aggregated Results: MMLU-DE (5-shot) (Hendrycks et al., 2021)
6 Results
We focus on continual pre-training when incoming datasets are large (200B tokens+). In such settings, training is expensive, thus, it is critical to efficiently adapt to the large amount of incoming data. However, most performant LLMs (Rae et al., 2021; Hoffmann et al., 2022; Zhao et al., 2023; Touvron et al., 2023b;a) are trained with a linear warmup and cosine decay schedule with a relatively low minimum learning rate. We hypothesize that re-warming this learning rate relatively high value and subsequently re-decaying it is needed to efficiently adapt to the linear warmup duration, re-warming the LR, re-decaying the LR, and maximum learning rate magnitude adaptation and forgetting. Finding that re-warming and re-decaying increases both adaptation and forgetting, in section 6.2 we investigate whether replay can help miligate forgelting when the learning rate is re-warmed and re-decayed, Subsections 6.3 and 6.4 combine the strategies studied in the previous two sections and report their performance relative basclines for weak and strong distribution shifts and at large model scale. Finally, in section 7, we illustrate LR re- warming can cause unwanted forgetting, introduce infinite learning rate schedules as promising way to circumvent it, and compare these schedules to baselines.
dataset. To this end, in section 6.1 we study the effect of
10

---

Published in Transactions on Machine Learning Research (06/2024)
2,65
2.80
(PT on Pile) Linwer Warmup
2,60
2.70
60:50 SP (2T on Plie) 0.5% Linear Warmup
405M (PT Piini Umber Warmup
Pile Val. Loss
2.76
Pile)
Warmup
2.55
SlimPalan almmalama
2.50
2.72
2.45
2.70
2,40
2.68
2.35
10
Tokens (B)
Tokens (B)
(a)
Pile Val. Loss (300B Pile-+300 -3l0B Slim Pajama)
(b) Slim Plimaajama Val. Loss (300B Pile->300B Slim SlimPajama)
4.00
1.60
405M Ger, (PT on Pile) Linwer Warmup
3.75
German Val. Loss
1.55
4C5M Gez, (PT on File) 196 Lisear Wurmap
405M Ger. (PT on Pile) a 5% Lineer Warmup
Pile Vall Loss
3.50
Warmup
1.45
3.00
2.75
1.35
1,30
2.25
2.00
10
20
40
Tokens (B)
Tokens (B)
(c) Pile Val. Loss (300B Pilc-2200B German)
(d) German Val. Loss (300B Pile->200B German)
Figure 3: The effect of linear warmup for weak and strong distribution shifts. (a).(b) and (c), have the same legends respectively, shown in the right figures. We train 405M parameters models following a lincar warmup and cosine docay schedule with varying lincar warmup durations: 0%,0.5%, and 2% of training iterations. Each learning rale schedule decays to 0.17/aa by the end of training based on the size of the dataset. We report results for the first 50B tokens of training. In the settings explored, we observe that the duration of the warm-up phase does not appear to be impactful when continuing to pre-train.
6.1 Learning Rate Schedule
Given the influence that the learning rate can have on adaptation and the low final LR values of prominent LLMs (Rae al., 2021; Hoffmann et al., 2022; Zhao et al., 2023; Touvron et al., 2023b;a), we hypothesize that the LR should be re-warmed and re-decayed to promote adaptation during continual pre-training. In this section, we investigate the effect of linear warmup duration, re-warming the LR, re-decaying the LR, and the magnitude of the Maar when continuing to pre-train. Specifically, we evaluate their respective effects in the two-dataset weak shift setting (300B Pile - 300B SlimPajama) and the two-dataset stronger shift setting (300B Pile + 300B Slim Bajmaaama). Notably, the model trained on Do (300B tokens of Pile) follow a linear warmup and cosine decay schedule?, simulating many common open-source pre-trained LLMs.
6.1.1 The Effect of Linear Warmup for Weak and Strong Distribution Shifts.
We first investigate the effect of lincar warm-up duration on forgetting and adaptation in the two datasets, weak shift and two datasets, stronger shift settings (see Sec. 5.2 for details). The models are pre-trained 300B tokens of Pile (Gao et al,, 2020) (Do). We continue pre-train the models SlimPajama (weak shift) and German Common Crawl (stronger shift) for the first 50B tokens of training, We re-warm and re-decay the learning rate using cosine learning rate schedule set to reach its minimal value (1min = at 300B and 200B tokens, respectively. We consider warming up the learning rate for 0.5%, 1%, and 2% of Dj 's total training iterations (132366 and 86000 iterations, respectively). Since the decay happens over the
Forr all cosine decays in this paper, unless otherwise specified, we the cosine annealing phase the token budget, the linear warmup duration (Twarmup) to 1% of training iterations, and set Mmdea Tmaz
11

---

Published in Transactions on Machine Learning Research (06/2024)
remaining budget of iterations (so resp. 99.5%, 99% and 98% of the total iterations), note that this implics that the decay phase of longer warmups happens marginally faster. Additionally, we train model with no linear warm-up (0%) that immediately decays the LR from Mmaor All experiments are conducted on 405M parameter model.
Figure reports the validation losses for Do and Dj for all models throughout the first 50B tokens of continued pre-training on D1. The top row reports results for the weak distribution shift, while the bottom row reports results for the stronger distribution shift. Across both distribution shifts, we observe that models using shorter linear warmup initially forget and adapt faster than their longer warmup counterparts. This happens because they increase the LR faster which leads to faster forgetting and adaptation. In particular, the model without any warmup adapts and forgets the fastest even undergoing an initial chaotic phase (as seen in the contimual learning literature (De Lange et al., 2022)). Indeed, coupled with noisy gradients due to adapting to new distribution and the resetting of optimizer states, its large initial learning rate causes a transient spike in validation loss across both shifts. In all scenarios, however, these initial differences diminish throughout. training, leaving all models with relatively similar forgelting and adaptation after 50B tokens.
Thus, in the settings explored, the duration of the linear warm-up phase does not appear to offect forgetting adaptation measured by the validation loss when continuing to pre-truin, although it can prevent initial transient spikes in the loss.
With this in mind, we set a linear warmup duration of 1% of training iterations for subsequent experiments.
2.6
SlimPajamt Val. Loss
405M SP
Pile Val. Loss
2.75
405M SP (PT on Pile] Conadant Swit
405M SP (PI Ptin] Conatent 36-08
485M SP (PT Pile) Masi. 1.5w0d
2.70
405M SP (PT Pilee Maxd BoO4
2.4
2.60
2.55
2.2
2.50
50
100
150
200
250
300
50
100
150
200
250
300
Tokens (B)
Tokens (B)
(a) Pile Val. Loss (300B Pile-+3000 Slim Pajama))
(b) SlimPajams Val. Loss (300B Pile >30B (iimPajama)
4.00
406 Pie Ger. Piall
3.75
488 Ger.
Constant Sw.04
German Val. Loss
40630 Ger. IPT on PDo) Constant Be 05
Val. LOss
3.50
600 Ger, IPT on Pilei Maat. 1.50 04
3.25
1.35
406M Ger,
MuLL 30.04
406M Ger.
MaxLR
3,00
1.30
2.75
1.25
2.50
1.20
2.25
1.15
2.00
1.10
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
Tokens (B)
Tokens (B)
(c) Pile Val. (300B Pile-+200B German)
(d) German Val. Loss (300B Pile-+200B German)
Figure 4: The effect of re-warming and re-decaying the learning rate on adaptation and forgetting.
We consider two constant baselines and three models that re- warm and re-decay. One baseline continues training from Mmin of pre-training (3- 10-4) while the other warms up to Mmmaz from pre-training (3 10-1). For the models that re-warm and re-decay vary Tmx {1.5 10 10 10 100110040000 4,6 4,6-10 10-4). All models except the Mnii baseline use linear warmup for 1% training iteration. The non-baseline models cosine decay the learning to reach 0.1 Tmar by the end of training. We observe that re-warming and re-decaying the learning rate is needed to best adapt to the new dataset. Small increases or decreases in Mmaz allow to trade-off between more or less adaptation. stronger distribution shift seems to be a catalyst for both forgetting and adaptation.
12

---

Published in Transactions on Machine Learning Research (06/2024)
6.1.2 The effect of re-warming, re-decaying, and varying mmma for Weak and Strong Distribution Shifts.
We now investigate the benefits of re-warming and re-decaying the learning rate (e.g, following a cosine schedule) for different values of 7maz- Specifically, we compare these models to two natural baselines: a model that does not re-warm, staying constant at mmm (3 10- and a model that re-warms to the pre-training Tmas 3-10-4) but does not re-decay. We use the same two two-dataset settings: we first pre-train on the Pile (Do) for 300B tokens and continually pre-train our model on SlimPajama (weak shift) German Common Crawl (strong shift) our Di datasets. The continual pre-training is conducted for the full size (300B and 200B tokens, respectively) of the datascts. The models that re-warm and re-decay the LR consider three strategies: re-warming to half the pre-training's Mmaz (1.5 10 4), re-warming to the same Maao as pre-training (3 10 4), and re- re-warming to twice the Mmar of pre-training (6- 10-"). In all cases, the learning rate is cosine-decayed after linear warmup to reach T/min = 0.1 - Mmar by the end of training. Finally, we consider models trained on Do U D1 as a third baseline umioontraaine, to provide an upper bound on performance.
Figure 4 reports validation losses for the Do and Dj datasets throughout the continual pre-training of all models. The top row of plots reports results for the weak distribution shift (300B Pile->30BB SP), while the bottom row reports results for the stronger distribution shift (300B Pile-200BB Ger.). For both shifts, the constant Amin learning rate model achieves the least forgetting on Do. It also adapts the least on Dj for the stronger shift, however, for the weak shift it adapts more than the constant Maar basclinc. When comparing these constant LR baselines to the models that, re-warm and re-decay on both shifts considered, we observe that the latter models adapt better to the new dataset by a significant margin for both distribution shifts, This shows that re- warming and re-decaying are necessary to maximize adaptation to the new dataset when contimually pre-training LLMs. Among the models that re-warm and re-decay the LR, we observe that varying the learning rate causes small differences in adaptation and forgetting: higher values of Mmaz lead to more forgetting and more adaptation while the opposite is true for lower values. When comparing the constant LR baselines to the union-traine trained baseline, we observe that the final validation loss for Dn is significantly higher than the distribution shift, but interestingly for the stronger distribution shift, the constant basclines achieve lower D, validation loss than the union-trained model. The stronger distribution shift appears to exacerbate the relative forgetting and ability of the models to adapt in the context of continually pretrained LLMs. When comparing models continually pre-trained with re-warming and re-decaying to the union baseline, we note that these models adapt better (lower final validation loss) to D2 than the union bascline. However, these models experience significant forgetting on Do, showing the need for replay to make these models competitive with the union baseline.
model's on both distribution shifts, This is also the case for Dj on the weak
In summory, continually pre-training LLMs, both re- re-wanning and re-decaying are necessary to marimize adaplation to the new dalasel; small increases or decreases in mmar allow to trade-off between more or less adaptation; stronger distribution shift between Do and Dj exacerbates forgetting and enhances adoptation; and the duration of linear warm-up phase does not appear to be impactful on forgetting or adaplation.
6.2 The Effect of Replay
In this subsection, we explore the effect of compute-oquivalent replay when continually pre-training models that re-warm and re-decay the learning rate.
Given the need to mitigate forgetting when re-warming and re-decaying, we move on to investigate the effects of replay in our weak and strong-shift continued pre-training scenarios. Specifically, we use compete equivalent replay (see Sec. 4.2 for details) where replay tokens from Do are added at the cost of removing the equivalent number of Dj tokens from the budget. Following the same two dataset settings, the model is pre-trained on Do (Pile) for 300B tokens. This is followed by continual pre-training a SlimPajama (weak shift) or German Common Crawl (strong shift). For more details regarding the sctup, plcase see Section 5.2. Our continued pre-training is conducted for the full size of the respective datasets, which is 300B tokens for SlimPajana (weak shift) and 200B tokens for German Common Crawl (strong shift). We consider 1%, 5%, 10,, and 50% replay for both shifts and add 0.5% and 25% replay runs for the weak and strong distribution shifts respectively. We consider two basclines to put these results into a broader context. The first bascline is model trained on Dj without replay. The second baseline model is trained from random initialization on
13

---

Published in Transactions on Machine Learning Research (06/2024)
2,55
2.85
405 (ix-SPNIDR Finel
2,50
2.80
600 SP (PT PE()
468 SP 6.5% Replay (PT Mil
2.45
2.75
SP Baglay
2.10
SlimPalam Val
2.70
600M0 SP 5% Reglay (PT Pho)
4:0 M4 SP 10% Reploy (PI on Plln)
2.35
2.65
SP 50% Replay Fücl
Pile
2.30
2.60
2.25
2.20
2.15
100
250
250
300
Tokens (B)
Tokens (B)
(a) Pile Val. Loss (300B Pile-x300B Slim Pajama)
(b)
SP Val. Loss (300B Pile->300B Slim Pajamaa)
1.00
1.6
Gar.
3.75
German Val, Loss
1.5
605M Ger, IPT on PMee
Ger 1%
(PT Pile
Pile Val. Loss
3.50
5% Roplay (PT on Pic)
605M Ger 10% Regloy (PT on Pio)
3.25
405M Gen 25% Beplay (PT on Pao)
3.00
Replay Pis)
1.3
2.75
2.50
1.2
2.25
2.00
1.1
0
25
50
75
100
125
150
175
200
0
25
50
75
100
125
150
175
200
Tokens (B)
Tokens (B)
(c) Pile Val. Loss (300R Pile-2000B German)
(d) German Val. Loss (300B Pile-+200B German)
Figure 5: The effect of replay at 405M scale for weak and strong distribution shifts. We report Pile validation loss (left) and SlimPajama//C GGenann validation (right top/botton) during training. Each model is trained from checkpoint pre-trained 300B tokens of Pile. The blue dotted line reports the final validation loss for models trained on FileLSSiiPaaaaaa or PileUGerman data, totaling 600B and 500B tokens datasets respectively. We observe that replay significantly reduces forgetting across both shifts, however, the stronger shift requires more replay to miligate forgetting to the same extent.
union of Do and Dj for 600B lokens (SlimPajama) and 500B Lokens (German Common Crawl). The latter bascline reflects the practice of fully re-training the model to update instcad of continually pre-training the existing model. All models wa-marm and re-decay the learning rate using cosine decay schedule fit to their token budget with the same Mmox (3 10 10 and Main (3 10-3) values during pre-training on Do-
Validation Loss Comparison The results in Fig. 5 (top and bottom) show the evolution of the validation loss during continual pre-training on the respective D2 datasets. Table reports the average final validation loss for each of these models. The final loss is averaged over the last 100 iterations of training sampled at intervals of 10 iterations. We consistently observe across both distribution shifts that even the lowest tested replay of 1% significantly reduces forgetting on Pile compared to the no-replay basclines. This effect is more pronounced in the strong-shift scenario due the larger amount forgetting in this setting. We observe little impact on downstream performance for 1%, 5%, and 10% replay when compared to the 0% bascline, showing that the forgetting benefits of replay come at little cost in our setting. However, when using an extreme amount of replay (50%), observe that the model adapts relatively significantly worse to Dj. Interestingly, for both datasets, the 50% replay models attain surpass the final average validation performance of the baseline training on D2 U Dy- This is curious as these model have seen 150B (for SlimPajama) and 100B (for German) fewer tokens of D, than their respective baselines.
In summary, we find that, when re-warming and re-decaying the LR in a continual pre-training contest, replay a useful tool for reducing forgetting. For both distribution shiftss, using an appropriate amount of replay yields simillar fenal validation loss to the Dj Do baseline. Moreover, for both shifts, the use of reploy
14

---

Published in Transactions on Machine Learning Research (06/2024)
Final loss of English-only 405M parameter models trained with varying amounts of replay. The loss is averaged over the last 100 iterations of training sampled at intervals of 10 iterations. The standard error for these measurements was computed but not reported as it was < 0.001 for all models. We observe that models using more replay achieve better odlatation-forgetting tradc-off (AVG Loss). Interestingly, the model using 50% replay archives nearly identical loss values while seeing 150B fewer tokens on SlimPajama.
Table 2:
Training Tokens
Validation
Dy Pile
SiimPajama/GGrmaa
AVG
300B Pile 300B SP
2.44
2.50
2.17
300B Pile 300B SP (0.5% Replay)
2.27
2.50
2.39
300B Pile 300B SP (1% Replay)
2.26
2.50
2.38
300B Pile + 300B SP (5% Replay)
2.23
2.51
2.37
300B Pile + 300B SP (10% Replay)
2.21
2.51
2.36
300B Pile + 300B SP (50% Replay)
2.54
600B Pile SP
2.17
2.53
2.35
200B Ger.
1.11
300B Pile 200B Ger. (1% Replay)
2.83
1.12
1.97
300B Pile 200B Ger. (5% Replay)
2.57
1.12
1.85
300B Pile + 200B Ger. (10% Replay)
2.46
1.13
1.80
300B Pile + 200B Ger. (25% Replay)
2.33
1.16
1.75
300B Pile + 200B Ger. (50% Replay)
2.21
1.73
500B Pile Ger.
2.26
1.25
1.75
seems negligibly affect adaplation to the downstreasm datasel, showing that reducing forgetling via reploy comes at very little cost when continually pre-training LLMs.
6.3 Continual Pre-training Final Performance for Weak and Strong Distribution Shifts.
In this subsection, we compare two continually pre-trained 405M parameter models to several baselines in the Iwo dalasel weak shift (Pile ++ SlimPajama) and two dalasel strong shift (Pile + German) settings. Our main goal is to determine how the differences in distribution shift affect final performance.
Continually Pre-trained Models To ablate the performance of combining LRR re-warming and re-decaying with replay, we opt to train one model that exclusively re- warmas and re-decays the learning rate and another that combines both techniques. Given results from the previous section showing that many replay percentages obtain similar average validation loss, we select 5% replay for the weak shift setting and 25% replay for the stronger shift setting because these percentages allow to more counterparts (due to compute-equivalent replay) with similar average final validation loss. For both models, we re-warm to the Mmar of pre-training (3 10-4) and re-decay it using cosine decay schedule set to reach Tmnin by the end of continual pre-training More hyperparameters are reported in Table 13 of the appendix.
tokens than their higher replay
Baselines We also train several baselines. Two baselines are trained on Dp and Dj respectively while the third is trained on the union of each dataset Do Dj. We consider the model trained on Do U to be an upper bound on performance as represents an expensive full re-training. The baselines trained on individual datasets can be seen as compute-equivalent alternatives to continual pre-training (e.g., one could opt to train model from random initialization on D) instead of continually pre-training it).
6.3.1 Final Performance Evaluated by Loss
Figure 6 reports the validation loss during continual pre-training of 405M parameter models for weak (top) and strong (bottom) shifts. Table 3 reports the average (over the last 100 iterations) final loss value for these models. Since the transition from English to German represents starker distribution shift than Pile to SlimPajama, training on German leads to significantly more forgetting on Pile (Do) for the contimually pro-trained model without replay (0.27 vs 1.39 for weak and strong shifts respectively). However, choosing 25% replay to handle the starker shift significantly reduces the amount of forgetting on Pile, a reduction of 1.23 in terms of final loss. When comparing continually pre-trained models to baselines trained exclusively
15

---

Published in Transactions on Machine Learning Research (06/2024)
2.8
3.1
Slim Pajama Val Loss
40034 Pile
4.EM SP
SP (PT Pis)
Pile Val. Loss
2.6
2.9
4.5M Replay (PT Pils)
4.6M Pile SP
2.8
2.4
2.7
2.3
2.6
2.1
2.4
100
200
300
400
500
600
100
200
300
400
500
600
Tokens (B)
Tokens (B)
(a)
405M Pile Val. Loss (300B Pile-+300B SlimPajama)
(b) 405M SP Val. Loss (300B Pile-330BB Slim Pajama)
3.0
4.0
German Val. Loss
Pile Val. Loss
3.5
2.5
405M Gun
40GM File
Pile)
3.0
2.0
405M 29% Replay (PT Pile)
405M File u Gar.
2.5
2.0
1.0
200
400
500
500
Tokens (B)
Tokens (B)
(c) 405M Pile Val. Loss (300B Pile +200 Corman]
(d) 405M German Val. Loss (300B Pile 200B German)
Figure 6: Final loss of 405M parameter models trained on two distribution shifts. Figures (a) and (b) are duplicated from Fig. for convenient comparison.
provided three baselines and two contimually pre-trained models. The baselines (light blue, dark blue, and maroon) are trained from random initialization on 300B tokens of SlimPajama, 300B tokens of Pile, and the union of both datasets (600B tokens). The continually pre-trained models (black and violet) start from a checkpoint pre-trained on 300B tokens of Pile (dark blue curve) and use 0% and 5% replay, respectively. We observe that for both distribution shifts, the combination of wa-warming the learning rate and using a small percentage of replay helps to strike a balance between forgetting and adaptation. Importantly, we note that the use of replay minimally affects downstream performance compared to the models using 0% replay.
Table 3: Final loss of continually pre-trained English-only & English- German models. All models have 405M parameters. The loss is averaged over the last 100 iterations training sampled at intervals of 10 iterations. The standard error for these measurements was computed but is not reported as was < 0.001 for all models. We observe that even for starker distribution shifts, the combination of LR warmup and 25% replay helps to match the average performance of the Pilc U German model.
Training
Validation Dj German/S5 SSP
D: Pile
AVG
English
HellaS wag- DE
Pile
2.70
2.44
33.
27.00
300H SP*
2.51
2.33
2.52
34.11
27.03
>
2.50
2.47
27.43
300D Pile 300B SP (5% Replay)
2.23
2.51
2.37
35.14
27.09
2.53
2.35
34.30
27.36
300B Pile
2.17
2.70
2.41
33.
27.09
200H German
3.97
1.17
2.57
27.74
20.53
300B Pile 200B German
3.56
1.11
2.31
29
31.23
3008 Pile Gorman (25% Replay)
2.33
1.36
1.75
32.48
31.04
500m Pile German
2.26
1.25
1.75
32.43
30.45
on DI, we obscrve that the continually pro-trained models always have lower validation loss across both distribution shifts. When comparing the continually pre-trained models with the Du U basclines we find that both models achieve nearly identical (weak shift) identical (strong shift) average final validation losses.
16

---

Published in Transactions on Machine Learning Research (06/2024)
This shows that for strong and weak distribution shifts, simple and scalable combination of LR, re-warming, LR re-decaying, and replay can achieve similar performance to the Do U DD baseline.
6.3.2 Final Performance Evaluated by Zero-shot and Few-shot Results on Popular LM Benchmarks
While final accuracy provides a good measure of performance on the pre-training objective, LLMs' abilities are typically judged by their performance on evaluation tasks. With the caveat that we use base models, that is our models have not been instruction-tune tuned, fine-tuned, or adapted to human preferences in any way, we present their evaluation on popular benchmarks in this section. Furthermore, evaluation of German-trained models. We refer the reader to Sec. 5.4 of the main manuscripi and Sec. A.6 of the appendix for a more detailed description of the chosen evaluation tasks.
also provide a qualitative
Table 3 reports the average accuracy of each model for our English evaluation tasks and the normalized accuracy for the German HellaSwag evaluation task. We do not report the average German evaluation score as it not informative due to evaluations having near-random chance accuracy (sce Table 11). We observe that English models consistently outperform German models on the English evaluations. However, the strong replay used with the 25% replay German model helps to reduce this gap. English models' English evaluation performance is very similar with a range of 1.19 between the highest and lowest values. We suspect that there is significant noise in the evaluation process for base models of this size and believe that the differences are likely not significant. That being said, the continually pre-trained model with LR re-warming, LR re-decaying, and replay does improve on the Do U D1 model. When evaluating German-trained models on English evaluation tasks, we see consistent improvements for models using more replay. We note that once again the model trained with LR re-warming, LR re-decaying, and replay does improve on the Do D1 model. Turning the German HellaSwag results we observe that German models consistently outperform their English counterparts. Among German-trained models, the continually trained models outperform the mmion-trained model and the model trained exclusively on German
Given the poor performance of German models on all German evaluation tasks except HellaSwag (the same as Euglish models on average), we further investigated their understanding of German by conducting a short qualitative study of model generations. In section A.5 of the appendix, we select five German prompts that contain various peculiarities of the German language (see Tab. 8 of the appendix). We then generate fixed token-length response for each of the models trained German Common Crawl. As bascline, we also evaluate the model trained only on the Pile. Despite the poor quality of generations at small model scale, we find that there is an observable improvement in the generative quality of German-language outputs from the models trained on German Common Crawl when compared to the Pile bascline, which tends to be systematically off-topic. This suggests that while our German-trained models have learned about the language, the evaluation tasks are too difficult to pick it up at the 405M parameter scale. Another reason is that the German dataset is smaller than the English datasets considered, and contains only web-scraped data, opposed to the more sophisticated English datasets used in this work.
In summary, for weak and stronger distribution shifts alike, it is possible to achieve competätive performance to a model trained on Do D,, by utilizing simple and scalable combination of LR re-warming, LR re-decaying, and replay. This is true for final validation loss and averaged language model evaluation scores, showing that this powerful combination of simple techniques can equip language models with new knowledge with little compromise to eristing knowledge.
6.4 Continual Pre-training Final Performance at Different Model Scales
In this subsection, we establish the effect of increasing parameter count by an order of magnitude on the final performance of continual pre-training. To accomplish this we compare two continually pre-trained models to several baselines at 405M and 10B parameter model sizes in the two dataset weak shift (Pile + Slimm SlimPajama) and two dataset strong shift (Pile + German) settings.
To ablate the performance of combining LR ro-warming and re-docaying with replay, we opt to train one model that exclusively re- re-warms and re-decays the learning rate and another that combines both techniques. Given results from (Sec. 6.2) for the weak distribution shifts, showing that.
Continually Pro-trained Models
17

---

Published in Transactions on Machine Learning Research (06/2024)
many replay percentages obtain similar average validation loss, select 5% replay for both model scales because these percentages allow us to see more new tokens than their higher replay counterparts (due to compute-equivvlent replay) with a similar average final validation loss. For both models, we re-warm to the Mmar of pro-training (3 10-4) and re-decay using cosine anncaling set to reach 7/mán by the end of continual pre-training. More hyperparameters are reported Table of the appendix.
Baselines We also train several baselines. Two baselines are trained on Do and D1 respectively while the third is trained on Do U-p We consider the model trained on Dn UD to be an upper bound on performance as it represents an expensive full re-training. The baselines trained on individual datasets can be seen as computeeeqquvvlenn alternatives to continual pre-training (e.g., one could opt to train model from random initialization on D instead of continually pre-training it).
6.4.1
Final Performance Evaluated by Loss
2.4
SlimPajama Val. Loss
2.6
108
2.3
108 SP
Pile Val. Loss
2.2
2.5
10B SP (PT Pik)
108 Replay
2.4
2.3
2.0
1.9
2.1
2.0
1.7
100
200
300
400
500
600
100
200
300
400
500
600
Tokens (B)
Tokens (B)
(a) 10B
Pile Validation Loss (300B Pile File-B300B Slim Pajuaajama)
(b)
10B SP Validation Loss (300B Pile +300B SlimPajama
3.1
Pajmaajaaa Val. Loss
2.7
3.0
4.65M SP (PT on Pūk)
Pile Val. Loss
4.05M4 SP 5% Beplay (PT on Pile.
2.5
2.8
2.4
2.7
2.6
2.2
2.5
2.4
200
Tokens (B)
Tokens (B)
(c) 405M Pile Val. Loss (300B Pile-+300B Slim Pajama)
(d) 405M SP Val. Loss (300B Pile-3300R Slim Pajama)
Figure 7: Validation loss during continual pre-training of 10B (top) and 405M (bottom) parameter
models. At each model scale provided three baselines and two continually pre-trained models. The baselines (light blue, dark blue, and maroon) are trained from random initialization on 300B tokens of SlimPajama, 300B tokens of Pile, and the union of both datasets (600B tokens). The continually pre-trained models (black and violet) start from a checkpoint pre-trained on 300B tokens of Pile (dark blue curve) and use 0% and 5% replay, respectively. We observe that for both model sizes, the combination of LR re-warming, LR re-decaying, and using a small percentage of replay helps to strike balance between forgetting and adaptation. Importantly, we note that the use of replay minimally affects downstream performance compared to the models using 0% replay (black and violet curves overlap in figures (b) and (d)).
Figure 7 reports the validation loss during continual pre-training for 405M and 10B models, while Table 4 reports the average (over the last 100 iterations) final loss value for each model. As expected, we observe that all basclines and continually pre-trained models consistently improve in perplexity on both datasets from increasing parameter count. For the 405M models, we observe that Pile U SP achieves identical validation loss each dataset to the baselines trained individually on them. In contrast, the 10B parameter model
18

---

Published in Transactions on Machine Learning Research (06/2024)
Table Final loss 10B and 405M parameter models. The loss is averaged over the last 100 iterations of training sampled at intervals of 10 itcrations. The standard error for these measurements was computed but is not reported as was 0.001 for all models. We observe that at both model scales, learning rate ro-warming combined with 5% replay approaches the average loss value of joint training.
Validation Loss
Model Size
Training Tokens
D&Pile
P.SSimPajaaa
AVG
300B Pile
1.75
2.21
1.99
300B SP
2.08
2.05
2.07
300B SP
1.88
300B Pile
300B SP (5% Replay)
1.79
2,00
1.89
000B Pile U SP
1.72
2.02
1.87
300B Pile
2.17
2.70
2:11
300B SP
2.51
2.53
2.52
405M
300B Pile 300B SP
2.44
2.50
2.47
300B Pile 300B SP (5% Replay)
2.23
2.51
2.37
000B Pile SP
2.17
trained on Pile U SP outperforms the models trained individually on each. We hypothesize that this happens due to larger models having more capacity, thus being capable of learning at a higher rate for longer. We observe that replaying 5% pile data when continuing to pre-train on SlimPajama reduces forgetting on Pile validation by 0.19 and 0.21 for 10B and 405M parameter models respectively. The negligible difference in recdutiong from replay despite the order of magnitude difference in parameters between both forgetting-recuction from replay. We
models suggests that model scale has limited negative influence believe this is because larger models forget. less by default. Indeed, the models trained without replay from a pre-trained Pile checkpoint forget 0.23 and 0.27 nats of Pile perplexity for 10B and 405M respectively. While the difference is small, this suggests that larger models forget less, confirming our hypothesis. When comparing the average final validation loss the models with 5% replay and baselines trained on the union of both datasets, we notice that there only a difference of 0.02 for both model sizes. This shows that for weak but realistic distribution shifts at two model scales, continual pre-training can achieve similar performance the expensive re-training baseline.
6.4.2 Final Performance Evaluated by Zero-shot and Few-shot Results on Popular LM Benchmarks
While final accuracy provides good measure of performance on the pre-training objective, LLMs abilities are typically judged by their performance on evaluation tasks. With the caveat that we use base models, that is our models have not been instruction-tune -unned, fine-tuned, adapted to human preferences in any way, we present their evaluation on popular benchmarks in this section. We refer the reader to Sec. 5.4 of the main manuscript and Sec. A.6 of the appendix for more detailed description of the chosen evaluation tasks.
Table 5: All Zero-shot and Few-shot results on popular LM benchmarks. Normalized accuracy is reported for HellaSwag and exact match (EM) is reported for NaturalQuestions and TriviaQA. All other tasks report unnormalized accuracy. MMLU and TriviaQA are evaluated 5-shot, while all other tasks are zero-shot. We observe on average, as expected, that 10B parameter models outperform their 405M counterparts and that the English-only 405M models outperform their German German-trained counterparts.
Midel Troinnng
HolloSang
ARC-e
BoolQ
MathQA
MMMLU
OBQA
PIQA
TSQAI
TR3A22
NQQ
T:QQA
AVG
20013
34.81
03..9
27.34
27.25
27:201
76.52
20.45
41.022
63.45
5001 SP
70.35
36.77
71.05
68.01
24.75
27.42
25:20
76.99
05.04
22:40
33.99
11.25
52.63
45.37
10B
3008 Pile 3008 SP
78.56
37.317
78.022
73.18
26.43
29.54
30.20
78.51
66.30
23.26
35.04
12.99
57.94
47.58
2003
2008 SP (5% Heplas)
74.24
26.83
30.601
78.02
68.67
13.32
57.96
47.68
0008
73.90
38.25
73.67
72.05
26.83
37.78
27.80
77.58
67.32
23.13
36.16
12.41
56.73
48.00
3003 Pile
41.95
22.01
51.77
59.24
24.12
26.18
19.50
06.30
53.83
24.85
42.11
0.91
3.97
33.55
50033 SP
46.22
21.76
54.08
59.63
22.71
26.18
19.60
08 23
49.50
22.64
38.65
1.69
14.18
$1.11
405M
900T3 Pile 500B SP
46.22
22,70
54.04
57.43
24.22
25.25
21.
00:20
51.46
23.13
36.01
2.02
15.25
34.53
2008 SP (5%) Replay)
55.01
24.22
25.94
20.60
23.38
15.70
33.14
6003 Pile
45.06
23.55
52.09
55.57
23.12
26.65
18.20
69:37
52.72
23.50
38.81
1.72
14.03
34.30
TIQA:
Truthful QA, WG: WinoGrande, NQ: Natural Questione, OBQA: OpenBook QA, TrQA-TriviaQA
Table. reports English-language LM evaluation results for our english-only continually pre-trained LLMs.
Normalized accuracy is reported for HellaSwag and exact match (EM) is reported for NaturalQuestions and
19

---

Published in Transactions on Machine Learning Research (06/2024)
TriviaQA. All other tasks report unnormalized accuracy. As expected, we observe that the larger (10B) models achieve stronger performance than their smaller counterparts and that models trained on more tokens always achieve better performance than models trained on fewer tokens. For both model scales, we observe that the models pre-trained continually using a combination of learning rate re-warming and 5% replay approach (10B) or surpass (405M) the performance of the models trained on the union of both datasets in terms of average accuracy. When comparing union-trained models to continually pre-trained models for different tasks, we observe for the 10B parameter models that the 5% replay model and union-trained model different tasks with notable differences being OpenBookQQ in favor of the
exchange best performance replay model and MML.U in favor of the union model. While this degradation in MMLU performance between both models could be cause for concern, we suspect it is due to the limited amount of training data used in our study. Following the initial release of this work, Glorioso et al. (2024) successfully applied our techniques without MMLU performance degradation; in fact, their performance on MMLU is improved during continual pro-training. For the 405M parameter models, the 5% replay model and union-trained model exchange best performance on different tasks with no notable differences. At both model scales, the replay model improves over the model only using re- warming though differences are small and may be attributable to noise.
In summary, we find that models continually pre-trained with a combination of LR re-warming, LR re-decaying, and replay ezceed the average performance (e.g., w.r.t. final validation loss and evaluation accuracy) of baselines trained from random initialization on individual datasets and achieve comparable evaluation performance on average to the expensive re- re-training boseline (trained on the union of both datasets). These results show that the benefits of continual pre-training hold at the 10B porameter scale, suggesting that this may also be the case for models with an order of magnitude more porameters (e.g. for 100 porameters).
7 Understanding and Circumventing the Pathologies of Re-warming
In this section, find that LR re-warming causes unwanted forgetting, introduce infinite learning rate schedules as promising way to circumvent it, and compare these schodules to basclines from the literature.
7.1 werming on the Same Data
In section 6.1, we have seen that contimuing to pre-train on new data initially leads to a quick increase of the loss on past data, which motivated the use of replay. The increase of the loss was, in particular, more pronounced for greater Mmar values. One hypothesis for the increase loss is that it is mostly due to a distribution shift between the pre-training datasets and associated negative transfer. To assess this hypothesis, re- ra-warm and re-decay over 300B tokens in setting with no distribution shift. That is, we follow a similar methodology as in our experiments from Fig. 4 but continue to pre-train on Pile as Dj
As seen in Fig. 8, independently of the distribution shift, rewarming the learning rate appears to be a significant cause of the increase in loss seen previously in Fig. 4 when starting to continue to pre-train, as evidenced by the increase in perplexity when re-warming the learning rate while training on the same distribution. For example, the re- re-warming leads to a peak increase of the Pile validation loss of 0.1 relative to its initial value with a 1/mar 3- 10-4 we continue pre-training on Pile, which might be contrasted with the Pile validation loss increase of 0.35 with the same learning rate schedule when continuing to pre- pre-train SlimPajama as in Fig. 4. It is noteworthy that the higher the ra-warming, the more pronounced this effect is, as seen with the Tmas = 6- 10-4 curve when continuing to pre-train on Pile (with a peak loss increase of 0.2) vs continuing to pre-train on SlimPajama (peak loss increase of 0.45).
In particular, after ro- ra-manning, models fail to recover quickly from the performance hit due to rewarming the learning rate even when training on the same dataset. This motivates finding alternatives to learning rate schedules requiring re-warming in order to improve the efficiency of continual pre-training.
20

---

Published in Transactions on Machine Learning Research (06/2024)
2.6
405M File (FT on Fil) Constant Od
File
Constant 3e
2.6
File IPT on File) MuLLR 1.5
Pile Val. Loss
2.5
405M File IF7 on File) MaaLR 30-04
Pile Val. Loss
2.5
405M Pile IPT on Pila) MiaLi 6w44
2.4
2.4
405 SP (PT on Piin] Constont Sw-0i
2.3
2.3
405M (PT on Pils) Constant 3a-05
(PT Pilet Maod Se-04
2.2
2.2
40S SP (PT on Piiel MaxUR de u6
20
40
60
100
40
60
8D
100
Tokens (B)
Tokens (B)
(a)
Re-warming (300B Pile-+300B Pile)
(b) Re-warming (300B Pile->300B SlinPajana)
Figure 8: Pile validation loss when continuing to pre-train on Pile (a) and SlimPajama (b). Each curve starts from the same checkpoint pre-trained on 300B tokens of Pile but is trained with a different. maximum learning rate. As we focus on the effect of re-warming the learning rate, we only show curves for the first 100B tokens. We observe that every model that re-increases its learning rate from the minimum learning rate of the initial pre-training (e.g., all models except constant) sees an increase in loss.
7.2 Infinite Learning Rate Schedules
In this subsection, investigate the use of learning rate schedules that intrinsically may not require warming. The motivations are twofold. On the one hand, a cosine decay schedule requires us to know the total number of tokens we want to pre-train on in advance. This limits the ability to continue to pre-train a converged checkpoint. On the other hand, we saw in the previous section that when continuing to pre-train a model that was initially pre-trained with a cosine decay schedule ending with a small learning rate, re-warming the learning rate from its minimumm value is needed to best adapt to the new dataset. However, as seen in the previous subsection, we observe that re-wanming the learning rate can exacerbate forgetting.
Thus, we explore "Infinite Learning rate schedules" (Zhai et al., 2022) which keep the learning rate at constant value across all new tasks. This can help prevent forgetting by avoiding re-warming the learning on new tasks. Additionally, this schedule is independent of the total number of tokens making it more suitable for continual learning selups compared to repeating the cosine decay schedule cyclically for each new dataset. As we saw, since a high constant learning rate is also suboptimal, we opt to perform a fast annealing of the learning rate at the end of pro-training, over a limited amount of tokens. We hope that this will recover the performance advantage of re-decaying the learning rate, while allowing the use of pre-annealing checkpoint. when continuing to pre-train.
The infinite learning rate schedules considered have 4 phases:
1. Linear warm-up phase - As before, the learning rate is initially increased to some maximum value Mmaz over T warrap timesteps, or equivalently until timestep End = T warrmu-- The Icarning rate undergoes warm-up only once (during the first task) and does not require re-warming for future tasks.
2. Cooldown phase - During this stage the learning rate undergoes a cooldown phase where the learning rate is gradually decayed to constant value Meonst according to some decay function for over Ted timesteps from timestep fnd to fconst fed + Ted. This stage also occurs only once during the first task.
Constant phasc The learning rate then remains constant for all future tasks over T Toonst timesteps from timestep tconst to tann Econst + coonst- The checkpoint obtained at the end of this phase the one should resume from when continuing to pretrain on new dataset.
Annealing phase- The learning rate is annealed a small value Mmin over T Tann timesteps from timestep lann to Lend Lann Tanny helping train the model to convergence before being deployed.
21

---

Published in Transactions on Machine Learning Research (06/2024)
Thus, the infinite learning rate schedules considered here can be written
t
Mmaz
E [0, Lea)
(worm-up)
T Tuarmup
Sea(t)
E (teds tconat]
(cooldown)
7/1
Tronst
E (Leousts lann
(constand)
Ammn
aand
Meonst
t E (tanns tend]
(annealing)
Aconst
In this work, we consider the two following functions for the cooldown phase's decay fea:
Cosine decay
fea(t) Aconst + Tmaz Moonst
(1 cos
- ted Loonst - tod
(3)
Inverse Square Root docay
Moonst Mmax h ( t ted
(ea(t) Tmao +
(4)
h(1)
t const tdd
where
h(x) = VI + ax
with a controlling the steepness of the inverse square root decay. We shift and stretch the Inverse Square root decay to adapt to the interval (hod, Lomat]]
The three different schedules are seen in Fig. 9 (b).
We now compare infinite learning rate schedules to a cosine decay schedule. We first explore a simple single-dalased, pre-training selup to evaluate the leasibility of the schedule for LLM pre-training. Subsequently, we explore its benefits in our three datasets, no shift setting.
7.3 Comparing Cosine Decay to Variants of our Infinite Schedules
Here we compare cosine decay schedule with infinite learning rate schedules in the common single-datasct pre-training setting. The aim of these experiments is to test if the infinite learning rate schedules can result in models that perform as well as models trained with a conventional cosine decay schedule.
The models are pro-trained on 300B tokens of SlimPajama from random initialization. Figure shows the training curves of 405M parameter models trained on SlimPajama with different learning rate schedules. We observe that all methods reach similar final validation loss showing that infinite learning rate schedules can be used for the common case of pre-training as well. These schedules additionally have the advantage that one can start annealing at any time in the constant phase to efficiently improves the loss when deciding to finalize pre-training, and a pre-annealing checkpoint can be loaded to continue pre-training.
7.4 Infinite Learning Rate Schedules: Scaling to Infinite Future Updates
We explore the role of the infinite learning rate schedules when multiple datasets seen in a continual lcarning setup. The models are trained from random initialization with different learning rate schedules on IID 100B subsets of SlimPajama (e.g,, our (hree dalasels no shift setting; see Sec 5.2). We focus on the no shift setting in these preliminary experiments and leave the weak and strong shift cases future work. This task simulates setting where large amounts of data from the same distribution are received at time increments and we wish continue pre-training our models on them (c.g., continuing pre-train the model the latest web-scrape). To make our results applicable to situations where previous
22

---

Published in Transactions on Machine Learning Research (06/2024)
3.0
Slim Pajama Pajama Val. Loss
40534 SE LurSert lnt
2.9
405M SP Cosine
Learning Rate
2.5
1.5
2.7
1.0
0.5
2.5
50
100
150
200
250
300
50
100
150
200
250
300
Tokens (B)
Tokens (B)
(a) Slim Pajamaa Validation Loss
(b) Learning Rate Schedule
Figure 9: Infinite learning rate schedules v.s. Cosine decay. We train 405M parameter model 300B tokens of SlimPajama from random initialization with two new schedules, Cosine Inf and InuSqrt Inf, and compare them to the cosine decay baseline. Cosine Inf and InaSgrt Inf first decay to a fixed constant LR value and stay constant thereafter until an abrupt final decay. These schedules, therefore, have the advantage that they can smoothly transition between one pre-training phase and the next without re-warming. We find that all methods reach similar final validation loss showing that Cosine decay not prerequisite for strong performance.
2.90
3.0
X30-1
SlimPajan Val. Loss
2.85
40534 SP 100B 3 Cosine
405M
2.5
409M SP L0GB 3 Ins
Leariing Rate
403M Costne
2.75
2.0
1.5
2.65
2.60
1.0
2.55
0.5
2.50
50
100
150
200
250
300
100
150
200
250
300
Tokens (B)
Tokens (B)
(a) SlimPajama Validation Loss
(b) Learning Rate Schedule
Figure 10: Infinite learning rate schedules evaluated on IID 100B token subsets of SP. The experiment simulates setting where new data from the same distribution arrives over time and the practitioner wishes to update their model the new data. The models trained from random initializttion dataset. For each dataset, train two checkpoints: checkpoint that continues the constant phase for all data in this datasct and a decayed checkpoint (c.g., phase 4). When transitioning to the new datasets, we select the former. We note that, in figure (b), the black and violet schedules overlap after ~ 80B tokens.
the first.
optimizer states are not available, we do not keep optimizer states across dataset boundaries. Fig. 10 reports training curves for 405M parameter models.
We observe that all schedules perform relatively similarly, however, the two infinite schedules have the start annealing at any time during the constant learning rate phase each split,
advantage that while the repeated cosine decays require knowing the number of tokens in advance. Additionally, we see negligible forgetting across dataset boundaries for the infinite LR schedules. While the losses initially increase sharply due to re-initializing the optimizer states, the infinite schedules models immediately recover from this.
In future works, would be interesting to study the impact of infinite learning rate schodules continual learning setups with distribution shifts, and investigate the stability of training over large amounts of tokens with long constant phase of the learning rate.
23

---

Published in Transactions on Machine Learning Research (06/2024)
In summary, saw that re-warming can hurt performance even when training on the same distribution, but that alternatives to cosine decay schedules might circumvent these issues. Furthermore, these infinite learning rate schedules provide a simple way to end or resume pre-training without being constrained to a particular token budget. That being said, settings with distribution shifts should also be explored to validate these schedules.
8 Limitations
While we have conducted a thorough empirical evaluation of continual pre-training for LLMs, there are some limitations to our work. In no particular order: 1) we only studied two model sizes (405M and 10B); 2) we did not run deduplication between the German training and validation datasets created from the German Common Crawl scrape (Taippala et al., 2022); 3) we primarily study the transition between two subsequent tasks; 4) we did not run our experiments over multiple sceds; and 5) our experiments infinite lcarning rate schedules are limited to 405M scale with no distribution shift. More explicitly, the first limitation is the number of model scales we consider. While we do consider 405M and 10B parameter model (much larger than most works), we could not extend the study to another order of magnitude duc to computational limitations (e.g,, 100B parameter scale). The second limitation of our work is that the German validation set was not deduplicated from the German training data. While we were careful to take distinct shards for training and validation, there may be some contamination between the two. Given that all baselines have access to the same dataset, however, we believe our results are still valid. The third limitation is that we did not run experiments updating models on more than two subsequent tasks. While we believe that studying this is important, our goal was to focus our compute on different distribution shifts and studying transitions between large datasets, rather than using a large number of datasets. The fourth limitation is that we did not run experiments over multiple seeds due to high computational cost, meaning that there is likely a stochastic element to some results. That being said, our LLMs are trained with a large batch size (2M+ tokens) and, thus, there is little variance in the gradient estimates. Coupled with the fact that the samples from each dataset are processed in the same order in all cases, we believe that our results should be relatively stable to changes in random initialization dictated by the seed. The fifth limitation is that it is very possible that over cnough tokens, the infinite schedules may end up being suboptimal due to only having a single phase of warmup and cooldown, as the learning on all subsequent datasets may just be equivalent to using constant. learning rate, which proved to be suboptimal (see Fig. 4). While Fig. 10 showed that the annealing phase helps recover from this suboptimality in the case of IID splits of the same datasct, it is unclear if this would hold over more tokens, or in the case where the different datasets have distribution shifts. Hence, experiments involving distribution shifts, and larger scale of models and datasets would be important to further test. these infinite schedules. Finally, another important consideration to explore at larger scale is the stability of pre-training with such schodules (in particular, during the constant lcarning rate phase without jP (Yang et al., 2022)).
9 Conclusion
In the context of continual pre-training of autoregressive transformer-based LLMs, we have seen that learning rate re-warming and re-decaying is important for adaptation and found that forgetting is easily mitigated with replay in this setting- at seemingly little cost to adaptation. Given their powerful ability to enhance adaptation and mitigate forgetting simultaneously, we proposed the simple and scalable combination of LR re-warming, LR re-decaying, and replay for continually pre-training LLMs at scale. We showed that these strategies enable continual pre-training to achieve average performance on par with expensively re- re-training from scratch all data, across two distribution shifts (weak & strong) and two decoder-only transformer LLM scales (405M & 10B). Upon further analysis, we identified pathology of LR re- re-warming and, inspired by previous work, proposed infinite learning rate schedules for continually pre-training LLMs. In initial experiments, our schedules achieve performance on par with cosine decay while circunventing the need for LR re-warming.
Our findings show that continual pre- pre-training is an efficient and promising alternative to re-training when updating decoder-only transformer L.Mss on new data. Equipped with our strategies, practitioners
24

---

Published in Transactions on Machine Learning Research (06/2024)
efficiently update their existing models (Rac et al., 2021; Hoffmann et al., 2022; Touvron et al., 2023b; Jiang et al., 2023; Gemma Team et al., 2024) on newly created higher-quality datasets. These strategies might also be relevant for pre-training curricula such as the ones used by Gemma Team et al. (2024). With the strong incentive for our community to continuc creating datasets of increasing quality, we only expect the need for continual pre-training to increase.
In follow-up work, it will be important to further investigate infinite learning rate schedules, growing models during continual pre-training (e.g., mixture-of-experts block expansion), and adapting the tokenizer to handle drastic changes to the data distribution. Moreover, we would like to explore continual pre-training in the context of multimodal or vision language models and other text-based generative modcls-we note that recently, Garg et al. (2023) concurrently replicated the success of the techniques discussed in this work in the context of CLIP models instead of LLMs. We also would like to explore replay buffer creating in the continual pre-training setting where an open-weight model does not disclose its dataset; we suspect using the available model for synthetic data or distillation may be a promising direction build the replay buffer.
Broader Impact Statement
Large language models have seen widespread adoption across a wide range of industry sectors due to their ability to perform very well after being trained on relevant datasets. Moreover, improvements in datasets (better filtering, updating knowledge, etc.) have been crucial to increasing the quality of the output of LLMs. As such, is reasonable to expect that organizations will spend significant amount of computing power and, thus, energy to create more powerful models. It is likely that some of this energy will come from non-renewable sources. While the experiments presented in our paper are environmentally costly, as argued in the paper, continuing to pre-train is promising method to significantly reduce the compute associated with updating model and, hence, the energy required to maintain foundation models.
Acknowledgements
We acknowledge support from NSERC Discovery Grant RGPIN- 2021-04104 [E.B.], the Canada CIFAR AT Chair Program [T.R.). and the Canada Excellence Research Chairs Program [I.R.]. We would also like to acknowledge funding from the FRQNT Doctoral (B2X) scholarship [B.T.], the scholarship for Artificial Intelligence of Université de Montréal's Éludes Supérieures et Postdoctorales [A.I.), and a fellowship of the IFI program of the German Academic Exchange Service (DAAD)JMR... This research was made possible thanks to the computing resources the Summit supercomputer, provided as a part of the INCITE 2023 program award "Scalable Foundation Models for Transferable Generalist AI". These resources were provided by the Oak Ridge Teadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-000R22225, In particular, we thank Jens Glascr for his help with the Summit supercomputer.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lone, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http::/ppapes..niie nips.ccc/ www.aa00--bbbttrrr----oeeeeeeeee
paper www.ffesssaee//ee.e http:///haaaa....aaaaaaa070cc...
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximal interfered retrieval. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing
25

---

Published in Transactions on Machine Learning Research (06/2024)
Systems 32, pp. 11849-11860. Curran Associates, Inc., 2019. URL httppp/////ppppssiii..ccc/p////p
online-continge aatae--oottmmall-eaamiinnaa
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étiemne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series open language models. CoRR, 16867, 2023. URL
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski Yejin Choi, and Hannanch Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms, 2019.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Jevy-Kramer Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregrossive Language Modeling in PyTorch, 8 2021. URL bettps:/wwwe wwwp:/wwtww..co//eeeee.. wttpeeeeeetgggtt-eeeeoo
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Cpt-neox-20b: An open-source autoregressive language model, 2022.
Tim Brooks, Bill Peebles, Connor Homes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Wing Yin Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL simulators.
Tom B Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are lew-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 1877-1901, 2020. URL
14165.
Pictro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Caldcrara. Dark experi- ence for general continual learning: strong, simple baseline. In Hugo Larochelle, Marc'Aurela Ran- zato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Newral Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https: http:/eedd.nnn.ooorr......o///ee 00046223387780077117767766666000
hashh/
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexandre Lacoste, David Vazquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation: new approach to continual learning. NewrIPS, 2020. URL https: wttpv/.wrrggaas///0....
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublincar memory cost. CoRR, abs/1604.061177 2016. URL http://aaaxivorg/aaa//0.....
Christopher Clark, Kenton Lee, Ming-Wes Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018.
Together Computer. Redpajama: an open dataset for training large language models, 2023. URL https:
// /tgggaab.cc//ogggtaarrpppp/e/eep
26

---

Published in Transactions on Machine Learning Research (06/2024)
Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, and Davide Bacciu. Continual pre-training mitigates forgetting in language and vision, 2022. URL https://arxiv..rttt 2205.09357.
Matthias De Lange, Gido van de Ven, and Tinne Tuytelaars. Continual evaluation for lifelong learning: Identifying the stability gap. arXiv preprint arXi::205511344 2022.
DeepSeek-Ai Al. Deepseek-v2: A strong, economical, and efficient mixture-of-experte language model, 2024.
DeepSeck-AT, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazno Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. Deepsock-coder-22 Breaking the barrier of closed-source models in code intelligence. CoRR, abs/ 0oss/2406.19993 2024. URL http///awwvvorrab////aaa/
Arthur Douillard, Qixuan Feng, Andrei RUsu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Aurclou Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-comuunictioon training of language models. arXiv preprint arXiv:2311 08105, 2023.
Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):228 135, 1999. ISSN 13646613. doi: 10.
URL https://ww..sciencddeeee.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101 00::2200100022/ .00027, 2020.
Saurabh Garg, Mchrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mchta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. Tic-clip: Continual training of clip models. ar preprial 2b01/22.
arXiv:2310112226 2023. URL
Thomas Mesnard Gemma Team, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, and et al. Gemma: Open models based gemini research and technology. 2024. URL https://storage wwwgleapis.com//eeeee.ee
aeriiiagemmagggee report.pds.
Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b SSM hybrid model. CoRR, abs/2405.16712, 2024. URL
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcce's mergekit: A toolkit for merging large language models. arXiv preprinl Xiv::24403. 13257, 2024.
Evangelia Gogoulou, Timothée Lesort, Magnus Boman, and Joakim Nivre. A study of continual learning under language shift. CoRR, abs/2311.01200, 2023. URL
http://axxvv.22..12200.00
Zheng Gong, Kun Zhou, Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. Continual pre-training of language models for math problem understanding with syntax- aware memory network, 2022. URL https://aclanthollog worge..or///.... www.orrg../aaa.eeoo acl-oon. 408/.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour, 2018.
Kshitij Gupta, Benjamin Thérien, Adam Tbrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. Continual pre-training of large language models: How (re)warm your model?, 2023. URL
27

---

Published in Transactions on Machine Learning Research (06/2024)
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association
Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. for Computational Linguisties, 2020. URL
Md Yousuf Harun, Thair Gallardo, Tyler I, Hayes, and Christopher Kanan. How efficient are today's continual learning algorithms?? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2430-2435, 2023a.
Md Yousuf Harun, Jhair Gallardo, Tyler L. Hayes, Ronald Kemker, and Christopher Kanan. Siesta: Elficient online continual learning with sleep, 2023b.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding, 2021.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. CoRR, abs/2102.01299, 2021. URL httpps///arrv..orraaaa////....00
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint ur 2rv: 2022. URL https:
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient. training of giant neural networks using pipeline parallelism, 2019.
Joel Jang, Sconghyoon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyconghun Kim, and Minjoon Seo. Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emérates, December 7-11, 2022, pp. 6237-6250. Association for Computational Lingnistics, 2022a. URL https: //abi./rg/10.1866///1//22// www.:mmmmmlpmm..mmmmmmmm 418.
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25- 29, 2022. OpenReview. net, 2022b. URL http::///ggaareeeeeeeeee///osets
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06825,, 2023. URL https://doi.on oorellorg//1..8/0 arXiv.2310.0692.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervisod challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Mecting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601- 1611, Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10.18653/vv 1//333/1/P77114/// URL http:///aaaanthooo..rrgg..
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/ abss/2001.08361, 2020. URL
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pro-training of language models, 2022. URL
28

---

Published in Transactions on Machine Learning Research (06/2024)
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:334,.02643, 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, 2019. doi: 10.1162/Lacl a 00276. URL
Veronika Laippala, Anna Salmela, Samuel Rönnqvist, Alham Fikri Aji, Li-Hsin Chang, Asma Dhifallah, Larissa Goulart, Henna Kortelainen, Mare Pámies, Deise Prina Dutra, Valtteri Skantsi, Lintang Sutawika, and Sampo Pyysalo. Towards better structured and less noisy web data: Oscar with register annotations. Tn Proceedings of the Fighth Workshop on Noisy User-generated Teat, W-NUT@COLIN 2022, Gyeongju, Republic of Korea, October 12 17, 2022, pp. 215 221. Association for Computational Linguistics, 2022.
URL https: / http:/acccaaaooooo.orrrooo org/ www.:oggg////////
Timothée Lesort, Oleksiy Ostapenko, Pau Rodríguez, Diganta Misra, Md Rifat Arefin, Laurent Charlin, and Irina Rish. Challenging common assumptions about catastrophic forgetting and knowledge accumulation. In Sarath Chandar, Razvan Pascanu, Hanic Sedghi, and Doina Precup (eds.), Conference on Lifelong Learning Agents, 22-25 August 2023, MeGill Universily, Montréal, Québec, Canada, volume 232 of Proceedings of Machine Learning Research, pp. 43 65. PMLR, 2023. URL https://proceedding mlr www.mmees..me.... press/v232/ /v232/ lesort23a.htm..
Timothée Lesort, Massimo Caocia, and Trina Rish. Understanding contlnual learning settings with data distribution drift analysis. arXiv preprint arXiv:2104.01678, 2021.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulga: Measuring how models mimic human falsehoods, 2022.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRevicw net, 2019. URL
Shirong Ma, Shen Huang, Shulin Huang, Xiaobin Wang, Yangning Li, Hai-Tao Zheng, Pengjun Xie, Fei Huang, and Yong Jiang. Ecomgpt-ct: Continual pre-training of e-commerce large language models with semi-structured data. CoRR, abs/2312.156995 2023. URL
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282. PMLR, 2017.
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training lifelong learning. http:///jaa.oor//pppprr/////2222
J. Mach. Learn. Res., 24:214:1-21::00 2023. URL
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plastiicl dilemma: iggtiig the continuum from catastrophic forgetting age-limited learning effects. Frontiers in psychology, 4 4(August): 504, 2013. ISSN 1664-1078. doi: 10. 201000000......00000...... URL http::/www..ummeecmmtaa nih.gov/ attrrpeeeeeeeattttaaaeeeeeeeeeee
Microsoft. RlegatronDeeeSSped February 28, 2024.
http://///mnocooote//eeeeaeereee 2020. Accessed:
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? dataset for open book question answering, 2018.
29

---

Published in Transactions on Machine Learning Research (06/2024)
Seyed-Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Görür, and Mehrdad Farajtabar. Wide neural networks forget less catastrophically. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Ma- chine Learning Research, pp. 15699 15717. PMLR, 2022. URL https: sttps://procceednng mlr pal...ress//v6/ mirzadeh22a.htan
Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learn- ing to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. URL httpp:://ppeaaccess.tt on Learning Represenlations, 2024. URL w.t.aaaaticcaaattiiiiiaeae
2019. Baaipennk_Leerninggtoe to CesaamiingttRRRmmmbeerr.r Rwwenbememee....ee A__ httpttoeeeeeeeeee__aaraaaa
htt.:/cc////ctteeeeeeeeeeeem _cPR 2019/htm1/ Btt___Driiie_eeeeee_ for_ hatpreoorrr_oooCoooiiaaaaaa... 2ttpaa____________pppe__
Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An
efficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference
Björn Plüster. GermanBenchmark.
German Benchmark Datasets,
8 2023.
URL attps://ggitnub.com/jjoee/
Martin Popel and Ondřej Bojar. Training tips for the transformer model. The Progue Bulletin of Mathematical Linguistics, 110(1):43 70, April 2018. ISSN 1804-0462. doi: 10.2478/ pttp:paaiili-2000222222 URL http: //xx.doi.org/10 20t.:/4a1/ppaann0000 20002000220
Ameya Prabhu, Zhipeng Cai, Puncet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener. Online continual learning without the storage constraint. arXie preprint arXiv:2305.092. 2023.
Yujia Qin, Cheng Qian, Xu Han, Yankai Lin, Huadong Wang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Recyclable tuning for continual pre-training, 2023. URL
wwtp:o//gg.......././. 2305.0870..
Alec Radford, Jong Wook Kim, Chris Hallacy, Adilya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Reseorch, pp. 8748 8763. PMLR, 2021. URL attp://proceedings..ll teresrpees///ee / vv39 ht/99///affooooe ord21..2m2.....
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11444 2021. URL 2112.11446.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter modols. In Christine Cuicchi, Irene Qualters, and William T. Kramer (ods.), Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanla, Georgia, USA, November 9-19, 2020, pp. 20. TEEE/ACM, 2020.
Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting neural networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nel, 2022. URL httplppp..ppppppeeesssee/////o/s
Sashank J. Reddi, Zachary Charles, Manzil Zahoer, Zachary Garrett, Keith Rush, Jakub Koncčný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive foderated optimization. In International Conference on Learning Representations, 2021. URL https: http:p///pppaaaeeeee.ee///////aa
30

---

Published in Transactions on Machine Learning Research (06/2024)
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2019. URL
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, pp. 348 358, 2019. URL https://arxiv.orgg http://rrrrabaa//..11111..
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent. diffusion models. In TEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10674-10685. IEEE, 2022. URL attpss//aas.orr///o.o0/////e....
Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk, and Gennady Pekhimenko. Moshpit sgd: Communication-effiientt decentralized training on heterogeneous unreliable devices. In M. Ranzato, A. Beygekzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 18195-18211. Curran As- sociates, Inc., 2021. URL
9771523344333996636442200
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schemaa challenge at scale, 2019.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint rxXi::2211.05100, 2022. URL 2211.05100.
Thomas Scialom, Tahin Chakrabarty, and Smaranda Muresan. Fine-tuned language models continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6107-6122, 6122, 2022.
Rico Semnrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Lingwistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: /111653//11PP1-111 baeclanthology.orggPPee
URL https:
Mohammad Sioeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Calanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.0805: 2019.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism, 2020.
James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be October 2021.
dreaming: A new approach for data-free dlas--nccemmnnaa learning. pp.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPa- jama: 627B token cleaned and deduplicated version of RedPajama. https: http://wwwwwwwbb..eee/ bet/blog/ ama, June 2023. URL
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith,
31

---

Published in Transactions on Machine Learning Research (06/2024)
Hannanch Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. CoRR, abs/ abs/2440.00159, 2024. URL https://doi.orr/10... 1tt.://008800/022112222222000000
Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 2.0: A continual pre-training framework for language understanding. In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAT 2020, New York; NY, USA, February 7-12, 2020, pp. 8968 8975. AAAT Press, 2020. URL https: http:////..................../.0
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2rXi::2302. 13971, 2023a. URL https://arxiv.or http///rwwv..rr/a/////././011// ab2/2302.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Tnan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Fric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.092228 09288, 2023b. URL https:////i...ooo ooooo.org/1... 005500/ar11v.307.0099
Tom Veniat, Ludovic Denoyer, and MarcAurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. In International Conference on Learning Representations, 2021. URL https:
Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ping Luo, and Ying Shan. Llama pro: Progressive llama with block expansion. CoRR, abs/2401.02411 2024. URL https://doi.org
10.48550/arXiv.2000
Yong Xie, Karan Aggarwal, and Aitzaz Ahmad. Efficient continual pre-training for building domain specific large language models, 2023. URL https://arxiv.org wttpp///..r///a///.......
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yav Yan Liu. On layer normalization in the transformer architecture, 2020.
Pratock Yadav, Qing Sun, Hantian Ding, Xiaopeng Li, Dejiao Zhang, Ming Tan, Parminder Bhatia, Xiaofei Ma, Ramesh Nallapati, Murali Krishna Ramanathan, Mohit Bansal, and Bing Xiang. Exploring continual learning for code generation models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (ods.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 782-792. Association for Computational Linguistics, 2023a. URL https::/ http::://dioooooooe www..oo//...../.1///o/ http:2222...111h....6...
Prateck Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolv- ing interference when merging models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: An- nual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023b. URL Abstract-Cont Cerrractt-Cooffeenneeettttt
32

---

Published in Transactions on Machine Learning Research (06/2024)
Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, David Farhi, Jakub Pachocki, Xiaodong Liu, Weizhu Chen, and Jianfeng Gao. Tensor programs Tuning large neural networks via zero-shot hyper- parameter transfer. In NeurlPS 2021, March 2022. URL
transffr
Xianjun Yang, Junfeng Gao, Wenxin Xue, and Erik Alexandersson. Pllama: An open-source large language model for plant science. CoRR, abs abs/2401.01600, 2024. URL https://doi.org// 10..///0/0//./...211....
Kaichao You, Mingsheng Long, Jianmin Wang, and Michael modern neural networks?, 2019.
Jordan. How does learning rate docay help
Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. CERT: continual pre-training on sketches for library-oriented code generation. In Luc De Raodt (cd.), Proceedings of the Thirty-Fits First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pp. 2369 2375. ijeai.org, 2022. URL https:
//doi.org/10.24990000 1111111.22993311 /tte////jjaii.////// 2022232929999
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In IEEE/CVE /CV Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 1204-1213. IEEE, 2022. URL
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023. URL
33

---

Published in Transactions on Machine Learning Research (06/2024)
Contents
Introduction
2 Main Findings and Takeaways and Examples our Method's Practicality
3
3 Related Work
5
3.1 Continual learning
5
3.2 Pre-training, Model Scale, and Contimual Learning
5
3.3 Domain Adaptive Continual Pre-training (DACPT)
5
3.4 Continual Learning for LMs Applied to Specific Domains
6
3.5
Tearning Rate Schedules
4
Background & Methodology
6
4.1 Linear Warmup and Cosine Decay Schedule
6
4.2
Compute-cquivalent Replay
5
Experimental Setup
8
5.1
Datasets
8
5.2
Continual Learning Settings
8
5.3
Training Setup
5.4
German and English LM Evaluation Benchmark
6
Results
10
6.1
Learning Rate Schodule
11
6.1.1
The Effect of Linear Warmup for Weak and Strong Distribution Shifts.
6.1.2
The effect of re-warming, re-decaying, and varying Amax for Weak and Strong Distribu- tion Shifts.
13
6.2
The Effect of Replay
13
6.3
Continual Pre-training Final Performance for Weak and Strong Distribution Shifts.
15
6.3.1
Final Performance Evaluated by Loss
15
6.3.2
Final Performance Evaluated by Zero-shot and Few-shot Results on Popular LM Benchmarks
6.4
Continual Pro-training Final Performance Different Model Scales
6.4.1
Final Performance Evaluated by Loss
18
6.4.2 Final Performance Evaluated by Zero-shot and Few-shot Results Popular LM Benchmarks...
19
7 Understanding and Circumventing the Pathologies of Re-warming
20
7.1
Re-warming the Same Data
20
34

---

Published in Transactions on Machine Learning Research (06/2024)
7.2 Infinite Learning Rate Schedules
21
7.3 Comparing Cosine Decay to Variants of our Infinite Schedules
22
7.4 Infinite Learning Rate Schodules: Scaling to Infinite Future Updates
22
Limitations
24
9 Conclusion
24
A Extended results
36
A.1 Domain Incremental continual pro-training
A.3 Replay for Different Dataset Sizes ................................
A.2 Model Merging v.s. Continual Pre-training
37
37
A.4 The effect of resetting optimizer states at dataset transitions
38
A.5 Qualitative evaluation of German models
38
A.6 Aggregated LM Evaluation Results
41
A.7 Aggregated average final accuracy
43
B Model hyperparameters
44
35

---

Published in Transactions on Machine Learning Research (06/2024)
Appendix
A Extended results
In the following subsections, we first present some new results in the Domain incremental continual pre- training sctting, comparing replay for different datasct sizes, and providing a qualitative analysis of our German language models. We also provide aggregated evaluation and final loss tables for all models in the paper.
A.1 Domain Incremental continual pre-training
We consider a domain incremental learning setting, where we train on a sequence of future datasets (Do, Dx..) each of which comes from a distinct domain. This setting particularly challenging due the distribution shift experience the transition between cach domain. Concretely, we consider datasct SlimPajama. However,
Do to be pre-training on the Pile (Gao et al., 2020) and Di to be pre-training instead of consuming the data in Dí using the sampling percentages from table 1, we process the dataset domain at time starting from the largest domain to the smallest. This simulates situation where new data is received from different domains at different times and we wish to update our models on the sequence while being robust to all distribution shifts.
Adapting replay to Domain Incremental continual pre-training In settings that span more than two tasks, we use a form of reservoir sampling (Buzzega et al., 2020), where the replay buffer updated discrete intervals. We refer to this technique discrele reservoir sampling. Specifically, given sequence of N datasets Do, Di...., DN of sizes 80, $1,...; that are trained buffer R at each dataset transition. Let R, correspond to the state of the replay bulfer the start of training on the i-th dataset. For replay ratio 0 < S 1, at any given i > 0, R, will contain data from all Dj for j < in proportions
in sequenee, update the replay
Pi.j = 8j 7 + -kj+1 Pk.j 8k
with Vj. Pi+1. =
(5)
El 8k
where i is the index of the dalaset on which we are currently pretraining, and gj if 0, and otherwise. The latter is because when pretraining Di, only see s (1- a) tokens of D, because use computo-oquivalent replay, except for the first datasct Dy where replay is not used, and where we hence see all so tokens of Do.
3.0
3.0
SlimPajama Val. Loss
405M SP 5% Duplay (PT on Pile)
Staridrrstange Hopiny
Pile Val. Loss
Wildpedia 556 Replay
Book 5% Replay
Arkiv Replny
2.6
2.8
Replay
5% Replay
2.7
Commoncravw 53% Beplay
2.2
2.6
2.0
2.5
50
100
150
200
250
300
100
150
200
250
300
Tokens (B)
Tokens (B)
(a) Pile Validation Loss
(b) SlimPajama Validation Loss
Figure 11: Ingesting data sequentially by domain. We explore an alternative approach consuming the available data. Similar task-incremental or class-incremental learning, we train on one domain at a time in sequential fashion. We I.R re-warming and I.R re-decaying for each domain well as discrete reservoir sampling.
36

---

Published in Transactions on Machine Learning Research (06/2024)
Figure 11 reports results for domain incremental approach to consumiig our 300B token split of SlimPajama, where each domain is processed in decreasing order of size. We use learning rate re-warming, LR re-decaying ing, and replay with discrete reservoir sampling. The total data processed is the same as the model trained with LR ro-warming and ro-decaying on the 300B tokens of SlimPajama (bluc dotted line). However, the order in which the data is processed and the learning rate schedule differs from the baseline.
We hypothesize that difficulties arise due to relatively strong distribution shifts and the many learning rate re-warmings. The number of consccutive distribution shifts with varying amounts of data from different domains makes it difficult to obtain strong final performance across all of them. In particular, it seems especially difficult to efficiently adapt to the new data distribution. We hypothesize part of these difficulties are aptimizatiio--elateer
Less frequent warming up and decaying of the learning rate may be beneficial in this setting, and the infinite learning rate schedules might help, but future research is needed to answer this question. At any rate, we believe this setting is unnecessarily difficult compared to consuming the data using a mixture (as we did in the main paper) and is therefore of limited practical use.
A.2 Model Merging v.s. Continual Pre-training
Some may be curions about the performance of model merging as a continual pre-training approach, and how it compares to contimuing to pre-train with the methods used in this paper. In preliminary experiments with TIES (Yadav et al., 2023b) using Mergekit v0.0.4 (Goddard et al., 2024), we consider merging models trained separately on each dataset (e.g.; merging 300B SP into 300B Pile and 200B German into 300B Pile) using 300B Pile the base model. We also merge continually pre-trained checkpoints into 300B Pile. English and German LM evaluation results can be found in Tables 6 and 7.
We find that, in general, continual pre-training outperforms merging models trained separately on each dataset and requires the same amount. of compute. Moreover, we find that merging checkpoints of a continually pre- trained model does not increase performance, likely due to replay being a more efficient strategy. However, we note that these experiments are by no mcans comprehensive; other techniques may yield better performance.
Table 6: English LM Eval Performance of Models Merged with TIES v.s. Continual Pre-training. Normalized accuracy is reported for HellaSwag and exact match (EM) is reported for NaturalQuestions and TriviaQA. All other tasks report unnormalized accuracy. MMLU and TriviaQA are evaluated 5-shot, while all other tasks are zero-shot. In (x% R), the R should be read as Replay.
TLAT
Balle A100
SCHEII
P0gA
Waatbeak Eiataage MCC
Sullaag 14.3
Sataokguntea (EM)
Twaag [PM]
ADC
30.00
25:31
34.5"
81.3
15.00
35.00
13:50
01:0
2008
3400
20.18
1.9)
1:31
17
35.00
15.00
27.38
17.4
19.25
35.00
15.00
01.75
11:14
11.05
(7/0
1.9)
3041
3103
18:31
3540
$2.1
25.51
$5.0
14.00
18.10
2168
450
5.8
6:31
2001
35.75
03
15.00
$1.09
33.00
15.0
18:00
02.99
11.7
35.1
(16
1.0
176
15.4
240 57
2135
18.31
28.2
3123
20.01
22.4
12:81
04.00
5858
3114
1.8)
1-6
$1.39
3135
3405 1h 2000 150 It)
26.55
45.72
6103
3212
34.13
18.00
03.30
5108
15.08
40/8
3.50
3.00
31.05
31.5
Te 2001 75 2018 (5
9035
28.00
1500
1755
1531
3000
1138
36.55
1.90
20.55
31.5
31.8
80.17
11.08
2018
18:01
71.11
0200
11.0
1.50
1.11
3117
7521
38.0
1521
3003
25.71
75.0
3000
11:32
(748)
TIQA: Teuthful QA, WG:WinoGrande, NQ Natural Questiono, OBQA: OpenBook QA, TAQ.AAAAAAVVv........... TriviaQQ
A.3 Replay for Different Dataset Sizes
To ablate the insights gathered from the experiments in Sec. 6.2 we vary the size of the continued pre-training on SlimPajama with 100B, 200B, and 300B tokens. The respective linear warmup schedules and cosine decays are fitted to the altered length of training. As expected, we observe a consistent decrease in loss at the end of training for SlimPajama with an incrcasod number of training tokens (see Fig. 12). We further observe similar trends concerning the effect of replay on forgetting and adapting to novel data. In conclusion, the simple combination of LR warmingngg, LR re-decaying, and replay is effective at different. dataset sizes.
37

---

Published in Transactions on Machine Learning Research (06/2024)
Table 7: German LM Eval Performance of Models Merged with TIES v.s. Continual Pre-training. Normalized accuracy is reported for HellaSwag and exact match (EM) reported for NaturalQuestions and TriviaQA. All other tasks report unnormalized accuracy. MMLU and TriviaQA are evaluated 5-shot, while all other tasks are zero-shot.
Model Ste.
Merging Method
Bese Model
Task Modelis)
Topk'S
MMLU DE
ARC-C DE
Hella-Sung DE
TruthfiQQA DE
AVG
26
22.86
20.65
24.78
21.05
22:33
2008 Ger.
76
24.35
19.54
28 67
25.83
24.00
TIES
300B Pile
50
21.08
23.50
400M
25
23.89
19.45
26. 76
26.70
23.95
90018 2008 (25% Replay)
75
23.91
18.94
30.81
24,00
24.57
Daseline
300B Pue
2000 Ger. (20%
25.78
25.56
Trushful QA, WG:WiaoGrande, Natural Quassion,
QA, T.QA/TiviiQA
2.80
400M Piie u SP Pinall loss
2.75
SP
Pile Val. Loss
2.5
2.70
40BM SP 3008 (PT ou Pik))
60GM SP 100B 55% Replay (2T on 200)
2.4
Slim HllmPajjan P
2.65
SP
(PT Pio)
Replay (2T 7030
2.60
2.3
2.55
2.2
2.45
50
100
150
200
250
300
50
100
150
200
250
300
Tokens (B)
Tokens (B)
(a) Pile Validation Loss
(b) SlimPajama Validation Loss
Figure 12: Replay v.s. no replay when re-warming the learning rate and continually pre-training on different amounts of data. We train 405M parameter models with and without 5% replay on 100B, 200B, and 300B tokens of SlimPajama data. Each model starts from a checkpoint pre-trained on 300B tokens of Pile and re-warms the learning rate using a linear warmup and cosine decay schedule fitted to its dataset size.
A.4 The effect of resetting optimizer states at dataset transitions
In all of our experiments, drop optimizer states between dataset transitions to show that our techniques can be used to continually pre-training open-source LLMs, which rarely provide optimizer states. In this section, we investigate whether keeping the optimizer states from pre-training can resolve instabilities seen during continual pre-training. Figure 13 reports validation loss curves during the continual pre-training of a 405M parameter decoder-only transformer on 40B tokens of SlimPajama data. Each model starts from a checkpoint pre-trained on 300B tokens of Pile and re-warms and re-decays the learning rate using a linear warmup and cosine decay schedule fit a dataset size of 300B tokens (Amas 3 10-4, Thmán 3- =-110-5) 10-5). We obscrve that, initially, both models follow slightly different trajectories. However, after 40B tokens of training both models reach similar loss values with differences allributable to randomness. We therefore conclude that, in this setting, keeping optimizer states does not affect continual pre-training compared to discarding them. We note that this is to be expected, as the relatively large momentum coefficient values used for LLM pre-training (0.9 and 0.95 for 31 and (2) cause the contribution of the starting optimizer states to quickly decay to For instance, after 1000 steps of optimization, the moment estimates from pre-training will contribute 0.0044% of the current moment estimates.
A.5 Qualitative evaluation of German models
In this section, we provide brief qualitative evaluation of the models trained on Gorman Common crawl (Sec. 6.3). We select five German prompts that contain various peculiarities of the German language (sce Tab. 8). We then generate fixed token-length response for each of the models trained continually
38

---

Published in Transactions on Machine Learning Research (06/2024)
2.76
SP Med 3m-0t (drap optimiow ebates)
2.74
400M SP MaxLR Ze (toop optimizoo statess
Pile Val. Loss
2.72
2.70
2.4
SlimPajama
2.68
2.3
2.2
10
15
20
30
35
40
Tokens (B)
Tokens (B)
(a) Pile Validation Losss
(b) SlimPajama Validation Loss
Figure 13: Keeping v.s. dropping optimizer states when transitioning from Pile to Slim Pajama. We train a 405M parameter decoder-only transformer on 40B tokens of SlimPajama data. Each model starts from checkpoint pre-trained on 300B tokens of Pile and re-warms and re-decays the learning rate using linear warmup and cosine decay schedule fitted to dataset size of 300B tokens (Mmer 3 10 :mmin = - 5). We observe that, initially, both models follow slightly different trajectories. However, after 40B tokens of training both models reach similar loss values with differences attributable to randomness.
pre-trained on German Common Crawl. As a baseline, we also evaluate the same model trained only on the Pile.
We provide the responses and a manual translation of the response in Table 9. While the relatively small 405M parameter models generally do not produce meaningful sentences, the model trained German Common Crawl generally produces grammatically correct sentences. This includes correct upper and lower casing, the generation of compound words, and the use of Umlaut characters. In all cases, responses tend to be repetitive after a few words. The incompleteness and fractional words of some outputs can be attributed to the number of tokens generated being very short. Longer sequences tend to provide syntactically correct sentences but also tend to repeat themselves. The limited nunber of samples and the generally low quality of the generated text does not allow us to make robust. statements the qualitative differences bet ween the individual models trained on German Common Crawl. However, the aforementioned problems of repetitiveness grammatical errors seem to be significantly stronger the Pile bascline, which generally fails to even respect the context given in the prompt. This is expected as the amount of German lext is significantly smaller on the Pile.
well
Thus we conclude that there is an observable effect on the generative quality of the German language from the models that used the data of German Common Crawl during training.
Table 8: German phrases used for qualitative evaluation alongside their translation (for readability) and justification.
Ref.
Prompt
Translation
Reasoning
Eine typisch deutsche Mahlacit ist
typical German meal
Conversational seutence.
Doner ist besser sls Poutine weil
Doener is superior poutine because
This prompt Umlaut.
Pferd keinen Gurkensalat.
The horse
eucumber salad
sentence rompound word
Handy deutsches
Cellphone german word
Prompe with anglicism (Handy Collphone)
Lieroolbruccsstelleeeenn
The ogg breaker
Long componnd word
39

---

mayy Published in Transactions on Machine Learning Research (06/2024)
Table 9: Generated text. from various prompt-model combinations. All lext was generated with a fixed token length and thus is sometimes incomplete. A manual translation of the text is also provided to give the non-German speaker an idea of the generative quality. The translation is as literal as possible to allow for adequate assessment.
Pooeet (tresdarkm))
Model
Dtavel Lenyth Reeperion
Trazalation
L0GM Pile
He Aptiltueeen
stel livert rieen Apd4kk
apple Hoe wath dow
Incomplee
ICGM 2001 Genown
UI Graell, de in der Rogel
H54 melimen Gieepnn seevts
crndete wetliipe coenos
(2A,, laai uanily
Py (A typreal Beran medl
IIM 2001 Mile 2000 Gernan 12005 Repian ac)
des Bix.
wire nit staen lnckernn
Boa.
World yas lis sceze dedicieess ler creams
potnto
Kartsdiiaatt
U&M J000 Pile
Die Kustrvunkk in wids
di Kind-
The doin have kib the wot linputtatt inosmpletd
Teddeie life
ICGM 2001 Pile SP
nel loser scauueal.
Ween du dies for dee Kirhe
If yeu am inriding webi is the Hivhen
undd leclsr.
10 Indox St.
& ai lealy.
ICM JNd Gemran
Asteortes
arperiee porttine
Scanie Kommetter
Whlte
40GM Pik:
Replao)
Wh linest B3
does [icompkee)
ICGM 300B Pik- XGB GeeaRn
Ween du dive for store Dieve erts
Fatks
If you opt Incempinte for Treee
466 SINB Pile
uus Kocllanaiinn oun Kachiaationss voa K
dae longttite lo Keahinlon
of oud iusii kas of outiin has of jassuglas) you are lnberwted in the canagory
JE totgpite coniiniii conlanntto
Pike
hulle dean Pied be
Wren dics be Kalepeie
40G2A
holes dean Pied be
comodayy
Br (Tkhe laben done oct vat cucember sabd))
110M S000 Pile J016 Gerzan (25% Hepiar)
Rinn Gate
lomn
Plan mibmn
41604 SANiI Pile 2016 Cerzani
de es usck uul cinan
Kran 21 Tiin nEi mage, dum Tome
If dinddie Bin Eir Tamm,
las you you alau (nieiig wild withu
Pik
Ple ansd ukde 1ar
sendem anch die Pla
The homes ane not cnle bowe,
homes (Jest inompiet))
frnkkkunkktt das
relpiien?
110M JAN Genzan
(se der deiieceer Spoctte
des Hards stede Es bn Wart
It sard, lisomplite)
the
Pr: Celiphoss is greas word
11604 SKGi Pile 2006 Cerzan (25% Bapias)
Asc Ri Alc 1ra3ii eilt 1s 9t
Lta Tolcian <ao uil dium Harly
Tha Eareliie to teeppire I8 tilgheme
300B Pik Cerama
Aie 20x4 den Engiaciess Wieretzt
"Healy"
which traddtie from Engltd "calppaace"
40G24 Pik
urdenn
die dest
'apggg but alio for
onle zoel troongles)
IIM SIND Pile SP
de wwwollllmoetnileeeeeeee
te linder Pog brraerr lecrppiee
E0004 20d8 Genzan
Ear Eisaelleu
de Celtrer zoer fatrer Untewdteed
Au oo Binzaa, asmplei
smoll but Impertar dlfiemss
PolTha spe
40GM4 Pik- Geram (35% Bapia))
de Mlnch Wean the Die Banttokk legrn.
gata womale Vorgong.
The mon. When xe plaze the ago
this uxtund proce.
40G24 Pl::
Birloshbrr
LOGM 300T Pil:
Rivardhane
da Eiaal
envegrary whrde retrggery jobike, Wrompiste
40

---

Published in Transactions on Machine Learning Research (06/2024)
A.6 Aggregated LM Evaluation Results
We evaluate our models on the following English and German LM evaluation tasks:
HellaSwag (Zellers et al., 2019) and HellaSwag DE: An English commonsense reasoning benchmark composed of multiple-choice questions that are deliberately designed to confuse language models. HellaSwag DE is German translation of the HellaSwag benchmark.
A12 Reasoning Challenge (ARC) (Clark et al., 2018): An English commonsense reasoning benchmark composed of science examination questions in multiple-choice format. The 7, 787 total questions have been divided into an easy subset with 5, 197 questions and hard subset with 590 questions. ARC-c DE is German translation of the challenge subset of questions.
3. BoolQ (Clark et al., 2019): An English reading comprehension benchmark composed of 15, 942 yes/no question-answering samples. Each example is split into question, relevant paragraph, and the solution.
MathQA (Amini et al., 2019): An English math word problem benchmark composed of multiple- choice questions across various areas of mathematics.
5. MMLU (Hendrycks et al., 2021) and MMLU-DE: An English benchmark designed to evaluate both zero-shot and few-shot scenarios, in order to evaluate both the general knowledge and on-the-fly fly problem solving of the model under test. MMLU covers a broad range of subjects. MMLU-DE is a German translation of the MMLU question set, which was translated by the OpenAI GPT 3.5 API.
6. OpenBookQA (OBQA) (Mihaylov et al., 2018): An English question-answering benchmark modeled after real-world open- book exams for assessing human understanding of particular subject. Questions about elementary science are paired with scientific facts and common knowledge, which the model is intended to use in multi-hop reasoning.
7. PIQA (Bisk et al., 2019): An English guestion-answering benchmark designed to test the physical commonsense reasoning abilities of the model. Most questions focus on applying uncommon solutions to everyday situations, which requires understanding of the physical world.
8. WinoGrande (Sakaguchi et al., 2019): An Euglish natural language understanding benchmark that involves determining when two or more expressions in a text refer to the same entity. The benchmark includes a diverse set of sentences and new evaluation metric that rewards models for making human-like predictions.
9. TruthfulQA and TruthfulQA DE (Lin et al., 2022): An English question-answering benchmark designed to evaluate the truthfulness of generated answers to questions. The questions are designed to contain common human misunderstandingg that lead to incorrect answers. TruthfulQA DE is a German translation of the TruthfulQA benchmark.
10. Natural Questions (Kwiatkowski et al., 2019): Is an English question-answering benchmark consisting of search queries submitted to the Google search engine.
11. TriviaQA (Joshi et al., 2017): an English question-answering benchmark comprised question- answer pairs provided by trivia enthusiasts. Its main focus to determine model's general world knowledge.
41

---

Published in Transactions on Machine Learning Research (06/2024)
Table 10:
All Zero-shot and Few-shot results on popular LM benchmarks. Normalized accuracy is reported for HellaSwag and exact match (EM) reported for NaturalQuestions and TriviaQA. All other tasks report unnormalized accuracy. MMLU and TriviaQA We observe on average, expected, that 10B param. models outperform their 405M counterparts and that the English-only 405M models outperform their German- German-trained counterparts.
evaluated 5-shot, while all other tasks are zero-shot.
Model Size Traciing Tokens
Hellasong
ARC²
ARC-
BoolQ
MarbQQ
MMLU
OBQA
PTQA
WG
TEQA1
TRQA2
NQ
TrQA
AVG
50018 Pile
68.46
34.81
02:19
08.20
27.54
27.25
27.30
76.82
62.53
20.44
33.68
6.05
41.32
43.45
20013
70.38
86.77
71.98
27.42
28.20
70.09
23.00
45.37
30013
37.37
T3.18
25.43
29.94
75.51
23.25
25.04
57.94
47.53
30033 Pile 200B SP (5% Replay'
73.24
30.42
74.24
70.50
26.85
25.79
30
78.02
08.67
21.01
35.02
13.32
57.86
47.68
00033 Pile U SP
73.39
30.25
73.57
12.05
26.85
37.78
27 50
67.52
23.15
36. 16
12:41
50.12
08.00
30013 Pile
00.55
22.01
51.77
09.24
24.12
26.38
19.50
66.09
53.83
21.85
42.11
0.01
8.37
33.95
18003
44.22
22.71
26.38
19.00
22.64
28.63
1.69
34.31
9001 Pile 300R
38.91
2.02
30038 Pile 300B
(0.0%
Replae)
55.18
60.38
24.19
25.50
08.72
23.38
1.86
15.58
35.16
(0015 Pile 20005
SP' (1% Renday'
06.11
28.55
61.88
67.95
23.42
25500
20.001
68.66
54.76
22.89
1.91
15.29
84.91
00013 Pile J0003
SP" (5%) Repday's
06.55
20.55
55.01
17.92
24.22
25.94
20.00
02.37
54.22
23.08
38.35
1.99
15.70
15.14
900B Pile 300B SP [105% Replay)
05.15
21.50
$4,42
58.44
23.62
25.53
21
65.77
54.54
23.75
37.87
2.38
15.00
34.91
3001
(50% Replay)
20.00
30
1.14
36.94
60003 Pile
45.00 06
93.55
52.99
65.57
23.12
26.65
15:30
09:37
32.77
23.50
39.81
1.72
14.00
34.30
40531
2009 Ger.
27:47
17.15
29.80
45.11
21.11
25:27
13:40
56.75
52.17
25.07
48.02
0.05
0.51
27.74
9001 Pile 200B Gar:
50.02
19.30
52.74
61.50
20,
25.34
15.40
56.05
49.995
21.48
44.01
0.19
1.583
23.20
8003 Pile 2008 Ges. (1% Raplay)
188000
14.50
50.52
23.65
8003 2008
Roplay)
322.8
21.25
60006
26.94
15.50
02:15
25.46
45.36
0.11
4.822
31.37
3003 Pile 2003
Ger. (10%% Replor]
34.10
20.63
43.73
52.60
22.35
25.41
18.40
63.06
30.04
25.34
44:331
0.19
4.59
31.14
3003 Pile - 2000 Ger. (25% Teephay)
56.05
21.50
47.56
600.0
20.82
25.00
17:20
63
51.14
25.70
45.84
0.36
5.97
32:48
6008 200R Ger. (50%) Rephoy]
900B Pile
58. 35
20.65
49.54
00.00
24.12
26.45
18:60
65.78
50.51
25.95
42.47
0.00
7.57
35.
96.99
17.40
100 SP InwStrt
43.20
90.31
51.35
60.70
25.38
18:30
68.34
52.01
19.71
35.91
1.94
14.35
33.33
100Bxx8 SP Cosindaf
43.22
22.61
51.05
50.57
22, 78
26.48
18.00
68.17
58.51
23.01
86.00
1.88
14.29
36.00
1003x3 SP Unsine
61.38
20.03
00.46
60006
22.75
25.21
18.00
07.57
52.17
21.18
29.74
1.44
13.40
18.65
TIQA:
Truthful QA, WG-WionGrande, NQ: Natural Queetione, OTQQA: OpenBook QA, TTQA::Tiviigg
Table 11: All Zero-shot and Few-shot results on popular German LM benchmarks. Normalized accuracy is reported for HellaSwag. All other tasks report unnormalized accuracy. MMLU is evaluated 5-shot, while all other tasks are zero-shot. Unexpectedly, we observe on average, that English language models match the performance of their German counterparts. Further inspection shows that performance is close to random chance on MMLU and ARC-C for all models, while German models perform better on HellaSwag and English models perform belter on Truthful QA. This has the effect of canceling out the perceived improvements in the overall score.
Training Tokens
MMLU DE
ARC-C DE
HellaSwag DE
TruthfulQA MC1 DE
AVG
300B Pile
24.92
18.77
27.09
26.93
24.43
300B SP
23.28
17.49
27.03
27.91
23.93
300B Pile 300B SP
25.34
17.32
27.43
26.81
24.22
300B Pile + 300B SP (0.5% Replay)
25.95
17.83
27.35
24.72
23.96
300B Pile 300B SP (1% Replay)
25.44
19.03
27.62
25.34
24.36
300B Pile + 300B SP (5% Replay)
24.82
16.89
27.09
25.95
23.69
300B Pile + 300B SP (10% Replay)
25.24
19.11
27.39
26.68
24.61
300B Pile > 300B SP (50% Replay)
25.11
18.86
27.51
27.17
21.66
600B Pile U SP
24.43
18.26
27.36
26.44
24.12
200B Ger.
23.74
18.26
29.53
26.32
24.46
300B Pile + 200B Ger.
24.29
19.62
31.23
24.85
25.00
300B Pile + 200B Ger. (1% Replay)
23.62
19.62
31.21
24.72
24.80
300B Pile + 200B Ger. (5% Replay)
24.82
18.94
31.03
26.68
25.37
300B Pile + 200B Ger. (10% Replay)
23.51
20.05
31.21
25.58
25.09
300B Pile + 200B Ger. (25% Replay)
23.78
19.20
31.04
25.58
24.90
300B Pile + 200B Ger. (50% Replay)
23.80
20.48
30.91
24.60
24.95
500B Pile U Ger.
24.53
18.43
30.45
25.70
24.78
42

---

Published in Transactions on Machine Learning Research (06/2024)
A.7 Aggregated average final accuracy
Table 12:
Final loss of models reported in sections 6.1, 6.2, 6.3, 6.4, and 7.4. The loss averaged the last 100 iterations of training sampled at intervals of 10 iterations. The standard error is < 0.001 for all models to Monte Carlo sampling over different random seems and is merely meant to reduce noise. We observe that models using more replay achieve a better adaptation-forgeting trade-off (AVG Loss). Interestingly, the model using 50% replay archives nearly identical loss values while sceing 150B fewer tokens on SlimPajama. All evaluation
don't report the specific value. We note, however, that this averaging does not correspond
Validation Loss
Model Size
Training Tokens
Max LR
Schedule
Dg (Pile)
Do (SP/German)
AVG
Constant
Cosine
2.42
2.55
2.48
1.5 x 10
Cosine
2.43
2.51
2.47
300B Pile + 300B SP
3 10 4
Cosine
2.44
2.50
2.47
10-4
Cosine
2.48
2.50
2.49
Constant
Cosine
3.22
1.21
2.22
10-4
Cosine
3.47
1.13
2.30
300B Pile - 200B Ger.
3 x 10
Cosine
3.56
1.11
2.34
6 10
Cosine
3.63
1.11
2.37
300B Pile
3 x 10
Cosine
2.17
2.70
2.44
300B SP
3 10
Cosine
2.51
2.53
2.52
200B Ger.
10
Cosine
3.97
2.57
500B Pile U 200B Ger.
3 x 10
Cosine
2.26
1.25
1.75
300B Pile U 300B SP
x 10
Cosine
2.17
2.53
2.35
405M
300B Pile + 300B SP
3 10
Cosine
2.44
2.50
2.47
300B Pile + 300B SP (0.5% Replay)
3 10-4
Cosine
2.27
2.50
2.39
300B Pile + 300B SP
(1% Replay)
3x 10
Cosine
2.26
2.38
300B Pile + 300B SP
(5% Replay)
3 10
Cosine
2.23
2.51
2.37
300B Pile + 300B SP (10% Replay)
3 10-4
Cosine
2.21
2.51
2.36
300B Pile + 300B SP (50% Replay)
x 10-4
Cosine
2.16
2.35
300B Pile + 200B Ger.
3 10
Cosine
3.56
1.11
2.34
300B Pile + 200B
(1% Replay)
10-4
Cosine
2.83
1.12
300B Pile + 200B Ger (5% Replay)
3 3
Cosine
2.57
1.12
1.85
300B Pile + 200B Ger (10% Replay)
3 10
Cosine
2.46
1.13
1.80
300B Pile 200B
(25%
Replay)
xx
Cosine
2.33
300B Pile + 200B Ger (50% Replay)
3 x 10
Cosine
2.24
1.22
1.73
300B Pile - 300B SP
10 4
Cosine
1.98
300B SP
1.2 x 10
Cosine
2.08
2.05
2.07
10B
300B Pile
1.2 x 10
Cosine
1.75
2.24
1.99
600B Pile SP
10
Cosine
1.72
1.87
300B Pile - 300B SP (5% Replay)
1.2 x 10-4
Cosine
1.79
2.00
1.89
Cosine Inf
2.51
2.52
300B SP
3 x 10-4
InvSqrt Inf
2.51
2.54
2.53
Cosine
2.51
2.53
2.52
405M
Cosine Repeats
2.53
2.55
2.54
100Bx3 SP
10-4
Cosine Inf
2.58
2.61
2.59
Cosine
2.59
2.60
43

---

Published in Transactions on Machine Learning Research (06/2024)
B
Model hyperparameters
Description
Value
10B Transformer Model
Parameters
9, 642, 249, 216 9, 408, 678, 912
Non- Non-Embedding Parameters
Num layers
36
Hidden size
4608
Num attention heads
36
405M Transformer Model
parameters
405, 334, 016
Non-Enin Non-Embedding Parameters
353, 822, 720
Num layers
24
Hidden size
1024
Num altention heads
16
Common
Optimizer
AdamW
$1,32
0.9, 0.95
Batch size
1104
Sequence length
2048
Hidden activation
GeLU
Weight decay
0.1
Gradient clipping
1.0
Decay
Cosine
Positional embedding
Rotary
GPT-J-Residual
True
Weight tying
False
Vocab Size
50432
Rotary PCT
0.25
Table 13: Hyperparameters of our 405M and 10B parameter transformer LLMs.
44

---

Published in Transactions on Machine Learning Research (06/2024)
Description
Value
10B Model- Cosine Schedule
Max learning rale (1maz)
1.2 10-4
Min learning rate (Tmms)
1.2 10-5
Warmup percent (Twarmup)
405M Model Cosine Schedule
Max learning rate (1mea)
3 10 10-4 4
Min learning rate (1min)
3 10-5
Warmup percent (Twarmuu))
405M Model Infinite LR Schedule Common
Max learning rate (1mez)
3 10
Min learning rate (1mma))
3 1015
Constant learning rate (Nconet)
1.65 10-4
Warmup percent (T (aarmmp))
1
Cooldown iters percent (Tea)
60
Constant iters percent (Tann)
25
Inverse Square root cooldown schedule Timescale (a:
10
Table 14: Hyperparameters of LR schedules. Unless otherwise specified in the text, we use these values.
45


Reviews:
[{"review_id": "FiMRTfANXl", "reviewer": "Reviewer_Uarn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* It is a simple yet effective approach in mitigating issues in continual training.\n\nWeaknesses\n\n* This work ignores the moment estimates in the Adam optimizer, and it is not clear how that will affect the effectiveness of training. Note that the instability issue might be coming from the inaccurate estimates in the early continual training, and simply preserving the estimates in the pre-training might solve all the issues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work challenges continual training of large language models by employing additional training data. The findings show that re-warming and re-decaying are necessary for better training with replay of prior data to avoid forgetting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "* I'd like to see the impact of using moment estimates in Adam optimizer to see if keep using the estimates in pre-training will be of benefit for the continual training.\n* Also, better to analyze how the moment estimates will change during training.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2MGnR6cYQW", "reviewer": "Reviewer_b7Jx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper presents a simple method and tested it on controlled and reasonably large scale dataset to show the effectiveness of this technique, which I really appreciate it. The method is clean and the paper is easy to follow. The technique could have high impact in the future if this holds effective in production scale and extend experiments to more distribution shift scenarios and show benefits in more challenging evaluation settings.\n\nWeaknesses:\n1. the learning rate scheduling rewarming or decaying has existed for a while as the paper indicated in Section 3.4, and the author claims their novelty to be setting up a controlled framework to verify this technique sounds incremental.\n2. The re-decaying method doesn't preserve the loss on the original dataset that the checkpoint is trained on. Therefore, the adapted model is weaker than the model trained on the union of the both datasets. Can this be one reason why the industry prefers to train from scratch on the large new pool of data instead of initializing from one previous ckpt and skip that data source?\n2. The replaying method looks good but the proposed method is sensitive to the hyper-parameter, i.e., randomly picking one hyper-parameter doesn't necessarily lead to better results than training on a union of the data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method to more efficiently and effectively adapt the pretrained checkpoint to the new data by re-warming the learning rate scheduling to avoid performance degradation brought by the distribution shift.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Can the author provide insights on how to choose the hyper-parameter like the replay rate to make the adapted model perform well on both original domain and the targeted domain, without running expensive hyper-parameter tuning?\n2. Can you find a hyper-parameter that out-performs the ckpt trained on the union of the both datasets?\n3. What if there're more new data coming, can this method be extended to >3 datasets?\n4. Can the author provide more evaluation results using popular lm eval benchmarks, especially some more complex tasks like QA, commonsense reasoning, etc.? To show this is a general methods that suits modern models and more challenging benchmarks", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "D8iLMYW5Hq", "reviewer": "Reviewer_uTNo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposed an interesting idea in the continual pre-training step. The proposed 3 phase steps are simple and it will be good to see if the steps can be expanded on larger models or datasets. Also, the experiments are comprehensive and the authors did an excellent job of providing detailed results. \n\nHowever, the result with the strong distribution shift dataset (German dataset) is lacking because there is no meaningful evidence. I understand that the authors have provided some possible explanations. The paper will be stronger if a different distributed shift dataset can be used where these results are more prominent.\n\nMoreover, the addition of an infinite learning rate schedule proposal is only tested in a low-size model. There is no discussion about whether the learning rate pre-warm which is an essential step mentioned in this paper is scalable or whether the forgetting problem is going to be increased in comparison.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a scalable alternative solution to continuous pre-training of Large language models. The common problems for continuous pre-training are poor adapting of new data or forgetting the learned data. To overcome these challenges the authors have proposed a three-stage process- learning rate re-warming, learning rate re-decaying, and replay of previous data. The authors have also noted forgetting can still happen due to the learning rate re-warming and propose an infinite learning rate schedule.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In Appendix A6 - there is no mention of the 10B model performance on the infinite learning rate schedule. You have also mentioned this test was only performed on the 405M model. Can you explain the reason for that?\n\n2. Can you please provide some details of compute consumption due to the replay rate change? Are those linearly related? From Figure 5b, it seems increasing the replay rate is lowering validation loss. So my question is - if changing the replay rate doesn't increase compute much- should we start with 50% replay than 5%?\n\n3. The proposed step relies on replay. Can this be used in private data or if the data privacy need to be protected?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IGZgd9yfu9", "reviewer": "Reviewer_663b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n- Very relevant and useful question and practical solution proposed \n- Thorough experimental setup  \n\nWeaknesses \n- My biggest concern is how general the conclusions were, but how limited the setup was. I understand it takes a lot of compute to do the pre-training, as evident from the training setup described in section 5.3 However, I do not think that pre-training on just 2 setups with GPT-NeoX is enough evidence for the general conclusion.    \n- The takeaways from the results and graphs in section 6 were nicely stated, but it was not easy for me to follow all the graphs and results and how they justified the takeaway.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the best way to continually pretrained LLMs. They argue that rewarming and redecaying the LR and replaying samples can help LLMs learn the new pre-training mixture w/o forgetting the old mixture. The main setup is the Pile to SlimPajama and the Pile to the German Common Crawl.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major\n- Though clearly stated the experimental setup, the conclusions should reflect the experimental setup and be more limited. For example, \n1.  Intro: “To answer this question, we conduct a large-scale empirical study of continual learning techniques for LLM pre-training.” The empirical study was for GPT-NeoX pre-training specifically \n2. Intro - “We establish the effect of learning rate re-warming and re-decaying for models pre-trained using a cosine schedule”. The effect was only shows for GPT-NeoX pre-trained.  \n\nMinor\n- A graph showing the LR vs timestep for cosine decay schedule would help the reader more easily understand what the scheduler is doing \n- “ therefore likely to have newer data within the overlapping domains” → What exact data sources are in SlimPajama and not the Pile?\n- How did you choose the subsampling percentages for the different datasets in SlimPajama?\n- In Figure 2a and 2c, why is there an initial spike in the validation loss for 0% linear warmup?\n- In figure 2b, why is there an increase in the validation loss between 2B to 10B tokens for 0.5%, 1%, and 2% linear warmup?\n- In 6.1.2, what about a baseline that starts at n_max and re-decays without rewarming (i.e. it uses the same scheduler as pre-training)\n- In re-warming and re-decaying, is the model re-warmed the first half and re-decayed the second half?\n\nTypos\n- 5.1 “and validaiton sets”  → validation \n- 5.3 - “adamW” → “AdamW” \n- 5.3 - “ batch size of 1104” → 1024?\n- “GPU nodes” → What type of GPU nodes? \n- 6.2 “However, when using an extreme amount of replay, … we observe that the model adapts less to D0” → Do you mean D1?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers overall agree that the paper meets the criteria for acceptance. Thus, I have chosen to accept the paper as-is. However, the reviewers still have some lingering concerns about generality and novelty, so I have not recommended this paper for further certification.
Decision: Accept (poster)