Investigating Continual Pretraining in Large Language Models: Insights and Implications
Authors: Çağatay Yıldız
Nishaanth Kanna Ravichandran
Nitin Sharma
Matthias Bethge
Beyza Ermis
Venue: Accepted by TMLR ()

Abstract: Continual learning (CL) in large language models (LLMs) is an evolving domain that focuses on developing efficient and sustainable training strategies to adapt models to emerging knowledge and achieve robustness in dynamic environments. Our primary emphasis is on continual domain-adaptive pretraining, a process designed to equip LLMs with the ability to integrate new information from various domains while retaining previously learned knowledge. Since existing works concentrate mostly on continual fine-tuning for a limited selection of downstream tasks or training domains, we introduce a new benchmark designed to measure the adaptability of LLMs to changing pretraining data landscapes. We further examine the impact of model size on learning efficacy and forgetting, as well as how the progression and similarity of emerging domains affect the knowledge transfer within these models. 
Our findings uncover several key insights: (i) continual pretraining consistently improves <1.5B models studied in this work and is also superior to domain adaptation, (ii) larger models always achieve better perplexity than smaller ones when continually pretrained on the same corpus, (iii) smaller models are particularly sensitive to continual pretraining, showing the most significant rates of both learning and forgetting, (iv) continual pretraining boosts downstream task performance of GPT-2 family, (v) continual pretraining enables LLMs to specialize better when the sequence of domains shows semantic similarity while randomizing training domains leads to better transfer and final performance otherwise. We posit that our research establishes a new benchmark for CL in LLMs, providing a more realistic evaluation of knowledge retention and transfer across diverse domains.

Published in Transactions on Machine Learning Research (04/2025)
Investigating Continual Pretraining in Large Language Models: Insights and Implications
Çagatay Yıldız
University of Tübingen
Nishaanth Kanna Ravichandran
nishaanthkamaae amwii:lmmmmmmmmmmmmmmmm.m
Cohere for Al Community
Nitin Sharma
vitinsharmaa31506 gmail.com
University of T'bhingen
Matthias Bethge
matthias, suatkias.bethgeeeuuittuubiiiiii
Tübingen Al Center, University of Tübiingen
Beyza Ermis
beyza@cohereee
Cohere for
Reviewed on OpenReview: http://p//w/ooo.///////////////a
Abstract
Continual learning (CL) in large language models (LLMs) an evolving domain that focuses on developing efficient and sustainable training strategies to adapt models to emerging knowledge and achieve robustness in dynamic environments. Our primary emphasis on continual domain-adapiin pretraining, process designed equip LLMs with the ability to integrate new information from various domains while retaining previously learned knowledge. Since existing works concentrate mostly continual fine-tuning for a limited selection of downstream tasks or training domains, we introduce a new benchmark designed to measure the adaptability of LLMs to changing pretraining data landscapes. We further examine the impact of model size on learning efficacy and forgetting, as well as how the progression and similarity of emerging domains affect the knowledge transfer within these models.
Our findings uncover several key insights: (i) continual pretraining consistently improves 1.5B models studied in this work and is also superior domain adaptation, (ii) larger models always achieve better perplexity than smaller ones when continually pretrained the same corpus, (iii) smaller models particularly sensitive continual pretraining, showing the most significant rates of both learning and forgetting, (iv) continual pretraining boosts downstream task performance of GPT-2 family, (v) continual pretraining enables LLMs to specialize better when the sequence of domains shows semantic similarity while randomizing training domains leads to better transfer and final performance otherwise. We posit that our research establishes a new benchmark for CL in LLMs, providing more realistic evaluation of knowledge retention and transfer across diverse domains.
1 Introduction
Recent advancements in the field of Natural Language Processing (NLP) have been significantly shaped by the development of large language models (LLMs) (Devlin et al. 2018 Radford et al. 2019: Brown et al.) 2020). These models, trained on vast corpora from diverse domains, have emerged as versatile tools for numerous NLP tasks. However, the increasing scale and complexity LLMs have raised concerns about the financial and ecological costs associated with training them from scratch (Luccioni al. 2022). This has necessitated more efficient approaches than retraining these models with each data stream. Continual Learning (CL.) emerges crucial strategy (Sun)

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: The details of the LI domains used in our experiments. Note that Art and Philosophy did not have any subdomains in M2D2 dataset.
Ll DOMAIN (ABBRV)
SIZE
#L2
#TOKENS
EXAMPLES OF L2 DOMAINS
Culture and The Arts (Culture)
1.8
265M
Mass Media, Sports and Recreation
History and Events (History)
1.2 GB
208M
Region, Period
Technology and Applied Sciences (Tech)
1.7 GB
268M
Agriculture, Computing Exercise, Nutrition
Health and Fitness (Health)
739 MB
99M
Religion and belief systems (Religion)
341 MB
48
Belief Systems, Major beliefs
General reference (GeneralRet)
196 MB
39M
Reference works
Philosophy and thinking (PhilThink)
721 MB
124M
Philosophy, Thinking
Art
578 MB
98 M
Philosophy
919 MB
156M
Quantitative Biology (Bio)
1,9 GB
336M
Biomolecules, Cell Behavior
Physics
4.1 GB
22
737M
General Physics, Biological Physics
Condensed Matter (C)ndMat)
3,5 GB
570M
Materials Science, Quantum Gases
Nonlinear Sciences (Nlin)
730 MB
134M
Self-Orgamizing Systems
Mathematics (Math)
4.5 GB
1.4B
Topology, Number Theory
Statistics (Stat)
2.4 GB
6
450M
Applications, Methodology
Economics (Econ)
67 MB
11M
Ecomometrics, Theory
Computer Science (CS)
4.5 GB
39
1.1B
Machine Learning, Gruphics
Astrophysics (Astro)
3.1 GB
562M
Earth/ Parth/Planetary, Cosmology
Total
32.4 GB
159
6.6B
et al. 2019 Biesialska et al. 2020) to reduce both financial and environmental costs while keeping models up-to-date. A critical aspect of this adaptation is ensuring that knowledge transfer occurs seamlessly across domains without catastrophic forgetting (French, 1999) and operates effectively without explicit domain identification for each task.
Continual learning works in L.LSs can broadly be divided into two sub-categories: continual fine-tuning and continual domain- adduiivn-ddaiiveeeeeeeee pretraining. The former incrementally fine-tunes an LLM on a sequence of downstream tasks (Wu)
et al.
2021 Ramasesh al. [02T Scialom al. 2022 Mehta et al. 2023 Gururangan et al.] [02 Khan et al.) 2022
Zhang et al. 2022 Razdaibiedina et al. 2023 Luo al.
2023b). this work, we focus on the latter- adapting LLMs
to new domains through incremental updates, thereby avoiding the need for exhaustive retraining whenever new data
becomes available (Xu et al. 2019; Gururangan et al. 2020; Ke et al.) 2023b). Despite growing interest in this area,
prior studies have primarily examined small-scale models or a limited number of training domains (Qin et al. 2022
Gupta et al. [023 Luo et al. 2023a; Ke et al., 2023b Gogoulou et al. 2024 Cossu et al. 2024) (see Wu et al. 2024
for a recent survey and Section 5 for related works). The most relevant work. Gururangan et al. (020), evaluated the
transfer capabilities of a RoBERTa model continually pretrained across four large domains. However, comprehensive assessment of forgetting and knowledge transfer in LLMs across diverse architectures and scales remains lacking. Furthermore, conventional CL benchmarks such as split CIFAR- 100 (Krizhevsky et al.) (2009) and Tiny ImageNet (Le) & Yang 2015) operate at much smaller data and model scale and thereby fail to capture the unique challenges of continual pretraining in L.Mss.
To bridge this gap, we conduct an extensive benchmark by pretraining LLMs across a wide range of domains and evaluating their performance throughout the learning process. Unlike prior works that focus on a narrow set of domains, our study leverages the Massively Multi-Domain Dataset (M2D22) (Reid et al. 2022), which spans 236 hierarchically organized domains from Wikipedia and Semantic Scholar, enabling detailed investigation of forgetting and knowledge transfer in large-scale setting. Our study systematically explores the core dynamics of continual pretraining in LLMs by (i) comparing continual pretraining vs. standalone adaptation across various architectures and model scales, (ii) analyzing forward and backward knowledge transfer and forgetting at different stages of continual learning, and (iii) investigating how perplexity is influenced by the order of training domains, batch size, and data imbalance. By quantifying key CL metrics, we reveal fundamental differences between LLMs and traditional small-scale CL setups, emphasizing that insights from prior CL research may not generalize to large-scale continual pretraining. Rather than proposing solutions to mitigate forgetting, our goal is characterize the unique challenges of CL in LLMs and provide realistic benchmark for future studies. Our findings uncover several key insights:
2

---

Published in Transactions on Machine Learning Research (04/2025)
Condmat
Histery
Culture+
CS *
Nlin
Tech+
OpenWebText
Religion
Generalief
Blo
Health
Stat
Physics
Econ *
Art-k
Math
Philosaphy
Figure 1: Cosine similarity between our L1 training do- mains. We also include Open WebText (ookaslan & Cohen 2019). an open- open-source replication of the GPT2 pretraining data set. The two big square blocks along the diagonal correspond to Wiki and S2ORC portions.
Figure 2: Average LI -omaai embeddings visualized using t-SNE. Wiki domains and natural sciences form two clear clusters. Note that Art and Philosophy are from S2ORC portion, but they are closer to Wiki due to they are social sciences and the rest S2ORC natural sciences.
i) Continual pretraining improves GPT2 models of all sizes while Llama2-7B does not benefit from it. This is because the majority of the training domains are too small for Llama2-7B to be adapted to.
ii) Continual pretraining outperforms domain- addativee pretraining across all model families and sizes. Importantly, pretraining on related domains enhances knowledge transfer, leading to better specialization when developing expert models for given domain.
iii Final model performance correlates positively with model size. Surprisingly, smaller models consistently show the highest degree of learning and forgetting.
iv) Randomizing the order of training domains enables positive transfer and reduced forgetting.
Continual pretraining curriculum determines the downstream task performance.
2 Methodology
In this section, describe training process, provide overview of the tasks (domains) used for continual pretraining and assessment of models, and explain the evaluation pipeline.
Training We initiate our process with a pretrained LLM that has been already trained on a comprehensive corpus, which generally represents a broad or general domain, such as a book corpus or web content. We then perform continual pretraining on series of unlabeled domain corpora. Our goal to continuously pretrain an LLM on these sequential domain corpora by using the original training objectives, e.g., the next token prediction likelihood for autoregressive LLMs. Note that once domain corpus is used for training, it is no longer available.
Tasks Our experiments are conducted on the M2D2 dataset (Reid al. 2022). which is extensive and finely
categorized corpus specifically designed for exploring domain adaptation in language models. It comprises 8.5 billion tokens and covers 236 domains, sourced from Wikipedia and the Semantic Scholar (S2ORC) database Lo et al. (2019). This dataset is unique in its combination of fine domain granularity and a human-curated domain hierarchy, set within a multi-domain context.
The corpus is divided into two levels: LI-domains and L2-domains. In the context of the S2ORC corpus, Li-domains refer to broad fields of academic research, such as Computer Science and Physics, while 1.2-domains correspond to
3

---

Published in Transactions on Machine Learning Research (04/2025)
specific arXiv categories within these fields, like "Artificial Intelligence" under Computer Science. For Wikipedia, 1.1 L.--domains represent major categories, and 1.2-domains encompass category pages within each L. domain. To maintain balance and computational efficiency in our experiments, we excluded domains exceeding 5GB of data, such as Medicine. Ultimately, we utilized 159 domains in our study (see Table | for details).
To show the cross-domain similarity, we first computed the task embedding by using Sentence-BEi (Reimers & Gurevych 2019) with 1OK samples from each domain and 50K samples from OpenWebText (Gokaslan Cohen 2019). open-source reproduction of GPT2 training dataset (Radford et al.] 2019). Then we computed cosine similarities between each task pair (Figure I For the similar-order experiments detailed in the next section, we order the training domains based on their similarity, starting with the Culture domain, which is the most similar to Open OpenWebText, and then proceeding to the next domain by sampling from the remaining ones based on similarity. Also see Figure 2 for the average L1 embeddings visualized using t-SNE.
Evaluation Each domain in the M2D2 dataset is split into train, validation, and test sets with no data leakage, outlined in [neiidd et al. (2022). Each validation and test set includes over 1 million tokens, allowing accurate evaluations within specific domains. We measure the effectiveness of all methods by testing perplexity on L2 domain test sets. For continual domain dodain-adaptive pretraining experiments, after completing training on a domain for one epoch, we checkpoint the model, and compute the test perplexity for current and previous domains.
3 Experimental Setup
Models and training We benchmark continual learning of existing pretrained L.MSs with different architectures and sizes. In particular, we consider (i) decoder-only models (GPT2-S, GPT2-M, GPT2-L and GPT2-XL, Llama2-7B) as well (ii) encoder-decoder models (RoBERTa-base and RoBERTa-large (Liu et al.) 2020)). We trained the models with Adam optimizer (Kingma & Ba 2015) with batch size of 16 sequences on NVIDIA A100 GPUs. We used DeepSpeed (Rasley et al.) 2020) with auto configuration, which assigns a dropout rate of 0.2 and automatic learning-rate selection.
Task ordering In order to investigate how the order of training domains impacts our domain-incremental continual learning setup, we ordered the tasks in experiments in two different ways: (i) similar-order where semantically related domains follow one another as described in the previous section, and (ii) random-onder, where the domains are shuffled.
Metrics for assessing continual learning efficacy To evaluate the effectiveness of continual learning, we begin by setting two baselines for comparison: zero-shot perplexity (ZS) which measures the innate ability of the original, unmodified models to predict outcomes without any domainn-seciinn tuning, and domain adaptive pretraining perplexity (DAPT), where we first pretrain original models on a particular domain and then evaluate it on the test portion of the training domain. In the same vein as Gururangan et al.] (2020), our continued pretraining on a corpus of unlabeled domain-specific text arguably expected to lead to better expertise on that particular domain. ZS acts as a fundamental baseline, ensuring that our models have basic level of competence and DAPT sets a targeted performance standard for our continual learning approach surpass. Achieving better perplexity than the DAPT baseline is the primary objective for continual pretraining, signifying that longer training horizons are more favorable than stand-alone pretraining that adapts model to a single domain.
To assess continual learning performance, we compute continual pretraining perplexity (CPT) where we evaluate the performance of a model M; on the most recent training domain D;, This measure helps us understand how well the model adapts to new information over time. Moreover, we compute the last checkpoint (LC) My against all the training domains to examine the final model's 's ability retain and transfer knowledge across broad range of subjects. We calculate forgetting (FG) previous domain by taking the difference between the perplexity of the current checkpoint that domain and the best perplexity achieved on that domain so far. Finally, we evaluate checkpoints on previously seen/ been/unseen domains to measure backward/forward transfer.
Metric expressed explicitly Let My denote the pretrained LLM that has been already trained on a comprehensive corpus Dr- Our continual pretraining corpora SN [Di, Dxx consists of N domains. At each stage i, the
4

---

Published in Transactions on Machine Learning Research (04/2025)
Zero dhoe
Dom, adaat peetr.
Cont. pret: ISiilaad
Last ckpt (Simila/)
Fong (Simlari
Comn pret: (Random)
Lost ckpt (Randem)
Forp [Random)
AA.2
21 7878
39.57
16.09
33.3
5.63
2.33
GPT2 SMALL
GPT2 MEDIUM
GPT2 LARGE
GPT2 XLARGE
LLAMA2 7B
Figure 3: Above panels show test perplexities (L) with different model sizes and training orders. For reference, we include the zero-shot and domain adaptation perplexities. Please see Figure T6 for results obtained on Wiki and S2ORC domains.
LLM M;-1 is trained on new corpus D., resulting in an updated model M, Hence, our continual learning setup generates MiNN checkpoints. Let In and fn denote the zero-shot (ZS) and domain adaptive pretraining perplexities (DAPT) computed on n th domain. Note that 2n is obtained by evaluating Mg on Dis whereas DAPT involves first training MM on DIr and then testing on Distt Further, let Ps denote the perplexity of c'th checkpoint Mc on Dn (notice that e > n and c < n correspond to backward and forward transfer). Then the main metrics of our interest computed as follows: 7S- median(z1.w) DAPT median(Ji.s) CPT median(p):
p LC = median(pi FG median ......NN)... gN).ge Ge = Pf min (p 1. Note that checkpoints sometimes achieve arbitrarily high perplexity on a single domain while perplexities on other domains are within the usual range. Hence, we chose median for aggregation instead of mean to deal with such outliers. We also verify that median and mean aggregations differ by less than percent except the outlier samples.
4 Findings and Analysis
In this section, we discuss main findings. We first discuss the main results (Section 4.1). followed by how the model scale impacts continual learning (Section 4.2). Section 4.3 examines the implications of the order of training domains. The subsequent Section 4.4 discusses our positive forward transfer findings. Then we analyze fine-tuning performances on benchmark tasks in Section 445 and Section connects these findings with a novel analysis based on prediction ranks. Finally, we list our remaining observations and ablation studies (Section 4.7).
4.1 What is the efficacy of continual learning?
Continual pretraining consistently improves GPT2 family We start by investigating whether continual learning is beneficial or not. Comparing the "last checkpoint" panels against "zero-shot", we observe that GPT2 models obtained at the end of continual learning always achieve better perplexity on the trained domains, implying significant knowledge accumulation. The overall improvement is overall more pronounced in random-order training. Further, the perplexities achieved by "continual pretraining" are always superior to the ones by "domain adaptive pretraining" Hence, in order to obtain collection of expert models diverse range of domains, continual learning and checkpointing at domain shifts is superior strategy to adapting a base model to each domain separately.
Llama2-7B perplexity does not improve by CL since the domains are too small When we turn to the Llama2-7B results, most strikingly notice that additional training (including domain adaptation) always degrades the perplexity, which conflicts with the GPT2 findings presented above. Diving deeper into the model details, we notice that the largest GPT2 variant (GPT2-XL) and Llama2-7B have comparable model sizes and architectures whereas their training data substantially differ: GPT2-XL is trained on 9B tokens data curated from Web pages while Llama2-7B has a foundational, Internet-scale training dataset with 2T tokens. Hence, we conjecture that additional domain adaptive and/or continual pretraining on relatively small corpus causes Llama2-7B to perform worse. Please note that we did not observe any training issues during learning, i.e., training perplexity always improved.
5

---

Published in Transactions on Machine Learning Research (04/2025)
perplexity (4) asimiar order]
Backward perplexity [random
24
20
35
33
25
25
12
13
Continual training domains
Condinual pre-training domains
SPAL-SMALL
GPTZ-MEDIUM
GPIZ-LARGE
GPTZ-XLARGE
Figure 4: Median backward transfer perplexity during continual pretraining. The grey background highlights Wiki domains. Larger models always achieve better perplexity, which is also aligned with the initial (zero-shot) perplexity. Please see Section 4.2 for detailed analysis.
To inspect our hypothesis, we investigate how domain size impacts the improvement of domain adaptive pretraining for Llama2-7B and GPT2-XL. Each point in Figure 18 corresponds to one domain adaptation experiment. The g-axis corresponds to domain sizes and the y-axis is the perplexity after domain adaptation (normalized by zero-shot). Positive values are the domains on which domain adaptive pretraining degrades perplexity. The left panel shows that and it converges to zero after 100 MB. On the contrary, additional pretraining improves GPT2-XL perplexity on all but six domains. These findings imply that additional pretraining of Llama2-7B model is useful only for relatively larger domains.
Llama2-7B perplexity very rarely improves if the domain is smaller than
4.2 How does model scale impact learning, forgetting, and final performance?
Larger models always perform the best agreement with the recent research on scaling laws (Kaplan et al. 2020 Bahri et al. 202T). Figure 3 shows that the model size perfectly correlates with the model performance. This holds true for all metries (DAPT, CPT, LC) and both orders (with random order achieving better final performance). Further, for each checkpoint obtained after training an L2 domain, report the median backward perplexity (computed all previous domains), and visualize it in Figure 4 The figure reveals that the backward perplexity at any stage of the training correlates with the model size.
Smaller models benefit more from continual learning In relation to the previous analysis, investigate how much CL improves overall model performance. For this, we subtract zero-shot perplexities from perplexities achieved during CL and visualize the results in Figure We can see for random-order training that the improvement is inversely correlated with the model size, which we find intuitive since larger models, already achieving good perplexity, have less room for improvement. The curves corresponding to similar-order training (Figure left) are rather cluttered; therefore, we inspect the average improvement over learning trajectories. The averages show the same performance improvement trend (GPT2-S> GPT2-M GPT2-L) while GPT2-XL surprisingly comes at the second place.
Forgetting inversely correlates with model size Finally, we quantify how much models forget due to continual learning. The "forgetting" panels in Figure3shoor that the smallest/largest model forgets the most/least while GPT2-M and GPT2-L are flipped when the training order changes. While following the trend, we believe Llama2-7B forgetting numbers should be taken with grain of salt since perplexity does not improve overall. Nevertheless, overall, we conclude that increasing model size could be way alleviate forgetting although even the largest model forgets some extent.
4.3
When does training in similar-order help?
demonstrate the positive impact of randomized training order on the model performance and transfer. While these findings consistently favor random order training, CPT panels in Figure interesting pattern: larger models seem to attain bettrr CPT perplexity in similar order training than random order.
As touched upon previously, Figures 3

---

Published in Transactions on Machine Learning Research (04/2025)
Improvement in backward perplexity zero-shot [similar order]
Improvement perplexity zero-shot frandom order)
-10
-15
Continuel pre training domeins
Continual pre-training domains
GPT2-SMALL
GPIZ-MEDIUM
GPT2-LARGE
GFTZ-XLARGE
Zero-shot
Figure 5: Median improvement in the backward transfer perplexity, normalized by zero-shot perplexity. The grey background highlights Wiki domains. Values smaller than zero indicate negative backward transfer. The backward transfer is at its lowest when the portions are switched in similar-order training (left), then it improves again. Positive backward transfer remains throughout learning in random-order.
UPT2-SMALL
GPT2-MEDEE MMEDMM
GPT2-LARGE
10T2-XLARGE
similar
random
Figure 6: Median improvement in backward perplexity (normalized by zero-shot, y axes, higher is better) as a function of the number of tasks between the checkpoint and the tested domain (æ axes). The diamonds and circles correspond to the similar and random order training and colors denote the models. The benefits of continual learning in similar-order steadily degrade with the transfer distance while it is stable for random order.
To further analyze this, instead of reporting aggregated performance all previous domains, contrast the perplexity change with how old the evaluation domains are. In particular, æ-axis of Figure sloowws how many tasks have passed between a checkpoint and domain it is tested on. For all model sizes, similar-order training impacts backward transfer to recent past (up to 40 domains) more positively than random-order. Naturally, the improvement in the case of similar-order training degrades over time since the recent training domains become significantly dissimilar to tested domains while random-order training does not worsen. Also, very interestingly, similar-order training larger models seems degrade less, indicating less forgetting, which agrees with the finding the previous section that larger models forget less.
4.4 When do we observe positive forward transfer?
Our analysis far has focused mainly on backward transfer and forgetting. In this section, we investigate whether models exhibit positive forward transfer by evaluating models on future domains.
Random-order training consistently leads to positive forward transfer For our first analysis, we evaluate all GPT2-S checkpoints saved after L2 domains on all unseen domains. Figure how forward transfer perplexity improves upon zero-shot performance. Noticeably, positive forward transfer is possible only to the $2ORC portion, i.e., no perplexity improvement when the model evaluated on the Wikipedia portion. This discrepancy expected as the S2ORC portion is about five times larger than the Wikipedia portion. Further, test perplexity on the S2ORC portion consistently improves with the number of pretraining tasks, i.c., longer training improves forward transfer. We conclude that the model accumulates knowledge that is on average beneficial to predict next tokens unseen finer-grained domains.
Positive forward transfer in similar training order is possible only semantically related domains To examine how training similar-order domains affects forward transfer, evaluate model checkpoints all subsequent I.I
7

---

Published in Transactions on Machine Learning Research (04/2025)
Improvement in forward transfer wrt zero-shot (T) (GPTT/GPPLLLPLLLLI SMALL-
Forgetting on the last 50 tasks (4)
Amnn Awne
-20
20
40
60
B0
100
Number of tasks the model trained on prior forward transfer
GPT2-small GPT2-mediu GPT2-large GPT2-xlarge
Wikipedia
Zero-shot
Checkpoints (50-100)
Checkpaints (100-150)
Figure 7: Forward transfer results with random training order. The T axis shows the number of domains the model is trained on be- fore forward transfer. Curves show the perplexity improvement over zero-shot. Clear positive/negative forward transfer to S2ORC/Wiki portions is observed.
Figure 8: order training into two groups based on their recency, showing that earlier checkpoints trans- fer better the recent past.
We divide the checkpoints in random-
Arthvmetic
14
Know edge
Nerma Approcate Scre
Colore
Boch
Resiion
Pommis
OPTcetation
Calber
Bef
Amgiion
4000
Figure 9; GPT2-M performance on two downstream tasks, captured at LI domain transitions (the higher the better).
The initial points represent the baseline performance of GPT2-M. Please see Figure 12 for other tasks.
domains immediately after completing training on given L.1 domain. Each plot in Figure 23 shows the forward transfer performance the domain stated the title. Most notably, panels in the first row reflect that pretraining on handful of domains leads to significantly worse performance compared to zero-shot (the dotted horizontal lines). In contrast, extended pretraining across a variety of domains occasionally leads to positive forward transfer (panels 2 and 3). Further, we notice a recency effect where the forward transfer perplexity improves if checkpoint is transferred to a domain that is conceptually similar to the most recent training domain: as anticipated, the most successful forward transfer to Astrophysics domains is attained after training on Physics.
4.5 Does continual learning influence downstream performance?
Until now, our evaluation has centered on assessing the language modeling capabilities of our models, specifically using perplexity as our performance metric. While training and evaluation performance metrics in CL have traditionally been the same (e.g., classification accuracy), LLMs are often benchmarked on downstream tasks whose metrics (e.g., question-answeringg accuracy) often differ from the pretraining objective (e.g., perplexity). Likewise, moving forward, we assess the performance of our continually trained models on different tasks. Overall, our findings reveal that continuing pretraining domains relevant these tasks generally enhances model performance, while pretraining on unrelated domains often leads to forgetting, thereby negatively affecting the model's initial task proficiency.
We choose five tasks aligned with our benchmark domains from BIG-Bench (bench authors 2023): Arithmetic, General Knowledge, Physics, CS Algorithms, and Few-shot Natural Language Generation (NLG). We report Normalized Aggregate Score, that is, the normalized preferred metric averaged over all subtasks under that particular task (c.g., Arithmetic task has twenty subtasks). In (bench authors) 2023), they specify that the best pert-reeformin language models achieved a score below 20, and model scores can be less than zero on some tasks. Comprehensive task descriptions, metrics, and additional outcomes provided in the Appendix A.3
8

---

Published in Transactions on Machine Learning Research (04/2025)
Median rank (1)
1
400
th UM A Urn prverngg & //y
M AN
200
Liama Carth Scierices
Trans cxSt EaSY
Engieeeing
MLl
Sttt.4
Senf ccae
CuEV
Alaah
cs cHy By region
Major beliefs af the egrit
MM
os ristory and events By continent
Perterning ars
C5.5E
History and events
Figure 10: Our rank-based knowledge transfer analysis, where continually pretrained Llama2-7B checkpoints are tested cs. AI domain. The red and green dots highlight some domains which the rank is significantly high and low. We see an increasing trend, implying a decrease in performance. We also notice that domains that are semantically similar/distant to cs. AI consistently lead to better/worse rank.
GPT2-M downstream performance is determined by continual pretraining domains As depicted in Figure 2 consistent decrease in Arithmetic task performance noted when models were continually trained on Wiki domains which then improves upon switching to S2ORC domains, with the exception of the Nonlinear Sciences and Astrophysics domains. The performance trends for the CS Algorithms and Physics tasks align with those observed for the Arithmetic task, with peaks reached after training on the CS and Physics domains, respectively (see Figure 12 for the results). In contrast, performance on General Knowledge tasks improved with Wiki domain training but went below the base performance led by training on S2ORC. Overall, we conclude that the performance gain/degradation measured by perplexity in the context of CL transfers downstream performance.
Continual Llama2-7B checkpoints consistently achieve chance-level performance Next, we repeat the same analysis with Llama2-7B checkpoints. As shown in Figure [3 performance drops drastically, often falling below GPT-1, regardless of the task or training order. Notably, the decline in downstream performance after the first task is catastrophic, starkly contrasting with our findings for GPT2-M. Further analysis reveals that continual pretraining severely degrades the model's ability to perform in-context learning and generate coherent dialogues. This aligns with observations from (bench authors 2023), which highlight the brittleness of language models their sensitivity natural language phrasing. Taken together, our results continual Llama2-7B checkpoints do not provide immediate insight into the extent of forgetting.
4.6 Prediction rank-based analysis for knowledge accumulation
Next, address the limitations of downstream task evaluation presented in the previous section, which stems from our continually pretrained Llama2-7B checkpoints' inability to be zero/one/few shot prompted. In particular, rank-based method provides an alternative metric for assessing domain knowledge retention, independent of prompting ability. We also demonstrate that the ranks achieved by Llama2-7B show much more interpretable pattern than the downstream task performance.
Our proposed pipeline builds on Geva et al. 2023), starting with text corpus preprocessing, including citation removal, LaTeX markup cleaning, and special character filtering. Next, we embed all sentences using Sentence-BERT (Reimers) 2019] and cluster the embeddings. Each cluster center is then associated with representative keyword an n-gram extracted from the sentences within the cluster. For example, "prior distribution" and "multi-layer perceptron" are cs.AI AI corpus. For each cluster, we extract target words- frequently occurring terms with high semantic similarity to the keyword (e.g.." "activation" and "forward pass" for "multi-layer perceptron"). We then retrieve sentences from the test corpus containing these keywords. Our continually pretrained LLMs process these sentences up to the keyword, where the keyword itself serves as the ground truth prediction target. Our key metric the rank of the target token, which reflects the models' ability to complete domain domain-specific sentences. By focusing on domain and subject-specific contexts, approach quantifies knowledge accumulation in a text corpus without being influenced
keywords related to the cs

---

Published in Transactions on Machine Learning Research (04/2025)
by prompt variations. We remind that our pipeline is domain invariant and can be applied any text corpus. This is particularly useful for niche domains (e.g., Cosmology and Nongalactic Astrophysics) for which there may be no corresponding downstream task.
Figure I0 presents the average rank of a Llama2-7B model continually pretrained on randomly ordered domains, checkpointed after each 1.2 domain and evaluated across all target words associated with all keywords in the es. AI domain (see Figure 14 for Llama2-7B model trained in similar order). Strikingly, the original Llama2-7B model achieves the lowest rank, i.e., the highest performance. Over the course of continual pretraining, we observe clear upward trend in rank, indicating a decline in prediction quality with further training. Knowledge transfer also follows distinct pattern: semantically similar domains, such as stat. ML and cs.CV, consistently achieve better ranks (signifying positive transfer), whereas more distant domains, such as Transport and Self care, lead to significantly worse transfer. Figure [ shows the findings of the same rank analysis on a GPT2-M model trained randomly. We notice an overall rank improvement, hinting at knowledge accumulation. Further, the training domains that achieve the best ranks are all CS domains; therefore, we conclude that our rank-based analysis is somewhat informative on the model's 's knowledge on a partieular topic. These findings align with our downstream analysis GPT2-M, reinforcing that training domains strongly influence test performance.
4.7 Additional observations and ablations
RoBERTa results differ substantially from decoder-only results To broaden our analysis and gain deeper insights into the behavior of different architectures, we have repeated all the experiments with RoBERTa and obtained somewhat surprising results. As shown in Figure 17. RoBERTa models do not seem to exhibit forgetting on old tasks, a finding that aligns with Cossu et al. (2024). This is also visible in Figure 24 where backward transfer perplexity remains similar to continual pretraining perplexity. Finally we observe positive forward transfer most of the time, which also conflicts with the decoder-only model findings. Please see Section A.2 for more detailed analyses.
LLMs forget more in the later stages of continual learning To complement the forgetting findings presented in Section 4.2 we analyze how the forgetting dynamics evolve during continual learning. We divide the checkpoints in random-ordes training into two groups based on their recency (checkpoints(50-100 and checkpoints| (00)) We evaluate all checkpoints on the last 50 domains and compute the perplexity change (caused by additional training). Histograms in Figure show that earlier checkpoints exhibit less forgetting, i.e., transfer better to the past. We informally hypothesize that in the earlier stages of training, the parameters that not important to the recently learned tasks are updated and once the models fill their learning capacity in the later stages, parameter updates become detrimental.
Batch size impacts learning dynamies As an ablation study, we increase the batch size from 16 to 64, thereby performing a quarter of gradient updates. Figure 20 compares the results obtained with different batch sizes. When trained in random-ordes continual pretraining and last checkpoint perplexities virtually remain the same despite varying the batch size. In similar-order, smaller batch size helps to improve continual pretraining perplexity but worsens the performance of the last checkpoint. We hypothesize that taking more gradient steps aids the model to better fit the current task while promoting forgetting the old tasks.
Balancing the data size across L2 domains does not improve performance We investigate whether the imbalance in training data sizes impacts the overall performance (see Table for L1 domain lengths). To address this, we set the number of maximum tokens to 100K for each L.2 domains (if they have fewer tokens, we used them all), and train the original model. Figure shows the resulting continual pretraining and last checkpoint perplexities. For both metrics, test performance on almost all L2 domains deteriorates after balancing the number of data points per domain. The results suggest using all data hand instead of leaving some out for the sake of balanced training.
Swapping Wiki and S2ORC portions verifies previous findings We swap the portions for similar-order training, i.e., training first on S2ORC, then on the Wiki portion. Arguably. this training order still follows conceptual similarity; hence, it allows us to see whether our previous findings still hold. The left panel in Figure shows that continual pretraining perplexity remains almost the same. Yet, the last checkpoint perplexity significantly changes: while the performance on the S20RC portion substantially degrades, we observe the opposite effect for the Wiki portion. Agreeing with our previous findings, conclude that the checkpoints perform worse when tested older domains/portion.
10

---

Published in Transactions on Machine Learning Research (04/2025)
Alternative random orders yield similar findings In random-order experiments, consider only one random- ized training sequence. To test whether the findings do not generalize to alternative randomized orders, we re-shuffle the dataset twice and repeat the experiments with GPT2-M. These experiments resulted in median CPTs of 16.4 and 16.78 while 16.69 obtained our main set of experiments. Given relatively much larger differences across different experiment setups, we conjecture that the standard deviation resulting from different random orders can be safely ignored.
5 Related work
We discuss two related but separate lines of research in the context of CL for LLMs: (i) continual fine-tuning, which aims at fine-tuning LLMs on a series of downstream tasks, and (ii) continual domain-adaptive pretraining, focusing on incremental updates to adapt an LLM to new domains without exhaustive retraining from scratch upon new data.
Continual fine-tuning A large body of CL. works for LLMs tries to mitigate forgetting during continual fine-tuning. [Luo et al. 2023a) investigate forgetting and distribution drift during continual learning on a series eight downstream classification tasks. In recent work, (Luo et al., 2023b) examines evolution of forgetting during continual fine-tumin fine-tuning.
Scialom et al. (2022) instruct fine-tune an LLM for eight tasks. KKaan et al. (2022) introduce an adapter-based fine-tuning strategy for three downstream tasks. Zhang ct al. (2022) propose add new modules to sequence generator (such an LLM) to continually adapt to five tasks. Razdaibiedina et al. (2023) introduce progressive prompts, where a growing number of prompts, are learned during continual learning, fine-tunes on 15 classification datasets. WWag et al. (2023) propose to learn orthogonal adapters to minimize interference between 15 classification tasks. QQn et al. (2022) propose efficient lifelong pretraining for emerging data (ELLE), where they expand a network during learning and include domain -inetiiiieniiiying prompts during pretraining to help the PLM identify the type of knowledge it is learning.
Continual domain- -ddative pretraining An alternative research direction, closer to our work, aims to continually pretrain LLMs to adapt them to new domains. In one of the earliest studies, Gururangan al. (2020) introduce growing
mixture of expert architecture for domain-adapticn continual pretraining Lazaridou al. (2021 presents dataset
to assess temporal generalization of language models when trained continuously. Jin al. (2021) continually pretrain research paper stream and a chronologically-orrr orderre tweet stream with different continual learning algorithms. In a similar study, Fernandez et al. (2024) proposes layer-specific pretraining and learning rates to mitigate forgetting when a GPT2-L model is adapted to (temporally changing) Wikipedia snapshots.
RoBERTa-base over a
Chen et al. (2023) study lifelong learning from a sequence of online pretraining corpus distributions based on a progressively growing mixture-of-experts (MoF) architecture. Likewise, Curuuragan et al. (20T)) introduce mixture architecture for continual adaptation, Ke et al. (2023a) show how a soft-masking mechanism for gradients of RoBERTa model could be useful for Nomainnadaptinn pretraining for eight tasks. Cossu et al. (2024) investigate the characteristics of the continual pretraining across ten domains. Duwal et al. (2024): Vo. (2024) study domain adaptation of Llama3- 8B model to a new language (Nepali) using quantized low-rank adaptation. The more elaborate study of Gogoulou et al. (2024) includes morphologically similar three (North European) languages and models up to 1.3B parameters. A very interesting technical report by Vo et al. (2024) shows that combining diverse pretraiming stages (an initial training of the embedding layers, pretraining of the entire network, and finally low-rank adaptation) yields impressive performance on Korean language. Di et al. (0024) continually pretrain a Llama2-7B model on 250k question-answe pairs but do not (2024) discover that LLMs suffer from forgetting in "continual memorization" tasks and replay methods cannot fully resolve the problem. Gupta al. (2023) examine different warm-up strategies for continual pretraining. [Öncel et al.
investigate forgetting. Chen al.
(2024) studies when and why domain- adaptive pretraining of LLMs may fail. Finally, Fisch et al. (003)) introduce a benchmark of task sequences that potentially lead to positive and negative transfer and further propose a simple strategy for robust forward transfer, which aims to pick the checkpoint with the biggest positive knowledge transfer among all past task checkpoints. Our work diverges from the others in that we continually pretrain the original model without any expansion on a much longer horizon of 159 domains, and further investigate the impact of domain order.
11

---

Published in Transactions on Machine Learning Research (04/2025)
6 Discussion
Prior studies in CL for LLMs have mainly focused on parameter-efficient fine-tuning or adaptation for limited selection of target domains or tasks, While beneficial, these methods often do not fully address the broader challenge of lifelong learning for LLMs. Our research diverges by exploring continual domain-adaptive pretaining of LLMs across an extensive set of domains to better understand the dynamics of knowledge preservation, new information retention and knowledge transfer. Below, we highlight three key insights and discuss notable observations from our research:
Continual pretraining benefits GPT models but degrades Llama2-7B While continual pretraining improves GPT models of all sizes, it negatively impacts Llama2-7B. This trend is consistent across all our metrics perplexity, downstream task performance, and output rank. Regardless of model size, continual pretraining outperforms domain- adaptive pretraining, underscoring the advantages of exhaustive training. Finally, we empirically show that Llama2-7B requires domains to be larger than 100 MB for improved adaptation.
Scaling laws for continual pretraining Our experiments reveal that larger models consistently achieve better perplexity and exhibit less forgetting. However, smaller models are more sensitive to continual pretraining, showing both the highest degree of learning and the most pronounced forgetting.
Continual pretraining boosts downstream task performance of GPT family For both GPT2-M and Llama2-7B. performance on BIG-Bench tasks follows the same trend as perplexity. Further, downstream task success is heavily influenced by the most recent training domains and correlates well with our complementary prediction rank analysis. A key takeaway is that additional generative pretraining before supervised fine-tuning can further enhance performance.
Randomizing training domain order improves knowledge retention With the randomized training order, we observe two benefits: that (i) the final checkpoint achieves superior average performance compared to similar-order training, and (ii) checkpoints exhibit positive backward transfer and reduced forgetting, suggesting that previously learned knowledge remains more intact.
Similar-order training strengthens domain specialization When consecutive training domains are semantically similar, continual learning enhances specialization. This is supported by two observations: (i) continual pretraining achieves lower perplexity in similar-order training, likely due to gradual knowledge accumulation, and (ii) models exhibit positive transfer to recent past domains but not to more distant ones the training sequence.
7 Limitations
Our research highlights CL as a powerful paradigm for learning LLMs, providing valuable insights into its mechanisms and benefits. However, we acknowledge several limitations in our study:
Since part of RoBERTa 's training data includes Wikipedia entries, potential overlap with our training set may have influenced its results.
To report average backward transfer perplexity, exhaustively evaluated all checkpoints across all past domains, totaling 12,561 evaluations model per setup. Forward transfer was assessed only after completing all L2 domains within a given LI domain, still requiring 171 evaluations per model per setup.
Computation time remains a fundamental limitation. With experiments spanning 159 tasks, the sequential nature of continual pretraining prevents parallelization of the training process.
Investigating how the original pretraining corpus (e.g., WebText for GPT-2) impacts continual learning of individual domains would be an interesting future work.
Acknowledgements Çağatay Yildiz and Matthias Bethge are members of the Machine Learning Cluster of Excellence, funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's 's Excellence Strategy EXC number 2064/1 Project number 390727645. This research utilized compute resources the Tübingen Machine Learning Cloud, DFG FKZ. INST 37/1057-1 FUGG!
12

---

Published in Transactions on Machine Learning Research (04/2025)
References
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprini 4rXiv:2102.06701, 2021.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. htattsss//ppeerrvvieeeen//rrrrro
uy
Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-Jussa. Continual lifelong learning in natural language processing: A survey. arXiv preprint arXiv:2012.0982. 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1 1877-1901, 2020.
Howard Chen, Jiayi Geng, Adithya Bhaskar, Dan Friedman, and Danqi Chen. Continual memorization of factoids in large language models. arXiv preprint 1rXiv:241..07175, 2024.
Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. Lifelong language pretraining with distribution--peciiaiieed experts. In International Conference on Machine Learning, pp. 5383-5395. PMLR, 2023.
Andrea Cossu, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, Tinne Tuytelaars, and Davide Bacciu. Continual pre-training mitigates forgetting in language and vision. Neural Networks, 179:106492, 2024.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint /8X1vv/810004805, 2018.
Sharad Duwal, Suraj Prasai, and Suresh Manandhar. Domain-adaptative continual learning for low-resource tasks: Evaluation on nepali. arXiv preprint arXiv:2412.13860, 2024.
Jared Fernandez, Yonatan Bisk, and Emma Strubell. Gradient localization improves lifelong pretraining of language models. arXiv preprint arXiv:24] 1,04448, 2024.
Adam Fisch, Amal Rannen-Triki, Razvan Pascanu, Jörg Bornschein, Angeliki Lazaridou, Elena Gribovskaya, and Mare' Aurelio Ranzato. Towards robust and efficient continual language learning. arXiv preprint arX5v:330.005711, 2023.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4): (284):22-11)) 1et::222-1335 135, 1999.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304. 14767, 2023.
Evangelia Gogoulou, Timothée Lesort, Magnus Boman, and Joakim Nivre. Continual learning under language shift. In Elmar Nöth, Aleš Horák, and Petr Sojka (eds.), Text, Speech, and Dialogue, pp. 71 71-84, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-70563-2
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://SkyLloo00///ttttttlo/oobp 2019.
Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats I. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. Continual pre-training large language models: How to (re) warm your model? arXiv preprint arXiv:2308.0400 04014, 2023.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't i stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004. 10964, 2020.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer. Demix layers: Disentangling domains for modular language modeling. arXiv preprint 108 000360.0.00......0 2021.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Yuhe Ji. Yilun Liu, Feiyu Yao, Minggui He, Shimin Tao, Xiaofeng Zhao, Su Chang. Xinhua Yang, Weibin Meng, Yuming Xie, et al. Adapting large language models to log analysis with interpretable domain knowledge. arXiv preprint arXiv:4412.01377 2024.
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wer Wen Li, Xiaokai Wei, Andrew Amold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging corpora. arXiv preprint arXiv:2110.08334, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08.0 1.0361, 2020.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. In The Eleventh International Conference Learning Representations, 2023a.
Zixuan Ke, Yijia Shao, Haowei Lin, Hu Xu, Lei Shu, and Bing Liu. Adapting language model while preserving its general knowledge. arXiv preprint arXiv:2301.08966 2023b.
Shadab Khan, Surbhi Agarwal, and PK Srijith. Lifelong language learning with adapter based transformers. In Continual Lifelong Learning Workshop ACML 2022, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega CA, USA, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cy- prien de Masson d' Autume, Tomas Kocisky, Sebastian Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems, 34:29348-29363, 2021.
Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23/N, 7(7):3, 2015.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2020. URL /tppp/tr////oeerreiieww
nett/forum?id=5yyS00t15.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Dan S Weld. S2ore: The semantic scholar open research corpus. arXiv preprint / 1.00782,, 2019.
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of bloom, 176b parameter language model, 2022.
Yun Luo, Zhen Yang, Xuefeng Bai, Fandong Meng, Jie Zhou, and Yue Zhang. Investigating forgetting in pre-trained representations through continual learning. arXiv preprint arXiv:2305. 059668 2023a.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, arXiv preprint arXiv:2308.087: 2023b.
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. Journal of Machine Learning Research, 24(214):1-50, 2023.
Firat Öncel, Matthias Bethge, Beyza Ermis, Mirco Ravanelli, Cem Subakan, and Çağatay Yıldız. Adaptation odyssey in LLMs: Why does additional pretraining sometimes fail to improve? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 19834-19843, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653//1/2024.cooom www.:4emmmlllammee 108. URL httttp:::s/aaatttttttat
org/ 2024. wtt..o////2....mmmm......... 1108/
Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Elle: Efficient lifelong pre-training for emerging data. arXiv preprint arXiv:2203.063/1 2022.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 18):9, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 2016.
Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2021.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong Hc. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2505-3506, 2020.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. arXiv preprint arXiv:2301./2314 2023.
Machel Reid, Victor Zhong, Suchin Gururangan, and Luke Zettlemoyer. M2d2: A massively multi-domain language modeling dataset. arXiv preprint urXiiv2210.073700 2022.
N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese hert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL
10084
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6107-6122, 2022.
Fan-Keng Sun, Cheng- Hao Ho, and Hung-Yi Lee. Lamol: Language modeling for lifelong language learning. arXiv preprint arXiv:1909.03322 2019.
Anh-Dung Vo, Minseong Jung, Wonbeen Lee, and Daewoo Choi. Redwhale: An adapted korean Ilm through efficient continual pretraining. arXiv preprint arXiv:2408. /294, 2024.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang. Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning, arXiv preprint urXiv:2310.14152 2023.
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi. and Gholamreza Haffari. Pretrained language model in continual learning: A comparative study. In International Conference on Learning Representations, 2021.
Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey, arXiv preprint 2rXiv:2402.01364 2024.
Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. Bert post-training for review reading comprehension and aspect-based sentiment analysis. arXiv preprint 1rXX::1904.02222, 2019.
Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. Continual sequence generation with adaptive compositional modules. arXiv preprint arXiv:2203.10653 2022.
A
Appendix
A.1 Key distinguishing aspects
We hope that our research marks a shift towards establishing more realistic benchmark for investigating CL in LLMs and hope that our interesting findings lay the groundwork for future studies in this realm. With this, we would like highlight some key distinguishing aspects of our work.
15

---

Published in Transactions on Machine Learning Research (04/2025)
GPT2-SMALL test perplexity (batch training)
GPT2-MEDIUM test perplexity (batch training)
23
20
20
17
100 200
300
600
700
100
200
300
400
500
600
700
Number training steps (x 1000)
Number of training steps (x 1000)
Figure 11: We mix all the samples in all the domains and train GPT2-S and GPT2-M models on the mixed dataset. One epoch corresponds to 700k iterations. We observe that the final checkpoint has comparable perplexity (19.2) to continual pretraining perplexities in the similar and random orders (19.38 and 19.67). Importantly, it significantly larger than domain adaptive pretraining perplexity (21.48) and the final checkpoints of continual pretraining in both orders (24.79 and 20.5). For GPT2-M, due compute constraints, we had end the training at 420k iterations. Extrapolating the perplexity curve reveals the same findings as GPT2-S.
A single continual pretraining run with the smallest (GPT2-S) and largest (Llama2-7B) models in our bench- mark required days and 4 months of training. respectively. Further, we compute the perplexity of all checkpoints on all previous domains, which adds up to 12,561 evaluations. This scale stands in stark contrast to conventional benchmarks such split CIFAR-100 (Krizhevsky et al. 2009) and Tiny ImageNet (Le & Yang 2015), which operate on much smaller scale.
The continual learning tasks in some of the previous works (Cossu et al. 2024 involve end-task fine-tuning to evaluate the performance of the continually trained LLM. On the contrary, most training domains considered in this work are not tied to a particular downstream task.
Our key findings in Figure highlight perplexity a primary metric. While this may seem at odds with standard LLM benchmarking practices, it aligns with the continual learning literature, where the primary optimization objective is consistently tracked and reported. Notably, we observe strong correlation between perplexity and downstream task performance, reinforcing its value as a proxy for measuring knowledge transfer and forgetting.
Our benchmark facilitates the study of how semantic ordering of training domains impacts performance aspect largely unexplored in previous research.
A.2 Perplexity in masked language models always increases with continual learning
To broaden our analysis and gain deeper insights into the behavior of different architectures, we have repeated all the experiments with RoBERTa and obtained somewhat counter- -nntutr- and surprising results. First of all, we want to point out that perplexity not well-defined for masked language models like RoBERTA In our experiments, we used the same perplexity computation for RoBERTa as the one we used for GPT2, which is equal to exp( Σ' log (1, (r.))) where X (Z....., .2,) is the tokenized sequence and log (x is the log- likeliholooo of the ith token conditioned the preceding tokens X according the model.
Forgetting in RoBERTa family is not evident In contrast to the GPT family, our analysis reveals that the RoBERTa family does not exhibit forgetting of old tasks during continual training. To illustrate this, we present a visualization of backward transfer performance across four randomly selected domains in Figure 24 Similar findings with encoder- decoder models were reported in (Cossu al. 2024). We conjecture that modifying the model architecture by including bottleneck layer plays significant role in this behavior.
16

---

Published in Transactions on Machine Learning Research (04/2025)
CS Algorithms
Moreatee hogrogaaa Soae
Prysics
Few Shot NLG
12
Cara Redtmn
Figure 12: GPT2-M performance on the remaining three downstream tasks from BIG-Bench, captured at L1 domain transitions. The initial point represents the baseline performance of GPT2-M. The CS Algorithms and Physics results align with the findings in the main paper. For the Few-shot NLG task, performance trends across Wiki and S2ORC domains do not follow a consistent pattern. A deeper inspection reveals that domains such as Culture, Art, Philosophy, Math, Stat, and Econ contribute positively to performance enhancement in this task.
RoBERTa-large always exhibits positive backward transfer while GPT2-L transfer performance depends heavily on the transferred domain Looking into Figure 24 we notice that backward transfer perplexity of RoBERTa-large remains relatively close to fine-tuning performance. Interestingly, we observe occasional jumps in perplexity when trained in random order, whose analysis is an interesting future work. On the other hand, Figure [9 demonstrates backward transfer the same four domains when GPT2-L trained. In agreement with our earlier findings, switching from Wiki portion to S2ORC causes significant perplexity degradation on Wiki domains when trained in similar order. Further, the characteristics of the test domain seem to determine whether the transfer is positive negative. Finally, we observe less fluctuating backward perplexity with random training order.
Encoder-decoder models require just a few L1 domains for good transfer (i) In stark contrast with the decoder- only models, pretraining even on the first L.1 domain helps to exceed zero-shot performance (comparing the dotted lines and the first point of each sequence). Interestingly, this holds when the pretraining and test domains belong different portions of the training set. (ii) We further notice the forward transfer perplexity tends improve for the first ten LI domains and later slightly degrade. Since it is still considerably above zero-shot performance, chose not to investigate this in detail. (iii) Lastly, the model size does not seem to influence forward transfer performance, which is again as opposed to decoder-only models.
A.3 BIG-Bench Experiments
Tasks.
We selected five tasks that align with our benchmark domains, as described below:
Arithmetic evaluates the model's ability in basic arithmetic operations - addition, subtraction, multiplication, and division - ranging from 1-digit to 5-digit numbers.
General Knowledge assesses the model's ability answer questions across broad spectrum of general knowledge, for example, "How many legs does a horse have?". It draws parallels with benchmarks focused general-knowlens question-answering such as those found in (Rajpurkar et 2016).
Physics aims to test the model's understanding of physics by asking it to determine which formula is needed to solve a given physics word problem, and evaluating the accuracy of the multiple choice responses. The decision to utilize a multiple-choice format concentrates on the model's 's comprehension of the physical principles each formula represents, addressing concerns that generating physics formulas through text might be overly challenging for current models.
CS Algorithums measures the model's performance two core algorithmic concepts: recursion (or stack usage) and dynamic programming, evaluating the model's 's computational thinking and problem-solving skills.
Language Generation from Structured Data and Schema Descriptions (Few-shot NLG) aims to assess the ability of a model to generate coherent natural language from structured data, supported by schema descriptions, within the framework of task-oriented dialogue system. The goal to determine whether a virtual assistant can learn to generate responses based on the textual description of structured data, enabling rapid adaptation to new domains with minimal additional input.
17

---

Published in Transactions on Machine Learning Research (04/2025)
Arthmetic (random)
Anthmetic (smilar)
8
Algorithms (random)
CS Algorithms (similar)
44
Physics (random)
Physies (similar)
Few Shot NLG (randorn)
Faw Shot NLG (similar)
8
General Knowledge trandam)
General Knowladge Isimilar?
Muyy WAAr My Wh
Figure 13: Llama2-7B performance on all downstream tasks from BIG-Bench. The initial data point represents the baseline performance of Llama2-7B: We notice that only after a few L2 domains, model performances immediately drop and never recover the base model.
Median rank (1)
700
200
Cutture
Hontoy
Bect
Health
Bo
Physic
Conditet
Minn
Math
Adtro
Figure 14: Our rank analysis for Llama2-7B 7B model trained on similar order. The grey/purple backgrounds show the Wiki and CS domains. We notice that the training domains clearly influence the transfer to cs. AI domain.
18

---

Published in Transactions on Machine Learning Research (04/2025)
Median rank (L)
60
40-
M
30
Way M M And M M
20-
10-
Sae2 mediom cond mattsti//lnnn
RD
c3 Al
stat.fl.
cs.c astro sh.co
math N7
CS.L
CsCC
astro HE
cc ke
math
Figure 15: Our rank analysis for GPT2-M. We that continual pretraining in random order, overall helps the model achieve better rank cs. AI test set. This finding is in agreement with the downstream task analysis presented in Section 4.5
Paro shot
Dwm. adeot portr.
Cont prets (Smiiar))
Lest ript (Smmlas)
Fong (aimlar)
Cont pret: [Random]
Last rkpt [Bendrm)
Farp [Randrmm]
M:31
12
20.00 02
23.66
42
0. 986
GPT2 SMALL
GPT2 MEDIUM
GPT2 LARGE
GPT2 XLARGE
LLAMA2 7B
Forg.
Forp. (Nandam)
20.03
22.64
18 82
20.29
22.24
3.741
GPT2-SMALL
GPT2-MEDIUM
GPT2-LARGE
GPT2-XLARGE
LLAMA2-78
Figure 16: more detailed analysis of our main results table. This time, we compute the test perplexities()) on Wiki and S2ORC portions separately, visualized in the upper and lower panels.
A.4 Domain sequences
How did we order domains? For random order experiments, we randomly shuffle all L2 domains and use the same domain sequence in all our experiments. For sanity check, we shuffle all 1.2 domains with different random seeds twice and train GPT2-M on these sequences.
For similar order experiments, and 50K samples from Open Web using Sentence-BERT (Reimers & Gurevych 2019). Then we set the similarity between two domains as the mean of the cosine similarities between all possible pairs of embeddings from these two domains. Figure shows the obtained similarity matrix. To form our similar order, we start with the Culture domain, which is the most similar to Open WebText. Subsequent domains are chosen from the unselected ones, where the probability of a domain being selected proportional to its similarity to the most recently selected domain. We believe this procedure contains some randomness while similarity ensured thanks to the said probabilistic selection procedure. Again, for sanity check, we consider two more similar orders and train GPT22S on GPT2-M on them.
first compute domain similarities by embedding 10K samples from each L1 domain
19

---

Published in Transactions on Machine Learning Research (04/2025)
Zero shot
Bem adapt peetr
Cont peetr iSimitei
Lest diptt 1Smmlar
Fong (Sinter
Cene. prem: (Random)
Last cxpe (Randem)
Fonp (Pandom))
1.57
1.5
1.25
1.28
ROBERTA BASE
ROBERTA LARGE
shot
adapt prate
Cont. poetr (Gimier)
Lave dept (Simlar))
Forp, (Smbar)
peetr. (Bandom))
Last cippt (Randam))
Tong. (Randam)
2.31
1.93
1.45
1.47
1.50
3.02
5..0
ROBERTA BASE
ROBERTA-LARGE
2.36
2.12
1.34
1.20
3.24
1.28
0,01
3.01
ROBERTA-BASE
ROBERTA-LARGE
Figure 17: Above plots show the test perplexities (L) with different RoBERTa sizes and training orders. For reference, include the zero-shot and fine-tuning perplexities. The first panel shows the cumulative results, and the remaining two show the results obtained Wiki and S2ORC domains, Inside the parentheses are the perplexity improvements zero (the smaller the better).
Domain edopt. perplexity (4)
LLAMA2-7B
Domain adapt. perplexity (4)
GPT2-XLL XLARGE
122
302
:00
Domain size (MB)
Domain size (MB)
Figure 18: Our analysis of how much domain adaptation improves the perplexity for Llama2-7B and GPT2-XL. Each point corresponds to one L2 domain. The x-axis is the domain sizes and the y-axis is the perplexity after domain adaptation (normalized by zero-shot). Positive y values are the domains on which adaptive pretraining cause degradation in perplexity.
Additional similar order experiments As noted in the previous paragraph, we train GPT2-S and GPT2-M models on domain sequences that are ordered by semantic similarity. Note that the first domain sequence starts with "Culture" (as in main experiments) while choose to start with "astro-ph" in the second domain sequence.
20

---

Published in Transactions on Machine Learning Research (04/2025)
ackoward transfer domain #4 (Culture and the arts Sports and Recreation) [simlier
Backward transter to domain (physlps. wlw.sssssccsss........ ph) [random order]
Parplaitty (4)
Pepplrt (4)
123
100
Continual pre domains
Continual training
Zero shot
Continual pretraining
Future checkpoints
Zero shot
Continus pretreining
Future checkpoints
Backwarrd trensfer to domein N34 (g.bic NC) |similer urder)
ckagrd transfer to demain +34 (lechnology, and applied sciences Transporti (rendom
Perpertyy
20
Perplentyy
300
Continual pre-training
Continual pre-training domains
Zero shot
Continual pretraining
Future checkpolitss
Zero shot
Continua pretraining
Future checkpoints
Backward trensfer to domein n #50 (physics on) /similar order]
Bockward transfer to comain #50 (cs ARI (rancom order)
30.0
22.5
60
300
120
140
100
60
60
100
pre-triining domeirs
Continuel pre-trainigg domeirs Continus pretmining
Zern-hot
Continual postrairing
Future cnerapoints
Zerc-shof
Future checkpoints
Backward transfer to domain #100 (math.QA) OA) Esimilar order]
Backward transfer to domain #100 (Art) (random order]
16
Parplastty (1)
16-
perplesity
13
20
62
D0
300
120
140
100
20
60
80
100
120
140
140
Continuzi pre-training domains
Continual pre training domains
Zero shot
Continual pretraining
Future eeecppplnts
Zaro-shot
Continua pretraining
Future checkpoints
Figure 19: Backward transfer illustration with GPT2-L trained in similar and random order (left and right columns). Each panel shows the backward transfer perplexity (pink) computed on particular domain as optimization proceeds. For baseline comparisons, we also plot zero-shot (yellow) and continual pretraining (green) perplexities.
Cant. pretraining perplexity (+) (similar order)
Last checkpoint perplexity (4) (similan order)
SMALLSMALL
GPTZ- SMALL
GPT2- 6PT2-SMAL164
GPTZ-SMALLES
20
Contirual pre trainini domains
Continual pre training domains
Cant. pretraining perplexity (+) frandom arder)
Last checkpo int perplexity (1) (random order)
GPT2 GPT2-SMALL SMALL GPT2-SMALLGR
GPT2 SMALL
30
GPT2-SMALL64
Contirual pre-trathing domains
Contrnual pre training domains
Figure 20: A comparison of GPT2-S training with batch sizes 16 (our default) and 64. For random and similar training orders (rows), we plot the continual pretraining and last checkpoint perplexities (columns).
21

---

Published in Transactions on Machine Learning Research (04/2025)
pretraining perplexity (1) (similar arder)
checkpoint perplexity order)
GFT2 SMALL
GPT2 SMMLL GPTA-SMALL-BALANCE
GPTZ-SMAAL-BALLANC
53-
Migh M
Continual pre trainini domains
Continaal pre training domains
Figure 21: comparison of GPT2-S training with all available data (our default) as well as subsample of data with equally many data points per L2 domain. We only train in similar orders and plot the continual pretraining (left) and last checkpoint perplexities (right).
Cont. pretraining perplexity (1) GGPT2-SMALL
Continual pre-training perplexity (1) (GPT2-SMAL1)
SIMILAR
SIMILAR ORDER
10
0.8m
d'aining
Figure 22: A comparison of GPT2-S training with our default similar training order (Wiki portion, followed by S2ORC) as well as an alternative version (S2ORC portion, followed by Wiki). We plot the continual pretraining and last checkpoint perplexities. Note that the x axis corresponds to the default training order.
The GPT2-S model trained on the new similar order yielded similar continual pretraining (CPT) perplexity (20.25) to what reported in main experiments (19.38). Nonetheless, the perplexity of the last checkpoint (38.4) is much worse than the results in the main paper (24.78), which shows that the performance of the final model depends crucially on the domain sequence.
Due to time constraints, we managed to evaluate the first 80 checkpoints of the continually pretrained GPT2-M model. The findings obtained so far align with the takeaways above: CPT (12.51) is quantitatively close to CPT attained on the same domains our main experiments (13.92). The last checkpoint we evaluated (checkpoint-80) achieved a perplexity of 18.77 on all previous domains while we reported 19.33 in the main paper. We believe these numbers will go up train more domains. In particular, switching from S2ORC to Wiki portion arguably will cause significant degradation on the last checkpoint perplexity since the final model will be most recently trained on Wiki while S2ORC constitutes a larger portion of the test domains. We will include and discuss the results obtained after the completion of the evaluation in our camera-ready.
22

---

Published in Transactions on Machine Learning Research (04/2025)
Forward transfer to Culture (4)
Ferward trandfer to History (4)
Forward trandfer to Toch (J)
Forward transfer to Health (4)
22.-
20.
203332155425500 16
15132335165 Forward transfer PhifThink (4)
101112 124412233315
Forward bransfer Religian (1)
Forward transfer (e)erafaa(())
Forward transfer Art
70
15
1011 2512
12 10
Forward transfer Philosophy (1)
Forward transfer Bio
Formed transfer Physic
Forward trendler to CondMMtt
10-
3 11
3 1011
15
1011
Forward (1m(1)
transfer Math
Forward transter
Forward transfer Econ
7 10115111111111
1
30 1511116611
0.
0 15 131141555551
1011
transter (5)))
Forward transter Astro
GPT2-SMALL
GPTZ-MEDIUM
GPTZ-LARGE
GPT2-XLARGE
Culture
Hiztory 1ech
Heelll
selpiee
Pi min
gio
Phyeie
condion
Nin
Natt
Satt
Aabro
Ciiture
Histrry
Health
Phi Think
Art
Wiloosaphy
Ehyrie
condil Ni
Ast
do
Forward transfer to Culture (1)
Forward transle to History (1)
Forward transler to Teth (L)
Forward transfer to Health (L)
Forward transfer to Religion 11)
Forward transfer to Generalite
Forward transfer to PhilThink
Forward transter Art
16
14
Farward to Philosophy (1)
Foward to (1
Forward transter to Physics (4]
Forward transter Conditat (1]
18
3.25
234560 14
15
01234567 12 1637
Forward transter Ninn
Forward transfer Math (4)
Forward transfer Stat (1)
Forward wanseer (40
30
150
1.5
10
Forward transfer to (1)
Forward transfer Astro
225
220-
ROBERTA-BAAE BASE ROBERTA-LARGE
158
Catuue
Hictory
Raligin
Generalnet
mirmirkk
Philesophy
go
Prydac
Math
stat
CS
Adtro
Culture
Histery
Toch
Heazh
Philosophy
go
paysis
Conditin
Mach
stat
Ecnn
Figure 23: Forward transfer results with similar training order. The checkpoints are saved after having trained on an LI domain (hence 18 checkpoints per model). The y'th panel shows the forward performance on i'th domain, obtained by evaluating all previous checkpoints on that domain. The dashed lines show zero-shot performance. x and y axes correspond to L1 domain names and perplexities, respectively.
23

---

Published in Transactions on Machine Learning Research (04/2025)
trensfer (Culture_and, aand_theeeeeaeeeeeeeaaeeereedaee Sports Recreetion)
Urensfer fphysics plesssicc.leesppph (rendom
Paplesty (1)
Perpleaboy 143
Continual pre-training domains
Continual pre-training domains
Zero-shot
Co Continull protraliving
Future checkpoints
Zaro-shet
Continual prett aettaining
Future chocntpoints
Backward tronsfer domain (q-ble. NC) Jalmilar order]
Ickwward transfer domain Michnology and apolied sciences
Transport) Irandom
2.15
Perplesity (1!
Depleeitt
2.10
100
Continual pre-training
Continual
Zero shot
Continual pretraining
Future checkpoints
Zero shot
Continus pretraining
Future checkooints
Beckward transfer to domain #50 (physits edi ph) [similer order]
Deckweerd trensfer to domein #50 (cs. ARI (randorn order)
Perperty
Perplesry
2.35
2.6
80
120
300
Cont pro-training domains
Continual
Zoro shot
Continual pretraining
Future checkooints
Zaro Shot
Condinus pretraining
Future checkooints
Backward trensfer to domein #100 (math OAI [similer order]
Backward trasfer to domain #100 MAti (randem order)
Parplxity
Perplexty
36
20
100
120
100
300
1.0
pre-reiting yg Future cherkquints
pretreirng
Zera-hot
Continual pestrannnn
Jwto-shot
Conrinaa pretraining
Future check hecittinnts
Figure 24: Backward transfer illustration with RoBERTa-large trained in similar and random order (left and right columns). Each panel shows the backward transfer perplexity (pink) computed on particular domain. For baseline comparisons, we also plot zero-shot (yellow) and continual pretraining (black) perplexities.
24

---

Published in Transactions on Machine Learning Research (04/2025)
Table 2: Order of the domains our experiments
Setup
Domains
Similar order in our main exper- iments
"Culture and and_the_aatt _ Humanities",
"Culture _ndde___n_____ the__rteets_________________eee_ arts Games aaannes_and____ _oys"
"Culture_and_the __t_anss__rffoomn__aans "Culture and arts The arts, and Entertainment"
"Cul-
ture and the arts Sports and Recreation".
"Culture lis__nddddhb saii_tte__a__a tit__ats_____ _istar_tsssll_________________la "History_ Lennd_eeess_____ Batils____ooottnttili "History_ane _aw_adddets__B_________
"History_ane and ann__veetts________e Beeeett_____errrreeeeeee___ hawpoooooyaandapppiieed_eee "agiincs__rrruuttturrr" Fingineriinnnnneeninnn
By
"Tech-
nology_ande snlogy_and_applied secenddsseeeeeeeee_ee Sasoinnee__oommiinniiege
"Technology_ane _adla_and______ Latpaioe___ad__ppeee__________ applied_sciences sciences _etessiiceeeeaaneeee Transport",
"Health_ane Sanith__ad__itnessss__s Exercise",
"Health and fitnsss "ecalth_scienee,,,
"Health and fitness Human_ medicine",
_Nurrtton,,",""""
"Health_and_fine _andddssssss_...m Public_healthy
"Health_ad _nd fitmess fitness_Selff Self_care".
"Religion eacligion_and__o belief sasttssssssmss__.e "aatems_Allah,,
"Reli-
foon_and_bellie belicf_systems Ltt_sssess__iliieesseeeeeeeeee Belief__sstems",, "Religion_ane wwwlee_sysseteeeeeeeeeeeeeeeeeee Majos
Major _ajor_beliicfs sfrloss_o__rro t___th_wwarr tho_dddld",
"General referece Further research tools and topies". "General referece Reference works".. "Philos-
ophy_ and mond_thinkingg_e Philosshhhilllphh" "Philosophy sactssophy_and__aaeee thinking__Tinkinng" "Art", "Philosophy", "q-bio", "q-bio.BMM BM", "q=" "q-bio.GN", "q-bio.MN", "q-bio.NC". "q-bio.OT", "q-bio.PE", "q-bio.QM" "Q="io.SC", "q-bio.TO", "physics.ac--hh, "physics.oo-hh", "phhs"..pppph" "physics.ammclus", "physicssttoooo atoo- ph". "physics.boo-ph". pphysics.chem-phee "physics.class-phee pphysics.comp-phe "physics.data- dataaan".
"physics.ed-ph",, "physies.flu-dyn "physics.gen-ph", "physics.geo-ph., "physies.hist-ph,, "physics.ins-det,
"physics.man" med-ph", "physics.optics", "physies.plasm-ph" "physics.pop-ph"" "physics.soe ph"", "physies.space-pp "cond-mas mat. din.
"cond-mat.mttl-se"" "cond-mat.other". "cond-mat.quant-ge"
"cond- soaud-mat.ooe soft" "cond-mas
"cond-mat.str-ell" "cond-mat.supr-com "nlin.AO", "nlin.CD",
Cn", PS", "I", "math.AC", "math.AG", "math.AP", "m"th.AT". "math.CA", "math.CT "math.CV" "math.D"" "math.DS", "math.FA", "math.GM", "math.GN", "math.GR", "math.GT"; "math.HO", "math.lT": KT", "math.LO", "Gath.MG", "math.NA", "math.NT". "math.OA". "math "math.PR", "math. QA", "math.RA", "math.RT", "math.SG", "math.SP". "math.ST", "stat.AP", "stat.CO", "stat.ME",
"stat.ML", "stat.OT", "stat,TH", "eCon.EM", "econ.GN", "econ.TH". "cs.Al", "es.AR", "es.CC", "cs.CE", "cs. CG", "cs.CI., "cs.CR", "cs.CV", "cs.CY", "cs.DB", "cs.DC", "cs.DL,, "cs.DM", "cs.DS", "cs.ET", "cs.FL", "cs.GL", "cs.GR", "es.GT", "cs.HC", "cs.R", "cs.IT", "cs.LO", "cs.MA", "cs.MM", "cs.MS", "cs.NA", "cs.NE", "cs.Nf", "cs.OIf", "es.OS", "cs.PF", "cs.PL", "cs.RO", "cs.SC", "cs.SD", "cs.SE", "es.SI", "cs.SY", "astro-ph.CO", "astro-ph.ER "astro-ph.HE", "astro-ph.IM", "astro-ph.SR"
25

---

Published in Transactions on Machine Learning Research (04/2025)
Setup
Domains
Random order in main experi- ments
"Sath.DS", "cs.CC", "Mat "Natural and plysical_sciencce/EEate sciences/Earth sciences". "physics.clas ph". "physics.fludynn' "physics.ed-ph". "cs.RO", "q-bio.PE", "stat.TH", "math.DG", "q-bio.BM", "physies.space-hh., "stat.CO", "cs.PF", "math "physics.bio-ph,, "math.GT", "es.HC", "Mathe- matics and logic/Logic", "q-bio.GNN" "nlin.CG", "math.OC" "physics.aco-ph", "nlin.PS", "es.PL", motod-ma.mmrrl-ef""" "physics.gen-ph",, Cood--matsupp-coon" "math.LO", "physics.optics", "cs.SD", "math.NT", "math.QA" "Technology and applied sciences/Transport "cs.GL", "math.SG", "cs.SY", "Philasophy", "math.ST", "math.SP", "cs.MA", "cs.CG", "Technology_ane lacilllyyy__d__aaee aad_applied___ accplppie__cienns///grrreee sciences//grrcuulure" Am"th "physies.ins-det", "math.NA", "Technology_aue _ad cedd__pplie__eeee sciences/Engiiceringg,, "physics.ao-
ph". "cs.IT", "cs.AR", "General RefeeeceeRefereeccee works". "cs.DS", "physics.hist-hh". "math.IT",
"stat.ML", "q-'io.MNN" "astro-ph.cO", wrooddmat.dis-maae "math.RT". "Philosophy thinking/Thinking"
"cs.IR", "cs.GR", "Health and fitess//Hcatt__xcieeee"" "cs.ST", "q-bio. QM", "cs.ET", "cond-mat.quant gas", "physies.med-ph,, "stat.OT", "math.CA", "math.CT", "ccon.GN", "math.CV", "physies.atm-clus" "statAPP" "Rath.PR". "physies.data-an.. "nlin.CD", "Rath.GR", "physics.oop-ph., "cs.DM". "es.SC", "Health_ann sai__nd___e Baa__iinsss/Sel_eeemm ccaeff, "q-bio.NC", "cs.L.O,, "cs.CY", "econ,EM", "q-bio.OT", "physics.gen ph". "Culture and_ the arts/The arts and Entertainment", "cs.DB", "q-bio.TO", "cs.NI", "math.RA", "cs "cs.CV", "Hcalhhcath_aam min__iinsssEeerciee" "physies.atom-hh", "cs.OS", "cs.Al", "Art", "nlin.AO", "stat.ME", "Religion and belief sitliessyyttmm/llaaa" "cs.NE", "nlin.SI", "cs.CE", "Culture and tet_ttthettttitieeeeeeee arts//issaal arts". "Mathematics and logic/Fields of mathematics", "physics.comp-ph", 'math.GM", "astro-ph.HE", "cs.OH", "Health and fitness/Public health", "physics.app-ph". "Culture and the arts/Culture and Humanities", "History_ane events/By_regioe "cond-mattother", "Culture_and the the_arts/Mass, MasiMMaaaaada,",, "math.GN", ccond-mat.soff"" "Natural and Shhssical_sienncss/Pyyye
"physics.plaam-ph"" "astro- ph.SR". "cs.MM", "Aath.FA". "q-bio.CB", "cond-mat.stat-mechh "astro-ph.IM"', "Mathemat- dos_and_logic/Maathmattt "Gath "math.AP", "math.AT", "cs.NA", "cond-mat.str-ll,, "Technol-
agy_anddappill senddd_ppied___eeee wttpapieessiicnceSCooonneee sciences/Computtng"" "cs.GT", "Religion aaniiioonnnd______ and belief mediiiineemdd" "cond-mat.mes- wtt.ees/Humma_mmdiiinneeee "Health_and_ and_fiinessNutrrtionn,, "es.CR", "cs.MS", "Health_ande Beealth__nddftnness/Human
beliefs world",
fitness/Human_
hall", "cs.DL", "cs.FL", "Culture and the arts/Performing arts", "Natural and physical sciences/Nature"; "physies.chem-ph". "Natural and httppppiccaliciiceeeeeeeeeeeeeee ph.EP" "History_and sat__anddeeeet//iipp Baantssy_coottnent.... "Culture_and_ie _he IT""".KT"". "Philosophy_ane ann__ttinkinn//hhloopppp""
"q-bio.SC", "cs.CI., "Cs.DC", "mmth.HO", "astro-
sattr_aandttt__ttt/Sprrtttt aatasr/poorssaant uedd_RReccatton, "physics.soc-ph"
"Religion_ane _al__anaddded___aaeeeeeeeeeeeee_ systems/Bclief_sytteeee "History and Leeeeeeeeeee_eeeeeeeee _erridd__edd"... "cs.SE", "Genneral_effereee/FFutl research_ tools and and_topics". "Cul- ture the arts/Games and Toys", "q-bio", "econ. TH""
26

---

Published in Transactions on Machine Learning Research (04/2025)
Setup
Domains
Alternative ran- dom order used in ablation
"cs.CR", "math.MG", "Mathematics, and log_i Logic", "cs.SI", "cs.HC", moond-mat.mtrl-cci "cond- mat-supr-con", "math.AP", "physies.med-phi" "nlin.CD", "Culture and the the_arts Culture and Humanities", "q-bio.CB", "cs.GL", "cs.RO",
"math.GN", "Philosophy_ande irr___nddttiinnnneg _hinkinng_Thinkinnn"" "stat.OT", "cs.AR", "q-bio.OT", "math.RT" "physics.ed-ph", "q-bio.GNN, "cond-mat mest mat.mes-hall", "cs.Al", "cs.PF", "econ.TH", "cs.CC", "q-bio.NC". "nlin.SI", "stat.ML", "Health mnd_fitness. Self_care", "physics.data-an... "physies.fu-dyn", "Culture and the arts The arts and Entertainment", "cs.ET", "physies.soc-ph., "cs.NA", cond-mmat.uanttgaas"" "Health_ama fitness fitmess_Health fhealth_science", "cs.SY", "math.AG", "cs.CG", "cs.CE", "math.KT". "Health_ann fitness Human_maan medicine", "math.AT", "math.DG", "General referece Further research tools and topics". "econ.EM", "math. "cs.DL", "Health and fitmess Exercise", "physics.plasm-ph" "Technology and a sciences Computing". "cs.CI." geeneral_refereee Reference works", "math.SG", "physies.atm-cls". "astro-ph.EP", "astro- ph.CO", "math.OC", "math.LO", "physics.gco-pe" "math.GR", "physies.gen-ph". "Philosophy", "physies.boo-ph". "Health and fitness Nutrition". "Technology and applied sciences Engineering", "m"th.NT", "cs.MM", "physics.ao-ph". ture and the arts Performeen arts", , "cs.GR", iphysics.atom-phi ""omd-mat.dis-an", "stat.AP", "cs.CY", "cs.FL" "Mathematics_ latisssannddd_d_____ee Mathematiis",, "math.IT"' paaturalandd_hhsicaa___ee Beatraa__ciiincs__iBiggege "cs.DM", "cs.PL", "Mathematics and logic __gie__oglllllell____lll_l__ee_e tory_and_evve _yy_and_eeents___ By_ continent", "Culture and the arts _ arts", "es.GT", "cs.IT", "math.NA", "q="io.TO", "math.PR". "nlin.PS", "es.CV", "cond-mat.streel", "econ.GN". "physics.comp-ph"" "math.RA", "math.CV", "q-bio", "q-bio.QM". "Culture_an _n____nd____ _at__tthe_r__________e_eeeess_s_ "as_MMMasiedaa" "math.QA", "Natural_and" pod__ppicaa_scinneeee Baciioss__hssslllssssee sciences". "Health_and Saalth_and_fitnesssss Lwll__uulllllllllllllleeeee "es.DD", "astro-ph.HHE", "cs.MS", "Natural and physical sciences Nature", "cs.DC", "Math.GM", "physics.pop- ph",
Thinking",
coaltur_aan__iee t_t__rte_______________ arts foiss_Gmms_amna and_Toys", "Cul-
Fields of mathemaatio"". "stat.ME", "q-bio.MNN. "His- _he_arts_______
mond-mma.stat-meede "q-bio.SC", "cs.OS", "Cul-
ture ture_and and tit__the_ss__________ _arartts___ tttss__ports__a_i caatissand__eeeeeee Let__anddeerttoon... "m", "math.CT". "History sarllaand______ and_events By By_period", the "cs.SC", "math.ST"; "physics.spacc-pht "cs.SE", Imath.DS", Tocchollgy_andd_ppppp scconncs__ccccss_e_eee "ransss""o""""""""""""""""" "m"th "physics.ins-de". "cond-mat.soff". "physics.hist-ph" "math.HO", "math.GT". "math.SP", "Art", "cs.NE", OA", "cs.SD", "Religion _eaei_anndddd______eeee_e___e beliee_eestteeeeeeeeeeeeeeeee_ee systems Major_beliefs______ belieee_llfffeeeeeeeeeeeeeeeeees of whrldd"world", "cs.OH", "astro- "physics.opties", "physics.app-hh", "Natu-
"astro-ph.SR", "cs.DB", TTechnology_and.1 and_applied__ _pppaicc_ssciiiccss_____ vociences__griculleeee Ih. " ral and ohd_icas ttlsspallssieeneeeeeeeeeeee sciences Earth sciences". "physics.aco-ph". "stat.CO", "cs.RR". "nlin.CG", "cs.MA", "cs.LO", "q-bio.BM", "q-bio.PE", "stat.TH', "physics.clas--h",, "physics.chem-ph", "His- iory_and_events, By__eion__ "Religion_and and_belief _ystefe_sssssemmms_____ Belicf systems",
Bysicc_systems"" "cs.NI", "nlin.AO"
27

---

Published in Transactions on Machine Learning Research (04/2025)
Setup
Domains
Alternative random order-2 used in ablation
"cs.SD", "q-bio.QM"" "physics.aco-ph", "cs.DS" "Art", "econ.TH", "math.CA", "math.SP", "Technol- ogy_and scienie
sciences Transport", "cs.MS", "physics.hist-ph" "Natural and phlssphlysssssheeee sciences Nature", "stat.OT", "math.AG"" "cs.GL", "Technology and aad_aadd__ppil applied_scieneeen Compptinng_
Ceeneral_refereoe_ Lateraarefrrre_Pheeeeeeeeeeeee _usrtherreeearchhheeeeeeeeeeetae sieearch_toossaad_aoooe
"math.LO", "math.KT". "math.OA". "Cul-
are_and_the aod_the_artt Liss_MMassmmdddd "math.FA", "math.PR", "cs.CG", "physics.ed-ph", "cond- mat.dis-nn", "cs.SE", "Health and fitness Public health", "Mathematics and logic Logic", "Philasophy", "Mathematies_and M Mathematics", "physics.gen-ph", "stat.ME", "His- jory_and_event, contineennn"" "math.CT" "physics.comp con". "Culture and arts Performing _ "physics.class-ph". "physics.app-ph". "q-bio", "Cul- ture and the arts Visual arts", "Culture and the arts Culture and Humanities", "cs.CE", "Cul- ture uure_and_a the arts ctiss_Spors__as _ndd_ Recreation", "nlin.SI", "Mathematics, and larrriiiis__e_iiiii____ _og_llogi___lddllllel Fields of mathemaaics"", Shhysics.eo--ph "physics.pop-ph". "History_ann eenoerttandeeeetts _ventss____reiioooe By_region",
"cond-mat.other" "cond-mat.supp
"cs. OH", Gxond-mat.mtrl-soe "astro-ph.EP", "cs.DL", "cs.DM". "cond-mat.stall natttmma.stattmcch.. "nlin.CD", "math. "math.MG", "cs.GR" "cs.CV", "cs.PI., "cs.CC", "math.NA", "math.QA", "cs.Al", "cs.CI.", "physies.space-phie "stat.AP", "q-hio.SC"" "Health and fitness Health science", "cs.MA", "cs.CY", "cs.ET", "q-bio.CB", "q-bio.GN", "physics.ao-ph", "q-bio.BM"", "math.AC", "cs.DB", "cs.DC", "math.HO", "O", "cs.NE", "math. "q-bio.PE", "econ.GN", "physics.atm-clus"" "cs.IT"" "Culture and the arts The_eheaae arts_ _nd Entertainment". "physies.chem-pphe "cs.MM", "cs.HC", "Philosophy, and_thinking Philosophy", "Health and fitness Human "ouman_medicine" "General referece Reference Reference_worrs"" "math.RA", "astro-ph.SR", "astro-ph.IM". "physics.boophi", "econ.EM", "cs.SC", "math.OC", "physies.ooc-ph" "stat.MI.", "cs.AR", "physics.med-ph", "cs.FI.', "Natu- ral ral_and_ physical_sciencess Seath_sciennes", "Religion_and besteesssssstessss..mmeees Alllah,, "q-bio,TO", "math.SG", "Technology and applied sciences Agriculture",
"Religion and belief_systems systems systems... "math.ST", "oond-mat.uaant-gas". "math.CV", "physics.atom-phh,, "physics.plasm-phee "Philoso- phy_and _ond_tthiiing__.. thinking_TThinknng"" Thinking", "nlin.CG", "es.GT", "physics.data-an", "cond-mat.mes-hall"," "cs.SY", "math.NT", "cond-mat.soff soft", ine "Natural_and phhsssndddphs_l_sss_ seccicclsciences____ "iannccs_BBllogy"" "Tech- nology and applied sciences Engineering". "physies.flu-dyn", "Fs.PF" "math GR". "q-bio.MNN, "cs.RO", "q-bio.NC", "cs.CR", shealth_and_ fitness fitneessSSel_eeeeeteeee "cs.LO", "cs.SI", "q-bio.OT", "astro- ph. "Religion_ani wand_belief_sseeeeeeeeeeeeeeeeee belief_systems. MMjasr_aao__________________ selleff_oo_thh_weeee
"cs.RR" "cs.NI", "S"th.DS", "astro-ph.CO", "History_ septory_anddeveee and__vents____teeeeeeeeeees____ _B___Brydd_"""""___ "cs.OS", "Culture_and_1 sand_the_arts... Lat_tttt_att__aams_____ "aa__aaamee__daaaaeaa Toys", ATTth tet "Math.GN", "Math.GM", "physics.optics,, "Health_an and fitness Nutrition". "S", "math.RT", "nlin.AO", "cs.NA", "math.AP", "math.GT", "Health and fitness Exercise",
"Natural aatural_and__aee pad_pchyscaal__ phhsiccl_scienseeee secsssccceeeeeeeeeeee Ptwsseaal_sciemeeeeeee sciences", "stat.TH"
28

---

Published in Transactions on Machine Learning Research (04/2025)
Setup
Domains
Alternative similar order (GPT2-S) used in our ablations
"Culture and_the_____ arts Games and Toys".
"Culture and the arts Culture and Humanities",
"Culture and the arts Mass media", gur_andd_thr tht_ttth_____________eere__ Lrot__SSotts__d___eee "Culture and the arts Visual arts", ral_reeerece Cenffrece__Futthrrreeeeeeee
"Culture and the arts Performing arts",
"Cul-
"Culture_ani satrrr__nn_thhe_e_e and__heearrss TaeeTheet__aaaee Entertainment". "History_ _nd events By_continent".
"Gen-
poool_and___eo and_topies". (General_reforcee__Re Roferencee_orks". works". "Art",
"Philosophy".. "Philosophy and thinking Philosophy". "Philosophy and thinking Thinking", "Religion_ande _ew_ieefssstemsss_________
"Religion_ame Siligiio_and_hellef___e_ _er__sssss_s_leef__eeeeee
"Reli-
and_events___y By_period"', "His- _ay__an__eeetts events By_region", "Health and fitness Exercise", "Health and fitness Health science", "Health and fitness Human medicine".
gion_and aau__and_beliff_____ www___eeeleeeeeeeeeeeeeeeeeeeeee systems, Major_belieff tftis_oo__eeet_eeeeeeee _f_th"_wrldd",
tory_am
"Health and fitness Nutrition", "Health and fitness SSel__care hatplooy___ad_pppieedd_eee "esiintes_Computinng nology and applied sciences Engineering", "Technology and applied sciences Transport", "cs.Al", "cs.AR", , "cs.CC", "cs.CE", "cs.CG", "cs.CL", "cs.CR", "cs.CV", "cs.CY", "cs.DB", "cs.DC", "cs.DI., "cs.DM", "cs.DS", "cs.ET", "cs.FL", "cs.GL", "cs.GR", "cs.GT", "cs.HC", "cs.R"', "cs.IT" "cs.LO", "cs.MA,,, "cs.MM", "cs.MS", "cs.NA", "cs.NE", "cs.NI", "cs.OH", "cs.OS", "cs,P",, "cs.RO", "cs.SD",, "cs.SE", "cs.SI" "cs.SY", "statAP", "stat.CO", "stat.ME". "stat.ML", "stat.OT", "stat,TH", "econ.EM". "econ.GN" "econ.TH", "math.AC", "math.AG", "math.AP", "math.AT" "math "math. "math.CV". "math.DG". "math.DS", "math.FA", "math. GM". "math, GN", "math.GR "math. "math.11", "math.lT". "math.KT" "math "math.MG", "math.NA", "math.NT", "Aath.OA", "mathhh"" "math.OC", "math.PR", "math.QA",
"Health _nd__atndss_____________________ fitmess Publec Public_health",, sedd__pplied_siieee AAriicttturr,,
"Technol-
"Tech-
"math. RA", "math.RT", "math.SG", "math.SP", "math.ST", "physics.acc-ph". "physics.ao "physics.app-
ph". pphysics.atm-cluss "physics.atom-ph" "physics.bio-ph", "physics.chem-ph". "physics.classphh,,
"physics.comp-hh" "physics.dataa-ne "physicx.ed-ph", "physics.fluddy"" "physicx.gen-hh., "physiex.eoo-ph",
histtp" "physsicsins-det", "physics.med-phh,, "physics.optics", "paysiix.plasmpph".. "physicx.oopph".,
"physics.soc-phr "physics.space-ph" "q-bio", "q-bio.BBM", "q-bio.CB", "q-bio. GN", "q-bio.MN", "q=bio.NNC"
"q-bio.OT", "q"" "q-bio. QM", "q-bio.SC", "q-bio.TO", "cond-mat.dis
wrond-mat.mtrl-scee "oone-matoother", "cond-mattqqant-gaa"" "comd-mat.softt.
"cond-mat.str-cll "cond- monddmmt.suuprcoo" "nlin.AO", "nlin.CD", "nlin.CG", "n"in.PS", "nlin.SI", "astro "astro-ph.CO", "astro-ph.EPP EP", bustro-ph.HE", "astro-ph.IM", "astro
29

---

Published in Transactions on Machine Learning Research (04/2025)
Setup
Domains
Alternative similar order GPT2-M used in our ablations
"astro-ph.CO", Destro-ph.EIP., iistro-ph.HEE, "astro-ph.IM", "astro "astro-ph.SR", "physics.acc-ph"., "physics.ao-
ph". "physics.app-ph.. "physies.alm-clus.,, "physies.atom-phhs "physics.bio-ph". "physics.chem-ph",
"physics.clase" class-ph", "physics.comp-ph",, "physies.data-an" "physics.ed-ph", "physies.fludyn", "physics.gen gen-pe""
prlysics..eo-pht "physssssssshhhsttppp"" fanysicc.inssder'"s "physics.mmd-ph"." "physics.opties,, "physics.plasm
"physics.pop-ph' "physics.soc "physics.space-pi" "stat.AP", "stat.CO", "stat.ME", "Ltat. "stat.OT",
"HH",, EM" "econ.GN". "econ.TII" "cond-mat.dis--r""
"cond-mal.mtrl-
sci", "cond-mat.other"" "cond-matt.uaattgaae "cond-mat.soff""
"comd-mat.str-el",
mutpdd--tttupp-com "math.AC", "math.AG", "math.AP", "math.AT", "math.CA" "math.CT", "math.CV", "math.DG "math. DS", "math.FA", "math. "math.GM", "math. GN" "m"th.GR", "math.GT", "math.HO", "math.IT", "math.KT". "math. "math.MG", "math.NA", "math.NT", "math. OA", "math. OC", "math.PR", "math. "math.RA" "math.RT", "math.SG", "math.SP"; "math.ST", "nlin.AO", "nlin. "nlin.CD", "nlin.CG", "nlin.PS", SI", "es.Al", "es.AR", "es.CC", "cs.CE", "cs.CG", "cs.CL", "cs.CR", "cs.CV", "cs.CY", "cs.DB", "cs.DC", "cs.DL", "cs.DM", "cs.DS", "cs.ET", "cs.FL", "cs.GL", "cs.GR", "cs.GT", "cs.HC", "cs.IR", "cs.LL,, "cs.MA", "cs.MM", "cs.MS", "cs.NA", "cs.NE", "cs.NI", "cs.OH", "cs.OS", "cs.PF", "cs,PI", "cs.RO", "cs.SC", "cs.SD", "cs.SE", "cs.SI", "cs.SY", "q-bio". "q-bio.BM", "q-bio.CB", "q-bio.GN", "q-bio.MN", "q-""""".""-.O""""""" "q-bio. "q=boo.."""""""" "q-bio.SC", "q-bio.TO", "Health_ame and__itness_____. _ii_fisss__xeerisee"" Exercise", "Health_an fitness Health _ealth_science", "Health_ and fitness NNtrrtionn" "Health and fitness Public_health", "Health fiinesseess__.. fitness__eel_ceeee Self__are"" care", "Culture and the arts Games and Toys". "Culture_and" the_arts_______eeeeeeeettettte ____arrttsssss_ Mass_media", ture gur_aan__hee ttt_tthh_rttt_ Sports_arttt RecreRReee...n. "Culture and the arts Visual arts", "Art", (Geeneralreefeeeee veferece__Refereeeeee Rewereeeeeeeeeeeeeeeeeeeeeeeeeee wworkk"" losophy_amn _nd thinking Thinking".
"Health and fitmess. Human Human__meiiane medicine".
"Culture and the arts Culture and Humanities", "Culture_ande _nd___d________ and_the___ the_arts___ inrforminggarts"
media",
"Cul-
"Culture_and" hht__anndd___rrtte_e Thhe_arts_Thh_arree The_arts_and Entertainment" "General referece Further research tools and topics",
"Phi-
"Philosophy".
"History_and_evee aaaaaadddeeeeeeeeeeeeeeeeeeaaeee By _y__contieet",
"History_and" aedry__and_eedss___e_eeeeeea_ese BB__________dd"""""
History_andd_evee Byveeetts___rrrgiinno"
"Reli-
gion and belief systems Allah".
"Religion and belief systems Belief systems",
"Reli-
gion_and hai___n__eeieee_eeeee Major beliefs of the the_woldd", "oidd"""" "Technology_ann lacplllogy__n__p_lllle land_apliedd_sieeee Lacences_Agriuluuree Caoosssoompp.n".
cecehollgy_andd_aplieedddcc secliedd__eeeeesssee Engincering",
Transport"
A.5 Learning rate selection
In our experiments, we used fixed learning rate of 5e-5. We acknowledge that learning rate selection plays a particularly critical role continual pretraining, where subtle changes optimization dynamics can significantly affect both knowledge acquisition and forgetting. In our experiments:
For GPT2 models, consistently observed that both training and test perplexities decreased, especially in larger domains (>50MB). Higher learning rates sometimes helped smaller domains converge faster, though not always consistently. However, we found that training loss behavior on the current domain did not reliably predict test perplexity behavior on other domains. In other words, sometimes rapid convergence on one domain led to performance degradation on others, and vice versa. These observations suggest that there is no universal or robust learning rate recipe for continual pretraining across heterogeneous domains.
For Llama2-7B, learning rate sensitivity was even more pronounced. Smaller learning rates often resulted in almost no model updates, while larger learning rates led to both larger improvements and larger degradations, depending on the domain. We settled on 5e-5 as balance: large enough allow meaningful adaptation, but not so large as to destabilize training. Still, we found that the rate and extent of perplexity change varied significantly across domains, reinforcing the difficulty of defining one-size-fits-- schedule.
30


Reviews:
[{"review_id": "9yEfqa1es8", "reviewer": "Reviewer_QXyD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper provided a detailed analysis of continual pretraining across a broad range of domains using the M2D2 dataset, offering a large-scale benchmark for LLM adaptability.\n2. Findings on different decoder-only models (GPT2-S, GPT2-M, GPT2-L, GPT2-XL, Llama2-7B) and different encoder-decoder models (RoBERTa-base and RoBBERTa-large) provide useful insights into improving continual learning strategies.\n3. The use of different and diverse evaluation metrics, including perplexity, forgetting, forward and backward transfer, and downstream performance, provides a comprehensive assessment of CL in LLMs.\n\nWeaknesses:\n1. Some key findings in this paper are not first proposed, such as “larger models always achieve better perplexity than smaller ones when continually pretrained on the same corpus”. Additionally, some findings may be too specific for individual model architectures, like Llama2-7B, and lack broader generalizability.\n2. The principle of designing the similar order is not clear to me. Moreover, this paper does not sufficiently clarify how tasks are positioned in different random-order settings. Besides, the authors mentioned in the limitations section that “in similar-order training, domains were ordered only once, meaning our findings may not generalize to other orderings”, but exploring different domain orders is crucial in continual learning, if similar-order only uses only a single specific order, the findings in similar-order may limit the generalizability.\n3. In the experiments, the authors primarily focus on comparing different models with the proposed random-order and similar-order, but I did not find the comparison with some baselines of other domain-adaptive pretraining approaches. Including some baselines would help contextualize the efficacy of the proposed benchmark more comprehensively.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the impact of continual pretraining in large language models (LLMs), distinguishing itself from existing works that primarily focus on continual fine-tuning. The study introduces a new benchmark to measure the adaptability of LLMs to dynamic pretraining data landscapes. By systematically analyzing continual pretraining dynamics, the paper examines knowledge retention, forward and backward transfer, and forgetting across various models. This paper leverages multiple evaluation metrics to provide insights into the efficacy of CL in LLMs. Moreover, it explores the impact of model architecture (decoder-only vs. encoder-decoder), domain ordering strategies, and model size on continual pretraining outcomes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Could authors please add tables to describe dataset arrangements in similar-order and random-orders? For example, in similar-order 1: 1. Computer Science -> 2. Physics -> 3.... Providing such details would improve clarity regarding task sequencing.\n2. Could authors please add more explanations on how to determine the ordering of the training domains based on their similarity? How is the first task selected? Additionally, how many different random-orders are explored in this paper?\n3. About Section 4.6, could authors explain why this analysis is relevant to continual learning insights? Additionally, how do the results of rank-based analysis support the findings on knowledge retention and transfer? This subsection lacks a clear explanation of its relevance to the paper, making it difficult to understand its significance within the broader context of the study.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6ajR0eMVZt", "reviewer": "Reviewer_F7Li", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper definitely explores an interesting problem from an experimental perspective. \n- The conclusions, such as for example those associated with model size or with different architectures, are potentially very useful for researchers in the field. \n- The paper is well-written and well structured. The experiments are wide enough for the type of conclusions made thorough the paper.\n \nWeaknesses:\n- As acknowledged in the conclusion, experiments were run for a single task order. This definitely impacts the results, as different tasks orders will affect forgetting, backward and forward transfer differently. Without this, it is hard to consider the results generalizable.\n- Related to the previous point, the \"dissimilar\" task order was ignored. \n- The writing of the metrics in section 3 is a bit confusing, as there are many concepts (i.e. metrics) explained at once. I believe this could be better structured. Also, later on in the experiments, connecting the things that are actually being measured to the definition of these metrics in section 3 is hard, since no acronyms are used in figures for example.  \n- Finally, it is also mentioned that experiments were limited to domains below 5GB. Why not simply subsampling these large domains to cover more variety?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an experimental study of continually pre-training a set of tasks. Experiments are run on benchmark datasets involving a large number of tasks and several domains. Several pre-training architectures are tested. This experimental paper provides conclusions around the behavior of continual pre-training under different architectures, for different tasks, for different model sizes. It investigates the roles of forgetting, backward transfer and forward transfer.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to weaknesses. I would like to see multiple task orders included in the experiments, and also (subsampled) datasets beyond 5GB. Also, please fix the explanation of metrics and how you use these to refer to results in the experiments.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2akU8e87vl", "reviewer": "Reviewer_XJRM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. Continual learning in LM pretraining is an important research area, and how we can inject additional knowledge into those models, while at the same time minimizing catastrophic forgetting, remains an important open research question. This paper makes progress towards answering some open questions in this space through more comprehensive continual LM pretraining benchmarking than what has been done in prior work.\n\n2. Some of the insights from the paper, particularly relating to (i) the best continual pretraining curriculum to maximize forward transfer (e.g. that presenting the domains in a random order is very competitive in terms of forward transfer and final performance, and should therefore be the default option); (ii) the minimum amount of data to successfully fine-tune different-sized models (and how some domains are too small to fine-tune the 7B-parameter Llama 2 model); and the fact that (iii) smaller LMs exhibit the most continual learning (i.e. the acquisition of new knowledge, as indicated by the largest performance improvement relative to the zero-shot baseline), while at the same time also exhibiting the most forgetting), would be useful for the broader community.\n\n3. The paper is overall well-written and very easy to follow.\n\n# Weaknesses\n\n1. Some of the findings are either well-known or not particularly novel, which would not be very useful for the community. For example, the fact that larger models perform better in terms of final performance (compared to smaller models, Section 4.2) is already well-known for a few years (Kaplan et al., 2021) and demonstrated many times over at this point. Furthermore, the fact that perplexity correlates very well with downstream task on related domains (Section 4.5) is also not particularly surprising, and is already well-established in the field. Lastly, the fact that positive forward transfer in similar training order is possible to only semantically related domains (Section 4.4) is also very intuitive and not surprising (basically under this semantically-related continual pretraining curriculum, the model can specialize hard on multiple new domains that are semantically very related to each other, which means that they also exhibit more forgetting & less transfer to older / unrelated domains).\n\n2. The paper is missing what in my opinion is an important baseline: The case with no continual learning at all, where the 236 new domain data are simply mixed together, and concatenated with the pretraining data. This setting can indicate the **upper bound on model performance**, where the model does not have to learn new domains in a sequential fashion, and can also potentially do multiple epochs over the new data. Basically: How much do we lose in the continual pretraining setup, as opposed to the case where we can just mix in all the new data together without having to do continual learning in a sequential fashion?\n\n3. I have concerns with the notation in page 4, Section 3. My main concern is that the FG / forgetting metric, which is defined as the median of $f_2, \\cdots, f_N$, crucially **overloads** the notation for $f$, which is already used for $f_n$ above it, which \"denotes the zero-shot and domain adaptive pretraining perplexities computed on the $n$-th domain\". This means that the symbol $f$ now has two overloaded meanings, which includes the $f_c$ definition used for computing FG / forgetting.\n\n4. Section 4.6, which computes the prediction rank-based analysis for knowledge accumulation, is done using the Llama2-7B model. This choice sounds questionable to me, because the Llama2-7B model does not seem to learn much at all from the continual pretraining stage, which worsens its performance by a big margin. I would suggest redoing these experiments with one of the GPT2 model sizes instead.\n\n5. The continual pretraining results with Llama2-7B seem to be quite bad for the vast majority of domains. Could this be a learning rate / batch size issue? In other words, how much hyper-parameter tuning is done to find a good set of continual pretraining parameters for Llama2 7B? This is particularly surprising given the fact that training perplexity always improved (Page 5, Section 4.1). Are the authors saying that this is an overfitting issue?\n\n6. There seems to be a pretty heavy reliance on the Appendix, where some Figures that are pretty central to e.g. Section 4.5 are in the Appendix (e.g. Figures 11 & 12). It'd be nice to have all the important figures in the main paper, which would make the paper more self-contained.\n\n# Questions & Suggestions & Comments\n1. For the DAPT baseline (page 4, Section 3), does it start from a model that has been pretrained? Or does it always pretrain on each domain from scratch?\n\n2. On page 6, there is a mention to \"... similar-order training (Figure 5, **right** ) ...\". This could be a typo, because the similar order training is in Figure 5, **left**.\n\n3. Pedantic comment: The model sizes explored in this work are pretty small (up to 7B). I'm not holding this against the paper because running larger LMs are very expensive, but I am not sure if they qualify as LLMs, which stand for **large** language models.\n\n4. On page 6, it is mentioned that \"increasing model size could be a way to alleviate forgetting...\". But at the same time, larger models also don't acquire as much new knowledge in the continual pretraining stage as smaller models (though they still perform better in terms of final accuracy). Given that learning & forgetting are intertwined, I'm not sure how much the recommendation to use larger models is a good conclusion.\n\n5. Typo: The first sentence of Section 4.3 is lacking a verb (the subject is \"Figures 3-5 ...\" but there is no corresponding verb, maybe something like \"showcase / demonstrate\" or something like that?)\n\n6. For figure 9, mention that \"higher is better\" or something like that to contrast with the perplexity evaluations, where lower is better.\n\n7. For the batch size experiments (page 10), are the learning rates adjusted accordingly (i.e. increased learning rate when the batch size is increased and we do fewer updates?).\n\n8. Related work: Mind the Gap: Assessing Temporal Generalization in Neural Language Models (Lazaridou et al., 2021), which also proposed a benchmark for continual learning of LMs.\n\n# Big Picture Suggestions\nGiven that some of the findings are already known / less novel, I would suggest exploring two additional research directions to increase the impact of the paper:\n1. Parameter averaging has been shown as a cheap way to do model combination, which would work here because we start from the very same base model. What happens if you average the parameters after training on each domain? For example, if there are $M$ domains and after doing continual pretraining on each domain, we have $M$ saved checkpoints, we can simply average these checkpoints in the parameter space. My guess is that this would mitigate the forgetting issue.\n\n2. One thing that would be very useful is training a model to predict: How much improvement & forward / backward transfer will there be from doing continual pretraining on a particular domain? This can tell us which data will bring the most value if we train on them, while at the same time allowing us to focus our efforts on identifying data that will be most useful for improving performance on a given dataset / domain.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new benchmark to assess the ability of LLMs to continually learn. Concretely, this paper focuses on the **continual domain-adaptive pretraining case**, where a base LLM is continually pretrained further on multiple new domains in a sequential fashion. To that end, the paper explores continual domain-adaptive pretraining on three models: The GPT-2 family of models (of various sizes), Llama2 (with 7 billion parameters), and also RoBERTa, with an emphasis on measuring both forward (i.e. generalization to new domains) & backward transfer (i.e. model forgetting).\n\nThrough experiments on various model sizes and 236 domains, the paper finds the following insights:\n- Continual pretraining helps different-sized models to different extents, whereby smaller models (<1.5B) acquire the most additional knowledge, albeit also suffer the most from forgetting;\n- Overall, larger models still achieve the best perplexity, even though the improvements (and forgetting) from continual pretraining tend to be less substantial than the improvements for smaller models;\n- Continual learning performance, as measured by language modeling perplexity, is indicative of downstream task performance on related domains;\n- The **curriculum** under which continual pretraining is done affects both transfer & final performance. To that end, conducting continual pretraining where new domains are added in a *random* fashion enables positive transfer and reduces forgetting, whereas presenting the new domains based on their similarity to previous domains improves performance more on related domains, albeit at the expense of less domain transfer to unrelated domains, stronger forgetting, and sometimes worse final performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Critical**: Address the first weakness, which is that some of the findings are either well-known, obvious, or not particularly novel.\n\n2. **Critical**: Add a baseline where we don't have to do continual pretraining in a sequential fashion, and can simply mix in all the new domain data and then train on it in an i.i.d. fashion (second weakness).\n\n3. **Critical**: Fix the overloaded notation (third weakness).\n\n4. **Critical**: Redo the experiment on Section 4.6 with one of the GPT2 models (rather than Llama2-7B which does much worse due to continual pretraining).\n\n5. **Recommended**: Dive deeper on why the Llama2-7B continual pretraining performance is bad, for example by using other hyper-parameters, etc.\n\n6. **Recommended**: Bring some of the Appendix figures to the main paper to make it more self-contained.\n\n7. **Recommended**: Address the questions & the big picture research suggestions.", "broader_impact_concerns": "No broader impact concerns come to mind.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LBfkXGtcsD", "reviewer": "Reviewer_dnFi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nComprehensive Empirical Study: The paper's primary strength lies in its extensive empirical evaluation. The authors meticulously conduct experiments across a wide range of models, tasks, and training configurations, providing a wealth of data to support their claims.\n\nNovel Benchmark: The introduction of a new benchmark based on the M2D2 dataset is a valuable contribution. This benchmark addresses limitations of existing continual learning benchmarks by operating at a more realistic scale for LLMs and encompassing a diverse set of domains.\n\nKey Insights: The paper identifies and articulates several valuable insights about continual pretraining. These insights are well-supported by the experimental results and provide practical guidance for researchers and practitioners in the field.\n\nClear Presentation: The paper is generally well-written and easy to follow. The experimental setup, evaluation metrics, and results are clearly presented, making the paper accessible to a broad audience.\n\n\nWeaknesses:\n\nLimited Background on Domain-Adaptive Pretraining: While the paper focuses on continual domain-adaptive pretraining, it doesn't explicitly define or describe the specific techniques used for domain-adaptive pretraining itself. Some readers may benefit from a brief overview of common domain adaptation methods employed during pretraining. \n\nLack of Discussion on Learning Rate Selection: The learning rate is a critical hyperparameter in LLM training, especially in continual learning scenarios where the model is adapting to new domains. The paper mentions automatic learning rate selection via DeepSpeed but doesn't provide details about the chosen learning rate range -- this is crucial for having the results put on a fair comparison perspective.. A brief discussion of learning rate choices and their potential impact on learning and forgetting would be valuable.\n\n-----------------\nOverall I agree with the takeaways of the paper and think it is of value to the community to know these takeaways.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive empirical study of continual domain-adaptive pretraining for large language models (LLMs). The authors introduce a new benchmark based on the M2D2 dataset, enabling a more realistic evaluation of knowledge retention and transfer across diverse domains than existing continual learning benchmarks. Through extensive experiments across various model sizes and architectures (GPT-2 family, Llama2-7B, RoBERTa), they uncover several key insights about continual pretraining, including its effectiveness compared to domain adaptation, the impact of model size, the influence of training domain order, and the resulting downstream task performance. The paper provides a valuable characterization of the challenges and opportunities associated with continual pretraining in LLMs, contributing significantly to the understanding of this important area.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper would benefit from improving the training details especially the meaning of what each of the compared \"baselines\" mean in implementation along with the final learning rates. \n\nThe pretrained model's learning rate and schedules affect the continual pre-training results a lot and would be helpful to keep in perspective when explaining things to the readers.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a thorough empirical study on continual domain-adaptive pretraining for language models, supported by a newly introduced benchmark and a wide range of experiments. Reviewers highlighted the depth and breadth of analysis, the value of the insights, and the clarity of presentation. The authors responded constructively to all critical comments and improved the paper accordingly. While some findings confirm expected behavior (e.g. larger models forget less), the contribution remains valuable for establishing best practices and a baseline in a relatively underexplored area. 
Recommend the authors implement the minor revisions suggested during the review process.
Decision: Accept (poster)