Overtrained Language Models Are Harder to Fine-Tune
Authors: Jacob Mitchell Springer
Sachin Goyal
Kaiyue Wen
Tanishq Kumar
Xiang Yue
Sadhika Malladi
Graham Neubig
Aditi Raghunathan
Venue: ICML 2025 poster (2025)
Keywords: pretraining
finetuning
catastrophic forgetting
transfer learning
Primary area: deep_learning->large_language_models

Abstract: Large language models are pre-trained on ever-growing token budgets under the assumption that better pre-training performance translates to improved downstream models. In this work, we challenge this assumption and show that extended pre-training can make models harder to fine-tune, leading to degraded final performance. We term this phenomenon \textbf{catastrophic overtraining}. For example, the instruction-tuned OLMo-1B model pre-trained on 3T tokens leads to over 2\% worse performance on multiple standard LLM benchmarks than its 2.3T token counterpart. Through controlled experiments and theoretical analysis, we show that catastrophic overtraining arises from a systematic increase in the broad sensitivity of pre-trained parameters to modifications, including but not limited to fine-tuning. Our findings call for a critical reassessment of pre-training design that considers the downstream adaptability of the model.

the recom- Overtrained Language Models Are Harder to Fine-Tune
Jacob Mitchell Springer' Sachin Goyal'
Kaiyue Wen 2 Tanishq Kumar
Xiang Yue
Sadhika Malladi Graham Neubig' Aditi Raghunathan
Abstract
Stage 1: Pre-train for Ttokens
Stage 2: Post-train (instruction tuning)
Large language models are pre-trained on ever-
growing token budgets under the assumption that
LLM benchmark performance
better pre-training performance translates to im-
proved downstream models. In this work, we challenge this assumption and show that extended pre-training can make models harder to fine-tune, leading degraded final performance. We term this phenomenon catastrophic overtraining, For example, the instruction-tun pre-trained on 3T tokens leads to over 2% worse performance on multiple standard LLM bench- marks than its 2.3T token counterpart. Through controlled experiments and theoretical analysis, show that catastrophic overtraining arises from systematic increase in the broad sensitivity of pre-trained parameters to modifications, including but not limited to fine-tuning. Our findings call for a critical reassessment of pre-training design that considers the downstream adaptability of the model.
58
58
56
56
54
Pre-trained
54
Post-trained
OLMo-IB IB model
1
3
1
2
3
Pre-training tokens (Trillion)
Pre-training tokens (Trillion)
Figure I. Language models with extensive pre-training can exhibit catastrophic overtraining, where the performance post-trained models degrades as the pre-training stage is ex- tended. We report the average performance of five common L.M benchmarks (ARC-Fasy, ARC-Challenge, PIQA, HellaSwag) for OL.o- intermediate checkpoints before and after instruction fine-tuning, with additional results in Section 2. We argue that catastrophic overtraining arises as a result a progressive in- throughout pre-training of model sensitivity parameter transformations, leading to greater forgetting of the capabilities acquired during pre-training after fine-tuning (Section 3). Overall, our results challenge the notion that scaling pre-training is strictly beneficial.
1. Introduction
Language models have achieved widespread success follow- ing a two-stage paradigm: (1) pre-training on a vast corpus of uncurated data, followed by (2) post-training on high- quality task-specific data, often confer targeted abilities reason-
mended ratio-and and Llama-3-88 scaled this further to 15T tokens. This trend is driven by consistent gains zero-shot performance (Gadre al.. 2024: Sardana et al., 2024), with few exceptions where scaling up not helpful (Wei et al., 2022; McKenzie et al., 2022a;b; 2023).
such as instruction-followie following, multi-modality, ing. Under the maxim "more data is better", there have been massive investments in scaling both pre-training and post-training.
In this paper, we demonstrate that the widely adopted strat- egy of scaling up language model pre-training does not universally translate to better performance after post- training. Through both theory and experiments, we uncover phenomenon we term catastrophic overtraining, where longer pre-training harms final model performance after instruction tuning other forms of post-training (Figure 1),
Hoffmann (2022) proposed a compute-optimal ratio of roughly 20 tokens per model parameter, yet recent models have far exceeded this. For example, Llama-2-7B (Touvron et al., 2023) was trained on 1.8T
'Carnegie Mellon University "Stanford University 'Havvrdd University "Princeton University. Correspondence Jacob Mitchell Springer ceppringrr@cmu.cde
Catastrophic overtraining is not an isolated curiosity; rather it emerges consistently across a range of models and tasks. As shown in Section 2, extensive empirical evaluations demonstrate the prevalence of this phenomenon in exist-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Overtrained Language Models Are Harder to Fine-Tune
OLMo-1B M.Moo--BAnttrropic-Hl (instruction fine-tuned)
OLMo-IB-LLaVA (multimodal fine-tuned)
ID: AlpacaEval
OOD: ARC-Challenge
OOD: ARC-Easy
ID: VLM Scure
OOD: ARC-Challenge
COD: ARC-Easy
Soore
55
65
Soore
45
32
30
30
30
60
60
OOD: PIQA
OOD: HellaSwag
OOD: Winogrande
OOD: PIQA
OOD: HellaSwag
OOD: Winogrande
Spore
75
60
60
Score
76
60
62
57
74
57
57
72-
55
2
2
2
3
Pre-training tokens (Trillion)
Pre-training tokens (Trillion)
Pre-training tokens
Pre-training tokens (Trillion)
Pre-training tokens (Trillion)
Pre-training tokens (Trillion)
(Trillion)
Base model
Fine-tuned model (IFT VLM)
Figure Extending pre-training can degrade performance after fine-tuning on Anthropic-HH (left) and LLaVA (right). We consider fine-tuning on various intermediate checkpoints from OLMo- IB pre-training. While the base model performance (before fine-tuning) improves with the pre-training token budget (black dashed curve), the performance after fine-tuning drops as we pre-train on more tokens. In the instruction-tunee tuning setting (left), we observe degradation the ID task ggreen)-AlpacaEvaall benchmarks (blue)-ARC, PIQA, and HellaSwag. In multimodal tuning setting, we observe degradation with overtraining on PIQA. and a larger gap between the fine-tuned and base model for ARC, HellaSwag, and Winogrande. We report average over three independent fine-tuning runs, plus error bars. Refer to Appendix F for additional models (OLMM---77B, L.LM360- Amber) and instruction-ttning datasets (extended results for Anthropic-HBi TULU).
well as on OOD
ing models. For instance, we show that the OLMo-IB model (Groeneveld et al., 2024a), pre-trained on 3T tokens and post-trained on AnthropicHHH (Bai et al., 2022), per- forms 3% worse on AlpacaEval (Li et al., 2023b) and 2% worse on ARC (Clark et al., 2018) compared to an interme- diate checkpoint trained on just 2.3T tokens (Figure 2).
ing. Regularization during fine-tuning can delay the onset, albeit at the cost of downstream performance.
2. Extended pre-training can hurt post-training
We study the effect of extended pre-training on two mon post-training setups-instruction tuning for instruction following capability, and multimodal fine-tuning (visual instruction tuning) with LLaVA (Liu et al., 2023a).
To understand why catastrophic overtraining occurs, we turn to carefully controlled experiments (Section 3). We find that modifying the parameters of a pre-trained model leads to forgetting of previously acquired capabilities, where the extent of this forgetting depends on the magnitude of the parameter modifications. However, another key factor in- fluencing forgetting is what we term progressive sensitivity: for modifications of equal magnitude, models that have un- dergone longer pre-training exhibit greater forgetting (Fig- ure 4). Catastrophic overtraining arises when this increased forgetting due to post-training modifications overtakes the improvement during pre- treitrain While constraining the magnitude of the parameter modifications that arise from post-training can mitigate this degradation, it can also limit the pre-trained model's capacity to adapt and learn. This reveals an inherent trade-off that shapes the feasibility of preventing catastrophic overtraining in practice (Figure 7).
2.1. Experimental setup
To analyze the effect of overtraining, we experiment on three language models with open-sourced intermediate check- points: OLMo-1B (Groeneveld et al., 2024a), OLMo-2- 7B (OLMo et al., 2024), and L.M330-AAmee--771 (Liu et al., 2023b). For each model, we perform post-training on in- termediate checkpoints. We investigate instruction tuning with two datasets: Anthropic-HH (Bai et al., 2022) and TULU (Wang et al., 2023), and we perform multimodal fine-tuning with the LLaVA visual instruction tuning frame- work (Liu et al., 2023a). We train each intermediate check- point on each dataset.
Finally, we present a theoretical analysis of a linear transfer learning setting in Section 4 that admits precise character- ization of catastrophic overtraining and progressive sensi- tivity. We study how incremental feature learning leads to progressive sensitivity and inevitable catastrophic overtrain-
We evaluate model performance along two key dimensions: the ID performance, evaluated on the fine-tuning task of interest (for e.g. instruction following), and the OOD perfor- mance, computed on suite of ten common LLM evaluation benchmarks, covering reasoning, QA, commonsense, and
2

---

Overtrained Language Models Are Harder to Fine-Tune
knowledge extraction. For each checkpoint, we tune the learning rate and select the model with the best ID perfor- mance.
We begin by defining the phenomenon, which call catas- trophic overtraining.
We refer the reader to Appendix C for further information on the pre-trained models, the specification of the fine-tuning process, and for details of evaluation.
Catastrophic overtraining is the phenomenon where extending pre-training beyond a certain token budget results in a decrease in the model's performance after subsequent modifications.
2.2. Results
We call this token budget where performance first begins to degrade the inflection point. Catastrophic overtraining can refer to a decrease of the pre-training performance or of the performance of other downstream tasks as pre-training is extended. Note that this performance drop can manifest differently across various downstream evaluation tasks, even for the same model.
Figure 2 compares the performance of various OLMo-1B models, trained to different pretraining budgets (x axis).
Extended pre-training always improves base models. In line with past work, we find that extended pre-training yields a monotonic improvement in the base models. The performance keeps improving on all the downstream tasks we evaluate (dashed line in Figure 2).
In Section 2. we see catastrophic overtraining when post- training OLMo-IB for instruction tuning or multimodal fine-tuning and evaluating on standard benchmarks. In the rest of this paper, we aim to answer two central questions:
Extended pre-training can hurt post-trained counter- parts. While the base model improves, we find sur- prising degradation when the base models are post-trained. Specifically, after fine-tuning on the Anthropic-HH dataset for instruction following, a base model pre-trained on 3T to- kens shows up to 3% lower response rate (AlpacaEval score) than one pre-trained on just 2.3T tokens (~~ 23% fewer to- kens). We see a similar drop on various OOD tasks such as reasoning and question answering, as evaluated on bench- marks such as ARC-Fasy, ARC-Challenge, HellaSwag and PIQA. Overall, after instruction tuning, models pre-trained on 3T tokens underperform compared to those pre-trained on 2.3T tokens, dropping to the level of models pre-trained with just 1.5T tokens (50% fewer tokens).
When and why does catastrophic overtraining occur?
2. What factors influence the inflection point?
To address these questions, we systematically study and build an intuitive picture of the effect of overtraining in the presence of Gaussian perturbations (Section 3.2) and then expand fine-tuning in a controlled but real-world setup (Section 3.3).
3.1. Catastrophic overtraining in a controlled setup
We documented several instances of catastrophic overtrain- ing in real-world scenarios. To gain a deeper understanding and explore more extreme degrees of overtraining, we inves- tigate simpler, controlled setup described below. Note that our real-world experiments used publicly available check- points from a single training run, which meant that each pre-training budget corresponded to a different final learn- ing rate due to the annealing schedule. In this section, we remove that confounding factor.
For multimodal fine-tuning, we see that extended pre- training translates to continuous improvements in the VLM score. However, models pre-trained on more tokens show greater forgetting and larger drops in performance across the various OOD benchmarks. On some datasets such as PIQA, the drop is so severe that extended pre-training actively hurts performance after post-traing post-training (Figure 2, right).
We present evaluations of additional pre-trained models different fine-tuning setups in Appendix F. Overall, while ex- tended pre- training always improves the pre-training perfor- mance, these gains do not always translate to post-training. There are several settings where extended pre-training ac- tively hurts post-training performance.
Pre-training setup. We pre-train models from seratch with sizes ranging from 15M to 90M parameters, spanning token budgets from 4B to 128B, on C4 web data (Raffel et al., 2019). We train with a cosine annealing schedule that an- neals every model to zero. In the main paper, we present results from the 30M model; see Appendix G for results with 15M and 90M parameter models.
3. Catastrophic overtraining
Modifications to the pre-trained model. We fine-tune the pre-trained models above. We fine-tune each model on various classification and language modeling datasets spanning QA, sentiment analysis, math, and code. Details on the datasets and hyperparameter choices are provided in Appendix D. We also consider simple modification of
In Section 2, we made a surprising observation where ex- tended pre-training can hurt post-training. In this section, we dig deeper into this phenomenon to understand why and when expending more compute by pre-training on more tokens can counterintuitively degrade performance.
3

---

Overtrained Language Models Are Harder to Fine-Tune
denote the base model weights, then we get
4.0
A perplexity
Perplexity
0.2
i 0=0+ + where ~ ~N 2²),
(I)
3.8
where E is the covariance matrix of the initialization dis- tribution of the parameters (prior to pre-training) and controls the magnitude of the perturbation.
0.0
10
10
10
10
Pre-training tokens
Pre-training tokens
Base model
Minimum y (0.0025)
Maximum (0.04)
First, we plot the change in C4 perplexity due to Gaussian noise, i.c. the difference between the C4 perplexity and @ in Figure 3 (left). We observe an interesting trend as we track the change in perplexity between the base model and the perturbed model function of the number of pre-training tokens:
Figure 3. Progressive sensitivity of Gaussian perturbations (left): extending pre-training progressively increases the degree to which Gaussian parameter perturbation degrades perplex- ity. Catastrophic overtraining (right): eventually, this leads to overall worse pre-training perplexity. We perturb OI Mo-330 models trained on various pre-training token budgets with Gaus- sian noise scaled by the factor 7 (color). The left plot shows the difference in perplexity between the perturbed and unperturbed models, while the right plot shows the absolute perplexity of the perturbed models.
Progressive sensitivity to noise: For a fixed magnitude of perturbation, the change in perplexity between the base model and the perturbed model increases monoton- ically with the number of pre-training tokens.
Simultaneously, we plot the absolute C4 perplexity of the base model (Figure 3, right, dashed line). We observe that the base model's perplexity decreases with the number of pre-training tokens.
adding Gaussian perturbations to the pre-trained weights warm-up in Section 3.2.
Our intuitive picture views post-training as some modifica- tion to the pre-trained model that is trained on large amounts of broad data. Such modifications are aimed improving some targeted performance (such as VLM score). However, as argued in (Kumar al., 2022), such modifications can inadvertently distort the pre-trained knowledge, leading to degraded performance cuitrf-diittibution or unrelated tasks.
In this setting, catastrophic overtraining arises from the interaction between the progressive sensitivity to noise and the monotonic improvement the base model as pre-training progresses. Early in training, the base model improves faster than the rate which sensitivity increases, leading to a net decrease in perplexity after Gaussian pa- rameter perturbations. Beyond a certain point, the rate at which sensitivity increases surpasses the rate at which the base model improves, leading to an increase in perplexity after the perturbation. This results in a U-shaped trend of the C4 perplexity after perturbation (Figure 3, right).
Downstream evaluation. While we evaluate real-world benchmarks in Section 2, we focus here on measuring the C4 perplexity of the modified downstream model indi- cator of how well the original pre-trained knowledge is pre- served. A decline in C4 perplexity may signal loss of this knowledge, potentially resulting in both out-of-distribttion performance degradation (due to forgetting or distortion). We also measure performance as perplexity on held-out from the same distribution as the fine-tuning data. We use perplexity rather than accuracy because is smoother and less noisy metric, and can often offer better measure of model quality than accuracy for small models (Schaeffer et al., 2023; 2024), Although our analysis centers on pre- training perplexity, we acknowledge that other factors may also contribute to downstream performance losses-2e a topic we leave for future work.
Tracking the inflection point. In Figure 3, larger per- turbations are associated with larger and more quickly increasing degradation of the pre-training loss. Thus, the point at which the degradation from sensitivity surpasses the improvement in the base model is accelerated for larger perturbations, leading to earlier inflection point.
Intuitive picture. Pre-training on more tokens improves the base model (as expected) but also makes the base models more sensitive noise. Progressive sensitivity leads to catastrophic overtraining the increase in perplexity due to noise eventually overwhelms improvements in the model. For large magnitude perturbations, this degradation sets in at lower token budget, while for smaller magnitudes of perturbations, catastrophic overtraining may not be observed until large token budget.
3.2. Warmup: Gaussian perturbations
We take base models pre-trained to various token budgets and add Gaussian noise of the following form. Let 0 E R
4

---

Overtrained Language Models Are Harder to Fine-Tune
GSMBK
SIQA
StarCoder- Python
At the inflection point at which sensitivity increases sur- passes the rate at which the base model improves, we ob- serve catastrophic overtraining. This results in U-shaped trend of the C4 perplexity after fine-tuning (Figure 5, top).
Faxx S 5.De
Prea 1.60-04 04
Aiaa -
4 perploctty
10
10
10th
16a
10
Tracking the inflection point for fine-tuning, Analogous to the Gaussian setting, since the rate of increase of degrada- tion is accelerated for larger learning rates, models trained with larger learning rates exhibit an inflection point at lower token budgets, and the degradation more pronounced.
Pre-training tokens
Pre-training tokens
Pre-training tokess
Base model
tuning (de-6)))
tueiinin LR (Araa)
Figure 4. Progressive sensitivity of fine-tuning: Extending pre- training progressively increases the degree to which fine-tuning degrades perplexity. OLMo-30M models trained on various pre- training token budgets are fine-tuned on downstream tasks using fixed hyperparameters: math (GSM8K), code (Starcoder-Pyie Python), and QA (SIQA). Lines connect models sharing hyperparameters, differing only in pre-training tokens. Learning rates range from 4e-06 to the dataset-specific maximum (fmax). We report the difference in perplexity between the fine-tuned and pre-trained models, a function of the number of pre-training tokens.
ID perplexity. While smaller learning rates generally re- sult less degradation to the C4 perplexity, the ID perplex- ity of the fine-tuned models shows a different trend: larger learning rates, up to a point, result in lower ID perplex- ity, though sometimes also exhibit a U-shaped trend in ID perplexity (Figure 5, bottom). This implies that tuning the learning rate can sometimes mitigate degradation only at the cost of fine-tuning performance. We explore in Section 3.3.2 when tuning the learning rate to minimize the ID perplexity can mitigate the degradation of C4 perplexity that arises as pre-training is extended, and when it cannot.
3.3. Fine-tuning pre-trained models
In the previous section, we studied how catastrophic over- training arises when adding noise to pre-trained models. While noise can be seen canonical modification, it is different from fine-tuning that might involve more struc- tured updates to the models. However, we see in this section that the intuitive story above also holds when we fine-tune models on real-world language datasets described above.
Intuitive picture. The intuition from the Gaussian per- turbation setting carries over to fine-tuning with a fixed learning rate. Pre-training on more tokens will improve the quality of the base model and at the same time make the model degrade more when fine-tuned. Beyond a certain point, pre-training on additional tokens will degrade the resulting fine-tuned model's C4 perplexity, and often the ID perplexity of the fine-tuning task.
3.3.1. FINE-TUNING WITH FIXED LEARNING RATE
3.3.2. BALANCING FINE-TUNING GAINS WITH DEGRADATION
First, analogous to how we quantify performance drop for fixed magnitude of Gaussian perturbation (7), we similarly need to regularize the fine-tuning in some way to ensure consistent degree of change across the pre-trained check- points. Fixing the learning rate is a simple and effective way to do so. While we do not provide a formal justification, we discuss our reasoning in Appendix D.
In Section 3.3, we showed that for a fixed learning rate, the sensitivity of pre- pre-trained models increases with the number of pre-training tokens, leading to catastrophic overtraining. In practice, however, the learning rate is tuned on a valida- tion set from the in-domain (ID) task. This tuning process may yield different optimal learning rates across pre-trained checkpoints, which can potentially mitigate catastrophic overtraining. The degradation depends on both the learning rate as well as the sensitivity. So if model pre-trained on more tokens can admit a smaller learning rate when fine- tuning to achieve good ID performance, it can compensate the increase in sensitivity.
For each learning rate, we plot the change in C4 perplex- ity from the pre-trained model to the fine-tuned model in Figure 4. In this plot, we track how the degradation in C4 perplexity evolves with the number of pre-training tokens. First, larger learning rates distort the model more and thus exhibit a greater increase in perplexity. Second, we observe a trend over pre-training tokens analogous to the behavior seen with Gaussian noise, but this time for fine-tuning.
However, this smaller rate does restrict the extent of neces- sary parameter updates, and might be insufficient to achieve good ID performance. This presents an interesting trade-off that we investigate empirically. We tune the learning rate to maximize fine-tuning ID performance. We track the optimal value as a function of the pre- pre-training token budget, and plot the ID performance and pre-train perplexity corresponding
Progressive sensitivity when fine-tuning: For a fixed learning rate, the change in perplexity increases mono- tonically with the number of pre-training tokens.
5

---

Overtrained Language Models Are Harder to Fine-Tume
GSM8K
SIQA
StarCoder- Python
MR
RTE
TREC
C4 perplexily
6
Ams 1,0e-03
maxx 2.4e-04
Trixx 3,0e-03
Qmxx = 9.0c-05
Tmax 9,0c-05
Tixx 1.0e-04
6
5
5
5-
4-
4
ID perplexilyy
1.0e-03
Amax 2.40044
Piiax 3.0e-03
Omx 9.0e-05
1.0
Timax 99.e-05
Tiax 1 1.0e-04
0.6
0.8
2
5
3
0.4
10tH
10
10tc
10
10
10a1
101
104t
10000
10
1010
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training takens
Base model
Minimum fine tuning learning rate (4n-6)
Maximum fine tuning learning rate (fmua)
Figure 5. Catastrophic overtraining for fine-tuning with fixed hyperparameters: extending pre-training can lead to overall increase in the C4 perplexity (top), and ID perplexity (fine-tuning task; bottom), when fine-tuning with fixed hyperparameters. OLMo-30M models pre-trained with varying token budgets fine-tuned downstream tasks using fixed hyperparametters: math (GSM8K), code (Starcoder--ytthonn
QA (SIQA), and classification (MR, RTE, TREC). Lines connect models sharing hyperparameters, differing only in pre-training tokens. Learning rates range from 4c-06 the datasct-specific maximum (fmaax). At sufficiently large learning rates (lighter colors), we observe performance degradation in both ID) and pre-training metrics beyond certain pre-training budgets. (See Appendices and G for ablations.)
to this optimal learning rate in Figure
learning rate would delay the inflection point. However, this would also result in lower ID performance.
Our findings indicate that the emergence of catastrophic overtraining depends on how the optimal learning rate evolves. We conceptualize this trade-off between ID per- formance and pre-train perplexity degradation into three scenarios, illustrated in Figure 7:
Summary. Overall, our experiments reveal that progres- sive sensitivity manifests under two types of modifications: unstructured Gaussian noise and structured fine-tuning, lead- ing us to conjecture that progressive sensitivity is universal phenomenon. With fixed perturbation magnitude or learn- ing rate, this sensitivity causes catastrophic overtraining. In practice, tuning the learning rate introduces a trade-off: its evolution determines if extended pre-training results in performance degradation.
1. Constant optimal learning rate: A constant optimal learning rate across token budgets leads to degradation in both ID and out-of-domain (OOD) performance for large pre-training budget T (Figure 7, left).
2. Slowly decreasing optimal learning rate: A slowly decreasing optimal learning rate may improve ID per- formance while OOD performance degrades (Figure 7, center).
4. A theoretical perspective of overtraining
Catastrophic overtraining contradicts the common belief that more pre-training improves model quality. To inves- tigate this, we analyze catastrophic overtraining for two- layer linear networks, focusing on identifying the inflec- tion point (Definition 4.2)-the point after which more pre-training harms final performance on the pre-training task. We first examine catastrophic overtraining via Gaus- sian perturbations, paralleling our empirical results (Sec- tion 3.2), and then demonstrate progressive sensitivity dur- ing extended pre-training a canonical fine-tuning scenario (Theorem 4.6). Next, formalize how restricting the mag- nitude of the updates can alleviate performance degradation, using regularization rather than reduced learning rates as in earlier experiments (Section 3.3.2). Without regularization, catastrophic overtraining inevitably emerges (Theorem 4.7).
3. Quickly decreasing optimal learning rate: A quickly decreasing optimal learning rate enables improvements in both ID and OOD performance as the pre-training budget increases (Figure 7, right).
Using a non-optimal learning rate to mitigate degrada- tion. In cases where catastrophic overtraining emerges when fine-tuning with the optimal learning rate, using non-optimal learning rate can sometimes mitigate the degra- dation or delay the inflection point. For example, in both cases where tuning leads to eventual degradation the OOD loss in Figure? 7, choosing to train with the smallest
6

---

Overtrained Language Models Are Harder to Fine-
GSMBK
SIQA
StarCoder-Pyee Python
RTE
TREC
ID perplexity C4 perplexity
5.5
4.25
5.3
6
10-
5.5
5.2
5.0
5.0
4.00
5.1
4
4.5
1.4
4.75
2.6
0.42
0.68
0.15
4.50
2.5
0.40
0.67
1.3
0.12
1016
10
1030
10
1030
10
10
10
101H
10
1010
1011
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-tzaining takens
Pre Pre-training tokens
Figure 6. Calastrophic overtraining after hyperparameter tuning: extending pre-training lead to eventual degradation of the C4 perplexity (top) and ID perplexity (fine-tuning task; bottom), even after hyperparameter tuning, OLMo-30M models pre-trained with varying token budgets are fine-tuned on downstream tasks: math (GSM8K), code (Stareoder-Pythonn QA (SIQA), and classification (MR. RTE, TREC). Lower is better. We tune the learning rate optimize ID performance. ID perplexity degrades with extensive overtraining (RTE, TREC); C4 perplexity degrades in GSM8K, Sarcoder-Pythooe MR, and RTE. Results averaged over three fine-tuning runs. (Additional ablations in Appendices D and G.)
While regularization can prevent this phenomenon, it can also impair fine-tuning performance by limiting adaptation (Theorem 4.7).
This theorem implies that E(t) UT ( is approxi- mately diagonal, and the vector of its diagonal entries (t) tracks which pre-training features have been learned by time t. In the ideal case, which we use in the main pa- per for brevity, we expect the first n elements of ott,, ) are of ,...., o and the remaining elements are zero.' There- fore, studying the evolution of over time and its impact the fine-tuning procedure allow to characterize how elongating the pre- rre-training period affects the pre-training and downstream performance of the final model. We will generally study progressive sensitivity and catastrophic over- training by characterizing the model at time steps fi,..., ta. We focus on studying the inflection point, the time at which catastrophic overtraining with respect the pre-training loss emerges.
4.1. Pre-training setting
We adopt the two-layer linear regression setting proposed by Saxe et al. (2018) as case where pre-training perfor- mance improves monotonically with training time via incre- mental feature learning. Precisely, we consider a regression problem where the data is generated by a full rank linear map y A x for x, y E Rd. with Aree E dddd and where sample x (O I). Denote the SVD of Apre as U'Pre'' with the diagonal elements of pre being strictly positive and monotonically decreasing. We will call these singular values the pre-training features, and denote them of > ... > o pre Let E be a diagonal matrix with the first singular values equal to those of pre and the remaining set O.
Definition 4.2 (Inflection point). Fix post-training modi- fication to the model .A. The inflection point with respect to the pre-training loss defined as the smallest r such that Epee (A((((()())))) (p (A(O(t++1))
We learn a two-layer network 0 W1 with W1, W2 E Rddd that minimizes the mean squared error Fpe the population of Gaussian inputs.
In the following two sections, we study the inflection point for two different post-training modifications: Gaussian pa- rameter perturbations and fine-tuning on a canonical family of tasks.
Spre (0()))
2
4.2. Gaussian perturbation setting
We initialize W, and W₂ with small values and train using gradient flow. Prior work has established that, as training proceeds in this setting, the model incrementally learns the spectrum of pre (Saxe et al., 2018: Gidel et al., 2019).
As a warm-up, we set A to be isotropic Gaussian param- eter perturbations, mirroring Section 3.2. Formally, let A(O(tn)) = ((t)) = (Wi(tn) + Zi) (12(W((t)) + Zz) where Z1, Zz ~ ~0 xx)), and let (t)) = E ((()))(()))) We characterize how the perturbed model
Theorem 4.1 (Informal statement of Saxe et al. (2018); Gidel et al. (2019)). There exists a sequence of timesteps t4 < ta such that timestep t,.
'Appendix contains the case when these coordinates are small but not exactly zero
(tt)) R UE!!V'
7

---

Overtrained Language Models Are Harder to Fine-Tune
Tuned LR constant with F
Tuned LR decrosses slowly with F
Tuned LR decressss quickly with P
training with respect the pre-training loss when the post- training modification consists of randomly perturbing the model parameters. In the next section, we study progressive sensitivity and catastrophic overtraining when fine-tuning on a family of canonical downstream tasks.
degradation
e
Degradation x
No Nodogradation' 's
4.3. Fine-tuning
We now consider the case where the fine-tuning algorithm A corresponds to learning another linear feature map with a shared structure. We define the fine-tuning task as learning y = A x, where A = UV Sharing U and V with A permits transfer learning to occur, even though the spectrum of A is not the same A We define the fine-tuning features of > ... of to be the singular values of A
Cop
Degradation
Degradation
No degradation
Pre-training tokens
Pre Pre-training tokens
Pre Pre-training tokens
Large 1.
Medium LR
Smalll
L. tureed on downstream val.
Let A(O(t)) = ( k) denote a model pre-trained for time and then fine-tuned with a small but finite learning rate r and a large batch size for k E [o, K] steps. The fine-tuning loss similar the pre-training loss with the new task A but introduce regularization term limit the deviation from the pre-trained initialization. This regularization term is standard design in meta learning literature (Chua et al., 2021; Denevi et al., 2018). [[((((t;)) k) - 1 Ca(((t; (); A) = TE [t;
Figure? 7. Schematic to illustrate how the scaling of the optimal learning rate can affect model evaluations as a function of the pre-training tokens T. The dashed lines indicate the hypothetical performance of fixed learning rate, while solid lines indicate the performance when using the learning rate that optimizes the ID performance. (Left) When the optimal learning rate is constant, we expect to observe degradation of both ID and OOD performance. (Center) When the optimal learning rate decreases slowly with T. we may observe degradation of only the OOD performance. (Right) When the optimal learning rate decreases quickly, we will not observe degradation of either metric of performance.
(3)
+ Al]O(t;k) (())
Analogous to the pre-training setting, our analysis pro- ceeds by tracking the vector of the diagonal elements o (ttt; of E(t; k) U V. We define ppro(t,) Sue (Ott, K)) - (0(,, 0)) as the change in the pre- training performance over the course of fine-tuning, and we characterize how Atre(t,) changes as the pre-training time tn increases. In particular, if (tre(t.)) is monotonically increasing, then we can conclude that progressive sensitivity is present.
pre-training loss E(r evolves as pre-training is ex- tended.
Proposition 4.3 (Informal version of Lemma A.4). Let
11,...,td la be defined as in Theorem 4. Then,
(p))(tn) - (t-e 2 (2dy²
(2)
The formal proof in Appendix A demonstrates that elon- gating pre-training introduces newly non-zero feature o, introduces a new dimension along which the perturbation degrades loss. The above proposition allows us to charac- terize the inflection point (Definition 4.2) in the Gaussian perturbation setting as the smallest u such that 2d-² > ore As such, smaller or more quickly decaying features will induce smaller inflection point.
To begin, we formalize the misalignment between the pre- training and downstream tasks in terms of their features. Definition 4.5. The pre-training task A and the fine- tuning task A are (a, miisaliigee when of > ao!re for all i r.
Our first result establishes that our setting exhibits progres- sive sensitivity when the fine-tuning task is different from the pre- pre-training one.
To establish catastrophic overtraining, we now illustrate that degradation proceeds monotonically beyond the inflection point. That is, elongating the training budget beyond the inflection point will increasingly degrade the pre-training performance of the model.
Theorem 4.6 (Progressive sensitivity; informal version of Theorem A.24). Assume that A and A are (a, 1)- misaligned with 1. Then, Apre(tn) 2 0 and Apre(ta) is monotonically increasing with the number of learned pre-training features
Theorem 4.4 (Informal version of Theorem A.3). For some ? > O, there exists an inflection point E [1, d) such that Eppe (n) increases monotonically for n 2 r.
Having established the prevalence of progressive sensitivity, we now turn our attention to understanding how and when
Our results establish the inevitability of catastrophic over-
8

---

Overtrained Language Models Are Harder to Fine-Tune
observe catastrophic overtraining with respect the pre-training loss. We first show that when regularization is not present and the downstream task is sufficiently distinct from the pre-trained task, then elongating pre-training will pre-training performance. Furthermore, we demonstrate that regularization can delay the inflection point at which pre-training performance starts to degrade (Definition 4.2), albeit at cost to the downstream performance.
forgetting- neural networks trained sequentially tasks tend to forget prior tasks-has also been well- documented in the literature (Kirkpatrick et al., 2017; French, 1999; Goodfellow et al., 2013; Kemker et al., 2018; Kotha et al., 2023). There have been several proposed miti- gation strategies, for example, Ahn et al. (2019); Hou et al. (2018); 8); Chaudhry et al. (2019a) propose using regulariza- tion to mitigate catastrophic forgetting. Other fixes include generative replay of examples from previous tasks (Shin et al., 2017) or maintaining a memory buffer of previous tasks (Chaudhry et al., 2019b; de Masson d' Autume et al., 2019). In this work, we show that catastrophic forgetting can become more severe with overtraining.
Theorem 4.7 (Catastrophic overtraining; informal version of Theorem A.25). The following true with high proba- bility:
1. Catastrophic overtraining is inevitable without reg- ularization. Let X O. There exists an og > 0 such that if Apre and A are (a.r))-mssllinedd, for a (x0, then the pre-training loss after fine-tuning (0 K)) monotonically increases for 2 r.
Scaling laws for optimal pre-training. In our work, we argue that training for fewer tokens can be beneficial for downstream performance after fine-tuning. Related to work, Isik et al. (2024) proposes scaling laws for certain downstream translation tasks after fine-tuning, but does not observe degradation with overtraining. In addition, optimal token budgets have been identified for fixed compute (Ka- plan et al., 2020; Hoffmann et al., 2022), extended to various contexts (Hernandez et al., 2021; Cherti et al., 2023; Muen- nighoff et al., 2023; Goyal et al., 2024; Liu et al., 2025; Bhagia et al., 2024). Existing laws sometimes inaccurately predict downstream performance (Diaz & Madaio, 2024), and U-shaped scaling trends have been observed (Caballero et al., 2022; Wei et al., 2022; McKenzie et al., 2022a). Practi- tioners often overtrain small models beyond optimal tokens to reduce inference cost (Sardana et al., 2024; Gadre et al., 2024).
2. Regularization can delay the degradation of pre- training performance at the cost of downstream per- formance. For any n, the inflection point r()) and the unregularized fine-tuning loss []0* A 1 increase monotonically with X.
Our results this section demonstrate that progressive sen- sitivity and catastrophic overtraining can arise in the rela- tively simple setting of training linear networks, which learn task-related features incrementally. We characterize the in- flection point (Definition 4.2) under various post-training modifications, including applying Gaussian perturbations and fine-tuning on a canonical task. Our main results demon- strate that elongating the pre-training period will inevitably result in progressive sensitivity and catastrophic overtrain- ing, and although appropriate regularization can delay the onset of these phenomena, this may come at the cost of the downstream task performance (Theorems 4.4, 4.6 and 4.7).
Transfer learning theory. Our theoretical analysis uses classical deep linear transfer learning setups (Gidel et al., 2019; Saxe et al., 2018). Such setups have been applied to knowledge acquisition (Wei et al., 2024; Arora et al., 201 8), downstream feature benefits (Saunshi et al., 2021; Wei et al., 2021; Shachaf et al., 2021; Chua et al., 2021; Wu et al., 2020; Tripuraneni et al., 2020), and fine-tuning- performance degradation (Kumar et al., 2022).
5. Related Work
Loss of plasticity. The idea that more training can degrade model's adaptability new tasks, termed loss of plas- ticity, has been primarily studied in small-model continual learning (Ash & Adams, 2020; Dohare et al., 2021) and re- inforcement learning (Kumar et al., 2020; Lyle et al., 2022: 2023; Ma et al., 2023; Abbas et al., 2023). Loss of plastic- ity has been attributed to loss curvature (Lyle et al., 2023; Lewandowski et al., 2023), increased weight norm (Nikishin et al., 2022), feature rank (Kumar et al., 2020; Gulcehre et al., 2022), and feature inactivity (Lyle et al., 2022; Do- hare et al., 2021). Multiple remedies have been proposed, including architecture changes (Lyle et al., 2023), param- eter resets (Nikishin et al., 2024; D'Oro et al., 2022), and regularization (Kumar et al., 2023; Ash & Adams, 2020).
We discuss additional related work in Appendix B.
6. Discussion
In this work, we uncovered a surprising trend: contrary to common belief, longer pre-training does not always lead to better post-trained models. We have shown that this is a consequence of a broader underlying phenomenon where models become more sensitive to perturbations as they are pre-trained on more tokens. Our theoretical analysis im- plies that this degradation of adaptability is especially catas- trophic when the pre-training and fine-tuning tasks are mis- aligned, and in such a case catastrophic overtraining may be inevitable, even if the fine-tuning process is regularized.
Catastrophic forgetting. The phenomenon of catastrophic

---

Overtrained Language Models Are Harder to Fine-Tune
Acknowledgments
Bhagia, A., Liu, J., Wettig, A., Heineman, D., Tafjord, O., Jha, A. H., Soldaini, I.., Smith, N. A., Groeneveld, D., Koh, P. W., et al. Establishing task scaling laws via compute-efficiens model ladders. arXiv preprint arXiv:2412.044400 2024.
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE2140739 Any opinion, findings, and con- clusions or recommendations expressed in this material are those of the authors(s) and do not necessarily reflect the views of the National Science Foundation.
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. Thirty-Fourdy AAAI Conference on Artificial Intelligence, 2020.
We gratefully acknowledge support from Apple, NSF and the A12050 program Schmidt Sciences (Grant #G2264481).
Caballero, E., Gupta, K., Rish, I., and Krueger, D. Broken neural scaling laws. arXiv preprint arXiv:2210.14891, 2022.
Xiang Yue was supported in part by Camegie Bosch Insti- tute Fellowship.
The authors would like to thank the following individuals for their helpful feedback and discussions: Christina Baek, Tianyu Gao, Gaurav Ghosal, Suhas Kotha, Vaishnavh Na- garajan, Chen Wu, and Ziqian Zhong.
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with a-gem, 2019a. URL https: / httpaaxxvvv...///..a/aaaaa org abs 1812. 02420.
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H. S., and Ranzato, M. On tiny episodic memories in continual learning, 2019b. URL https: http://axxvvvoroo abs///b///// 902. 10486.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, Reproducible scaling laws for con- trastive language-image learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 2818-2829. IEEE, June 2023. doi: 10. 10.1109/cvp:5222..000 2ttp:pppp22229.222202222222 URL http: /xx co..ccc9/0...111/////3377....... 2023.
References
Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. Loss plasticity in continual deep reinforcement learning. In Conference Lifelong Learning Agents, pp- 220-636. PMLR, 2023.
Chua, K., Lei, Q., and Lee, J. D. How fine-tuning allows for effective meta-learning, 2021. URL https: //arxiv. org/ abs / 22000 02221.
Ahn, H., Cha, S., Lee, D., and Moon, T. Uncertainty-baset continual learning with adaptive regularization, 2019.
URL wwtttttp//// /ttp/aaxivvv//////////..//...ava wrgg/bs///i 1atpb///99...11111....1111111111 11614.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT 2019, 2019.
Arora, S., Li, Y., Liang, Y., Ma. T., and Risteski, A. Linear algebraic structure of word senses, with applications to polysemy, 2018. URL https: : /ttp/aaxvvv/v.////////e////aaava org/ abs/ 1601.0371 03764
Ash, J. and Adams, R. P. On warm-starting neural network training. Advances in neural information processing sys- tems, 33:3884-3894, 2020.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan,
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
T., Joseph, N.. Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-
Z.. Hernan- dez, D., Hume, T., Johnston, S., Kravee, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J. McCandlish, S., Olah, C., Mann, B., and Kaplan, Training a helpful and harmless assistant with rein- forcement learning from human feedback, 2022. URL https: https://arxivy otg/abs////eooooooorot.gpt....rr abs 2204. 05862.
Cohen, J. M., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. Gradient descent on neural networks typically occurs at the edge of stability, arXiv preprint arXiv:2103.00065 2021.
10

---

Overtrained Language Models Are Harder to Fine-Tune
Conneau, A. and Kiela, D. Senteval: An evaluation toolkit for universal sentence representations. arXiv preprint VrXiv:1803.05440 2018.
Goodfellow, 1. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y. An empirical investigation of catastrophic for- getting in gradient-based neural networks. arXiv preprint arXiv: wtw.::3.2..2110 2013.
Dagan, I., Glickman, O., and Magnini, B. The pascal recog- nising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Goyal, S., Maini, P., Lipton, C., Raghunathan, A., and Kolter, J. Z. Scaling laws for data filtering data curation cannot be compute agnostic, 2024. URL https://
de Masson d' Autume, C., Ruder, S., Kong, L., and Yo- gatama, D. Episodic memory in lifelong language learn ing, 2019. URL https: /tt///axxiii arxiv. org/ abs /9906. 01076.
a arxiv. www.ooggg/bb//...ooo.o..../...er hts://bs//22 2404 07177.
Grattafiori, A. al. The llama herd of models, 2024. URL https:/ www.wiwwwgawaaaaeeeee.ee.oeeeeee / 2407 21783.
Denevi, G., Ciliberto, C., Stamos, D., and Pontil, M. Learn- ing to learn around common mean. In Bengio, S., Wal- lach, H., Larochelle, H., Grauman, K., Cesa-Bianchi N., and Garneu, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: httt:s:://proccedinngg... newsiis......... co
Groeneveld, D., Beltagy, I,, Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W. Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Pyatkin, V., Ravichan- der, A., Schwenk, D., Shah, S., Smith, W., Strubell, E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K., Zettlemoyer, L., Dodge, J., Lo, K., Sol- daini, L., Smith, A., and Hajishirzi, H. Olmo: Accel- erating the science of language models. Preprint, 2024a.
0it.2442222557777777900000333333 pdf.
Diaz, F. and Madaio, M. Scaling laws do not scale. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7. pp. 341-357, 2024.
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I,, Wang, Y., et Olmo: Accelerating the science of lan- guage models. arXiv preprint arXiv:2402.00833 2024b.
Dohare, S., Sutton, S., and Mahmood, Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.0632. 2021.
Gulcehre, C., Srinivasan, S., Sygnowski. J., Ostrovski, G., Farajtabar, M., Hoffman, M., Pascanu, R., and Doucet, An empirical study of implicit regularization in deep offline arXiv preprint wwxiv:2207.02092 2022.
D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L.., Belle- mare, M. G., and Courville, A. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer, 2021. URL https:// arxiv. srxiv.org/bbs.oo 2102. 01293.
French, R. M. Catastrophic forgetting in connectionist net- works. Trends in cognitive sciences, 3(4): -35, 1999.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., and Ji, R. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL https: //arxiv. org/ abs/ /ee//2366622 13394.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. I.., Hendricks, I.. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.085.4 2024.
Hou, S., Pan, X., Loy, C. C., Wang. Z., and Lin, D. Lifelong learning via progressive distillation and retrospection. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700- 6709, 2019.
Gidel, G., Bach, F., and Lacoste-Julien, S. Implicit regu- larization of discrete gradient dynamics linear neural networks, 2019. URL https: /ete://aarr /arxiv.. org/ abs/ 1.904 13262.

---

Overtrained Language Models Are Harder to Fine-Tum
Isik, B., Ponomareva, N., Hazimeh, H., Paparas, D., Vas- silvitskii, S., and Koyejo, S. Scaling laws for down- stream task performance of large language models. arXiv preprint arXiv:2402.04117, 2024.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instraction-folloiing www.tuww..mtttttttt//aeeeeee
models. https: a paca_eval, ,, 2023b.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language mod- els, 2020. URL https: arxiv. arg/abs/ aaaaabas/2000 /2001. 08361.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models, 2023c. URL eetps://arxe.
httg/bb/223055 2000/ 10355.
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth dozen images, 2016.
Liu, E., Bertsch, A., Sutawika, L., Tjuatja, L., Ferman- des, P., Marinov, L., Chen, M., Singhal, S., Lawrence, C., Raghunathan, A., al. Not-just-sealing laws: To- wards better understanding of the downstream impact of language model design decisions. arXiv preprini arXiv:2503.03862 2025.
Kemker, R., McClure, M., Abitino, A., Hayes, T., and Kanan, C. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Liu, H., Xie, S. M., Li, Z., and Ma, T. Same pre-training loss, better downstream: Implicit bias matters for lan- guage models, 2022. URL https: https://arxiv.orr
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins. G., Rusu, A.. Milan, K., Quan, J., Ramalho, T. Grahska-Barwinskk Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. Overcom- ing catastrophic forgetting in neural networks. Pro- ceedings of the National Academy of Sciences, 114 (13):3521-3526, March 2017. ISSN 1091-6490. doi: 10. 1.1073/ppas.111118351144 URL http: /txp/dx.i... org/10.1073/paas. 1a11835161111111
Abs/2210.1412
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tuning, 2023a.
Liu, Z., Qiao, A., Neiswanger, W., Wang, H., Tan, B., Tao, T., Li, J., Wang, Y., Sun, S., Pangarkar, O., Fan, R., Gu, Y., Miller, V., Zhuang, Y., He, G., Li, H., Koto, F,, Tang,
L., Ranjan, N., Shen, Z., Ren, X., Iriondo, R., Mu, C., Hu, Z., Schulze, M., Nakov, P., Baldwin, T., and Xing,
Kotha, S., Springer, J. M., and Raghunathan, A. Understand- ing catastrophic forgetting in language models via im- plicit inference. arXiv preprint arXiv:2309. 10105, 2023.
E. P. Llm360: Towards fully transparent open-source 1lms, 2023b.
Kumar, A., Agarwal, R., Ghosh, D., and Levine, Implicit paaeettaramtttiiiiatiinnn inhibits data-efficient deep rein- forcement learning. arXiv preprint arXiv:2010.144 14498, 2020.
Lyle, C., Rowland, M., and Dabney, W. Understanding and preventing capacity loss reinforcement learning. arXiv preprint arXiv:2204.0955 2022.
Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. Understanding plasticity in neural net- works. International Conference on Machine Learning, pp. 23190-23211 PMLR, 2023.
Kumar, A.. Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and un- derperform out-of-distribution, 2022. URL https: //arxiv. wttg/b//g//aa.. 10054.
Kumar, S., Marklund, H., and Van Roy, B. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308. 11958, 2023.
Ma. G., Li, L., Zhang, S., Liu, Z., Wang, Z., Chen, Y.. Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. arXiv preprint arXiv:2310.077111 2023.
Lewandowski, A., Tanaka, H., Schuurmans, D., and Machado, M. Directions of curvature as an explanation for loss of plasticity, arXiv preprint arXiv:2312.00244 2023.
Maggie, Culliton, P., and Chen, W. Tweet sentiment extrac- tion. https: com/ compet iipeeeitiiss/// ions/ tweet-sents iertt-sentiiemnn menntimmet--ettraaeeeee ion, 2020. Kaggle.
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023a.
McKenzie, I., Lyzhov, A., Parrish, A., Prabhu, A., Mueller, A., Kim, N., Bowman, S., and Perez, E. The inverse scaling prize, 2022a. URL https: / //tp//ggtbb//////////// com/ inverse- sca aesaliannaaa..............aaeil. inngingiiii
12

---

Overtrained Language Models Are Harder to Fine-Tune
McKenzie, I., Lyzhov, A., Parrish, A., Prabhu, A., Mueller, A., Kim, N., Bowman, S., and Perez, E. Inverse scaling prize: First round winners, 2022b. URL https:// irmckenzie.c co. uk/roundl.
Saunshi, N., Malladi, S., and Arora, S. A mathematical exploration of why language models help solve down- stream tasks. In International Conference on Learning Representations, 2021. URL httpt::tttpeeeeeeeeeeeee /opnrevie....eeeeee net/ / forrm?ormmmmmm j= TW3sFc1s
McKenzie, I., Lyzhov, A., Parrish, A., Prabhu, A., Mueller, A., Kim, N., Bowman, S., and Perez, E. Inverse scaling prize: Second round winners, 2023. URL https:// irmckenzie. co uk/ uk/roun round2.
Saxe, A. M., McClelland, J. L.., and Ganguli, S. A math- ematical theory of semantic development in deep neu- ral networks. CoRR, 0ts/1810010531,, 2018. URL
http:://axxiv..oooo http:///aaxx...oo/a//////// at...a////111...........
Muennighoff. N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C. Scaling data-constrained language models, 2023. URL https: //arxiv. htwzg/abs/33000 2355 6264.
Schaeffer, R., Miranda, B., and Koyejo, S. Are emergent abilities of large language models mirage? Advances in Neural Information Processing Systems, 36:55565- 55581, 2023.
Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In International conference on machine learn- ing. pp. 16828-16847, 16847. PMLR, 2022.
Schaeffer, R., Schoelkopf, H., Miranda, B., Mukobi, G., Madan, V., Ibrahim, A., Bradley, H., Biderman, S., and Koyejo, S. Why has predicting downstream capabilities of frontier ai models with scale remained elusive? arXiv preprint M.Xi::2406.0439/ 2024.
Nikishin, E., Oh, J., Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., and Barreto, A. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems, 36, 2024.
Shachaf, G., Brutzkus, A., and Globerson, A. A theoretical analysis of fine-tuning with linear teachers, 2021. URL https: httpaaaivvvvvo.... org/ aaaaabss/1000 2207 10111000000 01 641.
OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., Lambert, N., Schwenk, D., Tafjord, O., Anderson, T., Atkinson, D., Brahman, F., Clark, C., Dasigi, P., Dziri, N., Guerquin, M., Ivison, H., Koh, P. W., Liu, J., Malik, S., Merrill, W., Miranda, L. J. V., Morrison, J., Murray, T., Nam, C., Pyatkin, V., Rangapur, A., Schmitz, M., Skjonsberg, S., Wadden, D., Wilhelm, C., Wilson, M., Zettlemoyer, L., Farhadi, A., Smith, N. A., and Hajishirzi, H. 2 olmo 2 furious, 2024. URL https://arxiv: org/ abs//bb//////ee 2501 00656.
Shin, H., Lee, K., Kim, J., and Kim, J. Continual learning with deep generative replay, 2017. URL https:// arxiv. org/abs/ 1705. 08690.
Singh, A., Natarjan, V., Shah, M., Jiang, Y., Chen, X., Parikh, D.,, and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 8317-8326, 2019.
Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D. Scale efficiently: Insights from pre-training and fine-tuning transformers, 2022. URL https:/ / /atpiv.xiivv////i.///aa/aaaaaava org/ abs/ 10686.
Pang, B. and Lee, L. A sentimental education: Sentiment analysis using subjectivity summarization based on mini- mum cuts. arXiv preprint cs/0409058, 2004.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Sakaguchi, K., Bras, R. L.,, Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Tripuraneni, N., Jordan, M. I. and Jin, C. On the theory of transfer learning: The importance of task diversity, 2020. URL https: http//axvvvv.e///////////..a org/ abs / 2006.00000000000000 116.0.
Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y. Socialiqa: Commonsense reasoning about social interac- tions. arXiv preprint arXiv: 1wX.:0:9944097220 2019.
Vershynin, R. High-dimensional probability: An introduc- tion with applications in data science, volume 47. Cam- bridge university press, 2018.
Sardana, N., Portes, J., Doubov, S., and Frankle, J. Be- yond chinchillaoppimaal: Accounting for inference in language model scaling laws, 2024. URL https: //a rxxiv. orgo abs/ abc//4/1......... 00448.
Voorhees, E. M. and Tice, D. M. Building a question an- swering test collection. In Proceedings of the 23rd annual
13

---

Overtrained Language Models Are Harder to Fine-
international ACM SIGIR conference Research and development in information retrieval, pp. 200-207, 2000.
Wang, Y., Ivison, H., Dasigi, P.. Hessel, J., Khot, T., Chandu, K., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I. al. How far can camels go? exploring the state of instruction tuning open resources. Advances in Neural Information Processing Systems, 36:74764-74786 2023.
Wei, C., Xie, S. M., and Ma, T. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. Advances Neural Information Processing Systems, 34, 2021.
Wei, J., Kim, N., Tay, Y., and Le, Q. V. Inverse scaling can become u-shaped, arXiv preprint arXiv:2211.020) 2022.
Wei, S., Malladi, S., Arora, S., and Sanyal, A. Provable un- learning in topic modeling and downstream tasks. arXiv 2024.
preprint arXiv:2441.
Wu, S., Zhang, H. R., and Ré, C. Understanding and im- proving information transfer in multi-task learning, 2020. URL https:: // /aaaiviiiii..///.///.aaaa/aaitvv org/ abs/2005 00944.
Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N.. Pachocki, J., Chen, W., and Gao, Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.0344 000000033466.. 2022.
Zhang, Y., Backurs, A., Bubeck, S., Eldan, R., Gunasekar, S., and Wagner, T. Unveiling transformers with lego: a synthetic reasoning task, 2023. URL https:/ ///axi/a...aaa org/abs/2220
abs 2206. 04301.
14

---

Overtrained Language Models Are Harder to Fine-Tune
A. Omitted Proofs from Section 4
A.1. Formal Definitions and Assumptions
We provide formal definitions and assumptions underlying the theoretical analysis in Section 4. Throughout the text, we will use constant 8 to express a small probability.
The model consists of a two-layer linear network parameterized by 0 Wi W2, where W1, W2 Rdxd The network maps input Rd output y Wi W2 E Rd.
Model Architecture
Pretraining Task The pretraining data follows y A where A Rdxd is matrix with singular value decomposition (SVD) Apre V Here, U, V E Igdxd are orthogonal matrices, and y E Idd is diagonal with positive entries (Eprod 1= arranged in decreasing order. Inputs a ~ (0, Ia) are standard Gaussian.
Pretraining Process
The model is trained via gradient flow on the population loss:
Lprce (0) = En [| - A A [ = 0 A
(4)
with parameters initialized as Wi( = W2(0) = exp( --p( with large T > The gradient flow dynamics follow:
Wi(t) -2(0(t) - A2(")
(5)
W2(1) (t) (0(t) APr))
(6)
where (t)) = Wi (t))).
This setup is inherited from Gidel et al. (2019), where the authors consider a more general setup with a rank-R matrix A and show that the gradient flow dynamics converge to the optimal rank-r approximation of A sequentially for r R.
Theorem A.1 (Theorem 1 of Gidel et al. (2019)). Suppose A has rank R. There exists t,,... l and constant depending on Ard such that for (t) following Equations (5) and (6),
[4:) U (Epre.) 1/2 exp((-rr); [[+((t)) - (EPre.) 1/2 VT || exp(-Cr).
where Spree shares the first diagonal elements as Eree and the rest diagonal elements are
Finetuning Task The finetuning task follows y A where A UD shares the singular vectors of pre but has spectrum f.. The input distribution remains (N(0
Finetuning Process Starting from (0) (t,,) in Theorem A.1, the model is fine-tuned using gradient descent with learning rate 17, batch size m, and K iterations. We will call (0) the real initialization and denote the following initialization (0) as the ideal initialization.
(T) U (2pre, 1/2
(7)
W (0) = 1222 VT
(8)
The population loss is:
(0) = Eg [[00x ax
+ 0 (0)) = | - -- + A 0r (0)
(9)
We will estimate Ca using a batch of samples B3 with size m on every step.
Ln B+)
Σ
[z - x + -
oE Br.
15

---

Overtrained Language Models Are Harder to Fine-Tune
Denote the covariance of in batch Bx
1 x..' T
Σ
Then,
(0; B&)= = Tr -10-A -T-((--A))C (0 -41)=/)) + 2/10 (0))
The parameter update rule at step ks is:
WT + 1)= WT (k) (0)) 1 (WT)
(10)
W + 1) -2 (k) (n(W(())) 2/ (k" A (0"())
(11)
where 0'(() = (T(
We will denote the final finetuned loss as (n) La (n (K)).
We will use I to denote the upper bound of pre and tt as,
r max
xpre max E iSd
(12)
A.2. Formal Statement and Proof Theorem 4.4
In this section, we consider perturbations of the weights with isotropic Gaussian noise. For parameter 0 Wi W2, we will consider perturbations of the form (W1 + aW2 + 3) where a, 3 Iddxd are independent isotropic Gaussian noise matrices with ozj, BBy (N0 ?2 for some y O. We will define the perturbed pretraining loss as,
Cpre (0) = (- [I(W1 + ( + 3) pre
(13)
Under this definition, assuming pretraining initialization is sufficiently small, we have that the loss under a Gaussian perturbation is monotonically increasing.
Assumption A.2 (Small Pretraining Initialization). satisfies that, for in Theorem A.1.
(2
exp(-Cr) & min pre /2,1//
16d (2EPT 72)
Theorem A.3. Under Assumption A.2, if
a. dd there exists some N and s < d such thatfor all the loss under Gaussian perturbation (0 (0)) is monotonically increasing.
Proof. Choose as the minimum number satisfying 7.2 > "ppre d, for n >> s, then S d - 1, by Lemma A.4,
Spro (0" (0)) Lpro (0 (0))
Erre
By Lemma A.6,
Gi (0") Lpro (0*(0) > Spre 2
/2.
Cpre @n (0)) Epre - gg-((()) > (Irre) )² /2.
Combining the above,
(oro - Cpre (0n= 0.
The proof is complete.
16

---

Overtrained Language Models Are Harder to Fine-Tune
Lemma A.4. The following inequality holds for any n >>
(0m~l(()) (0)) 2 (2d7² - E p Epre (0" (0)) Cpre (
-,
(14)
Proof. We first expand the loss,
pre 0 (0)
|| [[," ( 3) pre 1
[|:]]
(U +0a) (
(()prem)) 1/2 V + =3) -U' 122
Allreaa + () 14 U
U (S
(gpenm) 1/2 +a) (2 + =3) prr F)
(Ieren + a (Fprem)) + (Iprem) 1/2 B + aB I
2
Fr)
Epre,
Spre
||a (EPrem) 1/2 / + +E [ 1/2
[1[2]
+ +B ||000|
(15)
where the fourth equality arises from the isotropy of the Gaussian noise, and the final equality comes from the independence and zero mean of the noise distributions.
Lemma A.5. For Gaussian noise matrix a E iRdxd where each entries has variance 72 and fixed matrix M, it holds that
a [2] = 1
Proof. It holds that
M] o B || d>211
The proof is then completed.
By Lemma A.5 and Equation (15),
Cpre( = (opr((") + 2d>2 (S/re 1/2 1/ E [|a,91
Taking difference with Epro 1)
(pr) Cpre (fm-1) =2d
" n,n
(ere y2,
We then proceed to bound the difference between the perturbed loss of the ideal initialization and the perturbed loss of the real initialization when the pretraining initialization is sufficiently small.
Lemma A.6. Under Assumption A.2, for any > 0, it holds that
Sprce (0)) Epre (0" (0)) S (2
Proof. By the definition of Cpre.
Epre (0) = (m,
[[(W] a + A 10
[[( a) + B) 1|
=En,a-N(0,99)
-IW A 1 + E [ + El| W 31133]] B2Ilaw
17

---

Overtrained Language Models Are Harder to Fine-Ture
By Lemma A.5,
E W1 W2 A 1 1W,
A + dy (1IWil (122l?)).
Taking the difference between Cpre (o)) and Spre (0 (0)),
Spre (0 (0)) Erre O (0)) < W1 W A
W, W A
F
+ +
W
|Will
+ d
W2
|all|
By Theorem A.I,
W Willr sexp(-Cr)
W₂ W2/1 S (CT)
Here exp(-Cr) & min E/T /2, 1/4]. Therefore,
W |1
|| |2: 2|2Th( ((W, (W Wi)) + [[ Wil
<2exp(- ) exp(-2CT)) 4exp(( pre
Similarly,
W - 1| <(2Tr( (WW W2)) + W2 - W21?
(W2
<2exp(-Cr CCT) exp(- 2CC)) I 4exp( -
pre
Finally,
W.W, - 1 W W₂ 1
(W W2 - W W2) W, W. + W,W, 2/
Here
W, - W.W5)lls - W. -
W: Wellr + [ - W1 Wilre/Ws - Wellr < 22dexp(-Cr))zi + expp(-200 < 4Vd2p.o exp(-CT)
And
W W. + W, 2A FF W W2W, W W W₂ A |F
<2Vdexp(-Cr) (0xeep((2211) + 2v 2v < 4V A
Combining the above,
W.W- 1 [W, W. A 16d (ERr)² (xnr))2xxpppx)x CT).
Combining all the above, we have
(prr (0)) - (0 $ exp(-Cr)8dEp. (2 + 72) s (Epra) 2/22
The final inequality follows from Assumption A.2.
18

---

Overtrained Language Models Are Harder to Fine-Tune
A.3. Dynamic Analysis of Finetuning Process
Before we proceed to the main result of finetuning, we will first analyze the dynamic of the finetuning process in this section.
We will introduce two auxiliary dynamics to help us track the evolution of the finetuning process.
The first auxiliary dynamic (t) is named as Ideal initialization dynamic, which is defined as the dynamic starting from the ideal initialization (0) in Equations (7) and (8) with the same update rule Equations (10) and (11) and data order as the finetuning process.
The second auxiliary dynamic 0' is named as Ideal initialization with infinite batch size, which defined as the dynamic starting from the ideal initialization 0" (0) in Equations corresponds to the case when the batch size is infinite and >" converges to the identity matrix.
(7) and (8) with the update rule Equations (16) and (17), which
W (A: + WT (R:) (x)00( A (W (k)) -2n (k) (0)) WT (k+ + Wr (A) (b)) (k) A - 2nA(((())) (k"( -0)))000(0)) (0))
(16)
(17)
We will show the following results about these three dynamics:
1. Lemma A.7 provides analytical expression for the ideal initialization dynamic with infinite batch size.
2. Lemma A.17 shows that the ideal initialization dynamic with finite batch size is close the ideal initialization dynamic with infinite batch size, with error bound depending on the batch size.
3. Lemma A.19 shows that the real initialization dynamic is close to the ideal initialization dynamic, with error bound depending on the scale of pretraining initialization (which then controls the distance between the real initialization and the ideal initialization by Theorem A.
4. We conclude our analysis by providing our assumption for the main result of the paper Assumption A.21 and show that the finetuning process tracks the ideal initialization dynamic with infinite batch size closely and eventually approximately converges to the minimum (Lemmas A.22 and A.23).
Throughout this subsection, we will call Wi and W2 as well conditioned if W| llop S 2vT and W/2lloop S 2VT.
1.3.1. ANALYTICAL EXPRESSION FOR THE IDEAL INITIALIZATION DYNAMIC WITH INFINITE BATCH SIZE
We will introduce the following function to better track the evolution of weight in the ideal initialization dynamic with infinite batch size.
n, (,,,oo,) = x -nzooo -x²) + 2n - =²
x²²)
(18)
Lemma A.7. For the ideal initialization dynamic with infinite batch size in Equations (16) and (17), we have
(K) (K))
W (A) = (E)
where
(2* 1(i m)i (+) (2DDe))
prey 1/2
:n. A, (E 1/2
(>200) 1/2).
Proof. Consider
E}) (T)))
E}( WF(AVV
We then have
(k 1) = (1) (21100000000000000000000000-0110
(0)) 20) (b)
21))) (ET - ) - (k)) ( (k) - (0)))))))
Through induction, we can prove that (k) = Eÿ (k) are diagonal for all k: This then follows from the definition of f. LI
19

---

Overtrained Language Models Are Harder to Fine-Tune
This suggests that (k) and W (k) always well bounded by r. Assumption A.8. We have that learning rate y and regularization parameter are upper bounded,
4ny(A + 2)T 1.
Lemma A.9. Under Assumption A.8, for the ideal initialization dynamic with infinite batch size Equations (16) and (17), we have that
( S VT
( S V
with D being the upper bound of Pre and thth as defined in Equation (12).
Proof. This a direct consequence Lemmas A.7 and A.28.
Next, we will show that (U (K) will converge to a weighted combination of " and Ift for finites steps K. Assumption A.10 (Large Enough but Finite Steps). We have that the step size K min Eop Se log 1000 for some constant 0.
Lemma A.11. Under Assumption A.8 and Assumption A. for the ideal initialization dynamic with infinite batch size in Equations (16) and (17), we have that for any i S n,
ADR
IU 6 (K)))s
3,
S.
1 1+1
Proof. By Lemmas A.7 and A.28, we have that
pre
prr
2, + !Σ
(WT(K))
1,
S(1 2n min
pre -,,1))
1)
1+X
1 1+1 A
This then suggests that once
1001 pre
pre 1,
K
log
20 min pre
E 1,1
E
It then follows that
prre 2,2
A 6,
(WF(K))
1+λ
1/0
Similarly, we have the bound for (K))aa Combining the two bounds, the proof is complete.
A.3.2. CORRESPONDENCE BETWEEN IDEAL INITIALIZATION DYNAMIC WITH INFINITE BATCH SIZE AND FINITE BATCH SIZE
We then proceed to bound the difference between the ideal initialization dynamic with infinite batch size and the ideal initialization dynamic with finite batch size.
Lemma A.12 (4.7.3 of (Vershynin, 2018)). For a fixed k, there exists a constant C₁ with probability 8, we have that when batch size m d + log(1/8),
|>{{] Lallop < Ci
+
log(1/8)
m
Assumption A.13 (Large Batch Size), We have that for constant C₁ defined in Lemma A.12 and e > 0. m 2 Cf(d 10g(100)))/e.
20

---

Overtrained Language Models Are Harder to Fine-Ture
Lemma A.14. Under Assumption A.13, for the ideal initialization dynamic with infinite batch size in Equations (16) and (17), we have that
vk < K, [2 - Iallop Se
with probability 6.
Proof. This is direct consequence of Lemma A.12 and Assumption A.13.
Lemma A.15. When the event defined in Assumption A.13 happens, for any k. S K, for the same well-conditioned
parameter 0(k) and (()), if applying the update rule Equations (16) and (17) yield 0 + 1) and applying the update
rule Equations (10) and (11) yield 0(k 1), then the difference between O(k: + 1) and 0 & 1) bounded by
W k 1) Wi(k + o) S 32me1
|| + W2(k: + 1)p S 32m/TI/22
Proof. Taking the difference between the two update rules, we have that
[| + 1) Wi (k + 1)p 2nl A (2(() Ia) W2(k) 'lop
s 2n 2nk A !lop |2((11 Iallop (2(k)
S 0(k)
A llope W2(k)llop
3/m
Similarly we can have the bound for [| + 1) Wa(k + 1)
Lemma A.16. When the event defined in Assumption A.13 happens, for the ideal initialization dynamic with infinite butch size in Equations (16) and (17), consider two different well-conditioned parameters O(k) and O (k) with the same initialization (0), denote Eμ max[ Wi(k))lop. (22(k) W2(k)]lopp: we have that
cx+1 $ (1 16m11')x..
Proof. Define A target A
Given the update rule, we have that
( (' + 1) (Wi(k) ((()) 2n [(0(k) A Larget W2(k) (0' A target W2(k)
We only need to properly bound the second term,
[k)(k) A target
W2(k) - (0'(k) A target
Ww'(k) "] 1lop
<||0(k) 0' (k) op Wa ooo + [10(k) target
loo W2(k) Wo(k)llop
The difference between (k) and ('kk) is bounded by []²(k) - 0' llop W1 (A)) Wi(
(2))k) [ [W]( W2(k) Ws(k))lop 4VTER.
Therefore, have that
|l (O(k) target
W2(k) - -('(()) A target [ 1lop S 16
We then concludes that This then concludes the proof.
G411 (1 + 161T)ck.
21

---

Overtrained Language Models Are Harder to Fine-Tune
Lemma A.17. When the event defined in Lemma A.14 happens for e 1(+ define the error between the ideal initialization dynamic with infinite batch size and the ideal initialization dynamic with finite batch size as EK = max(l]Wa(k)) (k)oollop: W2(k) W2(kk)loph: then we have that
ER 2(1 + 16n1' < 12
Proof. From Lemma A.9, we have that is well-conditioned, 0 is well-conditioned, combining Lemmas A.15 and A.16, we have that
Ek+1 S (1 +67nT)s 32mc13
Now we can inductively prove that for & E [0, K].
c S ((1 + 16nl1)* - 1) 2el
Given that < 11+166 we have that
EK < 14/2/4
This then concludes the proof.
A.3.3. ERROR INCURS BY DIFFERENT INITIALIZATION
Finally, will show that the real initialization dynamic is close the ideal initialization dynamic, with error bound depending on the scale of pretraining initialization (which then controls the distance between the real initialization and the ideal initialization by Theorem A.I).
Lemma A.18. When the event defined in Lemma A.14 happens for < 1(1+1611yx the ideal initialization dynamic with finite batch size in Equations (10) and (11), consider two different well-conditioeed parameters O(k) and O' (k) with the same initialization (0), denote e ma - W[(k))llpp: [[2(()) Wk)ppl we have that
ck+1 $ 32nD)).c.
Proof. The proof is similar Lemma A.16 and is omitted here.
Lemma A.19. When the event defined in Lemma A.14 happens for 4[1432g1]z consider two finetuning processes, with (t) starts from the real initialization (n) in Theorem A.1 and (t) starts from the ideal initialization ((n) in Equations (7) and (8). Then the two processes are close to each other for all k I K.
(k) (k) I (1 +1+ 32 exp(-Ct). (k) (k) S (1+ 32m exp(-Cr).
Proof. Define e max(1 WT()) (W [.) By Lemma A.17, 0 is well-conditioned, if well-conditioned, combining Lemma A.18, we have that
kt S +22n))EE.
This then suggests that
(1 32 3 exp(-Cr)
This then concludes the proof.
22

---

Overtrained Language Models Are Harder to Fine-Tune
A.3.4. COMBING TWO APPROXIMATIONS
Lemma A.20. Under Assumption A.8 and Assumption Assumption for II 1(1+1697) with probability 1 ò, we have that both
(k) and W (k) are well-conditioned and
[K] (K) S (1+ 31,)))"")
+ + +
(k) (k) S (1 11++-21)) 32nl)* exp(- + 2(1 + 167
Proof. This a direct consequence of Lemmas A.14, A.17 and A.19.
Given this lemma, we now present our main assumption and corresponding bound under this assumption.
Technical Assumptions. We will make the following technical assumptions to simplify the analysis. Assumption A.21. We will make the following assumption to control the regularity of training. For arbitrary constant Ag. for
min sal DDre D 2 p,n (Ao + 1)222
4000d
1. Finite regularization force: 0 S X λp-
2. (Assumption A.8) Finetuning learning rate bounded:
4ni(Ao 2) <
3. (Assumption A.10) The finite number of step K
log 1000
min Dop Stt
4. (Assumption A.13) Large enough batch size m,
Cd log(10dK))) (1 + 32,y))
m
for C1 defined Lemma A. 12.
5. Small enough initialization error exp(-Cr) <
K for defined in Theorem A.1.
We will first show this important lemma that the distance between the real initialization and the ideal initialization is bounded under Assumption A.21.
Lemma A.22. Under Assumption A.21, with probability - 8, we have that for every n S and S K,
min:gn (1 (1Epr E 12]
1 0" (A)]lE
1000(Ao + 1)
Proof. This is a consequence of Lemma A.20. However, to go from the operator norm bound on (k) and WT(k) to the Frobenius norm bound on 8th (k). we need the following two inequalities. The first one provides an operator norm bound on the difference between 0t (k:) and (k:),
10
0
+ [[ < 41 (I] (k) (A) + (k) - W (k))lop).
(W))
(k)llpp
The second one uses this operator norm bound bound the Frobenius norm of the difference between (*(k) and Qt (k).
100T (k)]l= & d 0t llop
Combining these two inequalities with Assumption A.21, we get the desired result.
23

---

Overtrained Language Models Are Harder to Fine-Tune
We continue to show that the finetunig process approximately converges the minimum.
Lemma A.23. Under Assumption A.21, with probability 1 8, we have that for every d,
Dit 2m
+ A
min.gn - -
IU 0t (K)V-
1 1+1 X
500(Ao + 1)2T
Proof. This a consequence of Lemmas A.11 and A.22.
A.4. Formal Statement and Proof of Theorem 4.6
Theorem A.24. Under Assumption A.21, with probability 1 1-6 6, For Apre(n) = (0 - Cpre(( and A (n) does not decrease with n.
4 pre 2
Proof. We will first provide a tight bound for (n). By Lemma A.22, we have that
minega [ 1)
110 (0)))=(0)]l
100(Ao + 1)2r
and by Lemma A.23, have that
ftt mn
+
A
minggn ( [ - 12]
1U (")
mn
1F S
1+
50(Au + 1
This suggest that
(prre(0 (0) Spre (0 (0)(
= (10 APM 1 110 - A 1
(0) (0)] 10000 (0) 2.4 ||op
< 3201 (0) - (0)l
minigal 1)
10(Ao + 1)
Similarly, we have that
D + AEpre
minsga DEpe Σtt (2]
(O)) Lpre (U
in vT)
1+ A
5(A0 + 1)
Combining these two inequalities, we have that
An - (Apoelu ftt mmn. + AEpre
vT) - Lpee Epre 1T))
3 3inn ( (12,, =2
1
10(Ao + 1)2
Meanwhile, we have that
S mn,n Cpee UU
+
pre nn VT)
(UEe VT) ( mnin
Dit pre 1,2
1,
Dpre)
2
1+ A
1
i=1
f i
Pre
1+ X
im
Therefore if we additionally define Ao = 0, we have that for 1 S n S d.
Erre
Σft m,n) )2
3 3min (1E1re Ett (2)
A
> 0.
(1 + A)2
5(Ao + 1)
This completes the proof.
24

---

Overtrained Language Models Are Harder to Fine-Tune
A.5. Formal Statement and Proof of Theorem 4.7
Theorem A.25. 1. Under Assumption A.21, when À with probability 1 if A and A are (4, (4isaiiiil (6" (K)) for S r.
then (0
2. Define the inflection point T' as the smallest value of r for which the pre-training loss E (n) increases monotonically for every n > r. Assume that regularization strength X X2 > 0 yields iterates O1 and Oz. Assumption A.21 holds for
minasd p. 1
Elt n.
}
min {
1 (T+ 22)a
A (1+
(1+ 12)² )
X
}.
E
4000d
(1+
then with probability 6, we have that rx2 S T)2 and the unregularized finetuning loss [(0((K) A ||? > [0% A for every
Proof. This the combination of Lemmas A.26 and A.27.
Lemma A.26. Under Assumption A.21, if 2 DEpe and À then Cpre (0 (O)) Spre (0n-1(K)) O.
Proof. With the same argument in Theorem A.25, we have that
mingen D2n 21
prr O0 (K)) pro (U ft mn VT)
5
Noted that
Cpee (U2., V²) Lpee ( (U in 1,m-1 V2) = =2#,,
(22
We further have that Et
Epre h,n
22Drr Therefore,
R,n
Cpre (0))))) (p (K))
2(Epre )
(UIft 2 Spe (U in,. VT)- - Spce (U -2 1
5 > 0.
This completes the proof.
Lemma A.27. Assume that regularization strength A x2 0 yields iterates 01 and 02, if Assumption A.21 holds for
<< 1 min 12 ftt
[2]
min {
(a+A)) (1+ x2)2 }
X7
X3
(1 x2)2 (1+x1))
4000d
then with probability we have that rX S Thg and the unregularized fineturing loss (00(KK - A > (00((K)
A 1 for every n.
Proof. Following similar proof as in Lemma A.23, we have that with probability 1-6 8,
[K) - U ft n. + A1 pre n.
min ( D
E
VT
1 1+1 A1
500r
(1+ (1+
(1+ A1
and
|0 (K) - 1 in.in + A2 Dpre
Sift
min:gn { -i
1
VT
1 + X2
500r
(1+
(1 + x1
This then implies that
(K)) A 12
ft
+ >1
pre
m..n
pree <
min:gn
ft 12] 4,4
1
1
1 + A1
50
(1 + A2)²
(1+ + A1
25

---

Overtrained Language Models Are Harder to Fine-Tune
Similar bound holds for ||0|
A
Combining these two inequalities, we have that
(1 (10 A 11?
1|
|102 10m-1 (K)
A 11=) - (K)
1
|
-1000-1
11?)
Σ
A2 Drre
2
mnn
+ A1 Apre
min.5n
Σ 23
Drre
Frr
1 A2
1 A1
25
(1+ x2)²
(1 11)2 (1)
min:gn ( (1
13
n.n 11?
>0.
(1 + 22)
(1+ 11)²
25
This then suggests that (K) showing that rAy Tag- Using similar argument, show that the unregularized finetuning loss [[0P(K) A A (K) A A
1 > 1105- (K)
1| when [0]
1 > 1090-((3 1 I.:
|1
1 P for every
A.6. Technical Lemmas
In this section, will first prove some of the technical lemmas function / defined in Equation (18). Recall that is defined
f(x; n, (, = x 2nzolo² x²) + 2n120(00 x²).
Lemma A.28. Vo > 0, E N. When (A + 2)m (2max 2 + x < 1, define o² f(l) (00; σo) in [min(o, oo), max o.)l. and
6 t
it holds that
1+A
(ov: o'1 [ 21 min oz) o¹1
Proof. Let det A) x(1² x(1²-o0) 0²) + Ax(x² -oox Then gheng(o"; We have that
f(000,,,0000) = -2ng(=; 0o. A).
For any a E [min oo), max
,12:0,00, A) x(2² o²) + xx(x=- oî) =x(x o²)(xx (A+ l)o").
(x;,,,,,,000)) a² a² 2n x: 0, ~o. A) + 2ng (a ; 0o
= (r a² (11 2z + (2 + 1)
When x [min(aroo) oo), max(o, ootl x(x + (A+ + 1)o*) 2 min On the other hand
x(x + (A + 1) < (A 4 2) max 20- ...)
This suggest that
1 - 21r(2 + (A+ 1)o") > 0.
Therefore,
ff. - 001 1 o² (1 2n min
Also f(x; n, A, 0o) o² has the same sign as r. a². This concludes the proof.
26

---

Overtrained Language Models Are Harder to Fine-Tune
B. Extended Related Work
Here we present an expanded and extended discussion of the related work.
Loss of plasticity. The idea that more training can be harmful performance has been studied before in other continual learning settings. Named loss of plasticity, this phenomenon refers to the degradation of the ability for model to adapt to new task. This has mainly been studied in the context of training on small models with small datasets (Ash & Adams, 2020; Dohare et al., 2021) or reinforcement learning (Kumar et al., 2020; Lyle et al., 2022: 2023; Ma et al., 2023; Abbas et al., 2023). Loss of plasticity has been attributed to the loss curvature (Lyle et al., 2023; Lewandowski et al., 2023), increased weight norm (Nikishin et al., 2022), feature rank (Kumar et al., 2020; Gulcehre et al., 2022), and feature inactivity (Lyle et al., 2022; Dohare et al., 2021). Multiple remedies have been proposed, including changes to the neural network architecture (Lyle et al., 2023), resetting model parameters (Nikishin et al., 2024; D'Oro et al., 2022), and regularization (Kumar et al., 2023; Ash & Adams, 2020).
While prior work focused reinforeement learning small-scale, synthetic setups, our work considers the large-scale autoregressive language modeling setting. Unlike prior work, where pre-training is often harmful for the downstream fine-tuning task, we show that overtraining on generic web data can also degrade fine-tuning performance despite being expected to help. Additionally, we highlight an increased sensitivity to degradation of the pre-training loss that arises with overtraining, an aspect largely overlooked in the literature.
Catastrophic forgetting. The phenomenon of catastrophic forgetling- neural networks trained sequentially on tasks tend to forget prior tasks-has also been well-documented in the literature (Kirkpatrick et al., 2017; French, 1999; Goodfellow et al., 2013; Kemker et al., 2018; Kotha et al., 2023). There have been several proposed mitigation strategies, for example, Ahn et al. (2019): Hou et al. (2018); Chaudhry et al. (2019a) propose using regularization to mitigate catastrophic forgetting. Other fixes include generative replay of examples from previous tasks (Shin et al., 2017) or maintaining a memory buffer of previous tasks (Chaudhry et al., 2019b; de Masson d' Autume et al., 2019). In this work, we show that catastrophic forgetting can become more severe with overtraining.
Relationship between pre-training loss and downstream performance. In our work, we argue that the degradation of the pre-training loss and the downstream loss may be related. Several works have tried to study the relationship between the pre-training loss in language models and their downstream performance. Liu et al. (2022) analyze the effect of pre-training beyond convergence and suggest that overtrained models exhibit better transfer to downstream tasks. Our work considers web-scale pre-training, which rarely converges in practice, so these findings do not contradict ours. Similarly, Tay et al. (2022); Zhang et al. (2023) highlight the effect of architecture on downstream generalization, given the same pretraining loss.
Scaling laws for optimal pre-training. In our work, we argue that training for fewer tokens can be beneficial for downstream performance after fine-tuning. Related to our work, Isik et al. (2024) proposes scaling laws for certain downstream translation tasks after fine-tuning, but does not observe degradation with overtraining. In addition, the optimal pre-training token budget has also been studied in other contexts. Notably, Kaplan et al. (2020); Hoffmaan et al. (2022) demonstrate that, given a fixed compute budget, there exists an optimal token budget for each model size. Subsequent works have extended scaling laws to broader contexts, including transfer learning, contrastive training, training under data constraints, and predicting performance from factors other than pre-training tokens (Hernandez et al., 2021; Cherti et al., 2023; Muennighoff et al., 2023; Goyal et al., 2024; Liu al., 2025; Bhagia et al., 2024). However, scaling laws are not always optimal for predicting performance. Diaz& & Madaio (2024) argue that existing scaling laws do not always predict downstream performance accurately. In addition, multiple works have observed U-shaped trends in performance as models scale (Caballero et al., 2022; Wei et al., 2022; McKenzie et al., 2022a).
To reduce inference cost, practitioners have turned to developing capable small models, which often requires overtraining beyond the compute-optimal token budget. In fact, Sardana et al. (2024) show that pre-training loss continues to decrease when trained for up to 10,000 tokens per parameter. Gadre et al. (2024) validated similar observations and propose scaling laws predict the model performance in this overtraining regime.
Transfer learning theory Finally, our theoretical analysis of catastrophic overtraining adopts classical transfer learning setup based on deep linear networks (Gidel et al., 2019; Saxe et al., 2018). Wei et al. (2024); Arora et al. (2018) use this setup to study how models learn and store knowledge. Another group of studies explain how transfer learning can improve performance after pre-training (Saunshi al., 2021; Wei et al., 2021; Shachaf et al., 2021). Chua et al. (2021); Wu et al.
27

---

Overtrained Language Models Are Harder to Fine-Tune
(2020); Tripuraneni al. (2020) specifically adopt similar deep linear network setting study feature learning during pre-training, and how these learned features benefit downstream tasks. Kumar et al. (2022) explores how fine-tuning can lead to degradation of out-of-distribution performance.
C. Experimental Details from Section 2: Large Model Experiments
In this section, we present all of the omitted experimental details from Section that are necessarily for replication.
C.1. Pre-trained models.
For our pre-trained models, we use checkpoints from three base models: OLMo-IB (Groeneveld et al., 2024b), OLMo-2-7B (OLMo et al., 2024), and LLM360-Amin (Liu et al., 2023b). choose checkpoints that have been released on each the model's HuggingFace pages, given by Table
Model
HuggingFace ID
Revision
Step
Token Budget
OLMo-IB
al alLenai/O1Mo-1B-1
step10000-tokens413
10k
0.04T
step117850-tooenss998
118k
0.5T
step3800--toeen115000
358k
1.5T
step44700-ttoeess1887BB
447k
1.9T
stop561255-tokees2555B
561k
2.4T
step738800-tokkes30998
738k
3.IT
OLMo-2-7B
allonai/0LMo-2-1222
stapeilsstep11000-tcccessgg
19k
0.08T
stagel=sstep2220000-cckkss3333
120k
0.5T
staggelsstee-411000-tttoees11113
441k
1.9T
584k
stagel
[taggll=sttgp2220000ttteemm33
727k
3.IT
stagll-steg5284444ttkee----333
929k
3.9T
LLM360-Amher (7B)
LLM360/ 1.M3360AAmber Amber
ckpt 040
40
0.12T
ckpt_102
102
0.31T
ckpt.244
244
0.75T
ckpt.306
306
0.94T
ckpt.358
358
1.IT
ckpt_410
410
1.3T
Table 1. Pre-trained models used in our experiments Section 2.
C.2. Fine-tuning setup.
We fine-tune with two different common post-training paradigms: instruction tuning and multimodal tuning. For instruction tuning, use the following datasets.
Anthropic-Hie (Bai et al., 2022). While Anthropic-HH is typically dataset designed for preference tuning the dataset includes both a "chosen" and a "rejected" response for each instruction- it can also be used as a standard instruction tuning dataset by treating the "chosen" response as the target. Anthropic-HH contains 180k instructions and responses.
TULU (Wang et al., 2023). We use version of the TULU SFT mixture, which contains 490k instructions and responses. However, for compute efficiency, we only use a randomly selected 200k subset.
LLaVA (Liu al., 2023a). We use the LLaVA visual instruction tuning framework to train multimodel models. The LLaVA framework involves two stages: first, fine-tuning adapter between vision model and pre-trained language model, and then fine-tuning the entire model to follow instructions in the presence of images.
When fine- mhe-ttning for instruction tuning, we use the standard SFT training algorithm with the following hyperparameters, as shown in Table 2. this table, we also present the hyperparameters we use with the LLaVA framework, using the defaults for all non-specified hyperparameters.
28

---

Overtrained Language Models Are Harder to Fine-Tune
Dataset
Batch size
Learning rates
Learning rate
Warmup
Optimizer
Weight
schedule
steps
decay
Anthropic-HH
256
1e-6, 5e-6 6, 1e-5, 5e-5,
Cosine
20
Adam W
8e-5, 1e-4, 2e-4
Alpaca
256
1e-6, 5e-6, le-5, 5e-5,
Cosine
20
AdamW
0
8c-5, 1c-4, 2c-4
TULU
256
1e-6, 5e-6, 1e-5, 5e-5,
Cosine
20
AdamW
0
8e-5, 1e-4, 2e-4
Visual (LLaVa)
256
1e-3
Cosine
50
AdamW
0
Stage 1 (Projec- tor training)
Visual (LLaVa) Stage (Inst.
256
8e-6, 1e-5, 2e-5, 4e-5, 1e-4
Cosine
40
Adam
tuning)
Table 2. Hyperparameters used for instruction tuning and LLaVA.
C.3. Evaluations
We evaluate the fine-tuned models in two settings: downstream evaluations- tasks that is representative of the goal of fine-tuning-ande generalist evaluations-taaks that are representative of the model's overall language understanding and inference capabilities. For downstream evaluations,
the following datasets.
AlpacaEval (Li et al., 2023b). To evaluate the downstream performance of instruction-tume models, we use AlpacaEval, a benchmark for evaluating the quality of a model's response to an instruction. The AlpacaEval benchmark contains 20k instructions, and measures the win-rate of the fine-tuned model against reference model. By default, AlpacaEval reports win-rate vs GPT-4 responses. However, we evaluate models that are weak by comparison to GPT-4. If we compare against GPT-4, the win rate so low that is difficult to see the differences between models. Thus, we compare against a weaker model. In particular, for each of models, use the model trained with seed with learning rate 10-5 This means that the on the same dataset. More specifically,
reference model of the same architecture that also fine-tuned
AlpacaEval scores are not comparable across different graphs, as the reference generations are different for each model and dataset. Additionally, the AlpacaEval score of the model trained with seed 0 and learning rate 10-5 is 50% by definition. Overall, we adopt these choices to ensure that the reference generations are comparable to each model output. We use LLaMA-3-70B-Insttuee (Grattafiori et al., 2024) as an evaluator to determine the win rate.
VLM Score. To evaluate the downstream performance of our LLaVA models, we use an average of the following five standard vision-language benchmarks: MME (Fu et al., 2024), GQA (Hudson & Manning, 2019), A12D (Kembhavi et al., 2016), POPE (Li et al., 2023c), and TextVQA (Singh et al., 2019), We report the average as the "VLM score".
Generalist evaluations. To evaluate each language model for generalist capabilities, we consider suite of ten commonly used LLM evaluation benchmarks. These tasks assess performance beyond the fine-tuning task. These tasks cover reasoning CARC_Challene and ARC (Clark al., 2018)), commonsense (PIQA (Bisk et al., 2020), Winogrande (Sakaguchi al., 2021)), natural language inference (BoolQ (Clark et al., 2019), COPA, SCIQ) and sentence completion (HellaSwag). For all of our evaluations, we report 5-shot performance.
D. Experimental Details from Section 3: Controlled Experiments
In this section, we provide additional experimental details for the controlled experiments presented in Section 3.
D.1. Pre-training and fine-tuning setup.
For our controlled experiments, we pre-train models using the OLMo codebase (Groeneveld et al., 2024b). We use muP parameterization for all of our experiments (Yang et al., 2022).
Pre-training, We train three different model classes: OLMo- 15M. 9LMM-30M, and OLMo-90M with 15M, 30M and 90M non-embedding parameters, respectively. We use the following hyperparameters for pre-training, as shown in Table 3. For
29

---

Overtrained Language Models Are Harder to Fine-Tune
each model, we train for tokens in the range 4B, 8B, 16B, 32B, 64B, 128B using the pre-tokenized C4 "high quality" web data distributed by OLMo (OLMo al., 2024).
We train with 8xA100 GPUs.
Hyperparameters
OLMo-15M
OLMo-30M
OLMo-90M
Layers
Heads
Number of unique tokens
50304
50304
50304
Hidden dimensions
192
384
576
Inner MLP dimensions
768
1536
2304
Max context length
1024
1024
1024
Activation type
SwiGLU
SwiGLU
SwiGLU
Attention dropout
0.1
0.1
0.1
Residual dropout
0.1
0.1
0.1
Embedding dropout
0.1
0.1
0.1
Optimizer
AdamW
AdamW
Adam W
Learning rate
0.0003
0.0003
0.0003
Betal
0.9
0.9
0.9
Beta2
0.95
0.95
0.95
Learning rate scheduler
Cosine
Cosine
Cosine training
Warmup steps
10% training
10% training
10%
Weight decay
0.1
0.1
0.1
Batch size
256
256
256
Table 3. Pre-training hyperparameters used in controlled experiments.
For each model, we anneal the learning rate to zero over the course of training, at the rate specified by the cosine learning rate scheduler.
Fine-tuning. For each our controlled experiments, we fine-tune the pre-trained models on a series of downstream tasks two types: classification and language modeling. These ten datasets are: classification- SUBJ (Pang Lee, 2004), BoolQ (Clark et al., 2019), MR (Conneau Kiela, 2018), CR (Conneau & Kiela, 2018), RTE (Dagan et al.. 2005), TREC (Voorhees & Tice, 2000), English Tweet sentiment (Maggie al., 2020), SIQA (Sap al., 2019), and language (Li et al., 2023a). For Starcoder-Python,
modeling- GSM8K (Cobbe al., 2021), Starcoder-Pytthe subset. To avoid confusion, note that despite the fact that GSM8k is often evaluated as a math reasoning benchmark, we treat it as language modeling task to evaluate how well the models can learn math-style text. We use the following hyperparameters for fine-tuning, as shown in Table 4.
use a 5k example
Hyperparameters
Values
Learning rate
4e-6, 8c-6, le-5, 20-5, 4e-5, 5e-5, 6e-5, 7e-5, 8e-5, 9c-5, le-4, 1.1e-4, 1.2c-4, 1.4c-4, 1.6e-4, 1.8c-4, 2c-4, 2.4e-4, 4e-4, 5e-4, 6e-4, 8e-4, le-3, 2e-3, 3e-3, 4e-3, 6e-3
Batch size Learning scheduler
32, 64", 256
Cosine Constant
Optimizer
AdamW
Weight decay
0.0
Warmup steps
10% of training
Epochs
4
Table Fine-tuning hyperparameters used our controlled experiments. tune over all specified learning rates. For other hyperparameters, when multiple are specified, the asterisks (*) indicates the default value which is used unless different hyperparameter specified. We perform carly stopping over the number of epochs.
Evaluation. For tuning, we use heldout validation set from each dataset, but report scores on separate heldout test set. In order to compute the perplexity for classification tasks, compute score for each class by measuring the length-normalized likelihood of the class, and then report the perplexity over the classes. For generative tasks, we use the standard language modeling loss. As measure of generalist capability, we report the perplexity on heldout C4 web data set.
Appropriate learning rate ranges for Figure 5. For visualization purposes, we choose to plot a subset of the learning rates which we evaluate in Figure In particular, we plot learning rates where the maximum pre-training perplexity, over all
30

---

Overtrained Language Models Are Harder to Fine-Ture
subj
siqa
g5m8k
starcoder-python-Sk
Ditancee (L2 norm)
1x
Distanze (L. norm)
3 10
Distance (L2 norm)
(L2 norm)
Pre Uia 101 JAROTS
10"
10"
10
10a
2 100
6
10"
615
10a
4 10
1255
4 10"
10
10"
10
x
6 10
1.20e-04 Learning rate
6 10
1.20p-04
x 10
1.20c-04
6 10
1.20a-04
Learning rate
Learning rate
Learning rate
Figure 8. Distance, as measured by L. norm, between the pre-trained and fine-tuned model as function of learning rate for OLMo- 30M. More specifically, Oye and 0n are the parameters of the pre-trained fine-tuned models, respectively, we plot ||0ge 0a |a as function of the learning rate. We observe that the distance between the pre-trained and fine-tuned model is not exactly, but approximately, directly proportional to the learning rate and independent the amount pre-training.
token budgets, is less than 6. This ensures that the learning rates plot are in range where the model is still retaining high perplexity which may not represent the more general case.
pre-training capability, and has not degenerated
Using learning rate as proxy for a fixed perturbation size. We report the distance between the pre-trained and fine-tuner model as function of the learning rate for different token budgets in Figure 8. Recall, from Section 3, that we specified that the learning rate is an approximate proxy for the size of the perturbation applied to the model. We observe that the distance between the pre-trained and fine-tuned model is not exactly, but approximately, directly proportional the learning rate and independent of the amount of pre-training.
D.2. Gaussian perturbations.
In this subsection, we outline the details concerning Gaussian perturbations applied during our experiments. particular, we perturb each parameter by random value sampled from a mnenn-zero Gaussian distribution and evaluate the degradation of pre-training perplexity in Section 3. Using isotropic Gaussian perturbation, i.e., perturbing each parameter by the same amount, would discount differences in parameter magnitude across different layers. To account for this, choose to scale the perturbation to each layer to be approximately proportional to the magnitude of the parameter in that hayerehowever, we want the magnitude to be constant for different pre- pre-training token budgets. Thus, we choose to normalize the magnitude of each perturbation to the same magnitude as the layer at initialization prior to pre-training.
E. Connection Between Progressive Sensitivity and Sharpness
In this section, discuss the connection between progressive sensitivity conjecture and the phenomenon known as progressive sharpening (Cohen et al., 2021) in greater detail.
Progressive sharpening. This phenomenon refers to the empirical observation that over training with a fixed learning rate, the spectral norm [|V) of the Hessian of the loss function L at the parameters 9 increases over time, at least early in training. In the case of of (full batch) gradient descent with a fixed learning rate 7, [[² specifically increases until reaches 2/m, which is discussed in detail in Cohen et al. (2021). In addition the spectral norm, other norms of the Hessian, such the trace norm, also exhibit similar behavior.
Relationship between progressive sensitivity and progressive sharpening when loss is quadratic. As it turns out, progressive sensitivity and progressive sharpening are closely related specifically in the quadratic setting. In particular, consider a quadratic loss function L())- = 10 Hθ + g 0 c, where E Rd, H E Rddd a symmetric matrix, E TRd and c E TR. We will look specifically at the sensitivity to Gaussian perturbation (0, A) = [C + Xε) - [(0)), where E ~ (O0 I) a unit Gaussian vector.
Proposition F.1. The sensitivity of L. to a Gaussian perturbation given by A(0 1T H.
31

---

Overtrained Language Models Are Harder to Fine-
328
328 hessian аpprox
328
hessian approx
3.74
648
spprox
648
approx
Pre- t--iirriinngg perplexity
1288
1286 hesslan approx
Pre-training perplexity
4.6
1286
1288 hessian approx
3.72
4.4
3.70
4.2
3.68
4.0
3.66
3.8
3.64
3.6
0.00
0.02
0.04
0.06
0.08
0.10
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Perturbation strength (A)
Perturbation strength (A)
Figure 9. Hessian approximation of the pre-training loss under a single interpolated Gaussian parameter perturbation. We randomly draw Gaussian perturbation and then compute the loss L(0 + As), where À is the scaling factor, for many different A (extremely close to zero on the left, and with wider range on the right). We then compute Hessian, and use it to render the quadratic approximation of the loss.
Proof. We have,
1 (0
E [C((0) - L(O Xε)]
((0 + Xε)
Ae)TH(()
H(0 Xe) + T Ao) + 2
-0 Hθ + g + +c)
(19)
E
X2E He
1² Tr
(20)
where the second equality follows from the linearity of expectation and the fact that R[e] =
This proposition establishes that the sensitivity under a Gaussian perturbation exactly related to the Hessian when the loss function is quadratic. This connection will hold, in general, when the loss function is well-approximated by its second-order Taylor expansion, such when À is small. In this instance, progressive sharpening and progressive sensitivity closely related.
Progressive sharpness is not sufficient to explain degradation when À is large. We plot the empirical loss of three different OLMo-30B models (trained on 32B, 64B, and 128B tokens) under a Gaussian perturbation with perturbation strength X, as well as the second-order Taylor approximation in Figure In particular, we draw the perturbation with the distribution described in Appendix D.2. We observe that while the loss apll-approximated by the Hessian when À is small (left), the approximation breaks down when À large (right), and the actual loss is substantially higher than the quadratic approximation.
Progressive sharpness is not sufficient explanation for fine-tuning sensitivity. Similar to the Gaussian case, we consider the loss of three BLMo-30B models as they are interpolated between the base model and the model fine-tuned ag news in Figure 10. In this example, perturbation strength of X = 0 corresponds to the base model, while a perturbation strength of À corresponds to the fine-tuned model. Similar to the Gaussian case, we observe that the loss not cell-approximated by the Hessian when À is large, and the actual loss is substantially higher than the quadratic approximation (right).
Progressive sensitivity generalization of progressive sharpness. Our results highlight that in addition to progressive sharpness, which specifically refers to progressive increase in the eigenvalues of the Hessian of the loss function with training, there is more global phenomenon where the loss becomes even more sensitive to perturbations than the quadratic approximation predicts.
32

---

Overtrained Language Models Are Harder to Fine-
3.74
10
328
648
Pre-training perplexity
Pre-training perplexity
128B
1288 hew.l-heessnnn-n approx
3.72
3.70
3.68
3.66
4
0.000
0.025 0.050 0.075 0.100 0.125 0.150 Fine-tuning perturbation ratio
0.0
0.2
0.4
0.6
0.8
1.0
Fine-tuning perturbation ratio
Figure 10. Hessian approximation of the pre-training loss under interpolated fine-tuning perturbation. We fine-tune each model on news yielding fine-tuning perturbation e, and then compute the loss L(0 + λε), where is the scaling factor, for many different X (extremely close zero the left, and with wider range on the right). We then compute Hessian, and render the quadratic approximation of the loss.
F. Omitted Figures from Section 2: Large Model Experiments
In this section, we provide the omitted figures from Section that show the results of the extended experiments with large models.
The following Table 5 lists the table of contents for the omitted figures.
Dataset (Variant)
OLMo-1B 1B
OLMo-2-7B
LLM360-7B
Anthropic-Hie (tuned learning rate)
Figure 11
Figure 13
Figure 15
Anthropic-HH (all learning rates)
Figure 12
Figure 14
Figure 16
TULU (tuned learning rate)
Figure 17
Figure 19
Figure 21
TULU (all learning rates)
Figure 18
Figure 20
Figure 22
VLM (tuned learning rate)
Figure 23
Figure
Figure 27
VLM (all learning rates)
Figure 24
Figure 26
Figure 28
Table 5. Figure references for each dataset (Alpaca, Anthropic-Hii HH, TULU, VLM) and model (OLMo-IB, OLMo-2-7B. LLM360-7B).
separated by learning rate tuning variant.
33

---

Overtrained Language Models Are Harder to Fine-Tune
AlpacaEval
arc challenge
arc easy
piqa
55
35
65.0
76
Score
50
62.5
75
30
60.0
74
boolq
winogrande
hellaswag
sciq
60
60
62.5
Score
92
60.0
58
50
57.5
90
2
3
2
3
1
2
3
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Fine-tuned model
Figure 11. Evaluation OLMo-1B post-trained on Anthropic-HH as function the number of pre-trained tokens, with tuned learning rates. We report the
eight different datasets: AlpacaEval is considered be the main evaluation of interest (corresponding with the downstream performance). and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We the intermediate checkpoints from Table for the evaluation. We tune the learning for each checkpoint to maximize the main evaluation (AlpacaEval). This figure is analogous to Figure 2.
34

---

Overtrained Language Models Are Harder to Fine-Tune
Evaa
arc challenge
arc easy
piqa
35
75
50
60
Score
30
70
40
50
65
25
40
boolq
winogrande
hellaswag
sciq
60
60
60
90
Score
80
55
50
50
70
40
3
2
3
2
3
1
2
3
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Minimum learning rate
Maximum learning rate
Figure 12, Evaluation OLMo-1B post-trained on Anthropic-HH as a function the number pre-trained tokens, for all learning rates. We report the scores eight different datasets: AlpacaEval is considered to be the main evaluation of interest (corresponding with the downstream performance), and the other datasets are considered out-of-disttibution (corresponding with the generalist performance). We the intermediate checkpoints from Table for the evaluation. We also compare the base model (dashed line). This figure similar to Figure 11, except we plot every learning rate, with a line representing a fixed learning rate.
35

---

Overtrained Language Models Are Harder to Fine-Tune
AlpacaEval
arc challenge
arc easy
piqa
68
57.5
84
81
Score
67
55.0
80
82
66
52.5
boolq
winogrande
hellaswag
sciq
76
80
7.25
Score
82
78
97.00
74
80
76
96.55
3
2
3
2
2
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Fine-tuned model
Figure 13. Evaluation OLMo-2-7B post-trained on Anthropic-HH as a function the number of pre-trained tokens, with tuned learning rates. We report the (corresponding with the downstream performance). and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We the intermediate checkpoints from Table for the evaluation. We tune the learning for each checkpoint to maximize the main evaluation (AlpacaEval). This figure is analogous to Figure 2.
eight different datasets: AlpacaEval is considered be the main evaluation of interest
36

---

Overtrained Language Models Are Harder to Fine-Tune
AlpacaEval
arc challenge
arc easy
piqa
85
60
Score
80
55
80
40
78
50
boolq
winogrande
hellaswag
sciq
76
80
Score
82
74
97
75
80
72
96
3
2
2
2
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Minimum learning rate
Maximum learning rate
Figure 14, Evaluation OLMo-2-7B post-trained on Anthropic-HH as function the number of pre-trained tokens, for all learning rates. We report the scores eight different datasets: AlpacaEval is considered to be the main evaluation of interest (corresponding with the downstream performance), and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We the intermediate checkpoints from Table for the evaluation. We also compare the base model (dashed line). This figure similar to Figure 13, except we plot every learning rate, with a line representing a fixed learning rate.
37

---

Overtrained Language Models Are Harder to Fine-Tune
Eval
arc challenge
arc easy
piqa
57.5
80
74
42.5
Score
55.0
79
40.0
72
52.5
78
37.5
70
boolq
winogrande
hellaswag
sciq
95.5
72.5
67.5
72.5
Score
95.0
70.0
65.0
70.0
67.5
62.5
94.5
67.5
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
PT tokens (trillion)
PT tokens (trillion)
tokens (trillion)
PT tokens (trillion)
Base model
Fine-tuned model
Figure 15. Evaluation LLM360-7B post-trained on Anthropic-HH as a function of the number of pre-trained tokens, with tuned learning rates. We report the scores on eight different datasets: AlpacaEval is considered be the main evaluation interest (corresponding with the downstream performance), and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We use intermediate checkpoints from Table the evaluation. We tune the learning rate each checkpoint to maximize the main evaluation (AlpacaEval). This figure is analogous to Figure 2.
38

---

Overtrained Language Models Are Harder to Fine-Tune
Alaa
arc challenge
arc easy
piqa
75
80
50
42
Score
72
40
40
30
70
78
37
boolq
winogrande
hellaswag
sciq
67
72
Score
70
95
65
70
62
67
94
60
1
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Minimum learning rate
Maximum learning rate
Figure Evaluation LLM360-7B post-trained on Anthropic-Hi HH as function of the number of pre-trained tokens, for all learning rates. We report the scores eight different datasets: AlpacaEval is considered to be the main evaluation of interest (corresponding with the downstream performance), and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We the intermediate checkpoints from Table for the evaluation. We also compare the base model (dashed line). This figure similar to Figure 15, except we plot every learning rate, with a line representing a fixed learning rate.
39

---

Overtrained Language Models Are Harder to Fine-Tune
Alaa
arc challenge
arc easy
piqa
50
32
65.0
76
Score
30
62.5
45
74
28
60.0
boolq
winogrande
hellaswag
sciq
60
60
93
Score
60
50
92
58
55
2
3
2
3
1
2
3
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Fine-tuned model
Figure 17. Evaluation OLMo-1B post-trained on TULU as a function of the number pre-trained tokens, with tuned learning rates. We report the scores eight different datasets: AlpacaEval is considered be the main evaluation of interest (corresponding with the downstream performance), and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We the intermediate checkpoints from Table 1 for the evaluation. We tune the learning for each checkpoint maximize the main evaluation (AlpacaEval). This figure is analogous to Figure 2.
40

---

Overtrained Language Models Are Harder to Fine-Tune
Alaa
arc challenge
arc easy
piqa
50
60
Score
30
70
40
40
60
25 30
boolq
winogrande
hellaswag
sciq
60
60
60
Score
75
50
55
40
50
40
50
2
3
2
3
1
2
3
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Minimum learning rate
Maximum learning rate
Figure 18. Evaluation OLMe-1B post-trained on TULU as a function the number pre-trained tokens, for all learning rates. We report the downstream performance). and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We the intermediate checkpoints from Table for the evaluation. We also compare the base model (dashed line). This figure is similar to Figure 17, except we plot every learning rate, with a line representing fixed learning rate.
eight different datasets: AlpacaEval is considered to be the main evaluation of interest (corresponding with the

---

Overtrained Language Models Are Harder to Fine-Tune
Eval
arc challenge
arc easy
piqa
82
60.0
86
Score
52.5
57.5
84
81
52.0
55.0
82
80
boolq
winogrande
hellaswag
sciq
97.5
76
85.0
80
Score
75
82.5
79
97.0
74
78
2
2
2
3
2
3
PT tokens (trillion)
PT tokens (trillion)
tokens (trillion)
PT tokens (trillion)
Base model
Fine-tuned model
Figure 19. Evaluation OLMo-2-7B post-trained on TULU as function of the number of pre-trained tokens, with tuned learning
rates. We report the scores eight different datasets: AlpacaEval is considered to be the main evaluation interest (corresponding with the downstream performance), and the other datasets are considered out-of-distribuiss (corresponding with the generalist performance). We use the intermediate checkpoints Table for the evaluation. We tune the learning rate for each checkpoint to maximize the main evaluation (AlpacaEval). This figure is analogous to Figure 2.
42

---

Overtrained Language Models Are Harder to Fine-Tune
AlpacaEval
arc challenge
arc easy
piqa
82
86
50
60
Score
84
81
40
55
82
80
30
boolq
winogrande
hellaswag
sciq
97
85
80
Score
76
82
78
97
74
76
3
2
3
2
2
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Minimum learning rate
Maximum learning rate
Figure 20. Evaluation OLMo--27B post-trained on TULU as function the number pre-trained tokens, for all learning rates. We report the downstream performance). and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We the intermediate checkpoints from Table for the evaluation. We also compare the base model (dashed line). This figure is similar to Figure 19, except we plot every learning rate, with a line representing fixed learning rate.
eight different datasets: AlpacaEval is considered to the main evaluation of interest (corresponding with the
43

---

Overtrained Language Models Are Harder to Fine-Tune
AlpacaEval
arc challenge
arc easy
piqa
70
42.5
80
74
Score
65
40.0
72
60
37.5
78
70
boolq
winogrande
hellaswag
sciq
75
96
67.5
72.5
Score
65.0
70.0
95
70
62.5
67.5
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Fine-tuned model
Figure 21. Evaluation LLM360-7B post-trained on TULU as a function of the number of pre-trained tokens, with tuned learning rates. We report the scores eight different datasets: AlpacaEval is considered to be the main evaluation of interest (corresponding with the downstream performance), and the other datasets are considered out-of -uitof-diistibution (corresponding with the generalist performance). We the intermediate checkpoints from Table 1 for the evaluation. We tune the learning for each checkpoint maximize the main evaluation (AlpacaEval). This figure is analogous to Figure 2.
44

---

Overtrained Language Models Are Harder to Fine-Tune
Alaa
arc challenge
arc easy
piqa
42
75
60
80
Score
40
72
40
37
70
78
20
boolq
winogrande
hellaswag
sciq
96
75
67
Score
72
65
70
95
70
62-
67
1
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Minimum learning rate
Maximum learning rate
Figure 22, Evaluation LLM360-7B post-trained on TULU as a function the number pre-trained tokens, for all learning rates. We report the eight different datasets: AlpacaEval is considered to the main evaluation of interest (corresponding with the downstream performance). and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We the intermediate checkpoints from Table for the evaluation. We also compare the base model (dashed line). This figure is similar to Figure 21, except we plot every learning rate, with a line representing fixed learning rate.
45

---

Overtrained Language Models Are Harder to Fine-Tune
VLM Score
arc challenge
arc easy
piqa
35
48
65.0
76
Score
46
62.5
30
74
60.0
boolq
winogrande
hellaswag
sciq
60
93
Score
60
60
92
58
50
91
55
2
3
2
3
1
2
3
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Fine-tuned model
Figure 23, Evaluation OLMo-1B post-trained on VLM as a function of the number of pre-trained tokens, with tuned learning rates. We report the scores eight different datasets: VLM Score is considered to be the main evaluation of interest (corresponding with the downstream performance), and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We the intermediate checkpoints from Table 1 for the evaluation. We tune the learning for each checkpoint maximize the main evaluation (VLM Score). This figure is analogous to Figure 2.
46

---

Overtrained Language Models Are Harder to Fine-Tune
VLM Score
arc challenge
arc easy
piqa
35
65
76
Score
45
74
30
60
40
72
boolq
winogrande
hellaswag
sciq
60
Score
60
60
92
57
90
50
50
3
2
3
2
3
1
2
3
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Minimum learning rate
Maximum learning rate
Figure 24, Evaluation OLMo-1B post-trained on VLM as function the number of pre-trained tokens, for learning rates. We report the eight different datasets: VLM Score is considered the main evaluation of interest (corresponding with the downstream performance). and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We the intermediate checkpoints from Table for the evaluation. We also compare the base model (dashed line). This figure is similar to Figure 23, except we plot every learning rate, with a line representing fixed learning rate.
47

---

Overtrained Language Models Are Harder to Fine-Tune
VLM Score
arc challenge
arc easy
piqa
82
Score
56
58
84
81
56
82
80
55
boolq
winogrande
hellaswag
sciq
85.0
76
80
7.25
Score
82.5
87.00
75
78
80.0
96.75
74
2
2
2
3
2
3
PT tokens (trillion)
PT tokens (trillion)
tokens (trillion)
PT tokens (trillion)
Base model
Fine-tuned model
Figure 25, Evaluation OLMo-2-7B post-trained on VLM as function of the number of pre-trained tokens, with tuned learning rates. We report the scores eight different datasets: VLM Score considered to the main evaluation interest (corresponding with the downstream performance), and the other datasets are considered out-of-distributio (corresponding with the generalist performance). We use the intermediate checkpoints from Table for the evaluation. We tune the learning rate for each checkpoint to maximize the main evaluation (VLM Score). This figure is analogous to Figure 2.
48

---

Overtrained Language Models Are Harder to Fine-Tune
VLM Score
arc challenge
arc easy
piqa
55
60
85
82
Score
50
55
80
80
45
boolq
winogrande
hellaswag
sciq
85
76
80
97
Score
96
80
74
78
95
3
2
2
2
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Minimum learning rate
Maximum learning rate
Figure 26. Evaluation OLMo-2-7B post-trained on VLM as a function of the number of pre-trained tokens, for all learning rates. We report the downstream performance). and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We the intermediate checkpoints from Table for the evaluation. We also compare the base model (dashed line). This figure is similar to Figure 25, except we plot every learning rate, with a line representing fixed learning rate.
eight different datasets: VLM Score is considered the main evaluation of interest (corresponding with the

---

Overtrained Language Models Are Harder to Fine-Tune
VLM Score
arc challenge
arc easy
piqa
55
42
Score
72
79
40
50
38
70
78
boolq
winogrande
hellaswag
sciq
67.5
72.5
95.5
72.5
Score
65.0
70.0
95.0
70.0
62.5
94.5
67.5
67.5
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
PT tokens (trillion)
PT tokens (trillion)
tokens (trillion)
PT tokens (trillion)
Base model
Fine-tuned model
Figure 27. Evaluation LLM366-7B post-trained on VLM as function of the number of pre-trained tokens, with tuned learning rates. We report the scores eight different datasets: VLM Score considered to the main evaluation interest (corresponding with the downstream performance), and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We use the intermediate checkpoints from Table for the evaluation. We tune the learning rate for each checkpoint to maximize the main evaluation (VLM Score). This figure is analogous to Figure 2.
50

---

Overtrained Language Models Are Harder to Fine-Tune
VLM Score
arc challenge
arc easy
piqa
Score
50
42
72
79
40
70
78
40
37
boolq
winogrande
hellaswag
sciq
75
96
67
72
Score
70
65
70
95
67
62
1
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
PT tokens (trillion)
Base model
Minimum learning rate Maximum learning rate
Figure 28. Evaluation LLM360-7B post-trained on VLM as a function of the number of pre-trained tokens, for all learning rates. We report the eight different datasets: VLM Score is considered the main evaluation of interest (corresponding with the downstream performance). and the other datasets are considered out-of-distribution (corresponding with the generalist performance). We the intermediate checkpoints from Table for the evaluation. We also compare the base model (dashed line). This figure is similar to Figure 27, except we plot every learning rate, with a line representing fixed learning rate.
51

---

Overtrained Language Models Are Harder to Fine-Tune
G. Omitted Figures from Section 3: Controlled Experiments
In this section, we provide the omitted figures from Section that show the results the extended controlled experiments.
G.1. Sensitivity
GSM8K
SIQA
StarCoder-Python
MR
RTE
TREC
max 5.00
2
max Ir=
max Ir=1
max
max
max
A perplexity
2
10tc
1041
103c
10
10tc
10
1010
10
1010
10
10.0
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Minimum tuning learning rate (4e
tuning learning (fiaa)
Figure 29. Sensitivity fine-tuned models with fixed learning rate in our controlled setup. This figure analogous to Figure from the main paper, but plots the difference in perplexity between the fine-tuned model and the base model for OLMo-30M. This figure illustrates that sensitivity increases progressively throughout training.
To supplement Figure 6 from the main paper, we plot the sensitivity of fine-tuned models with fixed learning rate in our controlled setup as function of the number of pre- training tokens in Figure 29. We find, across all datasets, that sensitivity progressively increases throughout training. Since this figure is sufficiently similar to Figure 6, we omit the corresponding sensitivity figures for the other settings we consider.
G.2. Extended fine-tuning experiments.
We now plot the extended fine-tuning experiments. We ablate the batch size, learning rate scheduler, and model size. Table 6 provides a reference to the figures that show the results of the extended controlled experiments.
Setting
Pre-training perplexity
Fine-tuning perplexity
Tuned pre-training perplexity
Tuned fine-tuning perplexity
Optimal LR
Batch size: 256
Figure 30
Figure 31
Figure 32
Figure 33
Figure 34
Batch size: 32
Figure 35
Figure 36
Figure 37
Figure 38
Figure 39
schedule: Constant
Figure
Figure 41
Figure 42
Figure
Figure 44
LR schedule: constant with warmup
Figure 45
Figure 46
Figure 47
Figure 48
Figure 49
OLMo-
Figure 50
Figure 51
Figure 52
Figure 53
Figure 54
OLMo-30M (extended)
Figure 55
Figure 56
Figure 57
Figure 58
Figure 59
OLMo
Figure
Figure 61
Figure 62
Figure
Figure 64
Table Table of contents for extended experimental settings. This table provides reference to the figures that show the results of the
extended controlled experiments.
52

---

Overtrained Language Models Are Harder to Fine-Tune
subj
boolg
MT
CT
rte
Pre-training (C4) perplecty
max 1rc1:.1-1..00 04
max lon_]r=99000000 05
max Ir- 1.1c 04
max Ir- 6.0c 03
12.5
max Ir= 1.4c 04
15
10
10.0
7.5
6
7.5
5.0
5.0
2.5
trec
tweet sentiment en
siqa
gsm8k max In= 6.0e-03
starcoder paaeeer--tth-o-kkk
Pre-training (C4)
max max_Ir= 1..e-00e04
max Le=1,4e-01
8-
max In=66 6,0e-03
max (r06-0e-000
perplexity
10
10
6
5
5
1010
10
1010
10
10.0
10
1020
10
1010
10
Pro-training tokens
Pre-training tokens
Pro-training tokens
Pre-training tokens
Pre-training tokens
Base model
Minimum learning rate (4o-6)
Maximum learning rate (max Ir)
Figure 30. Pre-training perplexity after fine-tuning as a function of the pre-training budget using the configuration specified Table but with batch size 256 for the OI.o--3M model. Each connected reflects series of models trained with fixed hyperparameters.
subj
boolg
mr
Cr
rte
Fine-tuning task
MAX [ne11.8m 1:4
MAX Ir=9.0e=05
1.1604
I1-6
0e-03
mAX 1r=1.4e-04
perplexity
0.6
0.6
0.9
0.4
0.8
0.8
0.4
0.2
0.7
trec
tweet sentiment en
gsm8k
siqa
starcoder- starcoder-python-
Fine- tuning task perplexity
max Im11 1.Be-0
XBX max_Ir= Ir- 164-0
XB8X Jt- 6.0e-03
mDx It 1r-6.0e-3
IDX 16-6006403
2
5
0.8
0.6
3
10
10
10
10
1010
10
1019
10
10t0
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Base model
Minimum learning rate (4e-6)
Maximum learning rate (max
Figure 31. Fine-tuning perplexity after fine-tuning as function of the pre-training budget using the configuration specified in Table but with batch size 256 for the MLM0-30M model. Each connected line reflects series of models trained with fixed hyperparameters.
53

---

Overtrained Language Models Are Harder to Fine-Tune
subj
boolg
Pre-training (C4) perplesiiy
20
15
10
10
15-
10
10
trec
tweet sentiment 6n
gsm88
siqu
starcoder-python--k
Pro-training (C4)
12.5
4.8
perplexity
10.0
10.0
5.25
4.2
4.6
7.5
7.5
5.00
5.0
4.75
1010
10t!
101
10t1
101h
1011
10
10
10
10
Pre-training tokens
Pre-training tokens
Pre Pre-training tokens
Pre-training takens
Pre-training takens
Figure 32, Pre-training perplexity after fine-tuning function of the pre-training budget with tuned learning rate to optimize fine-tuning performance using the configuration specified in Table but with batch size 256 for the OLMo-30M model. Similar to the untuned version but with the tin-ttuiigg-ppinel learning rate.
subj
boolg
Fine-tuning task
0.16
0.690
0.42
perplenit
0.35
0.685
0.66
0.15
0.680
0.64
0.675
0.38
trec
tweet: sentiment en
qsmm8k
siqu
starcoder-python-55
Pine-tuning task
0.22
0.56
3.0
perplexity
0.20
1.50
0.55
1.45
0.18
0.16
0.54
1.40
4.4
2.8
1010
10
10ta
10
10th
10
10th
10
10th
10
Pre-training tokens
Pre-training tokens
Pre-trainng tokens
Pre-training tokens
Pre-training tokens
Figure 33. Fine-tuning perplexity after fine-tuning function of the pre-training budget with tuned learning rate to optimize fine-tuning performance using the configuration specified in Table but with batch size 256 for the OLMo-30M model. Similar to the untuned version but with the fine-tuning-opee optimal learning rate.
54

---

Overtrained Language Models Are Harder to Fine-Tune
10-1 suby
boolg
110
learning
10-1
starooder python-55 5k
10-1
learning
10-1
Pre-traiiing tokens
Pre-trsining tokens
Pre-training tosens
Pre-training tokeess
Figure 34. The optimal learning rate for best fine-tuning performance function of the pre-training budget using the configura- tion specified Table but with batch size for OLMo-33M model. learning rate shown corresponds with those chosen Figures 32 and 33.
subj
boolq
mr
Cr
Pre-training (C4) perplexity
12.5
max 04
2.5
max 05
10.
max 05
2.5
maa 0a
max 1.
15
10.0
1..0
1..0
7.5
7.5
6
7.5
10
5.0
5.0
5.0
trec
tweet sentiment
siqa
gsm8k
starcoder-python-5k
Pre-training (C4)
12.5
Jr=7.0e:05
Ir-6 0c.43
Ir- 0e--03
Ir=6.00:03
10.0
perplexity
10.0
10
7.5
7.5
6
5.0
5.0
4
1010
10
10
10
1010
10
1000
10
10t0
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Base model
Minimum learning rate (4o-6)
Maximum leamiing rate (max Ir)
Figure 35. Pre-training perplexity after fine-tuning as function of the pre-training budget using the configuration specified Table but with batch size 32 for the OLMo-30M model. Each connected line reflects series models trained with fixed hyperparameters.
55

---

Overtrained Language Models Are Harder to Fine-Tune
subj
boolq
mr
CT
rte
Pine-tuming sree-tnning task perplexity perplexity
max Ir- 1r-0- 04
max 15-.00.0
max_ir 1wax_ir=9.0ee... 0c.05
0.5
max 11x.04
0.9
max mex_ir=1144.00
0.3
0.9
0,50
0.8
0.4
0.8
0.45
0.7
0.40
0.3
0.7
trec
tweet sentiment
gsm
siqa
starcoder-pytiee 5
Fine-tuning task
1.00
max [r=]
0.8
max In-7
max Ir-6
7-
MAX
max 6 0e-03
0.7
2
0.6
0.25
10
1041
10
10
10:00
10
10
10
1010
10a
Pro-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Base model
Minimum learning rate (4e-6)
Maximum leaminn rate (max Ir)
Figure 36. Fine-tuning perplexity after fine-tuning function of the pre-training budget using the configuration specified in Table but with batch size 32 for the OLMo-30M model. Each connected line reflects a series of models trained with fixed hyperparameters.
subj
boolq
mr
Cr
rtc
Pro-training (C4)
10.0
perplextty
7.5
6-
10
trec
tweet: sentiment en
gsmm8
siqu
starcoder-python-.
10
15
6.0
Pre-training (C4) perplexity
5.4
5.3
4.0
10
10t1
10L0
10t1
1010
10
1010
10
1010
10a
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Figure 37. Pre-training perplexity after fine-tuning as function of the pre-training budget with a tuned learning rate to optimize fine-tuning performance using the configuration specified in Table 3 but with batch size 32 for the OLMo-30M model. Similar the untuned version but with the fine-tunin tine-tuning-oe optimal learning rate.
56

---

Overtrained Language Models Are Harder to Fine-Ture
subj
boolg
0.44
0.69
Fine-tunie tiee-tuning task perplexity
0.68
0.350
0.16
0.66
0.42
0.325
0.15
0.64
0.40
0.300
0.67
0.14
0.275
trec
tweet, sentiment en
gsm8k
siqs
starcoder-pythenn Seriodee-pythoon-S5
Fine-tuning task perplexity
0.18
0.58
2.55
1.40
0.16
2.50
0.56
1.35
0.14
2.45
0.54
4.4
10
10
10t0
10
10
10
10
10
10
Pre-training
Pre-training
Pre-training
Pre-training
Pre-training
Figure 38. Finc-tuning perplexity after fine-tuning function of the pre-training budget with tuned learning rate to optimize fine-tuning performance using configuration specified in Table but with batch size 32 for OLMo-30M model. Similar the untuned version but with the fine-tuning-optimee learning rate.
sub)
hoolg
m1r
C7
rtc
10
learting
10
10-5
12e0
tweet sentiment
gammk
siqa
starooder-python-e
learning
1040
1021
10
10
10.0
10a1
1040
10
10
10
Pre-training tokons
Pro-training tokems
Pre-training toksnss
Pre-training tokons
Figure 39. The optimal learning rate for best fine-tuning performance function of the pre-training budget using the configura- tion specified in Table but with batch size 32 for the OLMo-30M model. The learning rate shown corresponds with those chosen in Figures 37 and 38.
57

---

Overtrained Language Models Are Harder to Fine-Tune
subj
boolg
mr
rte
Pre-training (C4)
12.5
15=9.0000
15
Tr=600e03
15
porplexity
10.0
10
7.5
5.0
trec
tweet sentiment en
siqa
gsmök
starcoder-python-3
Pre-training (C4) perplexity
max Inw De-05
max 1e=5.0e:05
10
maX Irw6 De-13
10
max Irwb Ir=6.0e-0'3 De-03
max Ie-60
8-
6
10
10
10th
10
10
10
10
10
10th
10
Pre-training tokens
Pre- tee-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
model
learaing (4e-6)
leamning Ir)
Figure Pre-training perplexity after fine-tuning as function of the pre-training budget using constant learning rafe scheduler (instead of Cosine) with the configuration specified in Table 3 for the OLMo-30M model. Each connected line reflects series of models trained with fixed hyperparameters.
subj
boolq
mr
rte
Fine- Fine-tuningg tee-ttning task perplexity
0-00.ee-05
0.9
In-7 0e-05
De-03
1.116
0.80
0.25
0.50
0.6
0.8
0.75
0.20
0.7
0.4
0.70
0.40
0.6
trec
tweet sentiment en
gsm
siqa
starcoder-pe pyaeerr--ythoo--k
Fine-tun toeettnnng task
0.8
10s. I--9 9. 9.0e-00
max 5m5..ee03
max, 1r 6.-03-333 De-03
max Ir- De-03
IDaX It- 6. 6.0e-03
7
0.7
3
6
0.4
0.6
2
5
0.2
1010
10
10000
10
10
10
1010
10
10tH
10a
Pre-trsining tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
madel
Minimum learning
leamiing Ir)
Figure Fine-tuning perplexity after fine-tuning a function the pre-training budget using a constant learning rate scheduler (instead of Cosine) with the configuration specified in Table 3 for the OLMo-33M model, Each connected line reflects series of models trained with fixed hyperparameters.
58

---

Overtrained Language Models Are Harder to Fine-Tune
subj
boolg
Pre-training (C4) perplexity
10-
15-
10
5
trec
tweet sentiment en
gsm8k
siqa
sercodderpythhore 5k
Pro- tre-trainnng (C4) perplexity
10
12.5
5.2
4.2
10.0
5.0
4.0
4.8
5.0
4.5
4.6
10
10
10
10
10
10
10
10
101h
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Figure 42. Pre-training perplexity after fine-tuning as function of the pre-training budget with a tuned learning rate to optimize fine-tuning performance using constant learning scheduler for the OLMo-30M model. Similar the untuned version but showing the performance with the fine-tuning-opimall learning rate.
subj
boolg
Fine-tuning task
0.44
0.350
0.69
perplexity
0.16
0.66
0.42
0.325
0.68
0.15
0.64
0.40
0.300
0.67
0.14
0.62
0.275
0.38
trec
tweet sentiment en
qsm8k
siqa
starcoder-python..
Fine-tuning task
1.50
2.65
perplxity
0.56
0.18
1.45
4.6
2.60
0.16
0.54
1.40
4.4
2.55
0.14
10t0
1041
10.0
1041
10
10
10
10
10
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Figure 43. Fine-tuning perplexity after fine-tuning function of the pre-training budget with tuned learning to optimize fine-tuning performance using a constant learning rate scheduler for the OLMo-30M model. Similar to the untuned version but showing the performance with the tuning-optimal learning rate.
59

---

Overtrained Language Models Are Harder to Fine-Tune
Best learning
10
tweet sentiment
gammk
staronder-py pyytonn--
learnnng rate
10
10-1
10
10
1010
10
10
10
10
Pre-training tokens
Pre-training tokens
Pre Pre-training tokens
Pre-training tokeens
Figure 44. The optimal learning rate for best fine-tuning performance function of the pre-training budget using constant learning rate scheduler the OLMo-30M model. The learning rate shown corresponds with those chosen Figures 43.
subj
boolq
mr
Cr
Pre-traiitig (C4)
12.5
max 1-1---
max 1r-7.0e-05
max Ir- 1.1e-04
maa 6e-00
max Ir- 1---0
perplextty
10.0
10.0
10.0
10
7.5
7.5
7.5
10-
5.0
5.0
5
trec
tweet sentiment
siqa
gsmök
starcoder -ythoo-k max 6.
Pro-training (C4)
10
max 1r11e004
10
max 7500.ee-5
max 1r=6
max
perplexity
8
7.5
6
5.0
4
1010
10t1
10tH
10
101H
10
1010
1041
10th
10a1
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Dase madel
Minimum learning rate (4c-6)
Maximum learning rate (max Ir)
Figure 45. Pre-training perplexity after fine-tuning as function of the pre-training budget using constant learning rate scheduler with warmup with the configuration specified in Table for the OLMo-30M model. Each connected line reflects a series of models trained with fixed hyperparameters.
60

---

Overtrained Language Models Are Harder to Fine-Tune
subj
boolq
mr
rte
Fine- teniiing task porplexity
max Ir-1 1r-1.1c04
0.9
max Ir=7 0c:05
0.55
max Ir- I.c44
0.8
max 6ww...6600000000 0c.03
0.9
max [r- 4c:04
0.25
0.8
0.50
0.6
0.20
0.45
0.4
0.7
0.15
0.40
trec
tweet sentiment
gsm 8k
siqa
starcoder-pytheen Sarooderrppthoo-555
Fine-tuning task perplexity
0.8
0e-03
0e.43
De-03
0p-03
0.7
3
0.4
0.6
2
0.2
1010
10
10
10
1010
10
1010
10
1010
1011
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Base model
Minimum learning rate (6e-6)
Maximum learning rate (max Ir)
Figure 46. Fine-tuning perplexity after fine-tuning function of the pre-training budget using constant learning rate scheduler with warmup the configuration specified in Table the OLMo-30M model. Each connected line reflects series models trained with fixed hyperparameters.
subj
boolg
Pre- trainrinnng (C4) perplexity
15
15-
5
10:
10
4
5
trec
sentiment
gsm8k
siqa
starcodder-yython-k
Pre- rre-trainnig (C4)
4.2
perplexity
5.4
5.0
4.0
5.2
4
4.5
5.0
10
10
10t0
10
10
10
10t1
10th
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-traies pre-training tokens
Figure 47. Pre-training perplexity after fine-tuning as function of the pre-training budget with a tuned learning rate to optimize fine-tuning performance using a constant learning rate scheduler with warmup for OLMo-33M model. Similar to the untuned version but showing the performance with the fine-tuning-optimal learning rate.
61

---

Overtrained Language Models Are Harder to Fine-
subj
boolg
0.68
Fine-taning task perplexity
0.350
0.16
W
0.66
0.42
0.325
0.68
0.14
0.64
0.40
0.300
0.67
0.62
0.38
0.275
trec
tweet sentiment 6n
gsmn8k
siqa
starcoder-python-k
Fine-tuning pre-tuning task
0.57
2,60
1.50
perplexity
0.18
0.56
1.45
4.6
0.16
1.40
2.50
0.14
0.54
1.35
10t0
10
101
10
10th
10
10th
10
10
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training takens
Pre-training takens
Figure 48. Fine-tuning perplexity after fine-tuning function of the pre-training budget with tuned learning rate to optimize fine-ture tunint performance using a constant learning rate scheduler with warmup for the OLMo-30M model. Similar to the untuned version but showing the performance with the fine-tumn toutttnnnniiinnn learning rate.
boole
10
learning
10-4
trec
twent sontiment
gam8k
staronder python- 5k
learning rate
10-3
10-1
10:1
10:9
10
19
10
1030
10
10:9 Pro: triiariinng tokons
10:1
10 Pre traintaiingg tolons
10a1
Pro: traintiiingg tokens
Pre Pre-training tokens
Figure 49. The optimal learning rate for best fine-tuning performance as function of the pre-training budget using a constant learning rate scheduler with warmup for the MLMo-30M model. The learning shown corresponds with those chosen in Figures 47 and 48.
62

---

Overtrained Language Models Are Harder to Fine-Tune
subj
boolq
rte
Pro-training (C4) perplexity
max
max 04
12.5
max,
I0aX 1rax_ir=2.0e-06
12.5
max 0a
10
10
10.0
10.0
7.5
5.0
trec
tweet sentiment en
siqa
gsm8k
Starcoder-pythone
Pre-trsining (C4)
max Jr-.:12222440
max 1r=1.60.04
max 1r-2.00222200 De.433
max Ir-1 0.o-022
max Ir- 1.00-22
perplexity
10.0
8
8
7.5
6
5.0
4
10
104h
1000
104
1049
10a1
10:00
1041
10
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
learning rate (4c-6) 6)
leamning (max
Figure 50. Pre-training perplexity after fine-tuning as a function of the pre-training budget using the configuration specified in
Table for OLMo-15M. Each connected line reflects series of models trained with fixed hyperparameters. Analogous to Figure (top) from the main paper.
subj
boolq
rte
Fine- tuning task perplexity
0,250
lr=x__r==,0000
1.0
max 1r-1.
0.55
maA Irx.-e-=
0.50
mrxx-e==200--0
maa Ir-2.e--
0.225
0.45
0.200
0.8
0.40
0.175
0.45
0.35
trec
tweet sentiment en
gsm8k
siqa
cerrcoder-python. -5k
Pins-tuning task perplexity
max Ir= 2.
max 1r-1__11ff 1r=1..o.04
max Jr=1.00:02
max 1r=2.0c-03
max 1ww.11=11.0000 02
0.6
0.70
2.5
0.65
2.0
0.60
0.2
10tH
1010
10
10
10
10th
10
10t0
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Base model
Minimum learning (4e-6)
Maximum learning (max It)
Figure 51. Fine-tuning perplexity after fine-tuning function the pre-training budget using the configuration specified in
Table 3 for MLM5-15M. Each connected line reflects a series of models trained with fixed hyperparameters. Analogous to Figure 5
(bottom) from the main paper.
63

---

Overtrained Language Models Are Harder to Fine-Tune
subj
boolq
Pre-training (C4) perplexity
10.0
10
7.5
10
trec
tweet sentiment en
gsmäk
siqa
starcoder-py -yttcodder--tthoo5555
Pre-training (C4) perplexity
12
6.4
5.80
10
10
5.75
4.8
5.70
4.7
5.65
10tH
10t1
1010
10t1
10
10
10
10t
10th
10th
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Figure 52. Pre-training perplexity after fine-tuning function of the pre-training budget with tuned learning rate to optimize fine-tuning performance using the configuration specified in Table 3 for OLMo-15M. Similar to the untuned version but showing the performance obtained with the ann--tuning-optimee learning rate, analogous Figure (bottom) from the main paper.
subj
boolq
Cr
0.75
0.38
Fine- tine-tunnng task
0.175
0.45
W W
perplexity
0.685
0.170
0.44
0.43
0.165
0.34
0.675
0.42
trec
tweet sentiment en
gsm8k
siqa
starcoder-pythan pyt--n--thhookk
Fine-tuning task parplexity
0.59
5.2
2.85
1.65
0.25
0.58
5.1
2.80
0.57
1.60
0.56
5.0
2.75
1010
10th
10
10t0
10
1040
10
10t0
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Figure 53. Fine-tuning perplexity after fine-tuning function of the pre-training budget with tuned learning rate to optimize fine-tuning performance using the configuration specified in Table 3 for OLMo-11M. Similar to the untuned version but showing the performance obtained with the fine-tuning-oppin optimal learning rate, analogous Figure (top) from the main paper.
64

---

Overtrained Language Models Are Harder to Fine-Tune
books
10-1
learning
trec
tweet sentament, en
gsm8k
siqa
starcoder python Sk
learning rate
10-1
10
10
10
10"
1010
10
10 Pre nee-training tokens
10
10 Pre- Pre-training tolenns
10"
Pre- pre-trainin tokens
Pre-training tokens
Figure 54. The optimal learning rate for best fine-tuning performance as a function the pre-training budget using the configura- tion specified in Table 3 for OLMo-15M. The learning shown corresponds with those chosen in Figures 52 and 53.
subj
boolg
mr
rte Ire 44-04
Pre-training (C4)
4e-04
Ir=1.0e:04
1.40-04
perplexity
10
10
10
10
5
5
5
trec
tweet sentt__tttiment en
siqa
gsm8k
starcoder- -yrtoddr--pyhho55kk
Pro-training (C4)
15
1r=1.4c-04
Ir=900e-05
Ir= 1 03
10
Ir= 6.0e 03
Ir=6 1e-660e-00 0e.-03
perplexity
10
6
5
1040
10
1010
10
10t0
10
1010
10a1
1010
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Base model
Mitimum learning rate (4e-6)
Maximum learning rate (max
Figure 55. Pre-training perplexity after fine-tuning function of the pre-training budget using the configuration specified in Table for Mo-3-30M. Each connected line reflects a series of models trained with fixed hyperparameters. Extended version of Figure 5 (top) from the main paper.
65

---

Overtrained Language Models Are Harder to Fine-Tune
subj
boolq
mi
Cr
rte
Fine- tee-tuuigg Lask perplexity
0.4
MAX In De-0-4
MBX [P=1.0e-04
max 10-044
IDaX 4e-004
0.9
max Ir- 1:444
0.5
0.9
0.3
0.8
0.4
0.8
0.2
0.7
0.3
0.7
tweet sentiment en
gsm8k
siqa
starcoder stareoder-python-
Fine- tee-tuunng Lask perplexity
maX 1----
XB8X 1100900-05
Tba Jr- 6,06-03
IDAX It 1r.1.---00
IDeX 18-6,0e-00
1.0
0.5
0.6
3
1030
10
101
10
10.6
10
10to
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Minimum learning (1e-6)
Maximun learning (max
Figure Fine-tuning perplexity after fine-tuning as function pre-training budget using the configuration specified Table for OLMo-30M. Each connected line reflects series of models trained with fixed hyperparameters. Extended version of Figure 5 (bottom) from the main paper.
subj
boolq
mr
rtc
Pre-training (C4)
6-
10.0
perplexity
15
5
10
7.5
6
10
5.0
5
trec
tweet sentiment en
gsm8k
siqa
starcodec-python-5B
Pre-training (C4)
5.5
perplexity
4.2
5.0
5.2
5.0
4.5
4.0
10
10
10.0
10
104H
10
1010
10
10
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Figure Pre-training perplexity after fine-tuning as function of the pre-training budget with a tuned learning to optimize fine-tuning performance using the configuration specified in Table for OLMo-30M. Similar to the untuned version but showing the performance with the mnn-tuning-optimall learning rate, analogous Figure (bottom) from main paper. Extended version Figure 55 from the main paper.
66

---

Overtrained Language Models Are Harder to Fine-Tum
subj
boolg
Fine-tunie tine-tuning task perplexity
0.66
0.15
0.42
0.325
0.68
0.14
0.300
0.67
0.275
0.13
0.38
trec
tweet, sentiment en
gsm8k
siqs
paarooder-yyhhon
2.60
Fine-tuning task perplxiiy
0.16
0.56
0.15
4.6
1.35
2.50
0.14
1.30
4.4
0.13
10
10
1010
10
10
10
10
10
10
Pre-training
Pre-training
Pre-training
Pre-training
Pre-training
Figure 58. Finc-tuning perplexity after fine-tuning function of the pre-training budget with tuned learning rate to optimize fine-tuning performance using the configuration specified in Table OLMo-30M. Similar the untuned version but showing the performance with the fine-tniing-optimal learning rate, analogous to Figure (top) from the main paper. Extended version of Figure 56 from main paper.
hoolg
10-1
learning
tweet sentiment
gamBk
staronder- Staronder-yython-- python-S4
learning rate
10-1
10
10
10th
1010
10
1010
10
10
Pre-training tokens
Pre-training Lokens
Pre-training tokens
Pre-training tokens
Figure 59. The optimal learning rate for best fine-tuning performance as a function of the pre-training budget using the configura- tion specified in Table 3 for Mo-3-3... The learning shown corresponds with those chosen in Figures 57 and 58.
67

---

Overtrained Language Models Are Harder to Fine-Tune
subj
boolq
mr
CT
rte
Pro- Pre-training (C4) perplexity
max 1r- 20
15
max 10-200
max 2r4-04
maX It 2.4e-04
max Ir- 2.4e-04
5
4
5
4
trec
tweet sentiment en
siqa
gsm8k
pearcoder-python
Pro ree-training (C4) perplexity
1r-2.4
1r-2.4
2.0e-03
1.0e-02
1.0e-022
10
5
4
4
4
1010
10
1010
10
10.0
10
10.0
10
10.0
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Base model
Minimum learning (4c-6)
Maximum learning (max Ir)
Figure Pre-training perplexity after fine-tuning as a function of the pre-training budget using the configuration specified in Table for OLMo-9UM. Each connected line reflects series of models trained with fixed hyperparameters. Analogous to Figure 5 (top) from main paper.
subj
boolg
mr
cr
rte
Fine- Fine-tuning tuning task perplexity
1r=2.4e-04
2.00e-00
0.50
2e-02404
2ex
0.3
1.0
0.4
0.8
0.45
0.2
0.8
0.40
0.3
0.7
0.35
0.1
trec
tweet sentiment en
gsm8k
siqa
starcoder-pythout
Fine- line-tuning task perplexity
1.0
MaX 1r-2.1e-04
0.7
Y18X 1r= 224-0
Ir- 0.0e022
I1-2 0e-03
max 1r- F-=00s-02
0.5
0.6
0.5
10t0
10t
1010
10t1
1010
10t1
1010
10
10tc
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Base model
Minimum learning rate (4c 6)
Maximum learning rate (max Ir)
Figure 61. Fine-tuning perplexity after fine-tuning function of the pre-training budget using the configuration specified in Table for MoMo-90M. Each connected line reflects series models trained with fixed hyperparameters. Analogous Figure (bottom) from the main paper.
68

---

Overtrained Language Models Are Harder to Fine-Tune
subj
boolq
4.0
Pre-training (C4)
3.8
15
perplexity
3.7
3.7
3.6
3.8
3.6
3.6
3.5
3.5
trec
tweet sentiment en
gsmök
siga
starcoder-pythee -aatoododr--ytoo-S--- 5k
Pre-training (C4)
4.6
4.2
perplexity
4.0
4.4
3.7
4.5
3.8
3.8
3.6
3.6
3.6
1040
10
1040
10th
1010
10t1
1010
10t1
10.0 Pre-training tokens
10t1
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Figure 62. Pre-training perplexity after fine-tuning as function of the pre-training budget with a tuned learning rate to optimize fine-tuning performance using the configuration specified in Table 3 for OLMo-90M. Similar the untuned version but showing the performance fine-tuning-optimall learning rate, analogous to Figure (bottom) from the main paper.
boolq
Fine- mnn-tuning task perplexity
0.28
0.14
0.24
0.64
0.12
0.60
0.34
0.22
trec
tweet sentiment en
gsm8k
siqa
starcoder-python--5
4.3
2.35
Fine-tuning task
0.14
0.53
perplesity
2.30
1.20
0.52
2.25
0.12
1.15
4.1
2.20
0.11
10
1041
10.0
1041
10
10
10
10
10
10
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Pre-training tokens
Figure 63. Fine-tuning perplexity after fine-tuning function of the pre-training budget with tuned learning rate to optimize fine-tuning performance using the configuration specified in Table 3 for OLMo-90M. Similar to the untuned version but showing the nerformance with the fine-tuiing-optimal learning rate, analogous Figure (top) from the main paper.
69

---

Overtrained Language Models Are Harder to Fine-Tune
subb
hoolg
mB
Cr
rtc
learning rabe
10-1
10-4
tweet sentiment.cc cnm
gam8k
siga
starosder python-5k
learning rass
10-1
1010
1021
10
10
10
10ah
10 Pro trainrainng tokens
10:1
1012 Pro pro-training tolons
10a1
Pro traintiinngg tokens
Pre-training tokens
Figure 64. The optimal learning rate for best fine-tuning performance function of the pre-training budget using the configura- tion specified Table for OLMo-90M. The learning rate shown corresponds with those chosen in Figures 62 and 63.
70

---

Overtrained Language Models Are Harder to Fine-Ture
OLMo-15M
-LM0-30M
OLMo-90M
Perturbstion (A)
Perturbation (A)
Perturbstion (A)
4.4
Base
0.05
- 0.00
-
0.1
-
Base
- 0.05
0.08
---
0.1
Base
0.05
0.08
-
0.1
4.2
4.2
Perplexity
4.0
4.0
5.0
3.8
3.6
4.5
3.4
10a
10a
10
10a
10
10a
Pre-training tokens (B)
Pre-training tokens (B)
Pre-training tokens (B)
OLMo-1B
OLMo-2-78 2-7B
LLM360 (7B)
3,4
Perturbstion
Perturbation (A)
Perturbation (A)
0.05
0.033
3.2
20
15
3.0
10
3.0
2.8
10a
10a
2
xx 10a
103
Pre-t Pre-training tokens (B)
Pre-training tokens (B)
Pre-training tokens (B)
Figure 65. Pre-training perplexity models with parameters perturbed by Gaussian noise, as a function of the number of pre-training tokens. We report the C4 web data perplexity of different models where each parameter is perturbed by Gaussian noise scaled by the factor X (color). This figures is an extension of Figure 3 to additional models: OLMo-15M, OLMo-90M, OLMo- IB, OLMo-2-7B, and LLM360-Ambe (7B).
G.3. Extended Gaussian perturbations experiments.
Here, we present extended experiments with Gaussian perturbations on additional models: OLMo-15M, OLMo-90M, OLMo-IB OLMo---7B, and LLMe60-Amber (7B). We perturb each parameter by Gaussian noise scaled by the factor X. Figure 65 shows the pre-training perplexity of models with parameters perturbed by Gaussian noise function of the number pre-training tokens. Refer to Appendix for more details the experimental setup.
71


Reviews:
[{"review_id": "CEzrjjU4ds", "reviewer": "Reviewer_gBdh", "summary": "The paper demonstrates how overtraining language models (training on more than the compute-optimal number of tokens) affects their ability to be fine-tuned on new data. For example, the authors perform experiments on OLMo-1B models and show that models pretrained on 3T tokens performed 3% worse on AlpacaEval and 2% worse on ARC reasoning after instruction fine-tuning compared to models trained on 2.3T tokens. Similar degradation is observed over several other evals (PIQA, BoolQ, Winogrande) and fine-tuning setups ((instruction tuning on TULU and multimodal fine-tuning). They perform further controlled experiments on 15M-90M parameter models and provide a theoretical linear model to characterize \"catastrophic overtraining,\" where fine-tuning performance first improves and then degrades with excessive pre-training.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback! We are happy to hear your positive comments that our experiments “support the main finding” and that our theory \"helps explain the observed phenomenon”.\n\nSummary of changes: To address your main concerns: (1) we clarify that our fine-tuning setups do lead to consistent improvements on the target task; (2) we will rewrite to clarify the scope of our findings; (3) we will add the per-token accuracy experiment you propose; (4) we will make specific changes as outlined below.\n\nPlease find our responses to each concern:\n\n\n> The authors describe \"catastrophic overtraining\" as universal\n\nThanks for bringing this up—in our revised version we will clarify the scope and remove the use of “universal”.\n\nTo summarize: Our core finding is that there are some important settings where additional pre-training hurts fine-tuned performance, despite helping the base model (Figures 1 and 2). We believe that finding *any* example of this is surprising. We do not claim that *all* settings exhibit this degradation with overtraining.\n\n\n> The evidence is limited to only two fine-tuning setups\n\nWe want to clarify—we consider more than just two fine-tuning settings.\n\nFor the our large model results (OLMo-1B):\n* Anthropic-HH (Instruction tuning)\n* Multimodal tuning\n* TULU (instruction tuning)\n* Alpaca (instruction tuning)\n\n(Figure 1 and Appendix B). We observe catastrophic overtraining for Anthropic-HH, multimodal tuning, and TULU. \n\nFor the small model results (OLMo-30M):\n* GSM8k\n* SIQA\n* Starcoder-Python\n* MR\n* RTE\n* TREC\n* CR\n* Tweet sentiment\n* SUBJ\n* BoolQ\n\n(Figures 2 & 3, plus Appendix C).\n\n\n> Both [IFT & VLM fine-tuning setups] fail to produce consistent improvements in the evals measured compared to the base model.\n\nFor Anthropic-HH, we aim to improve response quality (measured via AlpacaEval), and fine-tuning boosts scores by ~35–45% over the base model (see table below).\n\nFor VLM fine-tuning, the goal is to enable image input via an adapter—something the base model can’t do. The fine-tuned model reaches ~45% VLM score (Figure 1).\n\nWe also evaluate OOD tasks (ARC, PIQA, BoolQ, Winogrande, HellaSwag, and OpenBookQA). Fine-tuning on an unrelated task isn’t expected to help performance on these tasks and can often harm performance [1].\n\nTogether, these evaluations help us assess how overtraining impacts the ability to learn new tasks (measured by main eval) and the tendency to degrade unrelated capabilities (measured by OOD eval).\n\n\nHere are the base model (and instruction-fine-tuned) scores for AlpacaEval:\n\n| Tokens | Base model AlpacaEval (%) | Instruction-tuned AlpacaEval (%) |\n|---|-|---|\n| 0.5e12      | 10.57                 | **47.10**                            |\n| 1.5e12      | 14.43                 | **54.97**                            |\n| 1.8e12     | 11.94                 | **56.03**                            |\n| 2.3e12      | 12.39                 | **56.05**                            |\n| 3e12      | 11.07                 | **53.52**                            |\n\nTable: Comparing AlpacaEval for OLMo-1B base model vs instruction tuning with Anthropic-HH. We will add this to the main paper to clarify, apologies for the confusion.\n\n\n> Do your findings still hold if you measure token prediction accuracy instead of perplexity?\n\nWe ran this experiment, and yes, our findings hold. Below report per-token accuracy of GSM8k as a function of pre-training tokens (analogous to Figure 3).\n\n| Pre-training Tokens (B) | Pre-training data per-token accuracy (%) | GSM8k per-token accuracy (%) |\n|---|----|---|\n| 4 | 15.4     | 65.9    |\n| 8  | 15.8  | 67.2    |\n| 16  | 16.7    | 67.7    |\n| 32 | 16.4  | 68.0  |\n| 64   | 15.4  | 67.8 |\n| 128  | 13.4   | 67.1   |\n\nNote: GSM8k per-token accuracy is more predictable than web data because the examples are quite structured.\n\nResults corresponding to other figures & datasets similarly hold true.\n\n\n> The simplification [of the theoretical model] makes a number of assumptions that may not hold for LLMs.\n\nTo clarify—We do model discrete gradient descent explicitly during fine-tuning (see Appendix A.3).\n\nWe aren’t aware of any theoretical tools that would enable us to study the dynamics of LLMs without at least some degree of simplification such as the ones we describe. We fully acknowledge that there are limitations that arise from these simplifications but we hope our analysis is a starting point to understand this phenomenon that may lead to principled mitigation strategies in the future.\n\nAdditionally, since catastrophic overtraining is so counterintuitive (to us), seeing it emerge from incremental learning in a simpler setting reassures us it's not just an artifact of our LLM setup and offers intuition for a possible mechanism.\n\n[1] Goodfellow, Ian J., et al. An empirical investigation of catastrophic forgetting in gradient-based neural networks. 2013.", "claims_and_evidence": "- Experiments on 1B-7B parameter models and two fine-tuning tasks (Anthropic-HH SL on instruction-tuning data, multimodal fine-tuning) support the main finding that overtrained language models can perform worse after fine-tuning.\n- The authors provide a theoretical model that helps explain the observed phenomenon in a simplified setting.\n- The paper investigates how the learning rate during fine-tuning affects the degradation, showing that different learning rates are optimal for models with different pre-training durations.\n- To eliminate the confounder of the learning rate schedule being at different points for the different OLMo model checkpoints, they perform controlled experiments and train models from scratch for various pre-training budgets with a cosine annealing schedule.\n\nHowever, while the authors describe \"catastrophic overtraining\" as universal, the evidence is limited to only two fine-tuning setups. Both fail to produce consistent improvements in the evals measured compared to the base model, which could indicate issues in the choices of fine-tuning data mix.", "methods_and_evaluation_criteria": "Overall the methods and benchmark datasets used make sense. However, the use of perplexity in the small pretraining experiments might be misleading (i.e. the results in Figure 2). For downstream tasks like GSM8K, we don't care about perplexity on the answer but rather accuracy. Because it may be hard to get non-trivial accuracy in small models, you could look at average per-token prediction accuracy instead of loss/perplexity. This would help decouple token prediction calibration/effective temperature from downstream task performance.", "theoretical_claims": "The theoretical model described in section 4 looks sound. However, the simplification makes a number of assumptions that may not hold for LLMs (e.g. linearity, assuming the pretraining and fine-tuning datasets share the same singular vectors, not accounting for discrete gradient descent algorithm).", "experimental_designs_or_analyses": "The experimental design is generally sound although I have some concern about the choices of fine-tuning datasets as mentioned in the other sections.", "relation_to_broader_scientific_literature": "The paper connects to:\n- Research on pretraining scaling laws (Hoffmann et al., 2022; Kaplan et al., 2020) that established optimal token-per-parameter ratios\n- Research on model plasticity, particularly studies in reinforcement learning that showed models can become less adaptable after extensive training (Kumar et al., 2020; Lyle et al., 2022) and work on feature rank and inactivity as mechanisms for reduced adaptability (Gulcehre et al., 2022; Dohare et al., 2021)", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. I am confused by the results in Figure 1. It looks like for most evaluations, fine-tuning hurts instead of helps performance on the eval measured (solid line is below dotted line). Possibly the fine-tuning datasets used are too narrow and would benefit from additional regularization (e.g. by mixing in a wider range of diverse examples from other distributions). Do you have empirical evidence showing that your results generalize to settings where the fine-tuning datamix robustly improves performance on the target evals beyond the capability of the pretrained model?\n2. For your perplexity-based experiments, do your findings still hold if you measure token prediction accuracy instead of perplexity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lVSJhN9KwX", "reviewer": "Reviewer_fdKm", "summary": "The authors study a phenomenon they observe where the more overtrained a pretrained language model is (as a function of pretraining tokens per parameter, w.r.t. training-compute optimal amounts), the more difficult it is to fine-tune the model. The study is motivated by an initial example for OLMo-1B, where the authors tune the SFT LR to optimize quality for a primary task (e.g. AlpacaEval) and use this LR rate to SFT other tasks (e.g. ARC, BoolQ). When doing this, they see that the strongest performance on all tasks do not come from the final overtrained checkpoint, but instead come from earlier checkpoints, calling into question the value of overtraining in the SFT regime. The authors recreate this behavior in smaller scale controlled settings (to up 90M model, 128B tokens) and observe similar behavior, across a large sweep of learning rates, schedules, and batch size. Finally, the authors provide a theoretical explanation for the phenomenon proved for a 2 layer linear network.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback! We are happy to hear your positive comments that there is “significant insight in this paper”, that it “would be of interest to the community”, and that we include a “wealth of empirical results”.\n\nWe were also glad to read:\n\n> My issues with this work mostly lay in the claims made about the results [...] rather than the results themselves.\n\nTo address this, we’ve revised our framing to clarify our claims. These changes will be reflected in the final version.\n\nSummary of changes: To address your main concerns, we will: (1) rewrite to clarify the definition of “harder” and the scope of our findings; (2) further highlight how checkpoint selection (tuning) affects results; (3) make specific changes based on the comments below.\n\n\n> How would you define what it means to be harder to fine-tune, and what is the main evidence in your paper that supports this specifically?\n\nWe will clarify in the updated manuscript—by \"harder\" we mean specifically that extending the pre-training phase, and then fine-tuning, can lead to worse performance in-distribution (fine-tuning task) and also out-of-distribution (unrelated tasks) compared to a shorter pre-training stage.\n\nOur evidence:\n* After instruction tuning OLMo-1B on Anthropic-HH, models trained on 3T tokens underperform those trained on 2.3T on AlpacaEval (in-distribution) and ARC/PIQA (out-of-distribution) — see Figure 1.\n* After multimodal tuning, the 3T-token model also performs worse on ARC and PIQA (Figure 1).\n* After instruction tuning on TULU, the 3T-token model scores lower on AlpacaEval (in-distribution) and on multiple OOD benchmarks (ARC, PIQA, Winogrande, HellaSwag, OpenBookQA) — see Appendix Figure 7 (right).\n* For OLMo-30M, models trained beyond ~32B tokens show degraded in-distribution performance on several fine-tuning tasks (RTE, TREC) and degraded C4 web-data performance when fine-tuning with GSM8K, Starcoder-Python, MR, RTE — see Figure 2.\n\nApologies for the confusion, we will clear this up in our revision.\n\n\n> This broad claim [overtraining => harder to fine-tune] is very difficult to exhaustively verify\n\nThanks for bringing this up—in our revised version we will properly clarify the scope of our paper.\n\nTo summarize: Our core finding is that there are some important settings where additional pre-training hurts fine-tuned performance, despite helping the base model performance (Figures 1 and 2). We believe that finding *any* example of this phenomenon is surprising. We do not claim that *all* settings exhibit this degradation with overtraining.\n\n\n\n> If the authors decided to pick a different reference task to optimize LR over [other than AlpacaEval], the model would have looked significantly more predictable to fine-tune with more pretraining tokens. \n\nAgreed! A key contribution of our work is precisely to analyze how the optimal learning rate chosen for one task impacts observed degradation on others — this is the focus of Section 3.4. \n\nIn this section, we argue that we specifically observe performance (ID or OOD) degradation when optimizing for a task that requires a larger learning rate for good performance — empirically, this is the case for AlpacaEval, and many other evaluation settings (see response below).\n\n\n> The choice to optimize AlpacaEval, on one hand is motivated as general, but ultimately a bit arbitrary.\n\nFor instruction tuning, we pick AlpacaEval because it is a standard benchmark to measure response quality—improving this is our goal during fine-tuning. We agree that any single metric could seem arbitrary, but we also evaluate on a collection of other evaluation metrics:\n\nWe consider 12 total evaluations across various settings:\n* AlpacaEval (for instruction-tuned models)\n* VLM score (average over 5 VLM benchmarks; for multimodal tuned models)\n* 10 individual fine-tuning datasets where we optimize for ID performance: SUBJ, BoolQ, MR, CR, RTE, TREC, Tweet sentiment, GSM8k, SIQA, Starcoder-Python (Figure 2, plus more in Appendix C.2)\n\nWe believe that, collectively, these experiments robustly illustrate the occurrence of degradation across many practical fine-tuning scenarios.\n\n\n> This difficult to SFT behavior doesn't quite exist for LLM360-7B [and] OLMo-7B [...]. Is it because of the overtraining?\n\nGreat question! This difference is likely explained by how many tokens per parameter each model has seen:\n* OLMo-1B (3T tokens ≈ 3000 tokens per param) experiences degradation by ~2.3T tokens.\n* LLM360-7B (1.3T tokens ≈ 185 tokens per param) and OLMo-7B (4T tokens ≈ 571 tokens per param) have seen significantly fewer tokens per parameter relative to OLMo-1B.\n\nIf degradation scales linearly with model size, we would only expect degradation for a 7B model after ~16T tokens (7B × 2300 tokens per param). Thus, current 7B models simply haven't been trained long enough to reach that regime yet.", "claims_and_evidence": "The claims in the paper are clear. This paper makes a relatively bold and counter-intuitive claim: that pretraining more can make SFT harder (or in some cases hurt SFT), and the paper provides a wealth of empirical results that provide significant insights into the interaction between overtraining and SFT. The findings here are interesting and would be of interest to the community.\n\nA criticism of the work might be that the author's claim is too broad and underspecified, as to what \"Harder\" means. I believe while there's significant insight in this paper, the framing of the work suggests that overtraining might be bad as it can hurt post-training -- but this broad claim is very difficult to exhaustively verify. It is also unclear to what level practitioners already expect to have to tune basic SFT hyperparameters to maximize performance given a fixed pretrained model. Whether this is \"harder\" than what's expected is ill defined. This paper would in general benefit from a more straightforward framing of the results.\n\nAn example: In Figure 1, we see that the optimal LR was selected based on AlpacaEval -- which truly benefits from less pretraining under the experimental settings of this paper, at OLMo-1B scale. However, in Figure 7 (Appendix) we get to see the full LR sweep of each task. If the authors decided to pick a different reference task to optimize LR over, the model would have looked significantly more predictable to fine-tune with more pretraining tokens. The choice to optimize AlpacaEval, on one hand is motivated as general, but ultimately a bit arbitrary -- especially as it uses an imperfect autorater (and maybe not tuned well for 1B model outputs). \n\nSimilarly, in Figure 9, we see that this difficult to SFT behavior doesn't quite exist for LLM360-7B as the model is less overtrained. However we don't quite see the same AlpacaEval-based result for OLMo-7B to prove this hypothesis. Is it because of the overtraining, or does this phenomena improve as we scale to 7B, for these particular datasets?\n\nFinally I particularly have issues with how the authors sometime conflate SFT for instruction tuning, SFT-until convergence over a single task, and general \"post-training\" -- especially as the intention and training dynamics for these vary widely (post-training typically also includes RL these days.)", "methods_and_evaluation_criteria": "The methods and evaluation are insightful as we get to see the full behavior of the model and how pretraining and finetuning interact over a wide set of hyperparameters. In some sense, given these empirical results alone -- the reader could come up with their own valuable conclusions as to what the relationship between overtraining and SFT is. \n\nSome of the methodology is contentious: Figure 1 selects a checkpoint based on IT performance, with the expectation that it generalizes to SFT-until-convergence settings. It's unclear how valid this is as the intent of the two types of SFT are quite different (IT tries to train to generalize over many topics, SFT on a single task tries to maximize just that task -- in both cases it is typical to assume that they would need different LR / hypers.)\n\nIn Figure 3, the controlled experiment fixes this issue by focusing on single-task settings. However the \"harder to SFT\" claim is still hard to verify: for most of the tasks, there does indeed exist an optimal hyper-parameter that maximizes performance, and is predictable with more pretraining. There are a few where this is not true, and this is interesting -- however it is unclear if this implies that overtraining makes the model harder to SFT in general. Figure 4 basically shows this, and if theres a widely accepted optimal learning rate per-task, I'm not sure how much \"harder\" this is. This seems quite reasonable and standard.", "theoretical_claims": "I followed the theoretical argument, but did not check the correctness of the proofs.", "experimental_designs_or_analyses": "Yes, see \"Claims And Evidence\" and \"Methods And Evaluation Criteria\" for particular cases I have issues with. Aside from these, the authors do indeed do a very thorough job of providing exhaustive empirical results and provide nice analysis of the results. My issues with this work mostly lay in the claims made about the results, given the experimental settings, rather than the results themselves.", "relation_to_broader_scientific_literature": "The authors provide thorough connections to related work. So far, no previous work has found that downstream performance could be harmed by overtraining, although there have been works that show that pretraining evaluations can improve with overtraining and in particular Gadre et al. 2024 show that a scaling does exist between overtraining and downstream performance, but this is more aggregated across a larger set of models and tasks.", "essential_references_not_discussed": "A relevant work might be https://openreview.net/pdf?id=vPOMTkmSiu.", "other_strengths_and_weaknesses": "Other strengths:\n- The paper is well written free of mistakes. It is easy to follow and read, although including more pertinent information from the Appendix in the main text would improve the writing and let the reader assess the claims easier.\n\nOther weaknesses:\n- The theoretical argument is quite limited (two-layer linear model), and is unclear what value it adds to the work. The space could be more useful to provide more detailed discussion of the results, methodology, and relationship to scaling model size.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How would you define what it means to be harder to fine-tune, and what is the main evidence in your paper that supports this specifically?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3a5HzQNujH", "reviewer": "Reviewer_isiV", "summary": "This work challenges the widely held belief in the field that scaling pre-training robustly improves LM performance. The authors find that increasing token budgets during pre-training can actually lead to suboptimal performance on downstream fine-tuned tasks. They leverage popular open-source models and datasets to provide empirical evidence supporting this claim.\n\nThe authors conjecture that the suboptimal fine-tuning performance can be attributed to overly complex features of the pre-training distribution, which are learned toward the end of training. Drawing insights from the transfer learning literature, they provide theoretical evidence for this hypothesis using small models.\n\nTaken together, this work sheds light on how design decisions made during pre-training can have ripple effects on post-training outcomes.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback! We are happy to hear your positive comments that there is “compelling empirical evidence”, that “the studies involving intermediate OLMo checkpoints and newly trained models from scratch are especially persuasive”.\n\n\n> During pre-training, it may be difficult to identify precisely when overtraining has occurred.  [...] A helpful step the authors could take to operationalize their findings would be to suggest heuristics based on model weights or activations.\n\nGreat suggestion! Developing heuristics based on model weights or activations is indeed a valuable direction for future work. Another promising avenue would be creating scaling laws specifically for catastrophic overtraining. With accurate scaling laws, practitioners could significantly reduce computational overhead by performing fewer post-training experiments to identify optimal training durations.\n\n\n> In Section 6, the authors discuss their rationale for not conducting any staged pre-training (also known as annealing or mid-training) [1, 2]. It would be valuable to further expand on this point in the discussion or include additional experiments in the appendix that explore scenarios involving additional domain-oriented pre-training. It is possible that concluding pre-training with domain-specific data might mitigate much of the fine-tuning performance penalty associated with overtraining.\n\nThis is a great point. In our current experiments, our smaller-scale models are pre-trained on the C4 dataset (which includes some code), and we demonstrate catastrophic overtraining on Starcoder-Python fine-tuning (Figures 2, 3). This suggests domain-oriented data doesn't completely eliminate the problem.\n\nHowever, systematically understanding how the choice of pre-training distribution affects catastrophic overtraining remains important future work. Still, we believe adding domain-specific data cannot universally solve this issue: it’s practically impossible to include data from all potential fine-tuning domains. For example, catastrophic overtraining even arises in multimodal fine-tuning scenarios, where adding image data during pre-training (to a text-only model) simply isn't possible.", "claims_and_evidence": "The authors provide compelling empirical evidence supporting their claims. In particular, the studies involving intermediate OLMo checkpoints and newly trained models from scratch are especially persuasive.", "methods_and_evaluation_criteria": "The methods and evaluation criteria suite the research questions.", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper contributes to the literature on LM pre-training dynamics. There is growing interest in better understanding how decisions made during pre-training affect final model behavior after post-training. Through the lens of fine-tuning performance, this work suggests that overtrained base models may become more difficult to fine-tune for downstream tasks.\n\nThis paper relates to the literature examining the degree to which post-training robustly modifies LM behavior [1]. The authors' contributions suggest that the robustness of post-training behavior may be improved by mitigating overtraining.\n\n[1] - Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., & Henderson, P. (2024). Safety Alignment Should Be Made More Than Just a Few Tokens Deep. ArXiv, abs/2406.05946.", "essential_references_not_discussed": "I am not aware of any essential references that have been excluded from this work.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "During pre-training, it may be difficult to identify precisely when overtraining has occurred. One approach practitioners might adopt is applying post-training to intermediate checkpoints, similarly to the OLMo experiments. However, repeated post-training runs might be infeasible or prohibitively expensive. A helpful step the authors could take to operationalize their findings would be to suggest heuristics based on model weights or activations. These heuristics could help practitioners predict whether training should halt due to overtraining, without needing to rely on repeated post-training experiments.\n\nIn Section 6, the authors discuss their rationale for not conducting any staged pre-training (also known as annealing or mid-training) [1, 2]. It would be valuable to further expand on this point in the discussion or include additional experiments in the appendix that explore scenarios involving additional domain-oriented pre-training. It is possible that concluding pre-training with domain-specific data might mitigate much of the fine-tuning performance penalty associated with overtraining.\n\n[1] - OLMo, Team et al. “2 OLMo 2 Furious.” ArXiv abs/2501.00656 (2024): n. pag.\n\n[2] - Dubey, Abhimanyu et al. “The Llama 3 Herd of Models.” ArXiv abs/2407.21783 (2024): n. pag.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zeNDp472KJ", "reviewer": "Reviewer_YWVH", "summary": "This paper investigates the phenomenon of \"catastrophic overtraining\" in language models, where models trained on significantly more tokens than compute-optimal regimes exhibit degraded performance after fine-tuning, despite showing continued improvement in pre-training loss.  The authors present both empirical and theoretical evidence to support this claim.\n\n## update after rebuttal\n\nSatisfied with responses to questions : Rating remains \"4: Accept\"", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback! We are happy to hear your positive comments such as that our methodology is “appropriate, comprehensive, and designed [...] rigorously”, that our paper addresses a problem that is “highly relevant and timely”, and that our results are “novel and counter-intuitive, challenging conventional wisdom”.\n\n\n> Given the simplifications involved in the linear models, the theoretical results supply intuitions about the mechanism, rather than directly supporting the observed results\n\nWe aren’t aware of any theoretical tools that would enable us to study the dynamics of LLMs without at least some degree of simplification such as the ones we describe. We fully acknowledge that there are limitations that arise from these simplifications but we hope our analysis is a starting point to understand this phenomenon that may lead to principled mitigation strategies in the future.\n\nAdditionally, since catastrophic overtraining is so counterintuitive (to us), seeing it emerge from incremental learning in a simpler setting reassures us it's not just an artifact of our LLM setup and offers intuition for a possible mechanism.\n\n>The claim for $L_{\\mathrm{pre}}$ (base model capabilities) is a bigger and less definitively backed claim than the claim for $L_{\\mathrm{ft}}$ (fine-tuning performance).\n\nWe want clarify our claim regarding how extending pre-training can reduce the base model capabilities:\n\nOur core finding is that there are some important settings where additional pre-training hurts the fine-tuned model’s capabilities, despite helping the base model’s performance. We don’t intend to claim that all settings necessarily exhibit catastrophic overtraining after hyperparameter tuning.\n\nIn Figure 2, we find that the fine-tuned model’s “general capabilities” (blue lines) are degraded for PIQA, ARC Challenge, ARC Easy when instruction tuning, and PIQA, ARC Challenge when multimodal tuning. In Figure 3, we find that the fine-tuned model’s “general capabilities” (bottom) degrade for GSM8k, Starcoder-Python, MR, and RTE.\n\nHopefully this clarifies our evidence that the general capabilities can degrade with overtraining.\n\n\n> Beyond regularization, are there other potential techniques that could mitigate catastrophic overtraining and improve the fine-tuning adaptability of overtrained models? For example, could techniques like parameter resetting, learning rate scheduling adjustments during fine-tuning, or architectural modifications be explored?\n\nGreat question! We do explore various learning rate schedules during fine-tuning (constant, constant + warmup, cosine schedules), as detailed in Appendix C.2, and found catastrophic overtraining persisted in each case. Exploring additional mitigation techniques (e.g., parameter resets, architectural modifications) would indeed be valuable future work.\n\n\n> In the \"optimal learning rate trend\" analysis (Section 3.4, Figure 5): You categorize datasets based on whether the optimal learning rate is constant, slowly decreasing, or quickly decreasing. Could you provide further intuition or hypotheses about why certain datasets exhibit each trend? What characteristics of the datasets or fine-tuning tasks might determine the optimal learning rate trend and its relation to overtraining degradation?\n\nGreat question! While empirical datasets are difficult to analyze precisely, our theoretical results offer helpful intuition:\n\nTheorem 4.3 shows that extending pre-training increases the sensitivity (or forgetting) of pre-trained features during fine-tuning. This sensitivity is larger for tasks whose feature distributions significantly differ from pre-training (i.e., larger eigenvalue differences)—fine-tuning tasks which are much different from the pre-training task are more prone to catastrophic overtraining. In our revised version of the paper, we will clarify this with an appropriately updated theorem statement.\n\nOur intuition is that this theoretical insight translates roughly to practice. Empirically, tasks very dissimilar to pre-training likely require larger changes in model features, and larger learning rates naturally induce larger weight changes (even controlling for steps). We’ve empirically verified that larger learning rates lead to larger weight changes and will add a figure illustrating it clearly. Thus, tasks needing significant feature adaptation typically benefit from larger learning rates and, as a result, exhibit more pronounced catastrophic overtraining.", "claims_and_evidence": "* Claim: Overtrained language models are harder to fine-tune.\n  + This is well supported by experiments\n* Claim: Catastrophic overtraining exists, characterized by thresholds $T_{ft}$ and $T_{pre}$ (fine-tuning and pre-training performance, respectively) degrade when these thresholds are exceeded.\n  + $T_{ft}$ well supported by experiments, $T_{pre}$ less so.\n* Claim: Theoretical characterization in linear models supports the empirical findings.\n  + Given the simplifications involved in the linear models, the theoretical results supply intuitions about the mechanism, rather than directly supporting the observed results", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate, comprehensive, and designed to rigorously investigate the $T_{ft}$ research question.  The evaluations for the $T_{pre}$ element are weaker.", "theoretical_claims": "While I have not rigorously checked the proofs line-by-line, the informal statements and the overall theoretical approach seem reasonable.  The theoretical framework provides a valuable conceptual understanding of the overtraining phenomenon.", "experimental_designs_or_analyses": "* By varying pre-training token budgets and keeping model size constant, the authors effectively isolate the variable of interest (overtraining). Using intermediate checkpoints of pre-trained models and training models from scratch for controlled experiments further strengthens the design.\n* Evaluating on both downstream task performance and generalist performance provides a holistic picture of the impact of overtraining. Using multiple diverse benchmarks strengthens the generalizability of the findings.", "relation_to_broader_scientific_literature": "* Scaling Laws: The paper investigates a regime that deviates from compute-optimal scaling. \n* Overtraining and Generalization: The paper shows that overtraining in pre-training can negatively impact transfer learning ability, which is a novel and significant finding in the current paradigm of large-scale pre-training.", "essential_references_not_discussed": "N/a", "other_strengths_and_weaknesses": "* The paper addresses a highly relevant and timely question in the field of large language models. The finding that overtraining can degrade fine-tuning performance is novel and counter-intuitive, challenging conventional wisdom.\n* The claim for $L_{pre}$ (base model capabilities) is a bigger and less definitively backed claim than the claim for $L_{ft}$ (fine-tuning performance).", "other_comments_or_suggestions": "N/a - and I usually find typos...", "questions_for_authors": "1. Mitigation strategies (Section 6 and future work): Beyond regularization, are there other potential techniques that could mitigate catastrophic overtraining and improve the fine-tuning adaptability of overtrained models? For example, could techniques like parameter resetting, learning rate scheduling adjustments during fine-tuning, or architectural modifications be explored? \n\n2. In the \"optimal learning rate trend\" analysis (Section 3.4, Figure 5): You categorize datasets based on whether the optimal learning rate is constant, slowly decreasing, or quickly decreasing.  Could you provide further intuition or hypotheses about why certain datasets exhibit each trend? What characteristics of the datasets or fine-tuning tasks might determine the optimal learning rate trend and its relation to overtraining degradation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary: This paper studies how overtraining LLMs impacts their fine-tuning effectiveness. Through extensive analysis of small scale models, the authors find that pretraining on too many tokens can degrade downstream fine-tuning performance, even as pretraining loss continues to improve. The authors call this as “catastrophic overtraining” and support their claim with both experiments and theoretical framework.

Review summary:
All reviewers agree the paper presents an important insight into LLM training dynamics. The paper provides extensive empirical evidence across model sizes and tasks that overtraining can degrade fine-tuning performance. 

Recommendation:
I recommend acceptance.
Decision: Accept (poster)