Amuro and Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models

## Abstract

Abstract The development of large language models leads to the formation of a pre-train-then-align paradigm, in which the model is typically pre-trained on a large text corpus and undergoes a tuning stage to align the model with human preference or downstream tasks.
In this work, we investigate the relationship between pre-training and fine-tuning by fine-tuning multiple intermediate pre-trained model checkpoints. Our results on 18 datasets suggest that i) continual pre-training improves the model in a latent way that unveils after fine-tuning; ii) with extra fine-tuning, the datasets that the model does not demonstrate capability gain much more than those that the model performs well during the pre-training stage; iii) although model benefits significantly through supervised fine-tuning, it may forget previously known domain knowledge and the tasks that are not seen during fine-tuning; iv) the model resembles high sensitivity to evaluation prompts after supervised fine-tuning, but this sensitivity can be alleviated by more pre-training. 1 1 1 Code, results, and data to reproduce the experiments are available at https://anonymous.4open.science /r/AmuroCharRelease-DEC5 . All the model checkpoints resulting from this work are available at https://huggingface.co/KaiserWhoLearns/PTvsSFT_OLMo1b

## 1 Introduction

The rise of large language models (LLMs) as a general-purpose tool for a diverse range of natural language processing tasks has dramatically transformed the field, introducing new paradigms for data collection and model training
(Brown et al., , Biderman et al., , Touvron et al., ,
Jiang et al., ,
Chowdhery et al., , Groeneveld et al., , Wang et al., , inter alia).
Numerous models, training methods, datasets, and evaluation methods continue to be developed on an ongoing basis.
Nevertheless, a unified paradigm has emerged for training LLMs: pre-train on an enormous corpus of diverse documents, ranging from 250B Biderman et al. () to 15T AI@Meta () tokens, followed by an alignment stage to make the model more useful and performative for various tasks.

Figure: Figure 1: Illustration of the experimental scheme.
Refer to caption: extracted/5791979/figures/teaser.jpeg

Based on this paradigm, work has focused on improving these two stages.
Work to improve pre-trained models includes larger training sets Hoffmann et al. (); AI@Meta (); Touvron et al. (), different data selection mechanisms Xia et al. (), higher quality data Zhou et al. (), and various model architectures Su et al. (); Touvron et al. ().
Meanwhile, research on model alignment includes different training objectives Rafailov et al. (); Schulman et al. (),
new datasets Narayanan and Aepli (), more efficient training Hu et al. (); Dettmers et al. () and safety tuning Bianchi et al. (). The alignment stage usually involves either supervised fine-tuning for specific tasks or instruction fine-tuning for general-purpose usage.
Regardless, fine-tuning (almost always) comes at the end of pre-training and yields remarkable improvements on downstream tasks Touvron et al. (); Groeneveld et al. ().
Consequently, the benefits of each stage are largely explored independently, with improvements to pretraining being orthogonal to benefits from model alignment.

Rather than explore these two training regimes independently, we question: how do model pretraining and fine-tuning interact to affect the resulting model? Does more pre-training hinder better fine-tuning results? What does the model learn and forget during pre-training as well as fine-tuning?
Answering these questions requires us to examine how models learn during pre-training and how this affects fine-tuning. Therefore,
we fine-tune multiple pre-training checkpoints of a large language model (Figure 1), evaluating each checkpoint and its fine-tuned version on downstream evaluation sets.
We track model abilities during pre-training and compare them to improvements achievable after fine-tuning at the corresponding pre-training step.
We explore both supervised and instruction fine-tuning, testing the models’ memorization and forgetting when learning specific tasks and serving as general-purpose language-AI tools.
To the best of our knowledge, we are the first to explore fine-tuning intermediate model checkpoints.

Our experiments yield insights into LLM training.
We find that (1) continued pre-training can improve a model in ways that are only revealed after fine-tuning (§5); (2) tasks for which the model already performs well during pre-training benefit much less from fine-tuning than those where the model does not demonstrate capabilities (§4, §5); (3) although supervised fine-tuning can improve performance on in-distribution tasks, it can also cause the model to forget domain knowledge or tasks that it was previously capable of solving (§6); (4) fine-tuned models show high sensitivity to evaluation prompts, but this sensitivity can be alleviated by more pre-training (§6).
Our findings provide insights into model training and can inform methods for both pre-training and fine-tuning. Furthermore, our work shows the value of analyzing the training dynamics, in addition to analyzing the final LLM, as an aspect of interpretability, and we encourage model developers to release these checkpoints to aid future studies.

## 2 Background: Model Training

We begin with a brief survey of the core components of LLM training: pre-training, fine-tuning, and instruction fine-tuning.
We also discuss the related topic of in-context learning as well as different efficient fine-tuning strategies.

We use “model alignment” as a general term for techniques that align a model with a desired behavior, which can be accomplished by fine-tuning models after pretraining. The term is also associated with other definitions Shen et al. ().
We also note several related studies that explore training dynamics to understand model behavior Tirumala et al. (); Chen et al. (); Tian et al. ().
With this in mind, we conduct an empirical study on how the amount of pre-training affects the effectiveness of fine-tuning.

#### Pre-training

The first step of training a LLM is pre-training on a massive text corpus
Achiam et al. (); Touvron et al. (); Groeneveld et al. ().
For decoder-only models in the GPT family, the subject of our paper, work since the introduction of GPT-2 Radford et al. () has focused on scaling up model training.
Initial work increased model size to hundreds of billions of parameters Brown et al. (); Rae et al. (); Chowdhery et al. (), along with explorations in model size, training corpus size, and training data characteristics Hoffmann et al. (); Gururangan et al. ().
Since the push towards large models, work has shifted to increasing the amount of pre-training data Computer (); Soldaini et al. (), with new models now reaching 15 trillion tokens AI@Meta ().
Studies of model performance on various tasks at different model sizes introduced the idea of emergent model abilities Wei et al. (), with new model abilities being revealed as model training grows.

We also recognize a particularly important trend for this paper: model openness.
Early LLMs were proprietary models accessible only through an API.
The first large open model, Bloom Bloom Ström et al. (), allowed widespread LLM evaluation.
Subsequent open models, such as OPT Zhang et al. (), LLaMA Touvron et al. (); Keles and Bayraklı () and others Biderman et al. (); Gururangan et al. (); Almazrouei et al. (), have become the norm.
In this paper, we study OLMo Groeneveld et al. (), one of the only models to release individual pre-training checkpoints.

#### Fine-Tuning

Early work on instruction fine-tuning using reinforcement learning with human feedback (RLHF) Ziegler et al. (); Stiennon et al. (); Ouyang et al. () demonstrates the dramatic effect that model alignment could have on a pre-training model.
When a specific task of interest has been identified, supervised fine-tuning can improve a pre-trained model.
Task-agnostic tuning became popularized with the advent of T5 models (Raffel et al., ), where a pre-trained LLM is tuned using a general text-to-text solution.
When multiple tasks are given to the model, the model is commonly given a task-specific prefix or an instruction along with the task input, leading to the development of various methods of prefix tuning Li and Liang () and instruction tuning Wei et al. (); Mishra et al. (); Victor et al. ().

#### Instruction Fine-Tuning

Instruction fine-tuning is preferred when more general model behaviors are desired.
Popularized through reinforcement-learning with human feedback (RLHF) Christiano et al. (); Ziegler et al. (); Stiennon et al. (); Ouyang et al. () and reinforcement-learning with AI feedback (RLAIF) Lee et al. (), these methods utilize a reward model to simulate human feedback.
Others explore human preference tuning without a reward model Rafailov et al. (); Song et al. (); Xu et al. (), or study the effects of these tuning methods (Shen et al., ; Perez et al., ).
Sharma et al. () show that supervised fine-tuning can lead to similar performance as RLAIF.

#### In-Context Learning

While not the subject of this paper since it does not make changes to model parameters, in-context learning utilizes a small amount of supervised data to improve model performance.
ICL, also called few-shot learning,
is also used as an evaluation strategy where the model is given a prompt composed of examples of tasks expected to be solved. The underlying model is evaluated based on its response to the input.
ICL can benefit from a larger context window that adds more examples, which can spur work on the development of model quantization techniques Dettmers et al. () and the alleviation of hardware constraints Brown et al. (); Xie et al. (); Min et al. ().

#### Fine-Tuning Techniques

While model pre-training can be done by a few groups with large resources interested in developing new models, fine-tuning depends on the task and is of broad interest. Therefore, many techniques facilitate time-, memory-, and data-efficient model training through parameter-efficient fine-tuning (PEFT) Hu et al. (); Liu et al. (, ), quantization Jacob et al. (); Dettmers et al. (, ), and specialized data filtering Xia et al. (); Zhou et al. (); Attendu and Corbeil ().
This paper focuses specifically on full-parameter fine-tuning, while our findings suggest the potential for data-efficient and budget-friendly training by understanding the critical turning point of model training.
Our findings are closely related to the recent study on phase transition of model training Olsson et al. (); Wei et al. (); Chen et al. ().

## 3 Experimental Setup

In this section, we describe the model and datasets used.
The hyperparameter tuning procedure and setup for each fine-tuning setting can be found in Appendix A.

### 3.1 Model Choice

Our paper considers OLMo-1B Groeneveld et al. (), a high-performing open-source large language model.
Ideally, we would evaluate multiple models, but OLMo is the only model to release intermediate pre-training checkpoints, and thus the only model that supports our analysis(^2^22[https://github.com/allenai/OLMo/tree/main/checkpoints](https://github.com/allenai/OLMo/tree/main/checkpoints/official)) (^3^33We also experimented with RedPajama-INCITE ([https://www.together.ai/blog/redpajama-models-v1](https://www.together.ai/blog/redpajama-models-v1)), which is the only other model to release checkpoints. After extensive experiments, we found it performed worse than OLMo, given the training data available, and did not support our analysis. Several other models claim to release training checkpoints but have not done so.).
Despite being the only open model with training checkpoints, it fortunately has several desirable properties. First,
the model is fully open, including the training details, pre-training data, and fine-tuning data.
Second, the smaller model size allows us to train a model efficiently on a single A100 GPU. While evaluating a larger model would be desirable, we limit our study to the 1B model given the much larger computational demand of multi-GPU training. Our detailed analysis required significant GPU resources, which would have been prohibitive with a larger model.
We also note that OLMo-1B compares very favorably to the larger version, and recent work has shown that small models can compete with larger ones Riviere et al. ().

We select model pre-training checkpoints uniformly from the pre-training history along with the first and the final checkpoints.

### 3.2 Training Procedure

We fine-tune each of the selected model checkpoints using two different procedures to create fine-tuned models: supervised fine-tuning and instruction tuning.
The supervised fine-tuning is conducted separately for each model checkpoint and dataset, while the instructing fine-tuning is done once using the instruction dataset.
The instruction-tuned model is evaluated on a suite of LLM benchmarks.

#### Supervised Fine-tuning

We adapt the dataset choice from Yang et al., for supervised fine-tuning.
For each in-domain dataset, one to two cross-domain evaluation datasets are supplied.
Each pre-training checkpoint is fully fine-tuned for 3 epochs with a batch size of 8 and learning rates resulting from minimal hyperparameter tuning.
Each task is formatted using a default prompt-completion format (Table 4).

#### Instruction Fine-Tuning

We instruction-tune the model on TÜLU Ivison et al. (), following the decision of Groeneveld et al., .
Each model checkpoint is fully fine-tuned for 5 epochs with a batch size of 8 and a learning rate of $2\times 10^{-6}$.

**Table 1: Dataset information. For Generation tasks, ROUGE-L is used as evaluation metric, and accuracy is used for classification tasks.**
| Supervised Fine-Tuning |  |  |  |
| --- | --- | --- | --- |
| Task | Training | ID Test | OOD Test |
| Summary<br>Generation | XSum | XSum,<br>XLSum | CNN |
| Question<br>Generation | SocialIQa | SocialIQA | SciQ,<br>TweetQA |
| Natural Language<br>Inference | MNLI | MNLI1,<br>MNLI2 | RTE,<br>GPT3NLI(^4^44[https://huggingface.co/datasets/pietrolesci/gpt3_nli](https://huggingface.co/datasets/pietrolesci/gpt3_nli)) |
| Paraphrase<br>Detection | Paws | Paws | QQP,<br>STS-B |
| Instruction Tuning |  |  |  |
| Dataset | Description |  |  |
| TÜLU-v2 | A mixture of instruction datasets. |  |  |
| ARC | Grade-school multiple-choice QA. |  |  |
| OpenbookQA | Open book exam QA. |  |  |
| Hellaswag | Commonsense inference. |  |  |
| BoolQ | Reading comprehension. |  |  |
| SciQ | Science exam multiple choice QA. |  |  |

### 3.3 Evaluation

The evaluation challenge is to select a representative number of datasets for different types of tasks to test model abilities, recognizing that each dataset requires evaluating each model checkpoint and its fine-tuned counterparts.
We also select datasets based on the availability of in-domain and out-of-domain samples.

#### Datasets

The datasets are summarized in Table  1.
We evaluate the model with an in-domain test set and one or two out-of-domain test sets for each of the supervised fine-tuning tasks.
We conduct experiments on the tasks of summary generation Narayan et al. (); Hasan et al. (); Hermann et al. (), question generation Sap et al. (); Xiong et al. (); Welbl et al. (), natural language inference Williams et al. (); Wang et al. (); Dagan et al. (); Bar Haim et al. (); Giampiccolo et al. (); Bentivogli et al. (), and paraphrase detection Zhang et al. (); Wang et al. (); Agirre et al. ().
Each training set is sub-sampled to a size of 6,000 for fair comparisons.

In instruction fine-tuning, we base our downstream evaluation settings on Groeneveld et al., , as OLMo is found to have stable performance on these datasets.
The instruction-tuned models are evaluated on ARC (both arc easy and arc challenge) Clark et al. (), OpenbookQA Mihaylov et al. (), Hellaswag Zellers et al. (), BoolQ Clark et al. (), and SciQ Welbl et al. ().

#### Metrics

We use accuracy Pedregosa et al. () for classification tasks and Rouge-L Lin () for generation tasks.
We set the maximum amount of newly generated tokens to 5 for classification tasks and 60 for generation tasks.
Outputs are generated with greedy decoding.
For classification tasks, we experiment with both constrained decoding and logit-based predictions.
We find the best performance by selecting the label with the highest logit of its first subtoken (Appendix B).

Figure: ((a)) Learned during Pre-training.
Refer to caption: x1.png

Figure: ((a)) MNLI Matched
Refer to caption: x3.png

Figure: Figure 4: Amount of performance increase brought by fine-tuning between tasks that model can solve in pre-training (mandarin orange) and tasks that the model could not solve until fine-tuning (sage green). The exact number of mean increase is shown in Appendix G.
Refer to caption: x5.png

## 4 How does the model change across pre-training?

We begin our evaluation by considering how additional pre-training changes the BASE model. Typically, researchers track the value of the training or held-out loss during training.
However, performance improvements on downstream tasks do not always follow the same trend with the loss curves Groeneveld et al. ().

We evaluate the pre-trained checkpoints with few-shot examples, as models without alignment tend to do poorly in a zero-shot context.
Four shots are randomly sampled from the datasets, which are selected based on the highest performance shot amount reported in Yang et al., .
The model’s performance at each pre-training step is reported in Figure 2.

Broadly speaking, our results suggest that all datasets fall into one of two groups.
For the first group of datasets (Figure 2(a)), although the model shows clear improvement during the early stages of pre-training, performance levels off fairly early on and remains consistent. The dramatic improvements in the early stages of pre-training may result from larger steps in early optimization.
We find improvements stop increasing past step 342,000.
The second group (Figure 2(a)) shows tasks that are never learned during pre-training.
Performance remains constant throughout the whole pre-training process.
These datasets include MNLI, XSum, and BoolQ, and we found no difference between zero-shot and few-shot evaluations.
A natural hypothesis for this finding is potential data contamination in the pre-training data.
However, the evaluation datasets are selected based on the popularity of the task and the content of pre-training data.
All datasets that experience improvement do not exist in the model’s pre-training data Soldaini et al. (), while the more likely leaked datasets (MNLI, XSUM) never gain an improvement during the pre-trining process.

Overall, these results reveal an interesting dichotomy.
Some tasks can be learned during pre-training, while others are not.
Next, we explore what exactly the model is learning regarding this second group of datasets during pre-training by exploring the fine-tuned models.

## 5 Does more pre-training improve fine-tuning?

Groeneveld et al., compares OLMo’s performance on several tasks before and after fine-tuning the final checkpoint and finds that fine-tuning enables the model to do well on tasks for which the unaligned model does poorly. We observe (§4) that while some datasets improved
during pre-training, there is a group of datasets for which a pre-trained model does poorly.
Does the model learn anything that helps solve these tasks, and is fine-tuning required to do well on them?
Alternatively, does the model learn useful information for these tasks but cannot express it without fine-tuning? In this section, we further explore this dataset dichotomy by examining fine-tuned checkpoints for each of the datasets.

Our results appear in Figure 3 and Figure 4. First, we consider those datasets where the pre-trained models do well (Figure 2(a)).
These datasets do not improve with fine-tuning, suggesting whatever is learned during fine-tuning, which we discuss below, the model already gains the knowledge during pre-training. This effect is observed at all checkpoints; fine-tuning simply does not help.

However, a different story is observed for datasets that are not learned during pre-training.
For these, fine-tuning yields significant improvements at every model checkpoint, with Figure 4 showing the magnitude of improvement on these datasets compared to no improvement to the datasets already learned during pre-training.
Moreover, earlier checkpoints obtain more substantial gains from fine-tuning than later checkpoints.
The benefit of fine-tuning continues to increase until a certain threshold in pre-training steps is reached (approximately 424,000).

Figure 3 shows representative plots comparing the performance of a pre-trained versus fine-tuned model at different checkpoints for two datasets (full list in Appendix E).
For Hellaswag (learned during pre-training), fine-tuning does not benefit the model, even during early checkpoints when the model performs poorly on the task.
Nevertheless, for MNLI (not learned during pre-training), fine-tuning dramatically improves the model.
Interestingly, later checkpoints achieve better results after fine-tuning, even when the performance of the pre-trained model is unchanged.
This suggests that the model is, in fact, learning important information during pre-training, but it cannot express that information without fine-tuning.

Our findings suggest that early stopping in pre-training will not be detrimental to downstream fine-tuning performance, and the benefits of fine-tuning an LLM could exceed the benefits of continued pretraining, which sheds light on the potential of cost-effective training paradigm with less pre-training.
However, it is difficult to directly identify such a stopping criteria without fine-tuning intermediate checkpoints; the improvement trend is invisible before fine-tuning the checkpoints.
Future work may reveal other signals of pre-training behavior that correlate with downstream task performance after fine-tuning.
Overall, when resource-intensive pre-trained LLMs are not available, fine-tuning models on models with less pre-training may be a reasonable practical choice for obtaining a high-quality model.

## 6 Supervised Fine-Tuning: What does the model learn and forget?

What exactly is the model learning during fine-tuning such that it shows abilities in pre-trained models for some tasks but provides no benefit for other tasks?
We analyze the supervised fine-tuning process to understand what is learned and what is forgotten.
Specifically, we explore three dimensions: task format, task transfer, and domain knowledge.

Figure: ((a)) MNLI matched
Refer to caption: x6.png

### 6.1 Task Format

Sclar et al., show that LLMs are extremely sensitive to prompt perturbation in few-shot settings.
More broadly, extensive work on prompt engineering reveals the sensitivity of models to task format.
We hypothesize that fine-tuning fits the model to a specific task format, resulting in higher performance when the evaluation set matches this format.
To test this hypothesis, we vary the task format to either match the training format, use a different format, or rely on instructions.
We carefully construct three different prompt formats for the following settings. 1)
Default is the same format used for training, where we expect the model to benefit from learning the task format;
2) In contrast, IO format reflects a common way of performing supervised fine-tuning by incorporating only unprocessed input and output;
3) Instruct uses a human-readable instruction template to format the input.
Table 4 shows an example of each format.
Checkpoint performance before and after fine-tuning is shown in Figure 5.

In the early pre-training steps, aligning the task format with fine-tuning data seems to play a crucial role.
The model does not yet have enough information to overcome the differences between the training and test formats.
However, when fine-tuned on later pre-training checkpoints, the model gradually becomes more flexible with different task formats, suggesting that model sensitivity to prompt formatting observed may be resolvable with more pre-training and a fine-tuning stage. In this view, fine-tuning teaches the model how to format a response for the task.

### 6.2 Task Transfer

Numerous studies examine model forgetting, where further model training causes improvements on some tasks but degradation on others Mehta et al. ().
We evaluate model forgetfulness by examining whether the model does worse on some tasks after fine-tuning for other tasks.
Specifically, we divide our tasks into two types: classification and generation.
We notate the training datasets as $D_{T}$ and the evaluation datasets as $D_{E}$.
We represent the performance of a pre-trained model (BASE) on checkpoint $i$ as ${\text{Perf}}_{BASE}^{i}(d)$ where an evaluation dataset $d\in D_{E}$, and the performance of the i-th checkpoint fine-tuned on dataset $t\in D_{t}$ be $\text{Perf}_{t}^{i}(d)$. To normalize the effect caused by uneven performance across different datasets, we compute the mean ratio of change (MRC) in performance for each checkpoint as follows.

$$ $\text{MRC}=\frac{1}{|D_{E}\setminus\{t\}|}\sum\limits_{\forall d\in D_{E},d \neq t}{\frac{\text{Perf}_{t}^{i}(d)-{\text{Perf}}_{BASE}^{i}(d)}{\text{Perf}_ {BASE}^{i}(d)}}$ $$

Models fine-tuned on classification tasks and evaluated on generation tasks decrease on average 61.4% compared to models that are never fine-tuned.
In contrast, models fine-tuned on generation tasks can still perform the same as the BASE model on classification tasks, with a 0.3% MRC, which is not statistically significantly different from a 0% change.
Our findings on all pre-training checkpoints align with the findings of Yang et al. () on the final checkpoint of LLAMA-7B.

Regardless of the pre-training stage, a model can maintain classification abilities when trained for generation, but it loses its generation abilities when trained for classification.
This is perhaps not surprising given that classification tasks can be seen as a subset of generation, while the reverse is not true.
The model follows a simplicity bias and thus is more likely to memorize simple classification tasks than generation tasks with an exponentially larger search space.
Additionally, since we evaluate the classification tasks based on the output logits and the base model performs randomly on the classification tasks, it is much easier for the models to maintain the same performance as the BASE models.
Fine-tuning can cause a model to lose abilities when the desired fine-tuning behavior does not support those abilities.

Figure: ((a)) Paws $\rightarrow$ QQP
Refer to caption: x8.png

Figure: Figure 7: Ratio of out-of-domain performance change for each task, averaged across checkpoints
Refer to caption: x10.png

### 6.3 Domain Knowledge

Finally, we explore how a model’s generalization ability is affected by fine-tuning by inspecting whether the model forgets the domain knowledge it had before fine-tuning due to learning other abilities.
An example of OOD model performance is shown in Figure 6, and the mean change ratio by datasets is presented in Figure 7.

The model does not benefit equally from the in-domain fine-tuning: all NLI datasets experience a boost when fine-tuning on MNLI, while fine-tuning on Paws is detrimental to other paraphrase detection datasets.
This implies that both forgetting and learning are happening: the model learns to perform the task with in-domain knowledge, but it may, in turn, forget information more distant from what is learned in fine-tuning.
Questions remain, however, about whether there are different stages of learning and forgetting during fine-tuning and whether the model picks up different tasks in various stages, which requires further study of fine-tuning dynamics.

Overall, across these three lenses, we find that fine-tuning, although teaches a model how to perform a task, can sacrifice generalization abilities if such ability is not needed for the fine-tuned task.
For some datasets learned with pre-training alone, the model can easily understand the task format, and the nature of the task is probably supported by the pre-training objective.
For tasks that can only be learned with subsequent fine-tuning, the model may require additional examples to adapt to different task formats, or the task itself may be inconsistent with the pre-training objective.

## 7 Discussion

Our study uses fine-tuning of pre-training model checkpoints to understand the dynamics of pre-training and fine-tuning on model performance.
While our insights suggest directions for future work, we note important limitations inherent in our experiments.
This study considered a single, relatively small LLM on less than a dozen datasets, and still consumed thousands of hours of GPU training time at significant expense.
Future work needs to confront these issues on larger models and more datasets. We believe our experiments can focus future work on specific experiments with larger models.

Some datasets can be learned without fine-tuning.
We discover a dichotomy between datasets.
Some are learned during model pre-training, while others show no improvements during pre-training.
Furthermore, the datasets learned during pre-training do not benefit from fine-tuning.
This observation, combined with our study about what is learned during fine-tuning (Section 6) suggests that some tasks are presented in a manner that aligns with what the model sees during pre-training, and thus fine-tuning provides no additional information.
While we could identify what about the tasks placed them in the learned or not learnable during pre-training group, it may be possible to format tasks in a manner that better aligns with pre-training and makes them learnable.

Pre-training models can improve in undetectable ways without fine-tuning.
Some datasets are not learnable during pre-training but benefit significantly from fine-tuning (§4).
However, these datasets still benefited from additional pre-training, even though those benefits were not revealed without fine-tuning (§5).
Clearly, the model is learning important information about the task, even though it cannot express that information.
The identification of a measure available during pre-training that correlated with post-fine-tuning task performance could be used to guide pre-training and produce models that did better post-fine-tuning.
Perhaps there is a way in which information about these tasks can be included in pre-training, allowing the model to better utilize the massive amount of pre-training data.
For example, early stopping during pre-training could lead to better utilization of limited training resources if we knew when to stop.

Fine-tuning teaches task format but leads to forgetting unused abilities.
Our results show that fine-tuning guides the model to understand the format and complete a given task. As this information diminishes, the model’s overall ability improves.
However, fine-tuning comes at the expense of other model abilities, such as the capability of performing on tasks or domains that are unrelated to the fine-tuning task.
This insight can be helpful in our understanding of the multitask abilities of LLMs, where certain tasks can introduce conflicts during multi-task training Mueller et al. ().

## 8 Conclusion

Our experiments explore the relationship between fine-tuning and pre-training LLMs.
Our findings span from the latent benefits of pretraining to model learning and forgetting during fine-tuning.
Our results show that the model can rapidly pick up the datasets that it could not solve during fine-tuning with only a small amount of supervision.
In the meantime, we identify the aspects that LLM learns and forgets during supervised fine-tuning: task format, task solution, and domain knowledge.
Overall, our results demonstrate the value of analyzing language model training dynamics, and we would like to call for the release of pre-training checkpoints to aid future studies.

## Limitations

We discuss the weaknesses and limitations in the following section.

#### Computing Resource

Due to computational constraints, we can only conduct experiments on a 1B model and a limited amount of datasets.
The amount of GPU hours spent for each experiment in this study is listed in Table 3.

#### Availbility of Pre-training Checkpoints

This study would benefit significantly from including a broader spectrum of models, but the public pre-training checkpoint releases are limited.
Open-source LLMs with intermediate checkpoint release include OLMo Groeneveld et al. (), TinyLLAMA, RedPajama-Incite, OpenLM, and Pythia.
After a series of preliminary experiments, we select these models’ best-performing and robust families.

#### Scaling Law

Recent research shows that the model may resemble emergent capability Wei et al. () when scaled to a certain size.
Comparatively, Hassid et al., find that smaller model is capable of outperforming its larger variant when the computing resources is controlled.
To avoid potential confounding factors caused by quantization, our experiments are only conducted on the one-billion model, which may, therefore, conceal the emergent capability brought by larger models while at least giving insights about the potential of small models.

#### Analysis Protocol

Wu et al., show that the evaluation result may be affected by samples that have been memorized by the model during training instead of revealing the reasoning capability.
The only analysis protocol used is the downstream performance of a trained model.
More investigation should be done into model internals during pre-training dynamics and how they relate to the effects of fine-tuning.

#### Training Paradigm

Although multiple tuning strategies exist, to create a fair comparison environment where checkpoints received the same amount of training, models are fine-tuned with a fixed amount of epochs in this work.
On different pre-training stages, the model may converge at a different speed.
Further study can be done to study the effect of pre-training on different fine-tuning methods or fine-tuning dynamics in different pre-training stages.
We only explored the scenario of full-parameter fine-tuning.
Whether parameter-efficient fine-tuning or human preference tuning will lead to a different conclusion also remains an open question.

#### Randomness

In this study, we only assess uncertainty with Bootstrap during evaluation.
However, uncertainty may emerge during training, which poses optimizer initialization and data ordering.
Due to the computational constraints, we cannot reduce the randomness factor on this angle.

## Acknowledgments

The authors thank Saleh Soltan, Niyati Bafna, Fan Bai, Miriam Wanner, Xinbo Wu, Carlos Aguirre for their helpful feedback.

## Appendix A Hyperparameter Tuning

For both supervised fine-tuning and instruction tuning, we pre-set the effective batch size to 8, and tune the learning rate within {$2\times 10^{-5}$, $2\times 10^{-6}$, $2\times 10^{-7}$}.
Each model is fine-tuned for 3 epochs on the supervised fine-tuning tasks and 5 epochs on Tulu for instruction tuning.
In both settings, we adopt an AdamW optimizer with a linear learning rate scheduler.
The optimizer is warmed up for the first $3\%$ of the training time.

## Appendix B Prediction Generation Method

**Table 2: Performance of final checkpoint with different prediction generation method.**
| Dataset | Model | Free | Constrained | TokenProb |
| --- | --- | --- | --- | --- |
| MNLI | Fine-tuned | 0.786 | 0.791 | 0.792 |
| BASE | 0.0 | 0.0 | 0.327 |  |
| RTE | Fine-tuned | 0.658 | 0.662 | 0.66 |
| BASE | 0.0 | 0.0 | 0.241 |  |
| Paws | Fine-tuned | 0.871 | 0.878 | 0.878 |
| BASE | 0.0 | 0.0 | 0.558 |  |
| STS-B | Fine-tuned | 0.775 | 0.741 | 0.744 |
| BASE | 0.0 | 0.0 | 0.964 |  |

For classification tasks, we examine three different prediction generation methods: Free Generation (Free), Constrained Generation (Constrained), and Token Probability (TokenProb), the results are shown in Table 2.
In Constrained, we force the output to include at least one label in the acceptable label set.
In TokenProb, we compare the logits of acceptable labels, and select the label with the highest score as final output.
This ablation study is conducted only on the BASE and fine-tuned versions of final checkpoint of the pre-trained model.
We find that, although prediction generation methods bring less effect to the evaluation result of a fine-tuned model, BASE variants suffer much more from not knowing the desired output.
Therefore, we proceed all the classification experiments with TokenProb.

### B.1 Label and Tokenizations

Depending on the tokenizer variant, the label text may be tokenized differently, leading to evaluation unreliability.
For example, in paraphrase detection, the model could assign probability on both “yes" and “ yes" (the same label with a prefix space).
This behavior is reported and explored in various related work Sun et al. (); Batsuren et al. (); Singh and Strouse ().
In this study, we leniently regard all individual tokens that contains the whole label or part of the label along with some special charcters that do not affect the semantic as an acceptable target label.

## Appendix C Task Format

We adopt the task format from Yang et al. (), with an additional task format of input-output.
How each dataset is formated can be found in Table 4.

## Appendix D GPU Hours per-Experiment

We show a table of GPU hours spent for each experiment in Table 3.
The total number of GPU hours spent in this project is approximately 1067 A100 hours. We lose track of the GPU hours spent on preliminary experiments, so a lower-bound estimation is reported.

**Table 3: GPU hours for each experiment. The total amount of GPU hours spent in this project is approximately 1067 A100 hours.**
| Prelinminary Experiments |  |  |  |  |
| --- | --- | --- | --- | --- |
| Description | GPU Hours |  |  |  |
| Instruction Tuning on LIMA, TULU, and NaturalInstructions | $\sim$300 |  |  |  |
| Model Performance Verification, Dataset Selection | 120 |  |  |  |
| Instruction Tuning |  |  |  |  |
| Instruction Tuning | 360 |  |  |  |
| Evaluation | 10 |  |  |  |
| Total | 370 |  |  |  |
| Fine-Tuning |  |  |  |  |
|  | XSum | SocialIQa | MNLI | Paws |
| Training | 12 | 6 | 4.6 | 5.3 |
| Evaluation | 8 | 5.3 | 3 | 2 |
| OOD Evaluation | 96 | 32 | 11 | 25.6 |
| CrossTask Evauation | 5.2 | 6.5 | 7.7 | 8.15 |
| Task Format Evaluation | 16 | 12.8 | 6 | 4 |
| Total | 137.2 + 62.6 + 32.3 + 45 = 277.1 |  |  |  |

## Appendix E Per-dataset Figures

Figure: ((a)) ARC Easy
Refer to caption: x11.png

Figure: ((a)) MNLI matched
Refer to caption: x17.png

We show the model performance on each dataset after supervised fine-tuning and instruction tuning correspondingly in Figure 9 and Figure 8.
The datasets that already show improvement during pre-training do not benefit from fine-tuning, while performance improve drastically on the datasets that the model has never learned during pre-training.

#### Out-of-domain Generalization

The out-of-domain performance for each dataset with respect to pre-training steps is shown in Figure 10.
Overall, the model generalizes well after fine-tuning on NLI tasks, while its performance deteriorates when evaluated on out-of-domain paraphrase detection tasks.

Figure: ((a)) MNLI -> RTE
Refer to caption: x23.png

#### Cross-task Generalization

The cross-task performance for each dataset with respect to pre-training steps is shown in Figure 11 and Figure 12.

Figure: ((a)) Paws -> MNLI
Refer to caption: x30.png

Figure: ((a)) Paws -> XSum
Refer to caption: x36.png

#### Task-Format

The performance of models on evaluation sets formatted with different prompt formatting method is shown in Figure 13.

**Table 4: Formatting of the prompts**
| Task | Default Prompt | Instruction Prompt | IO Prompt | Expected Output |
| --- | --- | --- | --- | --- |
| Summary<br>Generation | ### Input: {document}<br>### Summary: | Please read the following text: {document}<br>Provide a summary: | {document} | {summary} |
| Question<br>Generation | ### Input: {context}<br>### Answer: {answer}<br>### Question: | Given the context: {context}<br>And the answer: {answer}<br>Generate a suitable question: | {context}<br>{answer} | {question} |
| Natural Language<br>Inference | ### Input_1: {premise}<br>### Input_2: {hypothesis}<br>### Inference: | Consider the following texts: Text 1: {premise}<br>Text 2: {hypothesis} The relation is | {premise}<br>{hypothesis} | {label} |
| Paraphrase Detection | ### Input_1: {sentence1}<br>### Input_2: {sentence2}<br>### Paraphrase Classification: | Let’s compare the two sentences:<br>Sentence_1: {sentence1}<br>Sentence_2: {sentence2} Are they paraphrasing?: | {sentence1}<br>{sentence2} | {label} |

Figure: ((a)) MNLI matched
Refer to caption: x6.png

## Appendix F License of Artifacts

We include the license of artifacts used in this paper in Table 5

**Table 5: License of artifacts used in this paper.**
| Name | License |  | Name | License |
| --- | --- | --- | --- | --- |
| OLMo-1b | Apache 2.0 |  | SocialIQa | CC-BY |
| TULU | ODC-BY |  | CNN/DailyMail | Apache 2.0 |
| ARC | CC BY-SA |  | TweetQA | CC BY-SA-4.0 |
| OpenbookQA | Apache 2.0 |  | MNLI | CC-BY-3.0 |
| Hellaswag | MIT |  | GPT3NLI | MIT |
| BoolQ | Apache 2.0 |  | RTE | N/A |
| SciQ | CC-BY-NC-3.0 |  | Paws | Free |
| XSum | MIT |  | QQP | Non-Commercial |
| XLSum | CC-BY-NC-SA 4.0 |  | STS-B | Other |

## Appendix G Performance Difference Numbers

**Table 6: Average performance change before and after fine-tuning for each checkpoint (Perf(Fine-tuned) - Perf(BASE)). The group that is never learned during pre-training is picked up by the model during fine-tuning.**
| Checkpoint | Learned in<br>Pre-train | Learned in<br>Fine-Tune |
| --- | --- | --- |
| 1000 | 0.048 | 0.062 |
| 18000 | 0.048 | 0.149 |
| 342000 | 0.004 | 0.286 |
| 424000 | 0.01 | 0.297 |
| 505000 | 0.03 | 0.304 |
| 592000 | 0.027 | 0.297 |
| 738000 | 0.021 | 0.264 |
| main | -0.005 | 0.290 |

The average performance change before and after fine-tuning for each checkpoint is shown in Table 6.
The data in this table is used to create Figure 4.

## Appendix H Full Performance Table

Due to availability of space and the amount of fine-tuned checkpoints, we omit displaying all exact metric values in the paper.
The performance of each fine-tuned variant on each dataset can be found in the csv file in the code base.

## Appendix I Generalization Taxonomy

Following the generalization taxonomy in Hupkes et al., , the evaluation card is included in Table I.

| Motivation |
| --- |
| Practical<br>Cognitive<br>Intrinsic<br>Fairness<br>$\square$ $\bigtriangleup$ |
| Generalisation type |
| Compositional<br>Structural<br>Cross Task<br>Cross Language<br>Cross Domain<br>Robustness<br>$\bigtriangleup$<br>$\square$ |
| Shift type |
| Covariate<br>Label<br>Full<br>Assumed<br>$\square$<br>$\bigtriangleup$ |
| Shift source |
| Naturally occuring<br>Partitioned natural<br>Generated shift<br>Fully generated<br>$\square$ $\bigtriangleup$ |
| Shift locus |
| Train–test<br>Finetune train–test<br>Pretrain–train<br>Pretrain–test<br>$\square$ $\bigtriangleup$ |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Could you provide more details on the selection criteria for the datasets and how they might influence the observed dichotomy between tasks learned during pre-training and those requiring fine-tuning?
2. How do you anticipate your findings would generalize to larger models or different architectures, given that your study was conducted on a relatively small model?
3. Can you elaborate on potential signals or metrics during pre-training that could indicate an optimal point to stop pre-training and begin fine-tuning?
soundness: 3
strengths: 1. The paper addresses an under-explored area by empirically studying the interplay between pre-training and fine-tuning stages in language model development.
2. It provides valuable insights that can inform more efficient training strategies, such as early stopping in pre-training when fine-tuning yields better results.
3. The study is thorough, involving experiments on 18 datasets across various tasks, enhancing the validity of the conclusions.
summary: This paper investigates the relationship between pre-training and fine-tuning in large language models by fine-tuning multiple intermediate pre-trained model checkpoints. The authors aim to understand how models develop during pre-training and how this affects their performance after fine-tuning on downstream tasks. The main contributions include empirical findings that continual pre-training improves models in ways only revealed after fine-tuning, that fine-tuning benefits tasks not learned during pre-training, that fine-tuning can cause forgetting of previously known tasks, and that prompt sensitivity after fine-tuning can be mitigated with more pre-training.
weaknesses: 1. The study focuses on a single, relatively small model (OLMo-1B), which may limit the applicability of the findings to larger models or different architectures.
2. Due to the scarcity of models with available pre-training checkpoints, the conclusions are based on limited data, potentially affecting the robustness of the results.
3. The paper primarily analyzes downstream performance without deep exploration of model internals or theoretical underpinnings of the observed phenomena.
4. The benchmark datasets (flan-style) seem too simple and out of date for modern LLMs. For example, MT-bench, alpaca-eval, and arena-hard.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: none
soundness: 2
strengths: This paper analyze the downstream performance of intermediate pre-training checkpoints and the corresponding fine-tuned models, and draws some insights that can help in developing more efficient and effective LLMs.
summary: This paper investigates the dynamics of capability acquisition in large language models (LLMs) and provides emprical analyses that reveal the contribution of the pre-training and fine-tuning stages to downstream capabilities. Multiple intermediate pre-training checkpoints were fine-tuned and evaluated, leading to four main findings:
1）the pre-training stage can enhance the performance of the fine-tuned model, even when such improvements are not apparent in the pre-trained model itself;
2）fine-tuning is more beneficial for tasks that have not been learned during the pre-training stage;
3）a model fine-tuned for specific tasks may forget knowledge and capabilities in other domains;
4）fine-tuned models show high sensitivity to evaluation prompts, but this sensitivity can be alleviated by more pre-training.
weaknesses: 1) The experiment employed only a single base model, which limits the generalization of the empirical findings. In addition to the five candidate models mentioned by the authors, Baichuan2-7B may also be considered a candidate that has released intermediate checkpoints. https://huggingface.co/baichuan-inc/Baichuan2-7B-Intermediate-Checkpoints
2) The parameters of the base model used in this paper amount to 1 billion, which does not include widely used model sizes of LLMs, such as 7 billiion.
3) The num of tasks for supervised fine-tuning is relatively limited, with only 4 tasks, including summary generation, question generation, natural language inference and paraphrase detection. This limits the generalization of the results.
4) The conclusions derived from the empirical analysis largely align with the established perspectives within this field, providing limited novelty.
5) There are no promising experiments demonstrating how these findings can inform the developing of LLMs.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: see Weaknesses
soundness: 2
strengths: (1)	This work explores an interesting topic in LLMs by investigate the relationship between pre-training and fine-tuning.

(2)	The authors conduct some experiments provide some observations in LLM training.
summary: This work analyzes the relationship between Pre-Training and Fine-Tuning of Large Language Models. The authors conduct experiments on multiple intermediate pre-trained checkpoints to analyze how models develop as they train. Through experimental results, they find i) continual pretraining improves the model in a latent way that manifests after fine-tuning; ii) fine-tuning most benefits datasets where the model does not show capability during pre-training; iii) although the model benefits significantly through supervised fine-tuning, it may forget previously known domain knowledge and tasks not seen during fine-tuning; iv) the model exhibits high sensitivity to evaluation prompts after supervised fine-tuning, but this sensitivity can be alleviated through more pre-training
weaknesses: (1)	There are some observations that are relatively easy to obtain (e.g., although the model benefits significantly through supervised finetuning, it may forget previously known domain knowledge and tasks not seen
during fine-tuning), which have limited impact on the literature.

(2)	The authors should provide a related work section to summarize the difference between this work and previous related studies.

(3)	The model backbone selected in this work is limited (only OLMo model). Have you tried other open-source models (e.g., OpenELM).

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: See weaknesses.

typos:
line 299: pre-trining->pre-training
soundness: 2
strengths: **(1) The problem that this paper seeks to respond is important and valuable.** E.g., How do pretraining and fine-tuning interact to produce the resulting model? Does more pre-training hinder better fine-tuning results? What does the model learn and forget during pre-training and fine-tuning? These questions are straightforward and valuable.

**(2) This paper is well written.** The author clearly clarifies the problem that each part tries to address, making it easy to understand.

**(3) The author clearly states the limitations of their work.** It is always good to see the authors states the limitations as it makes the paper more rigorous.
summary: This paper investigate the relationship between pre-training and fine-tuning by fine-tuning multiple intermediate pre-trained model checkpoints to understand how models develop as they train. The authors conduct experiments on 18 datasets and give following insights into LLM training based on the result: 
(1) continued pretraining can improve a model in ways that are only revealed after fine-tuning;
(2) tasks for which the model already performs well during pre-training benefit much less from fine-tuning than those where the model does not demonstrate capabilities;
(3) although supervised fine-tuning can improve performance on in-distribution tasks, it can also cause the model to forget domain knowledge or tasks that it was previously capable of solving;
(4) fine-tuned models show high sensitivity to evaluation prompts, but this sensitivity can be alleviated by more pre-training.
weaknesses: **(1) The experiments are insufficient.** To explore the relationship between pretraining and fine-tuning, it is necessary to ensure the generalizability of the conclusions. Verifying only one language model (OLMo-1B) is insufficient to provide convincing conclusions. I believe the author needs to validate their conclusions on more LLMs.

**(2) Some of the conclusions are not rigorous.** e.g. line 300-303, the authors state that "some tasks can be learned during pre-training, while others are not." This may be because the pretraining data possibly includes data from similar types of tasks (not necessarily contamination), whereas tasks that cannot be learned during pretraining (such as MNLI, XSum, and BoolQ) do not have such similar task data included in their pretraining datasets. In such case, the conclusion become completely meaningless. I suggest the author carefully examine the types of tasks included in the pretraining dataset before drawing conclusions.

**(3) Some insights are uninspired with limited practical guidance value.** E.g., the authors suggest that early stopping in pre-training and starting fine-tuning is an efficient way of utilizing the resource when the downstream datasets are never picked up by the model during pre-training. However, the practical issue is that if we want to train a specialized model, we should directly choose a well-pretrained model. We typically don't aim to start from the pretraining phase again. The primary purpose of pretraining is to equip the model with stronger foundational capabilities, providing a solid base for better specialization through further SFT.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: - In Section 5, the author claims that "the benefits of fine-tuning an LLM could exceed the benefits of continued pretraining", but in Section 7, the author also claims that "pre-training can improve models in unseen ways". These two viewpoints seem contradictory.
- During the fine-tuning process, the paper conducts experiments on different specific tasks. What if it is in a general setting (such as AlpacaEval, MT-Bench), would the conclusions be different?
soundness: 2
strengths: - Exploring the relationship between pre-training and fine-tuning is a valuable direction with significant implications for improving training efficiency and downstream task performance.
- The paper conducts a series of experimental analyses and summarizes some conclusions, which have some guiding significance for researchers who are new to the field.
summary: The paper explores the relationship between fine-tuning and pre-training LLMs through fine-tuning multiple pre-training checkpoints of large language models.

There are some findings based on experimental results:
- The pre-trained model may excel at some tasks without fine-tuning. 
- Continual pre-training improves the model in a latent way that is only observable after fine-tuning.
- The fine-tuned model may forget some unused abilities.
- The fine-tuned model exhibits high sensitivity to evaluation prompts, but this sensitivity can be alleviated through more pre-training
weaknesses: - The conclusion drawn from the paper is relatively superficial and has been discussed in many previous works or some industry consensus, which does not meet the bar of an ICLR paper.
- The paper lacks some deeper insights into analyzing the parameter changes or loss changes during the pre-training or fine-tuning stages, which would provide theoretical support for the observed experimental phenomena.
- The paper's layout is somewhat chaotic, with some figures/tables and related text not on the same page, which poses a significant obstacle to reading.