Revisiting the Superficial Alignment Hypothesis

## Abstract

Abstract The Superficial Alignment Hypothesis posits that almost all of a language model’s abilities and knowledge are learned during pre-training, while post-training is about giving a model the right style and format. We re-examine these claims by empirically studying the scaling behavior of post-training with increasing finetuning examples and evaluating them using objective task-specific standardized benchmarks.
Through experiments with the Llama-3, Mistral, and Llama-2 model families of multiple sizes, we observe that, similar to the pre-training scaling laws, post-training task performance scales as a power law against the number of finetuning examples. This power law relationship holds across a broad array of capabilities, including mathematical reasoning, coding, instruction following, and multihop-reasoning. In addition, for tasks like math and multihop reasoning, we observe that a handful of examples merely align the model stylistically but do not saturate performance on the benchmarks. Model performance is instead correlated with its reasoning ability and it improves significantly with more examples, illustrating the need for holistic evaluation programs leveraging objective benchmarks in addition to measurement of alignment to human preferences. We also observe that language models are not necessarily limited to using knowledge learned during pre-training. With appropriate post-training, a model’s ability to integrate new knowledge greatly improves on downstream tasks like multihop question-answering.
Taken together, these results shed new light on the Superficial Alignment Hypothesis, suggesting that it is, at best, an over-simplification.

## 1 Introduction

Large Language Models (LLMs) based on the Transformer architecture have achieved state-of-the-art performance on tasks that involve instruction following, problem-solving, and reasoning (Vaswani et al., , Achiam et al., , Dubey et al., ). The standard pipeline for building LLMs powered applications involves unsupervised training of a model on a giant corpus of data to gain general language understanding capability, referred to as pre-training (Radford et al., , Brown et al., ). The model is further improved using post-training, which involves finetuning it to excel at a particular domain or behave like a helpful chatbot. This process is also referred to as alignment. The predominant way to do this is through Supervised Finetuning (SFT) where the language model is provided with a prompt, and the model is finetuned to respond to the task (Wei et al., ). An additional step is Reinforcement Learning through Human Feedback (RLHF) where a model is trained using reinforcement learning to generate human-preferred responses, by being rewarded for good responses and penalized for bad responses (Ouyang et al., ).

To achieve the post-training goal of responding appropriately to various user queries, LLMs need to develop several task-specific capabilities, like mathematics, reasoning, utilizing knowledge, and tool use. To teach a model these capabilities, model builders collect human-annotated or synthetically generated data and finetune the model to obtain the desired behavior. Since data collection at scale is labor and cost-intensive, it is essential to understand the qualitative and quantitative value of obtaining additional post-training data. Studies like LIMA (Zhou et al., ) have hypothesized that post-training alignment is all about learning the style and format of the desired behavior. Specifically, it puts forward the Superficial Alignment Hypothesis, whose claims are:

- •
C1: A model’s knowledge is learned entirely during pre-training.
- •
C3: Post-training is largely about style and doesn’t does not teach a model new capabilities.
- •
C2: A small number of examples can saturate a model’s performance for a given task.

However, the experiments from LIMA and follow-up works (Lin et al., ) primarily evaluate chatbot style interaction capabilities - tasks that require mostly cosmetic changes to a model’s response style. It is unclear how these models improve on task-specific reasoning capabilities during post-training. They are also evaluated using a subjective win-rate comparison, over open-ended prompts. This doesn’t provide an objective pattern to analyze model behavior and thus fails to provide useful information about the nature of model performance and dataset size. For researchers and practitioners who finetune LLMs to perform specific tasks, understanding scaling behavior with more data is crucial in aiding data collection and annotation efforts. There is also a need to study if these LLMs are limited to the knowledge acquired during pre-training, or if we can introduce new knowledge and show how to utilize it effectively. So, we design three research questions to better investigate these claims:

- 1.
How does post-training model performance scale with dataset size?
- 2.
Is the model significantly improving on task-related capabilities or just learning the response style?
- 3.
Can a model integrate new knowledge from beyond the pertaining knowledge cutoff?

In the following section, we summarise the key results from the study, followed by sections that detail the experimental setup, results, and conclusions for each of the research questions outlined above.

## 2 Key Takeaways

- •
Post-training performance on a task has a power law relationship of the form
$P\propto D^{1/b}$ with the number of post-training samples, similar to scaling laws established for pertaining and inference (Kaplan et al., , Brown et al., ), across models of multiple families and sizes. (Section 3)
- •
Evaluating alignment models using win-rates as shown in Zhou et al. () could be misleading for reasoning-based tasks. For instance, LLM-based judges can prefer model generations that exhibit a chatbot-style answer for mathematical questions, even though the model might be poor at mathematical abilities as observed on math benchmarks. (Section 3.3)
- •
Through extensive error analysis on tasks like math and multihop reasoning, we see that when a model is finetuned for a task, the improvements in task-specific style and formatting saturate in just 100 examples, as hypothesized by the Superficial Alignment Hypothesis. However, the model’s performance on the task is directly correlated with its improvements in reasoning ability, which improves notably during post-training with more finetuning examples. (Section 4)
- •
Post-training a model for reasoning can also help a model integrate knowledge beyond its pre-training knowledge cutoff. Compared to pre-trained models, post-trained models learn and use new knowledge on downstream tasks effectively. (Section 5)

These experiments help frame the Superficial Alignment Hypothesis in a new light. Most importantly, the focus on post-training should not be entirely on stylistic alignment, but also on measuring downstream task metrics. When seen through this lens, we see that after finetuning on a few high-quality examples, LLMs behave in the right style and format, especially when evaluated through subjective techniques like win rate. However, this doesn’t necessarily warrant the conclusion that the model has been aligned for the task. When evaluated for their objective task-specific performance, we see that the models do improve significantly with additional data on many tasks during post-training over their pre-trained counterparts. In addition, these improvements are primarily driven by improvements in reasoning and analytical abilities during post-training. Good post-training is also an effective way for LLMs to learn and integrate new knowledge from beyond their knowledge cut-off.

## 3 Post-training Data Scaling

How does post-training model performance scale with dataset size?

The primary implication of the Superficial Alignment Hypothesis is that pre-training is all that matters, and with a rather small set of examples, we can align a model during post-training. However, this is a broad claim that is supported by a limited set of chatbot-style experiments. Post-training a model involves instruction following, problem-solving, and coding, and unlike chatbot-style dialogue whose evaluation is subjective and comparative, these capabilities can be judged using standardized benchmarks. For researchers and model builders who aim to improve performance on such tasks, it is important to understand performance scaling on such benchmarks with increasing fine-tuning data.

### 3.1 Experiment Design

**Table 1: Experiment training and test benchmark details**
| Task | Test Benchmark | Train Dataset | # train examples |
| --- | --- | --- | --- |
| Math | GSM8k Test | GSM8k Train | 7,500 |
| Multihop QnA | SubQA Test | SubQA Train | 2,700 |
| Coding | HumanEval+ | StarCoder Self-Align Train | 10,000 |
| Instruction Following | IFEval | Conifer Hard Messages | 5,000 |
| Instruction Following | IFEval | Dolly15k | 15,000 |

We look at four tasks (training datasets used are in parenthesis) - mathematical problem solving (GSM8k (Cobbe et al., )), instruction following ((Sun et al., , Conover et al., )), coding (StarCoder Self-Align (^1^11https://huggingface.co/datasets/bigcode/self-oss-instruct-sc2-exec-filter-50k)) and multihop question answering (SubQA (Tang et al., )). Starting from the base model and finetuning with increasing dataset size, we observe how performance scales during post-training.
For each task, we ran evaluations using a standard framework where available (Gao et al., , Liu et al., , Zhou et al., ). More details about the training data are available in Table 1 and additional dataset construction details can be found in Appendix A.3.

For our experiments, we finetune Llama-3, Llama-2 and Mistral model families on these tasks (Dubey et al., , Touvron et al., , Jiang et al., ). We chose base models because it is likely that instruct models are already extensively finetuned for these tasks.

We first finetuned the smallest (sub-10 Billion parameter) models from these model families, with the dataset splits of 0, 100, 500, 1000, 5000, and 10000 examples until the training dataset was exhausted. To study the effects of model parameter size, we also scale up the models in the Llama-2 and Llama-3 families from 7B to 70B parameters - a 10X increase. Since full-parameter fine-tuning of 70B models on a multi-node GPU cluster is resource-intensive, we limited the number of training runs for the 70B model sizes.

To ensure fairness, every model in a model family was trained with the same set of hyperparameters for a given task and dataset size, for 3 epochs over the base model with the default chat template from HuggingFace. More details about the hyperparameters for training and inference can be found in A.5.

### 3.2 Results

Figure: Figure 1: Performance improvements as finetuning data is scaled up, for models in the sub-10 Billion parameter range. The points are fitted with a power law curve of the form $P\propto D^{1/b}$. Model performance consistently scales in a power law fashion, across model families.
Refer to caption: extracted/5885555/images/data_scaling_log.png

##### Model performance for a task follows a power-law relationship with fine-tuning data.

Figure 1 shows the performance scaling with increasing post-training data for the smallest, sub-10 Billion parameter models, with the power-law fit line. Task accuracy $P$ closely follows a power-law of the form $P\propto D^{1/b}$ with the number of finetuning examples $D$, for all the models on all the tasks. This power-law relationship is in line with several other empirical scaling laws of LLMs with data size during pre-training, quantization, and inference (Kaplan et al., , Michaud et al., , Brown et al., ). The coefficients for the power law curves are in Appendix A.6.

In addition, model improvement curves do not cross each other. Better base models for a task are consistently better during post-training as well. This is in line with other works that relate pertaining performance with downstream task performance (Zhang et al., ). Performance scaling with data is also more consistent and predictable on reasoning-centric tasks like Math, Multihop QnA, and Coding, as opposed to subjective tasks like Instruction Following.
We also perform additional ablations on scaling curves with different dataset quality for Instruction Following in Appendix A.3.3, and evaluations on the GSM1k benchmark (Zhang et al., ) for math to check for dataset contamination, in Appendix A.1.

Figure: Figure 2: Performance scaling curves with increasing model size for models in the same family.
Refer to caption: extracted/5885555/images/model_scaling.png

##### Larger models in a model family learn better than smaller models with more data.

Figure 2 shows results for model scaling. All models of the family follow the same power law distribution. As expected, larger models consistently outperform smaller models for the same training data. However, the improvement is not always just an upward parallel shift. The performance of larger models curves upwards with more training datasets, indicating an increasing rate of improvements with additional data. The coefficients for the power law curves are in Appendix A.6.

Putting these results together, we see that larger and better base models scale even better during post-training. This highlights the role of pre-training in preparing a model to learn better during post-training.

### 3.3 General Purpose Alignment vs Task Specific Finetuning

The LIMA paper also introduced the LIMA dataset, a collection of 1,000 carefully curated prompts that was intended to align a pre-trained model to be on par with state-of-the-art post-trained models. It is based on the hypothesis that the model has inherently learned most of its capabilities during pre-training and thus, the model just needs to adopt a stylistic format to answer questions.

In this section, we finetune the Llama-3 8b model with the LIMA dataset, using the same training setup as the rest of the datasets. We call this model LIMA-1k. We also finetune a pre-trained model with 1,000 examples specific to a task - GSM8k for math and SubQA for multihop reasoning. This ensures that the LIMA and task-specific models are trained with a similar data “budget”. We call models finetuned with task-specific datasets Task-1k. We then evaluate the performance of the chat-bot style aligned LIMA-1k against task-specific fine-tuned Task-1k model on the task-specific benchmarks.

**Table 2: Comparision of task-specific finetuning v/s LIMA style stylistic alignment.**
|  | Task Accuracy | Win Rate |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | LIMA-1k | Task-1k | LIMA-1k | Task-1k | Neither |
| Math (GSM8k Test) | 14.7% | 46.5% | 84.4% | 0.24% | 14.2% |
| Multihop QnA (SubQA Test) | 21% | 36% | 20% | 57% | 23% |

##### Task specific post-training largely outperforms stylistic fine-tuning when evaluated objectively.

From the results in Table 2, we observe the marked difference that domain-specific data and post-training make against just chat-bot style alignment. For the same number of examples, domain-specific post-training greatly outperforms general-purpose chatbot style alignment.
In addition, since several works, including LIMA, report win-rate against other models as a performance metric, we also calculate the win-rate of the responses from LIMA-1k and Task-1k models on the GSM8k and SubQA test set prompts. We use the same prompt from LIMA to judge wins between responses and use GPT-4o (OpenAI, ) to predict wins. As seen in Table 2, the win-rate metric is an unreliable indicator of a model’s accuracy on a specific task. For instance, for the GSM8k Math task, Even though the LIMA-1k model generates a significant number of incorrect responses, it gets a higher win-rate than the task-specific fine-tuned model. This highlights the need for task-specific objective evaluations in addition to comparative win-rate metrics in foundation model evaluation programs.

## 4 Learning Reasoning and Style

Is the model significantly improving on task-related capabilities or just learning the response style?

In this section, we investigate what is driving the improvements in these tasks with more data, specifically aiming to delineate improvements in style/formatting versus improvements in reasoning. We do this by evaluating the generations of models with different finetuning levels.

### 4.1 Experiment Design

We evaluate finetuned model generations over math (GSM8k dataset) and multihop QnA (SubQA dataset). We took the Llama-3 8b base model as well as fine-tuned models using 100, 1000, and full training splits of the two datasets.

Both GSM8k and SubQA responses use the Socratic Method of generating subquestions to arrive at the final answer. So, the model is finetuned to follow this specific style of generating a subquestion-answer reasoning chain, a delimiter, followed by the final answer. Examples of expected model response styles and formats are in Appendix A.3.1 and A.3.2.

We then collect all the incorrect responses from these models on the test split and annotated them using GPT-4o (OpenAI, ). If the responses fail to stick to the previously specified format, it is annotated as Incorrect Formatting. If the responses contain an error in their subquestion-answer reasoning steps, we annotate it as Incorrect Reasoning. For math, we also check for Incorrect Arithmetic Calculations, since they are a major source of model errors. Each error category was evaluated independently for a response, using a tailored prompt. So, an error response can belong to multiple categories. More details about the prompt used for this categorization are in Appendix A.4.

### 4.2 Results

Figure: Figure 3: Breakdown of error responses by models finetuned with datasets of increasing data scales. The first group in each chart shows the Total Mistakes made on the test set by the models. Each error response is then independently evaluated for the different mistake types and thus can belong to multiple error types. There is a clear trend of models saturating on style and formatting improvements with just a few examples. However, reasoning and arithmetic errors continue to get better.
Refer to caption: extracted/5885555/images/style_substance.png

##### Style and formatting improvements saturate quickly.

From Figure 3 we see that the models get better at style and format errors with just 100 examples. If one just takes a passing look at the responses from this model, they could incorrectly conclude that the model is “aligned” to answer math or multi-hop questions. However, all of these responses are still incorrect for the task for which we fine-tune the model.

##### Reasoning performance continues to improve with more data.

Models continue to get better at reasoning and question understanding with more examples. The total number of mistakes a model makes highly correlates with reasoning errors ($r^{2}$ value of 0.98 for math and 0.99 for multihop QnA on Llama-3 8B) as opposed to total mistakes and formatting errors ($r^{2}$ value of 0.93 for math and 0.83 for multihop QnA). It also signifies that a model’s capabilities are not entirely learned during pre-training, because models can significantly improve their reasoning, or learn to apply it effectively, during post-training. This leads us to the idea that the superficial alignment hypothesis could be limited in scope to improvements on style-and-formatting alignment tasks. It doesn’t accurately characterize the improvements in capabilities that post-training is more effective at.

## 5 Learning New Knowledge

Can a model integrate new knowledge from beyond the pertaining knowledge cutoff?

In this section, we examine how post-training can help LLMs learn new knowledge after the pre-taining knowledge cutoff, and more importantly, use it correctly on downstream tasks.

### 5.1 Experiment Design

We first created Facts100, a hand-curated dataset of 100 news events that occurred after March 2023, the knowledge cutoff of the Llama-3-8b base model. The news events are from across the world and cover domains such as entertainment, sports, business, politics and science. We then created two questions for each event, as shown in example 4:

- •
Direct Question: A single-hop direct question related to the news event and its entity.
- •
Multihop Question: A multistep reasoning style question that first requires recalling what happened in the event in the first step and using it to answer in the second step. This checks if the model learns how to use the learned knowledge in the right way

Figure: Figure 4: Example of an Event and Question Pairs from the curated Facts100 dataset.
Refer to caption: extracted/5885555/images/facts_example.png

**Table 3: Accuracy of the models on the Facts100 dataset before introducing new knowledge.**
|  | Direct Qn | Multihop Qn |
| --- | --- | --- |
| Base Model | 8 | 12 |
| Post-trained Model | 32 | 27 |

To finetune and evaluate models on this task, we adopt the same task setting as the multihop QnA from the previous section. We evaluated the Llama-3 8B pre-trained model, as well as a multihop reasoning post-trained model. The post-trained model was finetuned on SubQARecall, a modification of the multihop reasoning QnA dataset SubQA from the previous section, augmented to recall the relevant event first before answering the question (more details in Appendix A.3.2. This post-training is meant to impart reasoning ability to the model while maintaining the same knowledge cutoff, since the SubQA dataset doesn’t contain any new information after the cutoff.
(^2^22We didn’t finetune the Llama-3 8B instruct model on the new facts because it performed poorly. This is because it is strongly aligned to refuse to answer questions beyond its finetuning cutoff data. Attempts to get the model to overcome this behavior through finetuning (ex: increasing learning rate) led to behavior degradation. Although unlearning methods can help undo this behavior, it is out of the scope of this study.) As seen in Table 3, both the models perform poorly on the Facts100 questions, since they have almost no knowledge of most of the new events that are crucial to answer these questions.

### 5.2 Introducing new knowledge

There are two primary ways to introduce new knowledge to a model - finetuning on the new events or during inference as part of the prompt. The latter is a simplified Retrieval Augmented Generation (RAG) setup Lewis et al. (). We investigate the role of post-training in both of these cases.

Event SFT: To train the model to learn this new information, we finetune a model on the Direct Question as the prompt and the Event + Answer as the response. The format is the same as the SubQARecall to keep the data in-distribution.

Event RAG-Oracle: In this method, we simulate a RAG setup in which knowledge is added in the prompt during inference instead of training on it. To isolate the errors introduced by the retriever component of RAG, we directly add the corresponding event in the prompt, emulating a perfect retriever i.e. an Oracle.

**Table 4: Accuracy of the models on the Facst100 dataset with new knowledge, introduced through Event SFT and RAG-Oracle.**
|  | Event SFT | Event RAG-Oracle |  |  |
| --- | --- | --- | --- | --- |
|  | Direct Qn | Multihop Qn | Direct Qn | Multihop Qn |
| Base Model | 65 | 37 | 49 | 34 |
| Post-trained Model | 81 | 55 | 86 | 71 |

#### 5.2.1 Results

##### Post-training a model for reasoning helps models learn and integrate new knowledge better.

As seen in Table 4, models post-trained for reasoning are significantly better at learning new knowledge (Direct Question) as well as integrating the new knowledge (Multihop Reasoning Question). However, SFT or RAG on the pre-trained model fails to show the same improvement on the harder multi-hop questions. Note that this answer is just one hop from the answer to the direct question but requires it to reason through the steps. This shows that the model can’t correctly use this new information in the right way on reasoning tasks.

However, if the model is first post-trained to do reasoning, it gets better at absorbing new information and using it in multihop reasoning tasks. This post-training was done on data from before the knowledge cutoff. Such post-training led to a marked difference in both SFT and RAG-based methods for introducing new knowledge.

##### Models hallucination is mitigated by post-training for reasoning, but is not eliminated.

Several studies have shown that LLMs hallucinate when introduced to new knowledge. Since all of the models in our experiments are finetuned to recall the event first and generate the answer based on it, we can easily check for hallucination in the recalled event. We also analyze the subsequent reasoning chain for reasoning errors. Both of these are done using GPT-4o and the prompt used is shown in the Appendix A.3.4.

Figure: Figure 5: Error Analysis on the New Fact Multihop Questions after fine-tuning. BM stands for the pre-trained base model and PM for the multihop reasoning post-trained model.
Refer to caption: extracted/5885555/images/facts_error.png

From Figure 5, we see that hallucinations about the event are a major factor in erroneous model responses. Even after introducing new knowledge, base models hallucinate significantly. This is in line with other works that demonstrate that finetuning models with new knowledge can lead to hallucinations (Gekhman et al., ).

However, models post-trained for reasoning make a marked improvement in both hallucination and reasoning errors. Although hallucination is not completely mitigated, the true value of finetuning is in preparing the model to reason with the new knowledge it receives. In addition, RAG-based methods are better than SFT for introducing new knowledge because the relevant information being in the context is a lot more useful. We also do a variety of ablation studies to compare against continual pre-training and LIMA-based finetuning, which are detailed in the Appendix A.2.

## 6 Related Work

Language Model Alignment: LLMs are first pretrained to gain general language understanding and world modeling, which is then followed by an alignment phase that involves SFT and RLHF to appropriately respond to user’s prompts (Ouyang et al., , Wei et al., , Dubey et al., ). Several recent works laid out their hypothesis on whether alignment is more about learning style and might not even be necessary because of In-Context Learning (Zhou et al., , Lin et al., ). More recent works by Zhao et al. () show that instruction tuning on better base models can outperform In Context Learning.

Scaling Laws: Several works like Hoffmann et al. (), Kaplan et al. (), Bahri et al. (), Michaud et al. () have studied and developed scaling laws for LLM Pretraining in terms of the dataset token count and the model parameter count against the cross-entropy loss. However, Schaeffer et al. () show that it is hard to predict scaling laws for downstream task performance because of constraints in the action space. Isik et al. (), Hernandez et al. () derive scaling laws for finetuning as a function of pretraining data and transfer learning respectively, and not post-training or instruction following. However, researchers and model builders improve model performance by collecting individual prompt-response examples rather than collective dataset tokens (Dubey et al., ).

## 7 Conclusions and Future Work

Conclusions: LLM post-training is a complex endeavor that involves improvements to instruction following, stylistic formatting, reasoning abilities, and general alignment to human preferences. LLMs can imitate the required style with “superficial” finetuning using a handful of examples, leading to the Superficial Alignment Hypothesis. However, a solely stylistic evaluation fails to characterize the many aspects of reasoning and task-specific capabilities that are key goals of finetuning. In fact, task-specific skills & reasoning significantly improve after post-training with more examples compared to the pre-trained model. These improvements closely follow a power law in our experiments with the number of finetuning examples across multiple model families and sizes. We also see that these improvements are driven by the model’s reasoning ability during generation, and are not limited to the model’s alignment to formatting or style. In addition, we see that the win rate against other models can be a misleading metric to measure tasks that require complex reasoning, signaling the need for holistic evaluation programs leveraging standardized, objective benchmarks, in addition to measurement of alignment to human preferences.

We also observe that good post-training can help LLMs overcome problems associated with knowledge cutoff, by enabling them to better utilize knowledge from beyond the pre-training corpus either via further finetuning or RAG. These results put together highlight the qualitative and quantitative characteristics of post-training, and the role of data scaling in this.

Limitations and future work: In this work we showed the performance improvement of a model on a task when it is finetuned with increasing task-specific data. However, frontier LLMs are trained to excel at multiple tasks, and we don’t thoroughly understand how finetuning for one task or domain affects the performance on others. A big open question would be investigating how to take advantage of this scaling behavior while preventing model degradation on existing capabilities. Similarly, we showed how models can learn new knowledge beyond their pre-training data cutoff, but the issue of hallucination isn’t solved. Further research in effectively introducing new knowledge, like continual learning methods during post-training can shed light on this.

In this work, we also limited the scope to supervised finetuning, on tasks that involve text generation. However, the implications from this opens up several interesting directions to explore further. Most notably, LLM post-training involves RLHF after supervised finetuning, and it would be interesting to see how RLHF can contribute to these improvements and how it scales with more data.

## 8 Acknowledgements

We would like to thank Hugh Zhang for their helpful reviews and feedback over the course of this project.

## Appendix A Appendix

### A.1 Math contamination ablation

Figure: Figure 6: GSM1k benchmark
Refer to caption: extracted/5885555/images/gsm1k_eval.png

Since there is a risk that the model might be contaminated with the GSM8k test set during pre-training, we cross-verified the GSM8k-trained Llama-3-8b models with the GSM1k private leaderboard. We observe a similar scaling trend on the new benchmark as well.

### A.2 Ablation studies for introducing new knowledge

We also try various ways of introducing new knowledge into the model.

Continued Pretraining: The base/posttrained model is first fine-tuned directly on the new events for 3 epochs with the same batch size. The idea is to continue pertaining on the new knowledge corpus.

LIMA: We also tried fine-tuning the base model with the LIMA dataset first to make it a general purpose Question-Answering model and provide the required new event as part of the prompt during evaluation, through RAG.

**Table 5: Ablation study of different ways to introduce new knowledge.**
| Method | Direct Qn | Multihop Qn |
| --- | --- | --- |
| Base Model + Continued Pretraining | 35 | 16 |
| Base Model + LIMA + RAG | 58 | 48 |
| Base Model + SFT + RAG | 100 | 51 |
| Posttrained Model w/ Continued Pretraining | 34 | 39 |
| Posttrained Model w/ SFT + RAG | 100 | 71 |

From table 5, we see that continuing pretraining with new knowledge on the base model or LIMA-style fine-tuning is not effective at introducing new knowledge.

Note that although SFT + RAG models have a perfect score on Direct Question, it is because the SFT models are finetuned on Direct Question-Answer in the first place and is thus, a result of memorization.

### A.3 Dataset Creation and Formatting

#### A.3.1 GSM8k Dataset

We chose the Socratic split of the GSM8k dataset since it contains well-developed subquestion-answer steps. This makes it easy to evaluate finetuned model responses for following this format, as well as evaluating the reasoning itself.

An example question-answer pair is given below:

#### A.3.2 SubQA Dataset

We modify the SubQA dataset to build a multi-hop reasoning dataset. The original SubQA dataset contains a select 1,000 subset of HotpotQA bridge-style questions, and two human-verified subquestions for it. For each of the thousand questions, we construct a multi-hop reasoning chain by chaining together the two subquestions, as shown below.

Subquestion 1:

Subquestion 2:

Generated multi-hop question and reasoning-based answer that the model is fine-tuned on along with the system message:

The SubQARecall version of the dataset was a simple modification of the response format, in which we added an event recall step first, which can be used to evaluate hallucinations. The event annotation was done using GPT-4o using the steps from the response. An example response format for this modified dataset is given below:

For evaluation, simple string matching ignores answers that are correct but differently worded. It also introduces false positives if the model just contains the final answer as part of its response instead of actually giving it as the answer. So, we use GPT-4o to evaluate the model response and provide the question, the correct answer, and the model-generated answer.

The evaluation prompt for this task is given below

A random subset of 100 multihop questions were held out for evaluation and the rest are used for training.

With this, we get a training dataset of 2,700 questions consisting of 1800 one-hop questions and 900 multi-hop questions. The evaluation set consists of the 100 held-out multihop questions. Since the models are fine-tuned specifically to follow the format of subquestion-answer reasoning steps, we can easily evaluate if they follow the expected reasoning format, as well as evaluate the correctness of the reasoning steps themselves.

#### A.3.3 Conifer and Dolly Dataset

The Conifer dataset has multiturn instruction following prompt-response dialogue examples and at each turn, the prompts gets harder, with more constraints and better responses.

For the Instruction Following Tasks, Dolly is of lower quality for the IFEval benchmark because the prompts are very different from the benchmark prompts and do not include constraints and directions. Conifer is a better dataset as it is targeted for constrained instruction following, similar to the IFEval prompts.

An example prompt-response pair from Conifer is given below:

An example prompt-response pair from the Dolly dataset is given below:

Figure: Figure 7: Model performance scales better with higher quality datasets
Refer to caption: extracted/5885555/images/quality_scaling.png

We also do an ablation study of fine-tuning Llama-3-8b against Conifer’s hard and easy split as well. We take dialogue examples with at least 4 turns and consider all first turns to create the ConiferEasy dataset and all last turns to create the ConiferHard dataset. Note that both of these are now single-turn prompt-response datasets. We also compare these models against the Dolly15k dataset. As seen in figure 7, for the same number of examples, we see that model performance scales better with better dataset quality. This highlights the importance of high-quality datasets.

#### A.3.4 Facts100 Dataset

The prompts used for evaluating hallucination and reasoning errors in the responses for the Facts100 multihop questions are given below:

### A.4 Error Analysis Prompt Templates

GSM8k error analysis prompts. The placeholders inside {} are replaced by the actual question, correct answer, and the model response.

SubQA Error Analysis prompt:

### A.5 Training and inference parameters

No PeFT methods were used, and the learning rate was set to 1e-5 with cosine decay to 0. We found that batch size has a big effect on model performance for smaller dataset sizes. The final batch sizes we used are in table 6

**Table 6: Batch sizes used.**
|  | 8b & 13b models | 70b models |
| --- | --- | --- |
| 0 - 100 | 2 | 16 |
| 101 - 1000 | 8 | 32 |
| 1001+ | 16 | 128 |

The evaluation was also done in a 0-shot setting to isolate improvements gained from adding few-shot examples. Sampling was done with a temperature 0f 0.4, top-p value of 0.95, and a repetition penalty of 1.1.

### A.6 Power law fit coefficients

**Table 7: Dataset scaling power law coefficients**
| Task | Model | a | b |
| --- | --- | --- | --- |
| Math | llama-3-8b | 19.47 | 7.66 |
| mistral-7b | 8.14 | 4.97 |  |
| llama-2-7b | 4.21 | 5.47 |  |
| Multihop QnA | llama-3-8b | 11.54 | 7.77 |
| mistral-7b | 5.40 | 7.75 |  |
| llama-2-7b | 8.27 | 9.39 |  |
| Coding | llama-3-8b | 27.14 | 23.43 |
| mistral-7b | 17.10 | 17.47 |  |
| llama-2-7b | 3.95 | 7.11 |  |
| IF-Conifer | llama-3-8b | 31.06 | 11.90 |
| mistral-7b | 21.76 | 9.28 |  |
| llama-2-7b | 17.83 | 10.00 |  |
| IF-Dolly | llama-3-8b | 30.94 | 21.12 |
| mistral-7b | 23.76 | 15.15 |  |
| llama-2-7b | 17.04 | 14.54 |  |

Coefficients for the power law curves of the form $P=aD^{1/b}$ are in Figure 1 are in Table 7.

**Table 8: Power law coefficients for Model size scaling experiment**
| Model Family | Task | Model | a | b |
| --- | --- | --- | --- | --- |
| llama-3 | Math | llama-3-8b | 18.18 | 7.28 |
| llama-3-70b | 32.07 | 9.07 |  |  |
| Multihop QnA | llama-3-8b | 10.81 | 6.91 |  |
| llama-3-70b | 23.58 | 8.37 |  |  |
| IF-Conifer | llama-3-8b | 33.35 | 13.52 |  |
| llama-3-70b | 37.85 | 11.83 |  |  |
| llama-2 | Math | llama-2-7b | 4.92 | 6.11 |
| llama-2-13b | 10.63 | 7.28 |  |  |
| llama-2-70b | 22.41 | 6.91 |  |  |
| Multihop QnA | llama-2-7b | 8.37 | 9.33 |  |
| llama-2-13b | 9.83 | 6.81 |  |  |
| llama-2-70b | 23.09 | 9.61 |  |  |
| IF-Conifer | llama-2-7b | 16.86 | 9.51 |  |
| llama-2-13b | 23.73 | 10.48 |  |  |
| llama-2-70b | 27.23 | 8.67 |  |  |

Coefficients for the power law curves in Figure 2 are in Table 8.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: see weakness
typos， the C3 and C2 at L055 are in the wrong order.
soundness: 3
strengths: This work presents in-depth analyses, offering insights for understanding the post-training process. It finds that the format-related errors are often saturated with a few samples, but the reasoning ability requries much more examples to improve.
summary: This work challenges the superficial alignment hypothesis, which is defined by three rules:  1) A model's knowledge is learned entirely during pre-training. 2) A small number of examples can saturate a model's performance for a given task. 3) Post-training is largely about style and doesn't does not teach a model new capabilities. This work illustrates a post-training scaling law and offers several insights, claiming that the model can learn new skills during the post-training and it also requires much more training data.
weaknesses: Since this work aims to study the scaling law of post-training, the choice of task is limited by the size of the training set. However, the training set is still small for building a reliable scaling law. One solution is to verify the idea of synthetic tasks. 

Except for several insights of analyzing the detailed post-training process, the main findings (the key takeaways section) are not a surprise; some of them have been discussed in the community. The contribution of this work is more like to be a verficiation and provide more evidence. Besides, this work uses the term "post-training" frequently, but the actual experiments are limited to SFT rather than RLHF.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How does the model's performance change when it is post-trained on datasets with diverse task formats, beyond those used here?
2. What are the underlying mechanisms that allow LLMs to integrate new knowledge effectively during post-training?
3. What are the long-term implications of extensive post-training on a model's performance across tasks? Does it risk overfitting to specific tasks or styles, potentially diminishing its generalization capabilities?
soundness: 3
strengths: 1. The paper conducts a comprehensive evaluation using well-defined benchmarks,  providing evidence that post-training improves more than stylistic alignment.
2. New insights: by analyzing the power-law relationship in post-training, it shows that model improvements align well with scaling laws observed in pre-training.
summary: This papre studies the superficial alignment hypothesis that  a language model's abilities and knowledge are learned during pre-training, while post-training is about giving a model the right style and format.  The authors investigate this hypothesis by assessing scaling laws in post-training across multiple tasks and model sizes, showing that fine-tuning enhances task-specific performance in reasoning and coding beyond stylistic alignment. Experiments show that while stylistic improvements are quick to saturate, task-specific reasoning, mathematical ability, and knowledge integration continue to benefit significantly with additional fine-tuning data.
weaknesses: - GPT-based Evaluation: Using GPT-based evaluation for alignment and error analysis introduces potential biases.
- Missing Citation: there are some related work that also demonstrate the scaling law in the post-training stage, sucu as this one (https://openreview.net/forum?id=MpCxUF8x61).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: The paper would benefit from expanding the size of the post-training dataset. Given that the authors are investigating the post-training scaling law, using only the Math dataset as an example for mathematical reasoning is quite limited. It is recommended that the authors consider augmenting the dataset through the use of synthetic data or by constructing a larger, more comprehensive dataset to study the scaling law (e.g., Evol-Instruct https://arxiv.org/abs/2304.12244 and WizardMath https://arxiv.org/abs/2308.09583). This would provide a more robust and comprehensive analysis of the scaling behavior in post-training.
soundness: 2
strengths: This paper aims to revisit the superficial alignment hypothesis, which posits that post-training primarily concerns style and format adjustments, while most capabilities and knowledge are learned during pre-training.The authors undertake comprehensive experiments to explore the scaling law in post-training for various reasoning abilities, thereby illuminating a feasible pathway for enhancing the advanced capabilities of large models.
summary: This paper presents an empirical investigation to re-evaluate the superficial alignment hypothesis, which posits that the majority of a large language model's (LLM's) abilities and knowledge are acquired during pre-training. By conducting extensive experiments with diverse pre-trained models (e.g., LLaMA) and fine-tuning tasks (e.g., mathematical reasoning), and incrementally increasing the number of fine-tuning examples, the authors demonstrate that the fine-tuning scaling law applies across a wide range of capabilities. This finding challenges the original Superficial Alignment Hypothesis, which asserts that post-training does not impart new capabilities to a model.
weaknesses: The superficial alignment hypothesis has not gained widespread acceptance in practical LLM pre-training. As stated in the Llama 3 technical report [1], the importance of improving performance for specific capabilities such as code/math/reasoning/tool use, etc., during post-training is emphasized. Similar conclusions have already been observed in existing technical reports or papers (e.g., the Figure 2 of paper [2]). This suggests that the re-evaluation presented in the paper may not be as novel as claimed.

Additionally, work [2] delves into how each ability varies due to the composition of supervised fine-tuning (SFT) datasets. The authors investigate this using varying amounts of mixed data and compare the results with individual ability performance, as illustrated in the Figure 3. However, the current submission only mentions multi-task settings as a potential future direction.

The distinction between pre-training and post-training has become increasingly blurred, especially with the widespread adoption of synthetic data. It is often challenging to definitively categorize synthetic question-answering pairs as either pre-training or post-training data. Moreover, there is typically an intermediary process between pre-training and post-training. Consequently, the exclusive focus on the scaling of post-training may be overly restrictive.

[1] The Llama 3 Herd of Models, https://arxiv.org/abs/2407.21783

[2] How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition, https://arxiv.org/pdf/2310.05492

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * Section 3.1; could author explain why do they put heterogeneous data in the same run? Might be good to have some scaling for individual tasks, to tease apart co-learning across tasks. But this is minor and doesn't change the overall weakness.

* I note that alignment is a bit overloaded term. The purpose of the alignment could be for human-like answer, but human-like doesn't imply stronger capability. It could be interpreted that LiMA conclude the hypothesis on a specific dataset (chat-style) and should not be taken too far from its original setting?

* In Table 2 and Figure 2, instruction tuning with LiMA also seems beneficial nonetheless, where Math 10->14.7; Multihop QnA 10->21. It's just that the training instances are not as much knowledge-intense and targeted as task-1k. Doesn't this shows that the benefit of Lima data?
soundness: 3
strengths: * the work study the scaling factors in post-training alignment to better counter-argue superficial alignment hypothesis. This seems a novel thing that other people haven't broadly tried. 
* the author basically revise the SAH to be conditional on the what it really takes to accomplish some downstream tasks. If the downstream tasks don't require more knowledge, then the alignment is only style-wise. This is a good message, although a bit obvious.
* showing that win-rate and task accuracy differs is an interesting observation.
* error analysis does seems good and comprehensive;
summary: The author revisit the validity of Superficial Alignment Hypothesis made in previous work (Zhou et al., 2024). Specifically, they try to show the ground for the hypothesis is shaky because it was established on running model with chat-style data, which do not requires more knowledge. Instead, the authors choose more knowledge-intense tasks, and shows that model do learn new knowledge (beyond what's in pretraining) and such observation scales w.r.t. task data, model sizes.
weaknesses: * Missing URIAL prompt [1] as a training-free alignment baseline, since the model is trained on LIMA-1k, it adds some complexity. 

* The authors should also make the investigation/claim a bit more systematic or formal. In terms of capability X, how does superficial alignment hypothesis not hold (+ some analysis on error), how does the observation in capability  X different from observation in capability Y. I think the problem statement is not so well-stated.

* Some claims are not well-supported and overselling:
Line 310-314: an alternative explanation is that the model is getting there? Also, how does the overall accuracy changes? Are the number of incorrect responses decreasing? I don't think the author could conclude much here.

Line 316: the claim in Line 312-314 and here are cyclic and contradictory. The author is essentially saying "the model do align in intermediate steps/format/style, but the improvement is nothing and superficial" (Line 312-314); but, "look, we are making improvement and this tell us even more about superficial alignment hypothesis is bad". Here the same observation of "making improvement" are interpreted in two completely different standpoint, without good rationalization.

Line 370 "this post training is meant to impart reasoning ability": but does it actually improve reasoning ability? It sounds more like the authors are building their own instruction-tuned model instead of using an off-the-shelf one.

Line 404 "reasoning helps", Line 482 "these improvements are driven by the model’s reasoning ability": reasoning is a broad term here, and authors only test one kind of reasoning. thus, the claim is stronger than the supporting evidence.

* The author also didn't do thorough survey for related work. For example, [2] also study the same topic (maybe not explicitly mentioning "superficial alignment hypothesis") in a more thorough and focused manner.

[1] The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
[2] Instruction-tuned Language Models are Better Knowledge Learners

Meta-review: This paper revisits and provides evidence against the superficial alignment hypothesis from the LIMA paper, which states that only a few samples are needed to "align" a language model, and that this alignment is only superficial. The paper finds that this is not really the case, and more substantial post-training alignment can change the model in substantial ways.

On the plus side, this paper is well executed, and the empirical results seem thorough (if somewhat too small to establish scaling laws with). On the negative side, the superficial alignment hypothesis itself was not really accepted as an established finding in the community, and thus the results aren't really surprising (my sense is that the null hypothesis in the community is still that post-training alignment changes the model capabilities substantially, and the LIMA paper, while interesting, did not show enough evidence to reject the null hypothesis).

Therefore, I am recommending that this paper not be accepted.

Decision: Reject