Training on the Test Task Confounds Evaluation and Emergence

Authors: Ricardo Dominguez-Olmedo, Florian E. Dorner, Moritz Hardt
Primary area: foundation or frontier models, including LLMs
Keywords: language models, benchmarking, emergence

Abstract:
We study a fundamental problem in the evaluation of large language models that we call training on the test task. Unlike wrongful practices like training on the test data, leakage, or data contamination, training on the test task is not a malpractice.  Rather, the term describes a growing set of techniques to include task-relevant data in the pretraining stage of a language model. We demonstrate that training on the test task confounds both relative model evaluations and claims about emergent capabilities. We argue that the seeming superiority of one model family over another may be explained by a different degree of training on the test task. To this end, we propose an effective method to adjust for the effect of training on the test task on benchmark evaluations. Put simply, to fine-tune each model under comparison on the same task-relevant data before evaluation. Lastly, we show that instances of emergent behavior disappear gradually as models train on the test task. Our work promotes a new perspective on the evaluation of large language models with broad implications for benchmarking and the study of emergent capabilities.

Decision: accept (oral)
Reviewer scores: 8, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1729798325109

thread_type:
official_review

<thread_object>
creator: reviewer_hvsb

<content>
summary:
### Paper Summary
This paper asks if recent open-source models are in fact better than previous models: Does exposure to task data during pre-training explain the difference in model performance? (Yes). The authors find evidence that the difference on task performance can be explained by exposure to these tasks during the end of pre-training, not that more recent models are fundamentally better LLMs.

### Review Summary
There could be some world where this paper is simply stating the known fact that most models adjust the type of training data towards the end of pre-training to include task-relevant data, and that that inclusion improves performance. However, I think this paper does a good job of showing that this inclusion in particular seems to be making a key/big difference in models' improved performance, and that the inclusion during pre-training itself (while its being mixed with other data) doesn't seem to have an outsized impact on the ability to answer task-relevant questions.

strengths:
The authors have a number of nice (creative/thoughtful) experiments showing how older models do equally well when given more task data, that changing the style of the question (not content) also closes the gap between models, and finally, that, when given more data, model families all seem to follow the same scaling law.

The figures are all great/clean/pretty/legible!

The writing is also all clear/careful!

weaknesses:
* some could say that this is an obvious finding / direct outcome of how models recently have been trained. I don't hold this view. I find it interesting that there is not an additional/additive benefit found in recent models (from adding task related data in the end of pretraining), and interesting that the results are so stark. [i don't read your paper as saying that there is no gain/improvement in recent models, but rather, that performance doesn't extend beyond what fine-tuning directly upon the task in question would yield]
* figure 3 is unclear. the added information/gain of figure 3 vs figure 1 is unclear, esp. when looking at figure 3 alone (without referring to the text). to start, if I understanding things correctly, I think the colors of the points/lines should not be the same as the other figures as the underlying data is different from the other figures. the shared colors (to me) is misleading.

### nitpicks
* line 244, "These results provides..." --> "These results provide..."
* line 259, "contrast Figure 3 and Figure 1 Quantitatively..." --> "contrast Figure 3 and Figure 1. Quantitatively..."
* consider turning "What does MMLU test for?" into an active statement / takeaway.
* line 429 R^2 is formatted differently than line 430
* line 37, end of second paragraph. I was tripped up by "of some" and "of others". I now believe it is "of some models" / "of other models". I would guess the elisions were made to shorten/clean the paragraph? When I first read it I thought it was a typo that meant "to some people" / "to other people" (like different people will interpret this phenomena differently.)

questions:
*  I didn't fully follow the final paragraph in section 4.2. Would your results not give an answer to your posed question? (i.e., doesn't task-training explain the improvements, and fall outside the scaling laws?)
* What eq do you fit in figure 8 bottom? Switching the equation used in the top row the one used in the bottom row didn't seem to be motivated in the text. Doesn't eq 1 also a capture a log-linear relationship? Spelling a bit more out here would help.
    * Giving a hint in the caption of figure 8 that the data are the same in both rows might help
    * Labelling the rows within the figure (not only the caption) might also help (like you do in figure 4).
</content>
</thread_object>

</thread>
<thread>
creation date:
1729979798102

thread_type:
official_review

<thread_object>
creator: reviewer_b3hy

<content>
summary:
The paper argues that newer language models only appear to be better on metrics like MMLU than earlier, iso-compute language models because of the presence of data in distribution to MMLU in the pretraining set. The paper demonstrates this by showing that when these models are finetuned on MMLU/GSM8K before evaluation, the performance differences between different model families go away. The paper also shows that emergent behaviors are less common under this new evaluation protocol.

strengths:
- The paper is very clearly presented and argues persuasively. The figures are easily understood at a glance.
- The conclusion argued for is surprising and novel, and has broad implications for evaluation methodology of LLMs and emergent behaviors in LLMs.

weaknesses:
- The p-values for signficance in figure 1 overestimate significance because of correlation due to models being from only a few model families.
- The paper never actually directly checks the hypothesis of more in-distribution data being the cause of higher MMLU performance out of the box, despite arguing for it via implication from the finetuning result.

questions:
I'd be interested to see (on open source models with available datasets, or by sampling unconditionally from base models) whether there is more MMLU-like data in model families which appear to do better on MMLU when using the naive evaluation method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668946934

thread_type:
official_review

<thread_object>
creator: reviewer_xml1

<content>
summary:
The paper explores an interesting phenomenon: training on the test task, which differs from training on the test data. Training on test tasks implies that task-relevant data is included in the pretraining stage of a language model. The paper highlights how newer models benefit from training on the test task, but when fine-tuning older models on task-relevant data, this performance advantage disappears.

strengths:
- This paper explores an ever-increasingly important area of LLMs: the "science of evaluations."
- It explores the implications of task-specific training for language models, providing evidence that training with task-specific data is beneficial. They find that many models from November 2023 gain advantages from this type of training. Additionally, they show that task-specific training is more likely responsible for recent boosts in model performance when FLOPs are held constant.
- The paper's implications (as stated in the discussion) suggest that training on task-specific data may be beneficial across models.
- The experiments and the paper are logical and well thought through.

weaknesses:
- How was November 2023 chosen as the cutoff? I wonder what would have happened if you had chosen November 2022. How would the results be affected?
- It is unclear how much hyperparameter tuning might affect results when training on task-specific data. How much could hparam tuning affect the results? How was the initial sweep in Appendix A.2 chosen? Although a full sweep might be expensive, additional clarity on how these were chosen will alleviate my concern. 
- Lack of consideration of instruction-tuning models. How do you think instruction tuning models might impact this study as often the newer datasets are comprised of examples to explicitly increase benchmark performance [1]? 
- Only a couple of main evaluation datasets are explored in the paper. 

[1] GenQA: Generating Millions of Instructions from a Handful of Prompts

questions:
- Do you the authors think LLMs should be part of the title as only LLMs are explored in the paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711870946

thread_type:
official_review

<thread_object>
creator: reviewer_zcyd

<content>
summary:
The paper defines *training on the test task* as a potential problem for evaluating LLMs, which could be common practice and strictly speaking not data contamination. They show that doing so may allow a model to achieve better results on a benchmark while not having better abilities overall. They suggest that further fine-tuning all models can help with better comparison across models, and show that doing so restores clearer scaling trends.

strengths:
* They test their method on on a large set of base models with different sizes. This also allows them to compare scaling on benchmarks.
* The proposed method is promising for better evaluations/comparisons and light on compute, requiring only fine-tuning.

weaknesses:
* They only do so for pre-trained base models. In practical settings, we often look at the benchmarks of fine-tuned chat models, or models further fine-tuned off base-models (e.g. LLama derivatives). It is not clear whether their findings would apply to already-finetuned models.
* They classify 'old' models as before November 2023, which seems somewhat arbitrary. I understand that choosing Nov 2023 reveals some sort of improvement given the same compute, but this could be due to reasons other than training on the test task. More justification is needed on why the improvement gap is a good heuristic for whether a model was trained on the test task. Alternative analysis on whether much more models after Nov 2023 include instruction data in the pre-training set could similarly justify why Nov 2023 is a good cut-off point.

questions:
* Ultimately, benchmarks are used to understand how good a model is at a general skill, like mathematical reasoning for MMLU and GSM8K. This method proposes to 'fight fire with fire' by fine tuning all models on the test task. Would your method merely reliably measure how well models performs on a specific benchmark, rather than how well models are at the ability it's meant to measure?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732317265723

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers. We will address in the general response the concerns raised by both reviewer Zcyd and reviewer xmL1.
### Instruction-tuned models
We have substantially expanded our experiments by considering an additional 36 “chat” and “instruct” language models, see Appendix E.1 of the revised manuscript. Our original findings generalize remarkably well, see Figure 18 in Appendix E.1. Newer instruct/chat models substantially outperform older instruct/chat models. After fine-tuning all models on the same amount of task-specific data, performance between newer and older instruct/chat models equalizes.

We find that the performance gap between instruct/chat models is smaller than the gap between base models (that is, $\hat{\theta}$ in Figure 1 vs Figure 18). This is unsurprising, as instruction-tuning datasets usually include some amount of benchmark data (e.g., [1]). We posit that the gap between newer and older instruc/chat models is nonetheless large because early-to-mid 2023 instruction-tuned variants (e.g., Vicuna, Alpaca, Koala, Llama 2 Chat) did not emphasize benchmark performance but rather human preference (e.g., “win-rate”) in a chat setting. For example, the Llama 2 technical report [2] includes no benchmark evaluations of their fine-tuned and RLHF’d models. This perspective has dramatically shifted in the last year and a half, and post-training interventions now explicitly aim to improve benchmark performance (e.g., [3, 4]).

### Choosing November 2023 as the cutoff

We chose November 2023 as a cut-off because the technical reports of models released starting late 2023 begin to mention the use of instruction data for training. We carefully examined the technical report of all model families evaluated. Among these models, none released before November 2023 mention the use of instruction data for pre-training. In contrast, most model families released after November 2023 mention doing so. Qwen 1.5, Olmo 1.7, StableLM 2, and MAP Neo explicitly include instruction data in the pre-training mixture. Llama 3 and Gemma state that their pre-training data was determined through ablations on benchmark evaluations, a mechanism through which models may implicitly train on the test task [5].

Our analysis is robust to shifting the temporal split by a few months. We include such a robustness analysis in Appendix B.2., specifically Figure 10 which uses Sept 2023 as the cut-off month and Figure 11 which uses Jan 2024 as the cut-off month. Our findings indicate that practitioners started adopting design choices around late 2023 that amount to models training on the test task more, which is consistent with models’ technical reports starting to mention the use of benchmark or instruction data for training. Choosing specifically the month of November as the cut-off is therefore not critical for our analysis.

[1] Mukherjee, Subhabrata, et al. "Orca: Progressive learning from complex explanation traces of gpt-4." arXiv preprint arXiv:2306.02707 (2023).

[2] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).

[3] Team, Gemma, et al. "Gemma: Open models based on gemini research and technology." arXiv preprint arXiv:2403.08295 (2024).

[4] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

[5] Hewitt, John, et al. "Instruction following without instruction tuning." arXiv preprint arXiv:2409.14254 (2024).Z
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors describe a confounder in evaluations, where the ability of a model to follow the task format is mixed with its underlying ability to do a task - for example, in a task like MMLU, the knowledge of the model is confounded with its ability to do few-shot multiple-choice tasks. The authors propose fine-tuning on a small, common set of task-related data to put all the methods on equal footing, which makes evaluation more stable gives consistent measurements even at small scales (handling emergence). 

This is a timely, important problem and the fine-tuning approach is novel (though I think concurrent with Snell et al at COLM) and the reviewer comments are addressed clearly.