Training on the Test Task Confounds Evaluation and Emergence
Authors: Ricardo Dominguez-Olmedo
Florian E. Dorner
Moritz Hardt
Venue: ICLR 2025 Oral (2025)
Keywords: language models
benchmarking
emergence
Primary area: foundation or frontier models, including LLMs

Abstract: We study a fundamental problem in the evaluation of large language models that we call training on the test task. Unlike wrongful practices like training on the test data, leakage, or data contamination, training on the test task is not a malpractice.  Rather, the term describes a growing set of techniques to include task-relevant data in the pretraining stage of a language model. We demonstrate that training on the test task confounds both relative model evaluations and claims about emergent capabilities. We argue that the seeming superiority of one model family over another may be explained by a different degree of training on the test task. To this end, we propose an effective method to adjust for the effect of training on the test task on benchmark evaluations. Put simply, to fine-tune each model under comparison on the same task-relevant data before evaluation. Lastly, we show that instances of emergent behavior disappear gradually as models train on the test task. Our work promotes a new perspective on the evaluation of large language models with broad implications for benchmarking and the study of emergent capabilities.

Published as a conference paper at ICLR 2025
TRAINING ON THE TEST TASK CONFOUNDS EVALUATION AND EMERGENCE
Ricardo Dominguez-Olmedo : Florian E. Dorner and Moritz Hardt'
Max Planck Institute for Intelligent Systems, Tübingen, and Tübingen Al Center 2ETH Zürich
ABSTRACT
We study fundamental problem in the evaluation of large language models that we call training on the test task. Unlike wrongful practices like training on the test data, leakage, data contamination, training the test task is not malpractice. Rather, the term describes a growing set of practices that utilize knowledge about evaluation tasks training time. We demonstrate that training on the test task confounds both relative model evaluations and claims about emergent capabilities. We argue that the seeming superiority of one model family over another may be explained by a different degree of training on the test task. To this end, we propose an effective method to adjust for the effect of training on the test task on benchmark evaluations. Put simply, to fine-tune each model under comparison on the same task-relevant data before evaluation. We then show that instances of emergent behavior disappear gradually as models train the test task. Our work promotes a new perspective on the evaluation of large language models, with broad implications for benchmarking and the study of emergent capabilities.
INTRODUCTION
The machine learning community has long recognized certain clear violations of the benchmarking protocol. Training on the test set the most notorious among them (Duda & Hart] 1973 Hastic ett al. 2017). Data leakage (Kapoor & Narayanan 2022) and data contamination (Roberts et al.) 2023 Jiang et al., 2024) are closely related problems linked to the rise of massive web-crawled tramiig datasets. Researchers can all agree that test data should never appear in the training set.
But it's been much less clear what to do about legitimate attempts to bring training closer to evalu- ation. There is obvious gap between next token prediction at training time and tasks, such as reasoning and question answering, at test time. Ongoing research and engineering efforts, fact, aim to narrow precisely this gap (MetaAl 2024). Why shouldn' 't training be informed by knowledge about the downstream test tasks? What's 's an unfair advantage for some may be the feature of others.
In this work, we group strategies utilize task knowledge at training time under the umbrella term of truining on the test task. Examples of training on the test task include the use of instruction- tuning data or question answering templates during pre-training (Bai al. 2023 Stability 2023] Groeneveld et al., (004)). Models may also implicitly train on the test task when their pretraining data selected through ablations on downstream benchmark evaluations (Gemma al. 20224] MetaAl 2024). We work from the premise that training on the test task is accepiable- or, at least, unavoidable.
In a nutshell, we show that training on the test task strongly confounds model comparisons across different scales and model families. Perhaps counterintuitively, we propose to mitigate the effects of training on the test task on benchmark evaluations by doing more of it. We show that we can effectively level the playing field by giving each model the same, sufficient task-specific fine-tuning before evaluation. This adjustment restores cleaner log-linear scaling and makes capabilities predictable based on much smaller model scales.
1.1 OUR CONTRIBUTIONS
We introduce the term training on the test task to group a growing repertoire of practices that utilize knowledge about evaluation tasks at training time. We study its impact present-day benchmark

---

Published as a conference paper at ICLR 2025
Base models trained after November 2023 outperform those trained before November 2023
MMLU
GSMBK
0.8
0.7
Difference 0= 0.003
Difference 0=0.191
Accuracy
0.6
Regression R² 0.943
Accuracy
0.6
Regression 0.901
0.4
0.4
0.2
0.3
0.0
1020
10
10a2
10
1024
10
10
10
10
1024
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
After fine-tuning all models on the test task, differences in model performance vanish MMLU
GSM8K
0.8
Difference å 0.005
Difference ê 0.005
Accuracy
0.6
Regression R² = 0.989
Accuracy
0.6
Regression R 0.958
0.5
0.4
0.4
0.2
0.0
1020
10
10
10
1024
10
10
10
10
10a4
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Models trained
Before November 2023
After November 2023
Figure 1: MMLU and GSM8K scores of 56 base models, with model sizes ranging from 70M to 70B parameters. Solid lines correspond to the regression fit of A = max(0, log C Cc-cc) Ge) + ON +r, where is accuracy, C' is pretraining compute, N is whether the model was trained after November 2023, and is random chance accuracy. The coefficient 0 denotes the average improvement of models trained after November 2023 when controlling for pretraining compute. Bold indicates statistical significance with p-value 0.05. (Top) We hypothesize that training the test task confounds benchmark evaluations, resulting in newer base models substantially outperforming older ones. (Bottom) We propose adjust for differences in test task training by fine-tuning all models on the same, sufficient amount of task-specific data before evaluation. After fine-tuning on the test task, differences benchmark performance between older and newer models vanish.
evaluations by critically examining the performance improvements of recent language models. Our analysis spans 56 different language models and major active benchmarks, MMLU and GSM8K.
We start in Section by dividing models into those trained before November 2023 and those trained after. We find that for the same amount of pretraining compute, newer models strongly outperform older ones, on average by 7 percentage points in MMLU and 19 points in GSMSK. We then fine- tune all models
the same amount of task-specific data before evaluation. After fine-tuning on the same task data, newer models no longer outperform older ones. Rather, their performance equalizes. See Figure D This outcome suggests that the main difference between newer and older models is the extent to which they train on the test task.
We propose a simple and effective method to adjust for the effect of training on the test task on benchmark evaluations. Put simply, to fine-tune each model on the same, sufficient amount of task-specific data before evaluation. To validate our method, we demonstrate its effectiveness in controlled setting: we take the older models and fine-tune them on the test task. Remarkably, this recreates the kind of performance differences observed between newer and older models, further suggesting that training the test task explains the improvements of newer models. We then show that we can undo the advantage of the fine-tuned models over the other models by further fine-tuning all models on the test task (Section 3.1 Figure
2

---

Published as a conference paper at ICLR 2025
Next, we provide evidence that training on the test task may be more dominant factor in benchmark performance than data contamination. To argue this point, we consider ARC and HellaSwag. Here, at first, there appears to be sign of newer models having unfair advantage over older models. But after reformulating these benchmarks as MMLU-style multiple choice question answering tasks (MCQA), we the same confounded results as for MMLU (Section 3.2 Figure 4. This suggests that the improvements of newer models on MMLU are likely not because of memorization of specific testing data, but rather due to an improved ability for MCQA tasks.
Then, we show how training on the test task distorts model family comparisons. Certain model families appear markedly superior to others before adjusting for test task training, but not after adjustment (Section 4.1] Figure O We then demonstrate how training on the test task has inflated the perceived progress made by recent model families. After adjusting for its effect, newer models only modestly improve the Pareto frontier of model performance relative to pre-training compute.
Finally, we demonstrate that training on the test task has profound implications for the study of emergent capabilities. The phenomenon of emergence disappears gradually as the amount of train- ing on the test task grows (Section [ Specifically, we can make capabilities visible and predictable from much smaller model scales, recovering cleaner log linear-scaling. Importantly, our adjustment also works in cases, like MMLU, where previous purported explanations of emergence, such as the choice of evaluation metrie, do not suffice.
Our work calls for major reorientation of large language model evaluation. Model comparisons and claims of emergence are strongly confounded by the choice of training data relative to the test tasks. When comparing models with different pre-training data, our recommendation is to give each model the same sufficient amount of fine-tuning on task-relevant data before evaluation.
2 ADJUSTING FOR TRAINING ON THE TEST TASK
We choose MMLU (Hendrycks et al. 2020) and GSM8K (Cobbe et al. (021 as case study for investigating training on the test task active benchmarks. MMLU tests for world knowl- edge, whereas GSM8K tests multistep mathematical reasoning. These two benchmarks are arguably the most influential of the 2022-2024 period under study. They are also included in the Hugging- Face (HF) Open LLM Leaderboard v1 (Beeching et al. (023), popular leaderboard that evaluates and ranks models with publicly available weights. We evaluate models using LM Evaluation Har- ness (EleutherAl 2024 in identical fashion the HF leaderboard
We evaluate 56 base models, ranging in size from 70M to 70B parameters. See Appendix B. for the full list. The HF leaderboard's 's FAQ makes the distinction between "base pretrained models" and instruction-tuned or chat models, arguing that this is necessary to ensure fair model comparisons. We select models that are categorized as "pretrained". We check that the technical report of each of the selected models makes no mention of the model being fine-tuned. We only consider models for which the number of training tokens is known. This allows us to estimate the total amount of pretraining compute in FLOPPs as C' = 6 N D. where C' is pretraining compute, N is the number of model parameters, and D is the number of training tokens.
While we focus primarily on MMLU and GSM8K due to their prominence, we find that the issue of training on the test task extends beyond these two benchmarks. Specifically, in Appendix F we evaluate and discuss the impact of training on the test task for five additional benchmarks: MMLU Pro (Wang et al. 2024), ) GPQA (Rein et al. 2023), BBH (Suzgun et al., 2023). MuSR (Sprague et al. 2023), and MATH Level 5 (Hendrycks et al. 2021), which form the OpenLLM Leaderboard
v2 (Fourrier et al. 2024a). Furthermore, in Appendix G we conduct similar experiments for an additional 36 instruction and chat models. We observe that our findings generalize remarkably well to instruction and chat models.
Recent models outperform older ones given the same pretraining compute. We evaluate mod- els on MMLU and GSM8K, and plot benchmark accuracy against pretraining compute in Figure [ top. We observe that performance correlates with pretraining compute for both benchmarks. How- ever, on the surface it appears that models released after November 2023 better leverage pretraining compute. For a given compute budget, newer models tend to attain better benchmark performance.
We also evaluate MMLU and GSM8K 5-shot, ARC 25-shot, and HellaSwag 10-shot.
3

---

Published as a conference paper at ICLR 2025
MMLU
GSMBK
Gain in accuracy after adjustment
0.2
0.4
0.1
0.2
0.0
0.0
1020
10
1022
1023
1024
1020
10sh
10a2
1023
1024
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Models trained
Before November 2023
After November 2023
Figure 2: Models trained before November 2023 tend benefit more from fine-tuning on task data.
We selected November 2023 as the temporal cutoff' for our analysis because the technical reports of models released from late 2023 onward start refereneing certain pre-training practices that may amount to training on test task. For example, Qwen (Bai et al. 2023). Olmo 1.7 (Groenvveld al., 2024) and MAP Neo (Zhang et al., 2024) explicitly include instruction data during pretrain- ing. StableLM [Stability 2023) reformulates some of its pretraining datasets to better resem- ble downstream tasks such as question-answering
More subtly, the pretraining data mixtures of Gemma (Gemma et al. 2024) and Llama 3 (MetaAl (202) were determined through extensive ab- lations on downstream benchmark evaluations. We validate that our findings are robust to adjusting the temporal cutoff by few months; Appendix of November as the cutoff is therefore not critical for our analysis.
Derennix[DIIieee [.I for details. Choosing specifically the month
This raises an important question: Do newer models outperform older mainly because newer models trained more on the test task? At first sight, an answer seems elusive. After all, the pre- training data of most recent models is not publicly available. Retraining all models with the same training data and compute budget would be both infeasible and cost prohibitive. In the next section, we propose a way get the answer by adjusting for the effect of training on the test task.
2.1 ADJUSTING FOR TRAINING ON THE TEST TASK BY TRAINING ON THE TEST TASK
We propose to adjust for differences in test task training by fine-tuning all models on the same, suf- ficient amount of task-specific data before evaluation. To do so, we need source of task-specific data for each the tasks we consider. For multiple choice guestioning answering, we use the aux- iliary training set accompanying the HF MMLU repository This training set is not an i.i.d. split of MMLU. Instead, it consists of the training sets from other multiple-choice question-answering benchmarks, comprising approximately 100,000 training examples and 30 million tokens. For math- ematical reasoning, we combine MetaMathQA (Yu et al., 20236) and Orca-Math (Mitra et al.) 2024), totalling approximately 600,000 training examples and 200M tokens. We fine-tune models for three epochs using standard hyperparameter choices, see Appendix B.2 The amount of compute required for fine-tuning is minimal compared to models' pretraining compute.
We plot model scores on MMLU and GSM8K after fine-tuning in Figure | (bottom). We observe that after fine-tuning on task relevant data, both newer and older models follow remarkably similar scaling trends. That is, newer models no longer appear to outperform older models.
Remarkably, we observe that older models tend benefit much more from fine-tuning on task- relevant data compared to newer models, see Figure 2 The improvements in older models are striking, often leaping from random chance accuracy to double-digit gains accuracy. In contrast, fine-tuning provides comparatively little benefit to newer models. This observation suggests that newer models have already been exposed to substantial amount of task-relevant data, making additional fine-tuning less impactful.
potential concern that our observations might result from our fine-tuning hyperparameters being systematically more favorable to older models. We verify that this is not the case by conducting robustness check on the fine-tuning hyperparameters, see Appendix B.3
4

---

Published as a conference paper at ICLR 2025
2.2 QUANTIFYING PERFORMANCE DIFFERENCES BETWEEN NEWER AND OLDER MODELS
We draw inspiration from scaling laws (Kaplan et al. 2020) in how we model benchmark accuracy A scale log- linearly with pretraining compute C. To account for emergence (Wei et al. 2022b). we assume that models perform at the task's 's random chance accuracy r up to scaling to some point of emergence c.. We let the variable N denote whether a model was trained after November 2023, and regress the model
A = a max(0, log C' - c.) + ON +r+
(1)
where a, 8 and ce are the fit's parameters, and e is random noise. We focus on the coefficient 8, which corresponds to the average difference benchmark performance between newer and older models after controlling for pretraining compute. We fit the model in Equation [ and report the regression coefficient in Figure I We obtain R² > 0.9 for all model fits. We use clustered standard errors to compute statistical significance, treating each model family separate group.
Before adjusting for test task training, the estimated difference in performance & between newer and older models are statistically significant, positive, and large. Specifically, recent models outperform older ones on average by over 7 accuracy points in MMLU and 19 accuracy points in GSM8K. These are remarkably large differences in benchmark performance. However, after the adjustment, the estimated coefficient is both small and not statistically significant. See Figure bottom. That is, conditioned on all models training on the same amount of task-specific data, we find no evidence for significant difference in benchmark performance between newer and older models.
Therefore, the performance of newer and older models equalizes when all models are exposed the same amount of task-relevant data. This suggests that the impressive benchmark improvements of newer models are primarily attributable newer models training more on the test task. We present causal interpretation of results in Appendix C outlying the assumptions necessary to establish causal link between training on the test task and the benchmark improvements of newer models.
3 RECREATING DIFFERENCES IN BENCHMARK PERFORMANCE
We have so far established that newer models strongly outperform older models for the same amount of pre-training compute. We now demonstrate how to recreate such differences in performance by actively manipulating how much models train on the test task. We do so in two ways. First, we fine-tune older models on task relevant data (Section B.T). Second, we reformulate certain test tasks to use MMLU-style multiple choice prompts instead of "cloze" evaluations (Section (.2). Both experiments recreate the kind of performance differences observed between newer and older models.
These results provide further evidence that the differences in performance between older and newer models are linked to test task training. They also demonstrate how test task training distorts bench- mark evaluations. Fortunately, in both cases, show that fine-tuning models task-relevant data before evaluation is an effective mechanism for mitigating the bias introduced by training on the test task. In doing so, systematically validate the proposed adjustment method.
3.1 FINE-TUNING ON THE TEST TASK
For this section, we only consider models trained before November 2023. We split the models into two cohorts: a control group and a treatment group. We take these older models as control group. We then create treatment group by fine-tuning the control group on the datasets of task-relevant data introduced in Section 2 We only fine-tune models with at least 7- 7- FLOPs, the pre-training compute of the smallest newer model. We fine-tune for a single epoch. We plot in Figure top the benchmark performance of the two cohorts.
Qualitatively, the differences in performance between the control and treatment groups resembles the differences observed between newer and older models, contrast Figure 3 with Figure L Quan- titatively, the estimated performance gain 0 from fine-tuning is similar to the difference between newer and older models estimated in Section 2.2 That is, fine-tuning older models on the test task produces both qualitatively and quantitatively similar confounding to that observed between newer and older models. This results further supports our running hypothesis that newer models are largely equivalent to older models that have trained the test task. They also demonstrate the large effect
5

---

Published as a conference paper at ICLR 2025
MMLU
GSM8K
Accuracy
0.6
Ô=
0.086
Accuracy
Ô 0.223
0.5
0.2
0.3
0.0
MMLU (adjusted)
GSM8K (adjusted)
Accuracy
0=
0.006
Accuracy
0
0.006
0.4
0.4
0.2
0.3
1020
1021
1022
10aa
1020
10a1
10??
102h
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Figure 3: Models trained before November 2023 without fine-tuning and (o) after fine-tuning on the test task. Their difference in benchmark performance & resembles that between newer and older models. After adjusting by training on the test task, their difference vanishes.
that training on the task can have on benchmark performance. Note that the gain in performance of the treatment group is slightly larger than the difference in performance between newer and older models. This to be expected, as all models in the treatment group are fine-tuned on the test task, whereas not all new models may train on the test task.
We then apply our proposed adjustment by further fine-tuning both the control and treatment groups on the test task, see Figure 3 right. After the adjustment, the estimated difference performance between the control and treatment group is both small and not statistically significant. We therefore validate a vital soundness property of the proposed adjustment procedure: after deliberately training some models the test task, we undo their advantage other models by further training all models on the test task.
3.2 REFORMULATING THE TEST TASK
In this section, we show that reformulating other benchmarks multiple-choice question answering tasks leads to similar differences in performance between older and newer models. We consider two additional benchmarks from the HF leaderboard vl: ARC Challenge (Clark et al. 2018) and Hel- laSwag (Zellers et al. 2019). Similarly to MMLU, ARC comprises grade-school level questions. HellaSwag instead tests for commonsense reasoning. Like MMLU, the questions in ARC and Hel- laSwag are accompanied by four possible answers, among which the model must differentiate the correct one. ARC and HellaSwag use "cloze" evaluations: models' answer is taken to be that with the largest completion likelihood given the input question. In contrast, MMLU formulates questions multiple-cooie:: all four answer choices listed, and the model is promoted pick one.
We first evaluate all models on ARC and HellaSwag using the standard cloze evaluation, and plot their benchmark performance in Figure left. We repeat the statistical analysis of Section 2.2 We find that the estimated difference in performance Ô between newer and older models small and not statistically significant: newer models do not outperform older models on ARC and HellaSwag.
We then reformulate ARC and HellaSwag as MMLU-style multiple-choiee questions, and plot the resulting benchmark performance in Figure I center. We observe large differences in performance between newer and older models. Specifically, we find the difference in performance â between newer and older models to be significant, positive, and large, and to be roughly similar in magnitude to that estimated for MMLU in Section 2.2 That is, reformulating the test task as multiple choice question answering leads to qualitatively and quantitatively similar confounding to that observed for MMLU. Therefore, newer models overperform MMLU likely not because of memorization of specific testing data (i.e., due to data contamination or leakage), but rather due to an improved ability for multiple-choice question answering.

---

Published as a conference paper at ICLR 2025
Cloze evaluation
Multiple choice
Multiple choice
(adjusted)
Accuracy
0.75
0.001
i 0.120
0.014
ARC
0.50
0.25
HellaSwag
Accuracy
0.75
θ
0.012
i 0.114
Ô 0.009
0.50
0.25
1040
1021
10a4
1023
1024
1020
1021
10
10
10a4
1020 1021
1022
1023
1024
Pretraining compute (FLOPs) Pretraining compute (FLOPs) Pretraining compute (FLOPs)
Figure 4:
Reformulating ARC and HellaSwag MMLU-style questions give rise large differ- ences between models trained (+) before November 2023 and (+) after November 2023. After
adjusting by fine-tuning on the test task, differences in performance vanish.
MMLU Multiple choice
MMLU Cloze
MMLU
Multiple choice
-0.4
Accuracy
0.068
Accuracy
0.008
-Brier score
0.6
-0.6
0.4
0.3
-0.8
1020
10
102h 1024
1020
10a1
1022
1023
1024
1020
10a2
10ah
1024
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Figure 5: When evaluating MMLU using "cloze" prompts, models trained (o) after November 2023 no longer outperform those trained (o) before November 2023 (middle). When using Brier score as the evaluation metric, we still observe sharp improvements in performance (right).
Lastly, we adjust for test task training by fine-tuning models on the MMLU auxiliary training set, and plot their ARC Challenge and HellaSwag scores in Figure 4 of a large nor significant difference in performance between newer and older models. Therefore, the proposed adjustment is effective in mitigating the bias introduced by evaluating models via multiple-choice question answering. Notably, performance on ARC and HellaSwag equalizes after fine-tuning the MMLU auxiliary training set. This indicates that the adjustment data need not closely resemble the test set, but rather the test task.
rright. We no longer find evidence
What does MMLU test for? We evaluate MMLU using the "cloze" methodology instead of the usual multiple-choice prompts. We plot the results in Figure 5areeleennee center. With cloze evaluations, the difference in performance between newer and older models is both small and not statistically signif- icant. This suggests that the standard MMLU evaluation conflates knowledge-testing with testing a models' ability to answer multiple choice questions]] Newer models therefore attain higher MMLU scores than older models largely because they are better at multiple-choice question answering, and not because they necessarily "know more".
4 IMPLICATIONS FOR MODEL COMPARISONS
So far, we have shown how training on the test task distorts benchmark evaluations. Next, we exam- ine its impact on the relative comparison of model families (Section 4.1) as well as its implications for accurately measuring progress in model capabilities over time (Section 4.2
For instance, smaller models suffer from particularly strong A-bias for multiple-choice ques- tions Olomidg 2023

---

Published as a conference paper at ICLR 2025
MMLU
GSM8K
Unadjusted
Adjusted
Unadjusted
Adjusted
0.7
0.8
Accuracy
Accuracy
0.4
0.3
0.0
10
10
102e
10ah
10
10
10
102h
102c
102h
1021
102c
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Liama
Qwen 1.5
Pythia
Figure 6: Training on the test task confounds relative comparisons between model families. After adjusting for test task training, none of the three model families appears to be superior.
MMLU
GSM8K
Unadjusted
Adjusted
Unadjusted
Adjusted
Accuracy
0.7
0.18
0.03
Accuracy
0.42
0.08
0.5
0.5
0.3
0.0
10a2
1023
1024
10
1022
10
10
1023
1044
1022
1023
10a6
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Pareto front
Models trained before November 2023
All models
Figure 7:
Training the test task overestimates the improvements made by recent base models. After adjustment, the area of improvement (green) reduces by sixfold.
4.1 COMPARING MODEL FAMILIES
We compare the performance of the Pythia, Llama 2, and Qwen 1.5 model families, which likely train on test task to very different extents. Pythia was trained on the Pile (Gao al., 2020), collection of curated datasets that are unlikely to contain much test task data. Llama 2 was trained mostly on web data, which is reasonable to assume may contain more test task data. Lastly, Qwen 1.5 explicitly includes instruction data in its pretraining mixture, thus likely training on the test task.
We plot the MMLU and GSM8K scores of the three model families in Figure as well as their adjusted scores (i.e., after fine-tuning task relevant data). Without adjustment, Qwen 1.5 appears to be the superior model family: it Pareto dominates both the Llama and Pythia models. contrast, all Pythia models perform better than random chance, making it unclear whether scaling Pythia offers any benefit at all. After adjustment, however, all three model families exhibit remarkably similar scaling trends. Therefore, after correcting for the confounding of test task training, of the model families appears superior to the others.
Training on the task therefore profoundly confounds relative model comparisons. Base models are rarely used "as is" and are generally adapted before deployment. Because of the confounding of training on the test task, performance before adaptation may not reliably predict performance after adaptation. It therefore makes little sense to compare base models at face value.
4.2 PROGRESS IN MODEL CAPABILITIES
Training on the test task substantially overestimates the progress in benchmark performance per unit of compute achieved by recent model families. In Figure 7 we plot the Pareto frontier of bench- mark accuracy against pretraining compute, both for models trained before November 2023 and for all models. We measure progress by considering the area of improvement of the Pareto frontier since November 2023, shaded in green. Without adjustment, the difference between the two Pareto frontiers is large, indicating very substantial progress since November 2023. After adjustment, how- ever, the area of improvement reduces by sixfold, showing only modest improvements. Therefore,
8

---

Published as a conference paper at ICLR 2025
MMLU
Task examples: 0
Task examples: 6.4k
Task examples: 16k
Task examples: 64k
0.70
Accuracy
Ce: 1.3e+22
Cei 5.6e+21
1.1e+21
Cei 5.6e+20
0.50
0.25
0.70
Accuracy
R² 0.632
R² 0.804
R
0.913
R² 0.950
0.50
0.25
1021
10
20
1024
10
102²
102h
1024
10 10 102a
1024
1021
1022
1023
1024
Pretraining compute (FLOPs)
Figure 8: Scaling on MMLU as models increasingly train on the test task. The point of emergence arises at lower scales (top). Training on the test task yields cleaner log- longlinear scaling fits (bottom).
training on the test task strongly overestimates the progress in benchmark performance per unit of compute achieved by recent model families.
On the other hand, recent models tend to be trained on more data than Chinchilla compute- optimal (Hotlmann et al.) 2022). Given the Chinchilla scaling laws, it is noteworthy that newer, smaller "over-trained" models match the performance of older, larger ones for the same amount of pretraining compute. Since inference and fine-tuning of smaller models is substantially cheaper, recent models can much more accessible to less well-resourced institutions, with little cost per- formance. For example, we find that Ilama 3 8B closely matches the performance of Ilama 2 70B (both have similar pre-training compute).
IMPLICATIONS FOR EMERGENCE
Throughout our evaluations, we observe emergent behavior for MMLU and GSM8K: models per- form at near random chance up to a certain scale of pretraining compute, followed by relatively sharper improvements in performance at larger scales (Wei et al. 20226). After training on the test task, however, emergence for MMLU and GSM8K appears We dedicate this section to more closely investigating the relationship between training on the test task and emergence.
at substantially lower scales.
Emergence arises at lower scales with increased test task training. We consider only models trained before November 2023, as we have established that these models train on the task less than newer models. We evaluate the models at intermediate checkpoints datasets of task relevant data introduced Section 2.1 We fit a and c, in Equation] to the different intermediate checkpoints, and report in Figure S top the corresponding points of emergence ce. We find that emergence arises at increasingly lower compute regimes as models train on the test task. For MMLU, models exhibit emergence around 10 FLOPs, the scale of Pythia 6.9B. After training on 64,000 examples, emergence arises around 6 1020 FLOPPs the scale of Pythia 410M. We observe similar results for GSM8K, Figure 19 in Appendix
we fine-tune them on the
Training on the test task yields increasingly better log-linear fits. The log-linear relationship between pretraining loss and compute well-established (Kaplan al. (020). We observe that training on the test task increasingly recovers log-linear scaling between pretraining compute and benchmark accuracy. Similarly to the earlier section, we evaluate intermediate checkpoints but instead fit log-linear functions in Figure 8 bottom. We observe that the R² of the fit improves substantially as the models train on more telk-rele data, jumping from 0.63 to 0.95 after training on 64,000 examples. Therefore, after training the test task. almost all the variation in benchmark accuracy is explained by linelincar scaling of pre- pre-training compute. We observe similar results for GSM8K, see Figure [9 in Appendix

---

Published as a conference paper at ICLR 2025
Recommendations. Schaeffer al. (2024a) argue that emergence appears due to the choice of metric. To mitigate emergence, they suggest considering Brier score instead of accuracy. We ob- serve, however, that emergence for MMLU does not disappear when using the Brier score, Fig- rrigtt nor that of ARC and HellaSwag when framed as multiple-choice questions, see Figure in Appendix[[] We discuss two practical solutions obtain predictive scaling while maintaining curacy as the evaluation metric.
For MMLU and multiple-choice benchmarks more broadly, cloze evaluations consistently yield smoother and more predictable scaling, even when using accuracy as the evaluation metric. Since the purpose of these benchmarks is knowledge-testing more so than testing multiple-chotee ability, cloze evaluations are preferable insofar predictive scaling at lower compute scales is an important consideration. This recommendation aligns with the concurrent work by Gu et al. (2024).
answering
More broadly, if sufficient task relevant data is available, then training on the test task can result in much more predictable scaling by shifting emergence to smaller compute scales, That is, scaling laws where models across scales are fine-tuned on the same, sufficient task-relevant data before evaluation. Such scaling laws correspond to those of specialist models, which for some tasks -e.g., legal annotation (Dominguez-Olmedo et al. 2024)- or purposes -e.g., safety- might be preferable to the scaling laws of generalist models.
DISCUSSION
The 1968 Olympics took place in Mexico City at the significant altitude 2340 meters, higher than Australia's 's tallest peak. Runners who had trained at altitude in their home countries were better prepared to compete in Mexico City's conditions, as it turned out. But the hotly debated results of the Games did not lead the organizers to prohibit training at natural altitude. Instead, they let everyone do it, and athletes came to consider altitude training an excellent way to train.
The anecdote holds lesson for the evaluation of large language models half century later. Knowl- edge about the evaluation conditions necessarily influences training practices under competitive pressure. It may be a fool's errand to prohibit the practice. Instead, we propose adjust for by giving every model the same task-specific preparation before evaluation. We work from the as- sumption that training on the test task, in general, cannot be effectively detected, disallowed, or disincentivized. Detecting what training data model has seen is a notoriously difficult problem existing heuristics achieve partial success at best. Researchers routinely acknowledge the futility of fighting data contamination. Moreover, we anticipate that the ways to effectively train on the test task will only grow in scope and adoption.
Our work demonstrates that comparisons of different models are confounded by the choice of training data and training practices. Different model families vary in the degree that they were- implicitly explicitly-trained on various test tasks. A relatively small amount of task data can have a disproportionally large effect on benchmark performance. therefore makes little sense to compare model performance face value without accounting for how the training data relate the test task.
Training the test task also has profound implications for the study of emergent behavior. Af- ter training on the test task, model capabilities become predictable at smaller scales. This greatly reduces the unpredictability associated with emergence, notably without any change in the metric.
Despite the daunting challenges that training on the test task poses for the fair evaluation of language models, it's also its own best remedy. Giving each model the same sufficient task-specific fine- tuning harmonizes model comparisons and linearizes the relationship between model capabilities and pretraining compute. We hope that our work informs stronger evaluation standards that address central challenges in the current evaluation ecosystem.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey data selection for language models. arXiv preprint arXi:2242.166277 2024.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.1686) 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leaderboard. Hug- ging Face, 2023. URL http::ttt/////tt //hggg wewhuggingface. ctlacce.co/aeee wwwpo//paa///aaeee hwtp///ggggggggggggggegeeee oope_ leaderboard
Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable Im 2 1.6 b technical report. arXiv preprint arXiv:2402. 17834, 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-24335 2430. PMLR, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.077 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are 2020.
few-shot learners. Advances in neural information processing systems,
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, al. Internlm2 technical report. arXiv preprint arXiv:2403. 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Leurning Research, 24(240): 1-113, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-fimetuee lan- guage models. Journal of Machine Learning Research, 25(70): 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
arXiv preprint arXiv:
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 11.......1.0 466, 2021.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research, 12:2493-2537, 2011.
11

---

Published as a conference paper at ICLR 2025
Andrew M Dai and Quoe V Le. Semi-supervised sequence learning. Advances in neural information processing systems, 28, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 200.CCCVRR..29952066555
Jacob Devlin, Ming- Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171 117664 2019.
Ricardo Dominguez-Olmett Moritz Hardt, and Celestine Mendler-Dünner. Questioning the survey responses of large language models. arXiv preprint wrxiv:2306.00955/ 2023.
Ricardo Dominguez-Olmeda Vedant Nanda, Rediet Abebe, Stefan Bechtold, Christoph Engel, Jens Frankenreiter, Krishna Gummadi, Moritz Hardt, and Michael Livermore. Lawma: The power of specialization for legal tasks. arXiv preprint arXiv:2407.1666 2024.
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. arXiv preprint arXiv:2402.15938 2024.
Richard O. Duda and Peter E. Hart. Pattern Classification and Scene Analysis. Wiley New York, 1973.
EleutherAL. Language model evaluation harness. hutppes//gttuub..cm//llletttttt
1m evaluation- 2024. Accessed: 2024-05-20.
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open
Ilm leaderboardd v2. https: : www.gggggacc..eecoooo// wwwppp/as//ooooocooooccoooo 11m- leaderboard/
open 1tel11mmmmm__a leaderboaadrrrrrrrr.. 2024a. Accessed: 2024-07-08.
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Perfor- mances are plateauing, let's make the leaderboard steep again. https: / h/wuggggnggg mnginngfcc... co/ spaces/ leade reodaerboaaae rbl 2024b. Accessed: 2024-07-08.
Samir Yitzhak Gadre, Georgios Smymis. Vaishaal Shankar, Suchin Gururangan, Mitchell Worts- man, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. CoRR, 2024.
Ruyi Gan, Ziwei Wu, Renliang Sun, Junyu Lu, Xiaojun Wu, Dixiang Zhang, Kunhao Pan, Ping Yang, Qi Yang, Jiaxing Zhang, et al. Ziya2: Data-centric learning is all Ilms need. arXiv preprint wrXiv:2311.03300 2023.
Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1747-1764, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002) 2020.
John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett. Darpa timit acoustic-phonetie continous speech corpus cd-rom. nist speech disc 1-1.1. NASA STI/Recon technical report n, 93:27403, 1993.
Team Gemma, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre. Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint 2433:22403.08295, 2024.
Shahriar Golchin and Mihai Surdeanu. Time travel LLMs: Tracing data contamination in large language models. arXiv preprint 0rXiv:2308.08493, 2023.
12

---

Published as a conference paper at ICLR 2025
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lam- bert, Kyle Richardson, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannanch Hajishirzi. Olmo: Accelerating the science of language models. Preprint, 2024.
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad. Jesse Dodge, and Hannanch Hajishirzi. Olmes: A standard for language model evaluations. arXiv preprint arXiv:2406.08444 2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306. 2023.
Moritz Hardt and Benjamin Recht. Patterns, predictions, and actions: Foundations of machine learning. Princeton University Press, 2022.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Corrected 12th printing). Springer, 2017.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689 2022.
Team I.. Internim: multilingual language model with progressively enhanced capabilities, 2023.
Minhao Jiang, Ken Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. Does data contamination make a difference? insights from intentionally contaminating pre- training data for language models. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arxiv preprint arXiv:2001.08361, 2020.
Sayash Kapoor and Arvind Narayanan. Leakage and the reproducibility crisis in ml-based science, 2022.
Jacob Devlin Ming-Wa Weig Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pp. 4171- 4186, 2019.
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In The Tenth Machine Translation Summit Proceedings of Conference. pp. 79-86. International Association for Machine Translation, 2005.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\." ulu 3: Pushing frontiers in open language model post-nga arXiv preprint 241 2024.
13

---

Published as a conference paper at ICLR 2025
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online). Available: http://yamn.leaae www.yaannnleee....mm www/exdb/mnisst 2. 1998.
Changmao Li and Jeffrey Flanigan. Task contamination: Language models may not be few-shot any- more. In Pruceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18471- I- 18480, 2024.
Jeffrey Li, Alex Fang, Georgios Smymis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-Im: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406. 1794, 2024.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar. et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Reprexentations, 2018.
Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. Are emergent abilities in large language models just in-context learning? arXiv preprint DrXXv:2300.01809 2023.
Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. arXiv preprinl (iX.x:2203.082444 2022.
MetaAL Llama 3: Advancing open foundation models, 2024. URL httphttt:///// htttps://ai.mee. com/
blog/ lllog//eta-am -ameet--llmma-- ama-37
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402. 14830, 2024.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orea: Progressive learning from complex explanation traces of gpt-4. arXiv preprint urXiv:2306.02707 2023.
OpenLlama. Openllama,
11ama
Judea Pearl. Causality. Cambridge university press, 2009.
Judea Pearl. Linear models: A useful "microscope" for causal analysis. Journal of Causal Inference, 1(1):155-17,, 2013.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. NAACI. 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-6) 2020.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEFF, 2020.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level googlc-proof q&a bench- mark. arXiv preprint arXiv:2311. 12022, 2023.
Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. Data contamination through the lens of time. arXiv preprint /006:223/0.00688 2023.
14

---

Published as a conference paper at ICLR 2025
Anna Rogers and Sasha Luccioni. Position: Key claims in Ilm research have long tail of footnotes. In Forty-first International Conference on Machine Learning. 2024.
Oscar Sainz, Jon Ander Campos, Iker Ferrra-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Encko Agirre. NLP evaluation in trouble: On the need measure LLM data contamination for each benchmark. arXiv preprint arXiv:2310.18018, 2023.
Erik Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language- independent named entity recognition. Development, 922: 1341, 2003.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage? Advances in Neural Information Processing Systems, 36, 2024a.
Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has predicting downstream capabilities of frontier models with scale remained elusive? arXiv preprint urXiv:2406.04391, 2024b.
Charlie Victor Snell, Eric Wallace, Dan Klein, and Sergey Levine. Predicting emergent capabilities by finetuning. In First Conference on Language Modeling, 2024.
Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of chain-of- of-thought with multistep soft reasoning. In The Twelfth International Conference on Learning Reprexentutions, 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alona et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 0rXiv:2206.046155 2022.
StabilityyI. Stablelm, 2023. URL /tl./t/tp////gghhbbb.../////aaat
Shane Storks, Qiaozi Gao, and Joyce Y Chai. Recent advances in natural language inference: A survey of benchmarks, resources, and approaches. arXiv preprint arXiv: 01 172, 2019.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging big- big-bench tasks and whether
can solve them. In Findings of the Association for Computational Lin- guistics: ACI. 2023, pp. 13003-13051, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-foolloinng llama model, 2023.
TogetherWeCompute. Redpajama incite, 2023. URL https://ww.together.cemt
redpajanas
models-ve le
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:23100 12.311...9444,. 2023.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: Billion Parameter Autoregressive Language Model. httpp::/ htttp:://github...
wtt.:hwww..om/ii.iiegee.c com
hts.//eshtttt wttp://ttattttooommmeee jax May 2021.
15

---

Published as a conference paper at ICLR 2025
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Generalization via declarative instructions 1600+ nlp tasks. In Pro- ceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp- 5085-5109, 2022.
super--aturalinstrueee
Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo. Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. Mmlu-pro: more robust and challenging multi-task language understanding benchmark. arXiv preprint wxXi:2406.001544 2024.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprini (rXii:2206.07682 2022b.
Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, et al. Skywork: more open bilingual foundation model. arXiv preprint 1rXx:22310.1941, 2023.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307. 024477770777 2023.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lx, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305 2023a.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. CoRR, 2024.
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples, 2023b.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.0460 04652, 2024.
Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen Anirudh Goyal, and Sanjeev Arora. Skill-mix: A flexible and expandable family of evaluations for ai models. arXiv preprint arXiv:2310.17567 2023a.
Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2023b.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2019.
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang Yinghao Ma. Yizhi Li, Ziyang Ma. Bill Lin, Emmanouil Benetos, Huan Yang. Junting Zhou, Kaijing Ma. Minghao Liu, Morry Niu, Noah Wang, Quehry Que, Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang, Yuelin Bai, Yuhan Zhang, Yuxiang Zhang. Zenith Wang. Zhenzhu Yang. Zijian Zhao, Jiajun Zhang, Wanli Ouyang, Wenhao Huang, and Wenhu Chen. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv: 2405. 19327, 2024.
16

---

Published as a conference paper at ICLR 2025
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instlwction-folloiingg evaluation for large language models. arXiv preprint arXiv:2311.09911 2023a.
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your LLM an evaluation benchmark cheater. arXiv preprint arXiv:2311.0196. 2023b.
17

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Benchmarks have played central role in both machine learning (Hardt & Recht 2022) and natural
language processing (Storks et al. 2019). Classically, benchmarks comprised both a test set and a reasonably large training set (Garofolo et al. 1993 LeCun et al., 1998 (Sang & De Meulder 2003
Koehn| 2005; Deng et al. 2009). Models were trained on the same training set, and then eva evaluated on the accompanying test set. The success of unsupervised language modelling (Peters al. 2018
Kenton & Toutanova| 2019 Radford et al. 2019), however, has changed this paradigm. Firstly, present-day language models differ in their training data, which is not standardized but rather treated as a design choice (Raffel et al., 2020; AAlbala et al., 2024 Li et al., 2024). Secondly, language models are a priori not trained with the explicit objective of maximizing any single benchmark.
Data contamination. Data contamination or test-set contamination refers to any overlap between the training and the test data such that test results overestimate model's 's generalization performance. The scale and often little curation of present-day pretraining corpora exacerbates data contamination concerns in language model evaluations (Jiang et al. 2024). Consequently, data contamination is usually discussed in the technical reports accompanying model releases (Radford et al.) 2019 [Brown] ctt al., 2020 Chowdhery et al., 2023 Touvron al.) 2023b). However, detecting and preventing data contamination is currently an open problem (Gunasckar et al. [2023] Yang et al.) 2023b] Golchin & Surdeanu )023). Roberts et al. (2023) and Li & Flanigan (2024) find that models often perform better on datasets that were publicly available during model training. While almost all models that we consider were released after MMLU and GSM8K, we nonetheless find that, controlling for com- pute, more recent models perform better. These performance gains are unlikely to be driven solely by test set leakage and require additional explanation. In Section 3.2 we find evidence that that training on the test task may be more dominant factor in benchmark performance than data con- tamination. A key insight of our work is that models can train on the test task (e.g., multiple-choice question answering) and do much better seeing any benchmark data. This defies traditional notions of data contamination, which strictly refer to the leakage of benchmark data (Magar & Schwartz 2022] Sainz et al.) [023) Dong et al.) 2024). Moreover, whereas the core concern with data leakage is that benchmarks may inadvertently find their way the training data, training on the test task is often a deliberate design choice (e.g.. pre-training on instruction data).
given benchmark (e.g.. MMLJU) without necessarily
Adaptation prior to evaluation. In the 2010s, language models had to be adapted to different benchmarks using supervised task data (Collobert et al.) 2011 Dai & Le 2015 Devlin et al.) 2019).
The purpose of such fine-tuned benchmark models was solely to facilitate the relative comparison of base models. Importantly, all models were adapted using the same supervised data (Collobert et al., 2011]. With GPT-3 (Brown et al., 2020). few-shot prompting emerged the dominant paradigm for adapting models to particular task prior to evaluation (Liang et al.) (202)). arguably due to its simplicity relative fine-tuning, Bommasani et al. (2021) argue that benchmark evaluations should account for the adaptation resources used by each model (e.g.. the adaptation data). Similarly, [iang al.) (2023) argue that the strategy for adapting the models to the benchmark evaluation should be controlled for. Clearly, few-shot prompting is weaker form of adaptation than training on hundreds of thousands of task examples, as newer models often do. Our proposal of fine-tuning on the same, sufficient amount of task-specific data prior to evaluation aims to effectively control for model adaptation, ensuring that all models are given equal adaptation resources.
Training on the test task. The effectiveness of fine-tuning on the training set accompanying LLM benchmarks is well-known (Wei et al. [002aa] Wang et al.) [002)] Chung et al.) 2024). Consequently, datasets contain or are partly derived from benchmark train data (Wei et al.) 2022a] Honovich et al. 2022, Mukherjee et al., 2023). Li & Flanigan (2024) identify small amounts of benchmark -specfffc data in the publicly available Alpaca (Taori et al.) 2023) and Vicuna (Chiang et al., (023) instruction-tuning sets. Zhou et al. (2023b) empirically analyze the effects of fine-tuning on benchmark-speccffe data and warn about its impacts on benchmark validity. To circumvent these issues, recent work has focused on indirect indicators of broader data contamination, such as lack of robustness to task transformations (Wu et al. 2023), or underperformance on benchmarks with novel task combinations (Yu et al. (023a).. In contrast, find evidence for training on the test task without the need for explicitly identifying specific data
many influential instruction-tuming
18

---

Published as a conference paper at ICLR 2025
points used training time, modifying tasks. In addition, proposed method of fine-tuning on task data before evaluation allows us to quantify and correct for the effects of training on the test task on benchmark performance.
Emergent abilities of language models. Emergent capabilities WWi et al. 2022b Ganguli et al., 2022) refer to levels model performance large scales that cannot be easily predicted by extrap- olating from smaller scales. Wei et al. (2022b) report emergent capabilities for various benchmarks including MMLU and GSM8K (Srivastava al. (202). However, Srivastava (rivasttsS al. (2022) [Schaeffer er al. (2024b) find that the log- gee-rbbability of the correct answer often improves smoothly, even when other metrics seem to show emergence. RRggers & Luccioni (2024) question the dominant definition of emergence and emphasize the importance of relating the training data to the test data before mak- ing claims about emergence, Lu et al. (2023) argue that most emergent capabilities can be explained by ii-context-learning
Gadre et al. (2024) find that model's perplexity its pre- vre-training data reliably predicts its average downstream performance. However, their analysis does not include the MMLU and GSM8K benchmarks, which are central our work. [Schaeffer et al. (2024a) argue that emergent capabilities are mostly an artifact non-linear and discontinuous evaluation metrics like accuracy. In contrast, like the Brier score. Similarly to our findings, [nell et al. (2024) show that increasingly fine-tuning on the task shifts the point of emergence to smaller compute scales.
find signs of emergence
MMLU even when using continuous metrics
B ADDITIONAL EXPERIMENTAL DETAILS
B. MODELS CONSIDERED
Model size in billions of parameters is indicated by N and pretraining data size in trillions of to- kens is indicated by D. Model weights were retrieved from the corresponding HuggingFace (HF) repositories.
Name
Train date
N
D
HF repository
Citation
baichuan- 135
2023-06
1.4
baichuaa-iic//Baccuuan
Yang
2023a
13B-Base
baichuan-7b
2023-06
7
1.2
Daichuan-inc/Baichuan
Yang et al.
2023a
7B Base
baichuan2-15
2023-09
2.6
baichuann-ic//Bacchuann
Yang et
2023a
13B-Base
baichuan2-7b
2023-09
7
2.6
Bicchuan-inc/Baichaan
Yang al.
2023a
7B-Base
falcon-1 11b
2024-05
11
5.0
tiuac/falcon-IIB
Almazrouci ct al.
2023
falcon-7b
2023-04
7
1.5
tiuae/falcon-7b
Almazrouet et
2023
gemma-22 2b
2024-02
3.0
google/gemma-222
Gemma
gemma-7b
2024-02
6.0
google/gemma-771
Gemma et al. (2024)
gpt-j-6b
2021-03
6
0.4
EleutherAl/gpt---bb
Wang & Komatsuzaki
(0021
internIm-200
2023-09
20
2.3
intemlm/internlm 20b
Intern (2023
internim-7b
2023-07
7
1.0
intemmlmiinternlm-77
InterL. (2025
(2025
internlm2-base-
2024-01
20
2.6
internlmmintermm22-mass
Cai et al.
2024
20b
20b
internim2-basee
2024-01
7
2.6
internlm/ing caaimmmmintenlmmmbassee base-7b
Cai ct al (024)
Ilama- 13b
2023-02
13
1.0
None
Touvron et al.
(203aa
1lama-2-13b
2023-07
2.0
meta-Ilama/l 1laamm/lama-2-111 -ama-2-13b-
Touvron el al.
2023b
hf
Ilama 2-70b
2023-07
70
2.0
meta- Ilama/Llanm 2-70b-
Touvron et al.
(003b)
hf
llama-2-1
2023-07
7
2.0
meta- Poeta-llama/Lantte 7b-
Touvron et al. MetaAl 2024
2023b
Ilama-3-8b
2024-04
8
1.5.0
Bertalllama/Meetllllaaaaa
8B
Ilama-300
2023-02
32.5
None
Touvron et al.
(023aa
Ilama 65b
2023-02
65.2
1.4
None
Touvron et al
2023a
Ilama-7b
2023-02
1.0
None
Touvron et
2023a
19

---

Published as a conference paper at ICLR 2025
map-neo-7b
2024-05
4.5
m-a-p/neo_70
Zhang al. (2024)
olmo-1.7-7b
2024-04
7
2.0
allenai/OLMoo-17-7555
Groeneveld ct al.
2024
olmo-1b
2024-01
2.0
AllenaiOOLMM-IB-.
Groeneveld 2024
olmo-7b
2024-01
2.5
allenai/OLLMo-BB.
Groeneveld 2024
openllama-13h
2023-06
13
1.0
openim-
Open1 (2023
rescarch/open llama 13b
openilama-3b
2023-06
1.0
openlm-
OpenLIama (2023
rescarch/open__lann
openllama-3b-v2
2023-07
3
1.0
openlm-
OpenLlama 2023
research/open Ilanch/a/opee_aaaaaaa _3__v2
openllama-7b
2023-06
1.0
openlm-
OpenLlama 2003
research/open.llanaa
1penllama-7b-v2
2023-07
7
1.0
openim-
OpenL.laaa 2023
rescarch/open Ilama .v_2
pythia-1.44 1.4b
2022-10 10
1.4
0.3
AleutherAl/pythia- 1.4b
Biderman ct al.
2023
pythia-
2022-10
12
0.3
EleutherA//pythia- 12b
Biderman al
2025
pythia- 160m
2022-10 10
0.16
0.3
AlputhorA//pythia- 160m
Biderman
2023
py pythia-Ib lb
2022-10
0.3
EleutherA//pyytha--l
Biderman et al.
2023
py 2yth
2022-10
2.8
0.3
EleutherA//yyttaa22.55
Biderman al.
2023
pythia-410m
2022-10
0.41
0.3
EleuthorA//yythha-41100
Biderman
py 6yth
2022-10
6.9
0.3
EleutherA//pyythaa-6.99
Biderman et al.
2023
90thha-70mm
2022-100 10
0.07
0.3
EleutherA//pythia-7000 70m
Biderman al.
2023
qwen-1.50 1.5-0.
2024-01
2.4
Qwen/Qwenl.5-0.5B
al. 2023
qwen-1.55 1.b
2024-01
1.8
2.4
Qwen/Qwenl.55
Bai el al.
2023
qwen- 1.5-14b
2024-01
14
4.0
Owen/Qwenl.5-1ad
Bai ct al. 2023
qwen-1.5-48
2024-01
2.4
Qwen/Qwenl.5-4B
al. 20233
qwen-1.5-7b
2024-01
7
4.0
Owen/Qwenn..5-11
Bai ct al. 20233
qwen2-0.5b
2024-06
0.5
12.0
Owen/Qwen2-0.5B
Yang al. (2044
gwen2-1.55
2024-06
7.0
Qwen/Qwen2-I.5B
Yang et al.
2024
qwen2-7b
2024-06
7
7.0
Qwen/Qwen2-7B
Yang ct al.
(2024)
redpajama-3b
2023-05
0.8
logedhercommutee/RePPaaaaa
OngetherWeCoppute
INCITE-Base
(2023)
redpajama-76
2023-05
7
1.0
logethercoomppterRRePiiiii
INCITE-7B-Base
(2023)
skywork-138
2023-10
3.2
Skywork/Skywork-13B
Wei 2023)
base
stablclm-2-11
2024-01
1.6
2.0
stabilityai/stabell--2 6bb
Bellagente ct al.
(0244)
stablelm-2-12b
2024-03
12.1
2.0
stablityai/stabell-22211
Bellagente
2024
stablelm-3b-4e
2023-09
2.8
4.0
stablittauistabiiim---ee1111
Stability
2023
stablelm-base-
2023-08
2.8
1.1
stabiliyyaisttabeln--osse
Stability Al 2023
alpha-3b-v2
alpha-3b-v2
stablelm-base-
2023-08
1.1
stabiltyaistaaleim-bass
Stability 2023
alpha-7b-v2
alpha-7b-v2
yi-6b
2023-11
6
3.1
01-ai/Yi-1.5-6B
Young et (024
riya2-13b-ba
2023-11
2.65
ICEL
Gan et all
2023)
Base
B.1.1 INSTRUCTION-TUNED AND CHAT MODELS
Name
Train date
Base
HF repository
Citation
model
Dllcon-bb-instruet
2023-04
falcon-7b
tiuue//falcom-7b-insmcce
Almazrouei et al.
2023
gemma-26 2b-
2024-02
gemma 2b
google/gemma- 2b-it
Gemma al.
2024
instruct
gemma-7b- instruct
2024-02
gemma-7b
googlefgemma-77hh
Gemma et al. 2024
internlm-chat-206
2023-09
internlm- 20b
intemlm/intemml-<hat-2200
InternIM 2023
internlm-chat-7b
2023-07
internim- 7h
intemlm/imtemmmmcchat77
Internl M 2023
20

---

Published as a conference paper at ICLR 2025
internm2-76
2024-01
internlm2-
inteminn/ntermmm2-776
Cai
2024
base-7b
internlm2-chat-
2024-01
internlm2-
inlemiim/intermmm22chaa
Cai et al.
2024
1_8b
base-7b
1_8b
intermi2-ccha
2024-01
intermm22-
intemlm/mnnemmm2-cchat
Cai et al.
2024
20b
base-20b
20b
Intemlm2-chat-11 7b
2024-01
internim2-
mmemim//ttemmmm2cchtt-77
Cai ct al. 2024
base-7b
llama-2-133 1-3-chhat
2023-07
llama-2-
Maa-llammallaaa2---11 chat-hf
Touvron et al.
2023b
13b
1aa-hh--
2023-07
llama-2-7b
meta-llama 2iaellama/LLlma-22770
Touvron al.
2023b
chat-hf
llama-3-8--
2024-04
Hama-3-8b
meta-llama/Meta- Llama-3-
MctaA 2024
instruct
8B-Instruct
map-neo-7b-
2024-05
map-neo- 7b
Zhang et al. (2024)
instruct
map-neo-7b-sft
2024-05
map-neo- 7b
heal-p/no..7..ff..v00
Zhang et al. 2024
olmo-7b-0724-
2024-01
olmo-7b
allenai/OLMo-77-07724
Groeneveld et al,
2024
instruct-hf
Instruct-hf
almo-7h-0724-5 hf
2024-01
olmo-7b
allena/OL.Mo-77-07722
Groeneveld et al.
2024)
SFT-hf
olmo-7b-instrus ib-instruct hf
2024-01
olmo-76
allona/OLLMo-7Bnnsttn
Groeneveld et al.
2024
hf
olmo olmo-7b-ste 7b-stthif
2024-01
olmo-78 7b
allnna/OLMMo-7BSFTTM. Qwen/Qwenl.5-um
Groeneveld et al.
2024
qwen- 1.5-0.5b-
2024-01
qwen-1.5
Bai al.
2023
chat
0.5b
qwen- 1.5--.bb- chat
2024-01
qwen-1.55
Qwen/(Ween.5-11.BB Chat
Bai
2023
1.8b
qwen- 1.on-115-111 1.b- 14b- chat
2024-01
qwen-1.5 1.5-
Qwen/(Wwen..5-14B-CCC
Bai et al.
2023
14b
qwen- 1.5--,,.--bchat
2024-01
qwen-1.5-
Qwem/QWwen..5-4BBCCh
Bai 2023
qwen- Jwwen-l.5-7b-chat
2024-01
qwen-1.55
1wcnn/Owen1.5-7B-CCh
Bai ct al.
2023
7b
hidpajama-7b-char
2023-05
redpajama-
fogethorrommpute///ePPPPaaaaa
TogetherWeCompure (2023)
7b
INCITE-7B-Cha
redpajama-chat-
2023-05
redpajama-
Rogetherconppue//ReePjjiaaa
TogetherWeCompee
3b-v1
3b
INCITE-Chat-3B-vl
2023
redpajama-
2023-05
redpajama-
logethercoomputte/ReePPiiii.nn
Togelher
instruct-3b-vl
3b
INCITE-Innttrut-3BB-v
(023)
redpajama-
2023-05
redpajama-
fogctheroomputt//RaPPjjai
logetherWeCompure
instruct-7b
7b
INCITE--7B-Insrruct
2023)
stablclm-2-1.6b.
2024-01
stablelm-
stabllityaistabbci----...6
Bellagente et al, (2024)
chat
2-1.6b
chat
stablelm-2-12l-
2024-03
stablelm-
stablityaistaabell--211b-
Bellagente et al.
2024
chat
2- 2-12b
chat
stablelm-2-
2024-06
stablelm-
stabilityai/stablllm- zephyr-1 6b
Bellagente et al
2024
zephyr-1.6b
2-1.6b
2023-11
stablelm-
stabilityaistablellm zephyr-
Tunstall et al. (2023
3b-4elt
vicuna- 13b-v1.1
2023-04
llama-1 13b
leeys/vicumaa33--vi....
Chiang et al. 2023
vicuna- 133
2023-06
laaa- 13b
minys/vicuna-..b-vi....
Chiang ct 2023
vicuna-7b- 7b-v1.1
2023-04
Пaaa- 7b
Imsys/vicuna- -b 1b.
hhang et al. 2023
vicuna-7b-vl.3
2023-06
llama-76 7b
Imsys/vicuma-7b-1..3
biang et al.
2023
B.2 FINE-TUNING HYPPERPARAMETERS
We fine-tune all model parameters. For models with less than 10B parameters, we fine-tune on single GPU with BF16 precision. For models between 10B and 30B parameters. we train single H100 node using DeepSpeed 7eRO-3 (Rajbhandari al. 2020) and full precision. For models with more than 30B parameters, train H100 nodes using DeepSpeed ZeRO-3 and full precision. Due the large compute cost of the
21

---

Published as a conference paper at ICLR 2025
Original hyperparameters
Best hyperparameters for newer models
0.7
Difference 0= 0.005
Difference ê = 0.005
(Post- (Post-adjius) aadfaaajujuntt
(Post cess-addustmentt
0.6
Regression R² = 0.990
Regression R² = 0.992
MMLU
MMLU
0.5
0.4
0.3
10
10
10
10
10a4
1020
10
102?
10
10a
Pre-training compute (FLOPs)
Pre-training compute (FLOPs)
Figure 9: We perform learning rate sweep using values [6 10-3, 2 10-5, 6 10-6, 2 10-6 6.10-J. The left plot shows MMLU performance after fine-tuning with the original learning rates detailed in Appendix B.2 The right plot shows MMLU performance after fine-tuning but using for newer models the optimal learning rate from the sweep that maximizes MMLU performance. Even with the advantage of higher hyperparameter search budget for the newer models, the estimated effect size of model recency on benchmark performance remains both small and not statistically significant.
experiments, perform minimal hyperparameter luning and use standard hyperparameter choices throughout. We use learning rate of 2- 10-6 for models with fewer than 10B parameters and learning rate of 2- 10 for models with more than 10B parameters. For Qwen 2 well as four the 7B models Gemma 7B, Olmo 7B, Olmo 1.7 7B, and Llama 8B- benchmark accuracy degraded after tiniitunng. For these models, we use a peak learning rate of 2- 10 instead. We use a cosine learning rate schedule with linear warm-up for 50 steps and decay to 10% of the peak learning rate. We use AdamW (Loshchilov & Hutter 2018) as the optimizer, with 81 0.9, 822 0.95, and 10- We fine-tune with batch size 64. We 0.1 and clip gradients at 1.0. We verify that the training loss decreases for models on both the fine-tuning datasets. To reduce the computation burden of fine-tuning, we train with context size 600. We verify that less than 5% of the fine-tuning examples have context length above 600.
weight decay rate of
We use internal cluster of A100 and H100 GPUs Fine-tuning all models required approximately 10,000 H100 GPU hours, whereas evaluating all models in the different benchmarks required approximately 400 H100 GPU hours.
B.3 ROBUSTNESS CHECK ON THE HYPERPARAMETER SEARCH BUDGET
We found the learning rate be, by large margin, the single most impactful hyperparameter. We perform sweep on the MMLU auxiliary training set with the following learning rates: [6- 10 2.10 6-10-6,7 1,10-6,2100 6 10 100") "1. We are unable to perform more extensive hyperparameter sweeps due to their large computational cost. We find that no models benefit from smaller learning rate than 2 10 and only one model benefits from larger learning rate than 2- 10 5, Pythia 70M (the model with smallest pre training compute). Thus, for every model except Pythia 70M, the optimal learning rate inside boundary of the hyperparameter sweep.
A potential concern is that older models may appear to match the performance of newer models because the fine- tuning hyperparameters could be systematically more favorable to the older models. To address this concern, recreate main experiment by fine-tuning older models with their original learning rates, while selecting for newer models the optimal learning rate in the sweep that leads to highest MMLU performance. That is, we deliberately give newer models systematic advantage in terms of hyperparameter search budget. We plot the results in Figure g The estimated effect size of model recency on benchmark performance shifts from â = -0.005 å 0.005. Therefore, despite giving newer models a systemic advantage terms of hyperparameter search budget, the effect size remains both remains small and not statistically significant.
C CAUSAL INTERPRETATION OF OUR FINDINGS
In Section 2.2 we established that models trained after November 2023 significantly outperform those trained before November 2023 for both MMLU and GSM8K. We then showed that fine-tuning all models in the test task equalizes the performance of newer and older models. We now present causal interpretation of
22

---

Published as a conference paper at ICLR 2025
C
N
A
T
Figure 10: Whether a model was trained after November 2023 (N) influences its pretraining com- pute (C) and how much it trains the test task (T). All three influence the benchmark accuracy (A) of the model.
findings, aiming quantify the extent to which the effect of model recency N on benchmark accuracy A is mediated by training on the test task
The key obstacle to our analysis is that test task training unobservable. Firstly, because practitioners are typically not transparent about their design choices, including the pretraining data. Secondly, because the extent to which different training practices might amount to test task training is unclear. Nonetheless, by fine-tuning intervene the extent which models train on the test task.
spec-specific data,
Figure I0 summarizes our causal assumption. The time which a model was trained determines the design choices made, such as pretraining data or pretraining compute C. These design choices turn affect how much the model trains on the test task. All these factors ultimately influence the pretrained model and thus its benchmark performance. We also admit that compute might influence test task training. For instance, pre-training larger datasets may lead models training more on the test task.
interpret proposed adjustment method as intervening on test training variable T'. Namely, by fine-tuning models on the same amount of task-specific data hefore evaluation. The external validity of our subsequent analysis hinges on the assumption that our controlled experimental setting -finc-tuning models after the pretraining stage- is reasonably similar to the natural settings in which practitioners might train on the test task during pretraining (e.g.. . by including instruction data in the pretraining data mixture). We provide evidence for this Appendix D.3
We model fine-tuning as a hard intervention do(7 = t) (Pear 2009p. The specific magnitude of the interven- tion f need not be quantificd. Instead, the key assumption is that by fine-tuning on the same, sufficient amount of task data, all models will have received the same amount of test task training. Since some base models may have already trained the test task before fine-tuning, this assumption only holds if test task training saturates, we train on enough task data to reach saturation. The fact that our task-specific datasets allow older models to match the performance of newer models provides some evidence that we train on enough task-specific data to reach saturation.
We draw inspiration from scaling laws (Kaplan al. 2020) and model the relationship between pretraining compute and its causal descendants piecewise log-linear:
[o]
f(C,a) = ou Σ a logC C. q]
simplicity, we consider three fixed knots at c2 0, 02 20 and c3 1023 FLOPs. We assume other variable relationships to linear, resulting in the structural assignments:
T= = S(C, 3) + + +6, 8~ ((0.00))
(3)
A A = +N + + + c~~(0,03)
(4)
We denote henchmark accuracy after fine-tuning as A Al To estimate the direct effect N A of model recency on accuracy, we regress the linear model
A| (CC, +N +
f(C, u te, n/=n+
(5)
where tb, 'f' are the fit's parameters and is random noise. The coefficient 1 corresponds to the direct effect N A of model recency benehmark accuracy. We additionally regress the difference in accuracy pre-
23

---

Published as a conference paper at ICLR 2025
Table 3: The indirect effect + T A me- diated by test task training T is positive, sig- nificant, and large: newer models attain higher benchmark scores primarily because of training on the test task.
Table 4: We find evidence of significant di- rect effect of model recency N on accuracy A, that is, of the improvements of newer models being attributable to anything else other than training on the test task.
MMLU
GSM8K
MMLU
GSM8K
0.071
0.168
-0,004
0,000
v/
(0.018)
(0.032)
(0.009)
(0.032)
R
0.530
0.503
R²
0.926
0.763
Standard errors in parentheses. Bold indicates
Standard errors in parentheses. Bold indicates
p < 0.05.
0.05.
and post-intervention
A - =
-
-(C,, + 70N + +70 76 7 +1 62
for B' = -0= = 62 (6)
where 3¹, o², b are the fit's parameters and é' is random noise. The coefficient ø' corresponds to the indirect effect N -> >> A of model recency on benchmark accuracy mediated by test task training Peaal 2013). That is, the improvements in accuracy of recent models attributable to training on the test task.
We fit the models in Equation S and Equation 6 and we report the coefficients pertaining to N + A and .V + ''' A in Table 4 and Table 3 We find that the indirect effect N + 7' + A of model recency on find evidence therefore find no evidence the
accuracy mediated by test task training T is significant, positive, and large. In contrast, of significant direct effect N - A model recency on accuracy. improvements of newer models being attributable to anything else other than training on the test task.
In conclusion, our causal analysis indicates that the differences in MMLU and GSM8K performance between newer and older models observed in Section 2.1 Jree largely attributable to differences in test task training. That is, the mechanism hy which newer models outperform older models is primarily by training more the task.
D ROBUSTNESS CHECK ON THE TEMPORAL SPLIT
D.1 ADJUSTING THE TEMPORAL CUTOFF BY A FEW MONTHS
We repeat the analysis of Section for additional temporal splits: September 2023 and January 2024, and present the results in Figure IT and Figure 12 respectively. Our results are robust to shifting the temporal cutoff by a few months. That is. our findings indicate that practitioners started adopting design choices around late 2023 that amount to models training on the test task much more, which is consistent with models' technical reports starting mention the use of benchmark instruction data pre-training time. Choosing specifically the month of November as the cut-off therefore critical for our analysis.
D.2 EN vs CN LANGUAGE DATA
Instead of diving models using temporal split, divide models based on whether they trained primarily on English (EN) data on mixture of English and Chinese (EN+CN) language data. While there is considerable overlap between the temporal split and the EN/EN+CN model split, there are notable differences. In particular, the Baichuan, Baichuan 2, and InternLM, and Skywork families were trained before November 2023 and trained on EN+CN data, Conversely, Gemma, Llama 3, StableI 2, Falcon 2, and Olmo were trained after November 2023 and trained on EN data.
repeat the analysis of Section 2 for the EN and EN+CN model split, see Figure 13 trolling for pretraining compute, models trained on EN+CN language data outperform those trained primarily on EN by 9 accuracy points on MMLU and 12 accuracy points on GSM8K. After the proposed adjustment, however, the difference in performance between models trained on EN data and EN+CN data is small and not statistically significant.
We observe that, con-
24

---

Published as a conference paper at ICLR 2025
Model split: trained before September 2023
GSMBK
0.8
0.7
Difference 0.074
Difference 6= 0.158
Accuracy
0.6
Regression R² 0.942
Accuracy
0.6
Regression R² 0.898
0.5
0.4
0.4
0.2
0.3
MMLU (adjusted)
GSMBK (adjusted)
0.8
0.7
Difference 6= 0.014
Difference ê= 0.005
Accuracy
0.6
Regression R² 0.991
Accuracy
0.6
Regression R² 0.957
0.5
0.4
0.4
0.2
0.3
0.0
1020
10
104
10
1044
10
10
10a2
10-2
10
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Models trained
Before September 2023
After September 2023
Figure 11: Robustness check with September 2023 as the temporal cutoff.
Model split: trained before January 2024
MMLU
GSMBK
0.7
Difference 0.056
0.8
Difference ê 0.146
Accuracy
0.6
Regression R 0.934
Accuracy
0.6
Regression R² 0.891
0.4
0.4
0.2
0.3
0.0
MMLU (adjusted)
GSM8K (adjusted)
0.8
0.7
Difference 0.011
Difference 0.009
Accuracy
0.6
Regression R² 0.990
Accuracy
0.6
Regression R² 0.957
0.5
0.4
0.4
0.2
0.0
10
10
10aa
1021
10a4
1020
10
10
10ah
10
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Models trained
Before January 2024
After January 2024
Figure 12: Robustness check with January 2024 the temporal cutoff.
The confounding and measured effect sizes for the EN and EN+CN model split resemble those obtained for the temporal split, which interpret valuable robustness check of results.
D.3 How SIMILAR ARE NEWER MODELS OLDER, FINE-TUNED MODELS?
Section 3.1 we fine-tune older models on test task, we demonstrate that the differences bench- mark performance between the tne-ttneed and non fine-tuned models resemble those between newer and older models. In this section we provide further evidence that newer models resemble older, fine-tuned models.
We take the older models and we fine-tune them with 64,000 training examples from the auxiliary training sets introduced in Section 2 We plot in Figure the benchmark scores of the older, fine-tuned models well newer models. qualitatively observe that both the older, fine-tuned models and the newer models exhibit similar scaling. That is, older fine-tuned models resemble newer models terms of performance per compute.
We perform quantitative analysis consisting in discriminating between the older models and the newer models based their pretraining compute and benchmark accuracy. That we construct tabular dataset where rows
25

---

Published as a conference paper at ICLR 2025
Model split: trained primarely EN data
GSMBK
0.8
0.7
Difference 0.092
Difference 6 0.125
Accuracy
0.6
Regression R² 0.951
Accuracy
0.6
Regression R² 0.876
0.5
0.4
0.4
0.2
0.3
MMLU (adjusted)
GSMBK (adjusted)
0.8
0.7
Difference 0 0.009
Difference ê 0.011
Accuracy
0.6
Regression R² 0.990
Accuracy
0.6
Regression R² 0.957
0.5
0.4
0.4
0.2
0.3
0.0
1020
10
104
10
1044
10
10
10a2
10-2
10
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Models trained
Primarily EN
both and CN
Figure 13: Models trained on both English (EN) and Chinese (CN) language data outperform those trained primarily on English data. After adjusting for test task training, find evidence of significant difference 0 in performance between models trained on EN data and EN+CN data.
MMLU
GSM8K
0.75
Accuracy
0.6
0.50
0.4
0.25
0.00
1020
1021
1022
1023
20
1020
1021
1022
10
1024
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Fine-tuned old models
New models
Figure 14: New models resemble old models that fine-tuned. Temporal cut-off:: November 2023.
Table 5: Accuracy in discriminating between older and newer models in terms of their pretraining compute and benchmark accuracy. Older, fine-tuned models are indistinguishable from newer mod- els.
Discriminator test
MMLU
GSM8K
Older models vs newer models
64.6%
73.9%
Fine-tuned, older models newer models
52.2%
52.5%
Random chance accuracy is 50%.
models and columns their corresponding pretraining compute, benchmark accuracy, and whether the model was trained after November 2023. then train classifier aiming to predict model recency from compute and accuracy. Intuitively, if the performance of older models is very different from that of newer models, then we would obtain high prediction accuracy (i.e., the two classes are highly separable). Note that prediction accuracy provides lower bound the total variation (TV) distance between the distributions of compute and accuracy of older and newer models.
26

---

Published as a conference paper at ICLR 2025
We train XGBoost classifiers and report balanced accuracy for leave-one-out cross-validation in Table We obtain close to gandom-chancs accuracy discriminating between older, fine-tuned models and newer models. That is, older fine-tuned models are indistinguishable from newer models in terms of their performance per pre-training compute.
E RESULTS FOR THE OPENLLM LEADERBOARD v2
HuggingFacc released on June 2024 a revision the OpenI.LM Leaderhoard (Fourrier et al. (202aa). The HF leaderboard v2 differs from v1 in the six benchmarks it considers: MMLU Pro (Wang et al. 20244), GPQA (Rein al. 2023). BBH (Suzgun et al., [003), MuSR (Sprague el al. 2023). the Level 5 subset of MATH (Hendryckss al. 2021 and IFEval (Zhou al. (2025)). MMLU and GPQA test for knowledge and are framed as multiple- choice questions. BBH and MuSR test for reasoning. MATH tests for mathematical reasoning. IFEval tests the ability of models to follow instructions.
The creators of the OpenLLM Leaderboard cite contamination as key motivation for releasing the v2 revision. They note that key criterion in choosing the benchmarks of the HF leaderboard v2 was lack of contamination in models as of today. In particular, FForrier et al. (2024b) claim that current models are not contaminated for GPQA, MuSR, and MMLU Pro: GPQA due to the gating of the test set, and MuSR and MMLU Pro due to their "youth". Fourrier et al. 2222bb succinctly express their concern as regards to data contamination in the HF leaderboard vl:
"Some newer models also showed signs of contamination. By this, we mean that models were possibly trained on benchmark data or data very similar to benchmark duta. As such, some scorey stopped reflecting the general performance of the model and started overfit on some evaluation datasets instead of reflecting the more general performance of the task being tested. This was, in particular, the case for GSM8K and TruthfulQA, which were included in some instruction fine-tuning sets."
Note that "models were possibly trained on benchmark data or on data very similar to benchmark data" " encompasses not only test set contamination but more broadly training on the test task.
We evaluate all models on MMLU Pro, GPQA, BBH, MuSR and MATH Lvl 5. We use the LM Evaluation Harness library in an identical fashion the HF leaderboard v2. We do evaluate on IFEval since tests for instruction following and we evaluate base models. We additionally evaluate the models that we fine-tumed in Section [2.] for multiple choice question answering and mathematical reasoning. This gives us models' adjusted benchmark scores after training on multiple choice question answering and mathematical reasoning. For MATH Lvl 5. we use the models fine-tuned mathematical data, whereas for MMLU Pro, GPQA, BBH and MuSR we use the models fine-tuned on multiple choice question answering. The fine-tuning datasets were not adapted to the new benchmarks in the HF leaderboard v2, thus giving a valuable insight into how well these relevreleannt datasets generalize beyond MMLU and GSM8K.
We plot in Figure 15 models benchmark scores pre- and post-post adjustment. We find that newer models significantly outperform older ones in all five benchmarks after controlling for pretraining compute. The dif- ferences in performance are smaller in absolute terms than those measured MMLU (0.073) and GSM8K (0.191). This is in part because these benchmarks are "harder", meaning also smaller differences in perfor- mance between the best and worst model. For this reason, we also report the difference between newer and older models relative to the difference between the hest and worst model. This relative difference is 13.7% for MMLU Pro, 14.5% for GPQA, 12. 1% for MuSR, 9.7% for BBH, and 10.0% for MATH Lvl 5, compared 15.3% for MMLU and 25.0% for GSM8K. Therefore, newer models overperform in MMLU Pro, GPQA and MuSR about as much as they do for MMLU, and somewhat less for BBH and MATH Lv1 S.
Fine-tuning on task-relevant data reduces the difference in performance between newer and older models for all five benchmarks. Therefore, we find evidence that training on the test task plays substantial role in newer models outperforming older ones in the benchmarks of the HF Leaderboard v2. For GPQA and MuSR, the difference in performance after adjustment is very small ((0) < 0.002) and not statistically significant. For BBH, the estimated difference in performance Ô reduces by 40% to 0,015 and is no longer statistically significant. For MMLU Pro and MATH Lv1 5 the difference reduces by 19% and 33% respectively but remains reasonably large (& ? 0.01).
One possible reason for the fact that the adjustment for MMLU Pro and MATH Lvl 5 is not as effective as for MMLU and GSM8K is that the fine-tuning examples are simply not relevant for MMLU Pro and MATH Lv]5. For example, the questions and answers in MATH Lvi contain much more LaTeX equation formatting than our mathematical reasoning fine- tuning dataset. Similarly, our multiple choice tine- mie-tuning dataset contains mostly questions with 4 answer choices, whereas all MMLU Pro questions have 10 answer choices. Thus, models are primarily fine-tuned to answer "A", "B", "C", and "D" but not "E", "F", "G" We modify MMLU Pro contain questions with 4 answer choices by randomly discarding 6 of the incorrect answer choices. We
27

---

Published as a conference paper at ICLR 2025
Unadjusted
Adjusted
θ 0.036
θ 0.024
MMLU Pro
0.3
0.2
0.1
0.35
Ô= 0.014
Ô= 0.002
GPQA
0.30
0.25
0 0.015
θ 0.000
0.45
MuSR
0.40
0.35
0 = 0.025
0.015
0.5
BBH
0.4
0.3
MATH Lvi 5
0.15
ê 0.016
0.0.33
0.10
0.05
0.00
1020
10
10
10
10201020
1021
20
1023
1026
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Models trained
Before November 2023
After November 2023
Bold indicates statistical significance with p 0.05.
Figure 15: Results for the OpenLLM Leaderboard v2. For all benchmarks, models trained after November 2023 significantly outperform models trained before November 2023 when controlling for pretraining compute. After fine-tuning models multiple choice question answering and math- ematical reasoning. differences in performance between newer and older models reduce for all five benchmarks. These differences significant for MMLU Pro and MATH Lv1 5.
longer significant for GPQA, MuSR and BBH, but remain
evaluate models pre- and post-adjustment and plot the results in Figure 16 performance between newer and older models after adjustment reduces from 0.024 to 0.016, and no longer statistically significant. This observation suggests that fine-tuning one more relevant task-data might further and older models in MMLU Pro and MATH Lv1
observe that the difference
reduce the gap between
Discussion. Fourrier et al. (2024b) cite newer models overperforming in the HF leaderboard v1 due to being "possibly trained on benchmark data or on data very similar to benchmark data" as major reason for the HF leaderboard v2 revision. We, however, find evidence that training the test task is also confounder for the newly included benchmarks. Specifically, the difference in performance between
and older models
28

---

Published as a conference paper at ICLR 2025
Unadjusted
Adjusted
0.6
(4 choices)
= 0.049
ô= 0.016
MMLU Pro
0.5
0.4
0.3
1020
10
1022
1023
1024
10
1021
1022
10
1024
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Models trained
Before November 2023
After November 2023
Bold indicates statistical significance with p 0.05.
Figure 16: We modify MMLU Pro to only contain questions with 4 answer choices by for every question randomly discarding the incorrect answer choices. After adjustment, the difference in performance between newer and older models is smaller and no longer statistically significant.
Model Rankings on MMLU
Worst model
Best model
Before adjustment
After adjustment
ei unemini skpaed stal i interiin) Balaary oo 2.14 (peen dea
7900
pyth
tte Belaiin To guee
openialnn
Figure 17: Training on the test task significantly alters model rankings on MMLU.
significant for MMLU Pro, GPQA, MUSR, BBH and MATH Lvl these differences reduce after adjusting by fine-tuning on the test task.
Fourrier et al. 2024b) explicitly highlight GPQA and MuSR as benchmarks likely unaffected by contamina- tion, the former due to being gated and the latter due to its "youth". Not only do newer models significantly outperform older ones in GPQA MuSR, but these differences in performance fully vanish after fine-tuni tine-tuning on the test task. That is, newer models likely overperform in GPQA and MuSR precisely due to training on test task.
These findings highlight that training the test task is distinct phenomenon from test set leakage. Strategies that aim to mitigate data contamination -e.g.. dynamic benchmarks- might not be effective in mitigating the confounding effect of training on the test task. In contrast, we extensively demonstrated our proposed adjustment procedure, that is, fine-tuning on sufficient task-relevant data before evaluation.
effectiveness of
F ADDITIONAL FIGURES
MMLU rankings Training the test task significantly alters model rankings MMLU, with average shift of 4.8 ranks and maximum shift of 16 ranks.
Reformulating ARC and HellaSwag as multiple choice In Figure 18 we show that ARC and Hel- laSwag do not exhibit emergence when using the standard cloze evaluation. When reformulating the task as multiple choice in the style of MMLU, however, we observe emergence around 1022 to 1023 FL.OPs, simi- larly MMLU. Emergence in this range of compute persists even when changing the evaluation metric from accuracy to Brier score -a continuous metric-, as suggested by [Schaeffer et al. (2024a).
Emergence for GSMSK as models train on the test task Similar to MMLU, we find that increasingly fine-tuning models mathematical reasoning makes the phenomenon of emergence gradually disappear, Figure 19 The point of emergence arises at increasingly lower scales, recovering cleaner log-linear fits.
29

---

Published as a conference paper at ICLR 2025
Cloze
Multiple choice
Multiple choice
Accuracy
0.6
Accuracy Accuracy
-Brier score -er--Brrrrrr score
0.25
0.75
HellaSwag ARC
-0.50
0.4
0.50
0.75
0.2
0.25
-0.25
Accuracy
0.6
0.75
-0.50
0.50
0.4
0.25
1021
1023
1021
1023
1021
1023
Pretraining compute
Pretraining compute
Pretraining compute
Figure 18: ARC and HellaSwag scores of models trained (o) before November 2023 and (0) ter. Middle: reformulating the test task as multiple-choice leads to emergence around 10 to 1023 FLOPs. Right: when using Brier score as the metric, we similarly observe sharp changes perfor- mance around 10 to 1023 FLOPs.
GSMBK
Task examples: 0
Task examples: 16k
Task examples: 64k
Task examples: 320k
0.8
Accuracy
Ce:
2.2e+22
Cei 9.1e+21
Cei 4.2e+21 21
Cei 1.3e+21
0.6
0.4
0.2
0.0
0.8
Accuracy
0.6
R = 0.515
R² = 0.673
R = 0.783
R² = 0.857
0.4
0.2
0.0
10
10
20
1024
10
20
102h
1024
10 20
20
20
10
2022
20
10
Pretraining compute (FLOPs)
Figure 19: Scaling GSM8K as models increasingly train on the test task. The point of emergence Ce arises at lower scales (top). Training on the test task yields cleaner log- linear scaling fits (bottom).
G RESULTS FOR INSTRUCTION-TUNED AND CHAT MODELS
We evaluated 36 instruct and chat models, Appendix B.I.I Our findings for base models presented in the main generalize remarkably well instruction-tunee and chat models, see Figure 20 Newer instruct/chat models substantially outperform older instruct/chat models. However, after finc-tuning all models on the same amount of task-specific data, performance between newer and older instruct/chat models cqualizes.
We find that the performance gap between newer and older instruct/chat models is smaller than the gap between newer and older base models, contrast the estimated effect sizes in Figure [ with Figure 20). This is perhaps be expected, as instruction-tuniin datasets usually include some amount of benchmark data.
posit that the gap between newer older instruct/chat models nonetheless large because early-to- mid 2023 instruction-tunee variants -e.g.. Vicuna (Chiang et al., 2023), Alpaca (Taori et al., (202). Llama Chat (Touvron et al, 2023b did not emphasize benchmark performance but rather human preference (e.g.. "win-rate") chat setting. For example, the Llama technical report (Touvron al., (203b) includes no benchmark evaluations for their chat models. This perspective has dramatically shifted the last year and half, and post- bost-training interventions now explicitly aim to improve benchmark performance (MetaAl 2024
Gemma et al. 2024 Lambert et al. 2024).
30

---

Published as a conference paper at ICLR 2025
Instruct and chat modelst
MMLU
GSM8K
0.8
0.7
Difference è= 0.063
Difference ô= 0.154
Accuracy
0.6
Regression R² = 0.950
Accuracy
0.6
Regression R² === 0.841
0.4
0.4
0.2
0.3
0.0
10
1023
1022
1023 Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
MMLU (adjusted)
GSMBK (adjusted)
0.8
0.7
Difference ê= 00.001
Difference 0.027
Accuracy
0.6
Regression R² 0.983
Accuracy
0.6
Regression R² 0.929
0.5
0.4
0.4
0.2
0.3
0.0
10
2023
10a2
1023
Pretraining compute (FLOPs)
Pretraining compute (FLOPs)
Models trained
Before November 2023
After November 2023
Figure 20: Reproducing Figure 1 for instruction-tune tuned and chat models.
31


Reviews:
[{"review_id": "CzrGnMmiPZ", "reviewer": "Reviewer_Zcyd", "summary": "The paper defines *training on the test task* as a potential problem for evaluating LLMs, which could be common practice and strictly speaking not data contamination. They show that doing so may allow a model to achieve better results on a benchmark while not having better abilities overall. They suggest that further fine-tuning all models can help with better comparison across models, and show that doing so restores clearer scaling trends.", "questions": "* Ultimately, benchmarks are used to understand how good a model is at a general skill, like mathematical reasoning for MMLU and GSM8K. This method proposes to 'fight fire with fire' by fine tuning all models on the test task. Would your method merely reliably measure how well models performs on a specific benchmark, rather than how well models are at the ability it's meant to measure?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* They test their method on on a large set of base models with different sizes. This also allows them to compare scaling on benchmarks.\n* The proposed method is promising for better evaluations/comparisons and light on compute, requiring only fine-tuning.", "weaknesses": "* They only do so for pre-trained base models. In practical settings, we often look at the benchmarks of fine-tuned chat models, or models further fine-tuned off base-models (e.g. LLama derivatives). It is not clear whether their findings would apply to already-finetuned models.\n* They classify 'old' models as before November 2023, which seems somewhat arbitrary. I understand that choosing Nov 2023 reveals some sort of improvement given the same compute, but this could be due to reasons other than training on the test task. More justification is needed on why the improvement gap is a good heuristic for whether a model was trained on the test task. Alternative analysis on whether much more models after Nov 2023 include instruction data in the pre-training set could similarly justify why Nov 2023 is a good cut-off point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X6ogTlvlAF", "reviewer": "Reviewer_xmL1", "summary": "The paper explores an interesting phenomenon: training on the test task, which differs from training on the test data. Training on test tasks implies that task-relevant data is included in the pretraining stage of a language model. The paper highlights how newer models benefit from training on the test task, but when fine-tuning older models on task-relevant data, this performance advantage disappears.", "questions": "- Do you the authors think LLMs should be part of the title as only LLMs are explored in the paper?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper explores an ever-increasingly important area of LLMs: the \"science of evaluations.\"\n- It explores the implications of task-specific training for language models, providing evidence that training with task-specific data is beneficial. They find that many models from November 2023 gain advantages from this type of training. Additionally, they show that task-specific training is more likely responsible for recent boosts in model performance when FLOPs are held constant.\n- The paper's implications (as stated in the discussion) suggest that training on task-specific data may be beneficial across models.\n- The experiments and the paper are logical and well thought through.", "weaknesses": "- How was November 2023 chosen as the cutoff? I wonder what would have happened if you had chosen November 2022. How would the results be affected?\n- It is unclear how much hyperparameter tuning might affect results when training on task-specific data. How much could hparam tuning affect the results? How was the initial sweep in Appendix A.2 chosen? Although a full sweep might be expensive, additional clarity on how these were chosen will alleviate my concern. \n- Lack of consideration of instruction-tuning models. How do you think instruction tuning models might impact this study as often the newer datasets are comprised of examples to explicitly increase benchmark performance [1]? \n- Only a couple of main evaluation datasets are explored in the paper. \n\n[1] GenQA: Generating Millions of Instructions from a Handful of Prompts", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "casvXAIbEx", "reviewer": "Reviewer_b3hy", "summary": "The paper argues that newer language models only appear to be better on metrics like MMLU than earlier, iso-compute language models because of the presence of data in distribution to MMLU in the pretraining set. The paper demonstrates this by showing that when these models are finetuned on MMLU/GSM8K before evaluation, the performance differences between different model families go away. The paper also shows that emergent behaviors are less common under this new evaluation protocol.", "questions": "I'd be interested to see (on open source models with available datasets, or by sampling unconditionally from base models) whether there is more MMLU-like data in model families which appear to do better on MMLU when using the naive evaluation method.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is very clearly presented and argues persuasively. The figures are easily understood at a glance.\n- The conclusion argued for is surprising and novel, and has broad implications for evaluation methodology of LLMs and emergent behaviors in LLMs.", "weaknesses": "- The p-values for signficance in figure 1 overestimate significance because of correlation due to models being from only a few model families.\n- The paper never actually directly checks the hypothesis of more in-distribution data being the cause of higher MMLU performance out of the box, despite arguing for it via implication from the finetuning result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tcafCpvnbl", "reviewer": "Reviewer_HVsB", "summary": "### Paper Summary\nThis paper asks if recent open-source models are in fact better than previous models: Does exposure to task data during pre-training explain the difference in model performance? (Yes). The authors find evidence that the difference on task performance can be explained by exposure to these tasks during the end of pre-training, not that more recent models are fundamentally better LLMs.\n\n### Review Summary\nThere could be some world where this paper is simply stating the known fact that most models adjust the type of training data towards the end of pre-training to include task-relevant data, and that that inclusion improves performance. However, I think this paper does a good job of showing that this inclusion in particular seems to be making a key/big difference in models' improved performance, and that the inclusion during pre-training itself (while its being mixed with other data) doesn't seem to have an outsized impact on the ability to answer task-relevant questions.", "questions": "*  I didn't fully follow the final paragraph in section 4.2. Would your results not give an answer to your posed question? (i.e., doesn't task-training explain the improvements, and fall outside the scaling laws?)\n* What eq do you fit in figure 8 bottom? Switching the equation used in the top row the one used in the bottom row didn't seem to be motivated in the text. Doesn't eq 1 also a capture a log-linear relationship? Spelling a bit more out here would help.\n    * Giving a hint in the caption of figure 8 that the data are the same in both rows might help\n    * Labelling the rows within the figure (not only the caption) might also help (like you do in figure 4).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors have a number of nice (creative/thoughtful) experiments showing how older models do equally well when given more task data, that changing the style of the question (not content) also closes the gap between models, and finally, that, when given more data, model families all seem to follow the same scaling law.\n\nThe figures are all great/clean/pretty/legible!\n\nThe writing is also all clear/careful!", "weaknesses": "* some could say that this is an obvious finding / direct outcome of how models recently have been trained. I don't hold this view. I find it interesting that there is not an additional/additive benefit found in recent models (from adding task related data in the end of pretraining), and interesting that the results are so stark. [i don't read your paper as saying that there is no gain/improvement in recent models, but rather, that performance doesn't extend beyond what fine-tuning directly upon the task in question would yield]\n* figure 3 is unclear. the added information/gain of figure 3 vs figure 1 is unclear, esp. when looking at figure 3 alone (without referring to the text). to start, if I understanding things correctly, I think the colors of the points/lines should not be the same as the other figures as the underlying data is different from the other figures. the shared colors (to me) is misleading.\n\n### nitpicks\n* line 244, \"These results provides...\" --> \"These results provide...\"\n* line 259, \"contrast Figure 3 and Figure 1 Quantitatively...\" --> \"contrast Figure 3 and Figure 1. Quantitatively...\"\n* consider turning \"What does MMLU test for?\" into an active statement / takeaway.\n* line 429 R^2 is formatted differently than line 430\n* line 37, end of second paragraph. I was tripped up by \"of some\" and \"of others\". I now believe it is \"of some models\" / \"of other models\". I would guess the elisions were made to shorten/clean the paragraph? When I first read it I thought it was a typo that meant \"to some people\" / \"to other people\" (like different people will interpret this phenomena differently.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)