Train-before-Test Harmonizes Language Model Rankings
Authors: Guanhua Zhang
Ricardo Dominguez-Olmedo
Moritz Hardt
Venue: ICLR 2026 Oral (2026)
Keywords: Evaluation
Large language model
Primary area: foundation or frontier models, including LLMs

Abstract: Existing language model benchmarks provide contradictory model rankings, even for benchmarks that aim to capture similar skills. This dilemma of conflicting rankings hampers model selection, clouds model comparisons, and adds confusion to a growing ecosystem of competing models. In this paper, we take a different perspective on model comparison: instead of relying on out-of-the-box performance via direct evaluation, we compare model potential by providing each model with identical benchmark-specific fine-tuning before evaluation. We call this approach train-before-test. Our primary contribution is a comprehensive empirical evaluation of model potential across 24 benchmarks and 61 models. First, we demonstrate that model potential rankings obtained through train-before-test exhibit remarkable consistency across all benchmarks. Whereas traditional rankings demonstrate little external validity under direct evaluation, they enjoy a significant degree of external validity when applying train-before-test: model potential rankings transfer gracefully from one benchmark to another. Second, train-before-test restores the connection between perplexity and downstream task performance, lost under direct evaluation. Remarkably, even pre-finetuning perplexity of a base model predicts post-finetuning downstream performance, suggesting that ranking consistency reflects inherent model potential rather than fine-tuning artifacts. Finally, train-before-test reduces the model-score matrix to essentially rank one, indicating that model potential is dominated by one latent factor, uncovered by train-before-test. While direct evaluation remains useful for assessing deployment-ready performance, train-before-test provides a complementary lens for understanding achievable performance of models after adaptation.

Published as a conference paper at ICLR 2026
TRAIN-BEFORE-TEST
HARMONIZES LANGUAGE MODEL RANKINGS
Guanhua Zhang, Ricardo Dominguez-Olmedo, Moritz Hardt Max Planck Institute for Intelligent Systems, Tübingen and Tübingen Al Center
ABSTRACT
Existing language model benchmarks provide contradictory model rankings, even for benchmarks that aim to capture similar skills. This dilemma of conflicting rankings hampers model selection, clouds model comparisons, and adds confusion to a growing ecosystem competing models. In this paper. we take a different perspective model comparison: instead of relying out-of-the-box perfor- mance via direct evaluation, we compare model potential by providing each model with identical benchmark-specifee fine-tuning before evaluation. We call this ap- proach train-before-test Our primary contribution is comprehensive empirical evaluation of model potential across 24 benchmarks and 61 models. First, we demonstrate that model potential rankings obtained through train-before-test ex- hibit remarkable consistency across all benchmarks. Whereas traditional rankings demonstrate little external validity under direct evaluation, they enjoy a significant degree of external validity when applying main-before-testt model potential rank- ings transfer gracefully from one benchmark to another. Second, wrai-before-ten restores the connection between perplexity and downstream task performance, lost under direct evaluation. Remarkably, even pre-finetuning perplexity of base model predicts post- nost-finetuning downstream performance, suggesting that rank- ing consistency reflects inherent model potential rather than fine-tuning artifacts. Finally, muin-before-tess reduces the model-scor, matrix to essentially rank one, indicating that model potential is dominated by one latent factor, uncovered by train- before-test. While direct evaluation remains useful for assessing deployment-ready performance, train-before-test provides a complementary lens for understanding achievable performance of models after adaptation
INTRODUCTION
Existing language model benchmarks provide contradictory model rankings, even for benchmarks that aim to capture similar skills (Liang et al., 2023] BBeehhin et al.) 2023) Fourrier et al. (0224). reliably compare, rank, and select models when different benchmarks yield conflicting information? While this ranking disagreement is often attributed the diverse capabilities of large language models (Ruan et al., (2024), it creates conundrum in practice that muddles model development decisions (Zhang & Hardt 2024))
This inconsistency poses serious challenge: how
Current evaluation methodology works from direct evaluation, probing models via black-box function calls. However, large language models are trained on diverse, often proprietary data mixes that vary significantly across models (Grattafiori al., 20224, Gemma al. 2024, Guha al. 2025). Recent work showed that this leads to the problem of training on the test task (Dominguez-Olmee Olmedo et al. y024): the extent to which a model has encountered data similar to the test task during training confounds model comparisons, rankings, and scaling laws (Kaplan et al.) (020). Put simply, an otherwise inferior model may have simply prepared better for specific task.
In this paper, we take fresh perspective on evaluation methodology: in contrast with direct evaluation, we compare model potential by giving each model the same task-specific fine-tuning. We call this approach train-before-tee Its goal is to achieve valid model comparisons by ensuring that all models receive equal preparation for the test.
"ode available at wtthttss:// htttpp:://githbb.coooooo attt:pp://gthhhb..c///aaaaalall ions/ m-harmony

---

Published as a conference paper at ICLR 2026
Direct evaluation
ARC-Challenge
NO-Open
Train-
ARC-Challange
NQ:Open
Figure 1: Rankings of 61 language models on two question-answert benchmarks: Natural Ques- tions Open and ARC Challenge. Top: Direct evaluation leads to inconsistent rankings. Although both benchmarks test for guestion-answering ability, the resulting model rankings show substantial disagreement. Bottom: Train-before-ter aligns model rankings. Note: For each of the two plots, we greedily align model rankings as much as possible without violating confidence intervals, thus revealing only those ranking changes that are statistically significant. See Appendix C.I
We envision trin-before-test as a tool for reliable model selection for downstream adaptations. Increasingly, practitioners select one from many available models with the goal of adapting for specific task. Under direct evaluation the best model to begin with may no longer be the best model after task-specific preparation. contrast, we show that suin-before-task yields model comparisons and rankings that enjoy broad external validity.
1.1 OUR CONTRIBUTIONS
Direct evaluation leads to ranking disagreement even between related tasks. We demonstrate that the prevalent direct evaluation scheme results in strong disagreement between model ranking across various benchmarks. We show that this strong ranking disagreement persists even when restricting to benchmarks that aim to capture similar tasks. Moreover, rankings still strongly disagree when evaluating models from the same family. The situation presents serious conundrum for model selection: Under direct evaluation, benchmarks fail to give reliable and actionable insights for model choosing among multiple alternatives.
Train- brein-before-tett leads to consistent model potential rankings. We comprehensively evaluate muin-before-ter across 24 benchmark datasets and 61 large language models. By fine-tuning each model on identical task-relevant data before evaluation, we uncover remarkably consistent model potential rankings. Ranking agreement between benchmarks, measured by Kendall's I's tau, improves 274 out of 276 benchmark pairs, with the average Kendall's т increasing from 0.52 to 0.76. Figure [ illustrates the result for one typical pair of benchmarks. This consistency suggests that model potential, unlike out-of-the-box performance, has external validity (Salaudeen et 2025 and transfers gracefully across different tasks.
Model potential aligns perplexity rankings with downstream tasks. Perplexity benchmarks used be popular, but fell out of favor public benchmarking and model comparison because of the apparent disconnect between perplexity and downstream task performance (Wei et al., 2022 Ganguli et al. 2022 Liu et al. 2023 Magnusson et al.) 2023 Lourie et al. 2025a). We indeed validate this disconnect when comparing model families under direct evaluation. However, train-before-test restores this fundamental relationship in two ways. First, we show that post-fine fine-tuning perplexity rankings align well with post-fine-tuie tet-f--ttttnngg downstream task rankings, creating consistency between training objectives and task performance. Second, and more remarkably, for base (non- nonninntruuction tune)) models, pre-fine-tuning perplexity predicts post-fine-tuning downstream performance.
2

---

Published as a conference paper at ICLR 2026
This suggests that the ranking consistency we observe reflects inherent model potential rather than artifacts of fine-tuning
Train-before-test sheds light on the latent factors of benchmark scores. Consider the large benchmark-model score matrix, where each entry (i.5) corresponds to the performance of model j on benchmark i. Several works have considered this matrix for different reasons and found that it is approximately low rank (Kuan et al. 2024] Owen 2024 Burnell et al.) (023), but not quite. The first singular value is dominant and correlates with pre-training compute. However, the other components aren't It negligible, and their interpretation remains unclear. We show that train-before-test clarifies this state of affairs. After train-before-te be-fore---et, the benchmark -model matrix is essentially rank one. The first principal component accounts for 86% of the explained variance across all models, and for 93% of the variance for a single model family. This suggests that model potential is dominated by a single latent factor, while the additional components observed in direct evaluation may reflect task-specific training exposure.
RELATED WORK
Benchmarking has played a central role in the advancement of machine learning (Liberman
2010
Hardt & Recht 2022). While absolute model performance is often fragile to even seemingly minor
changes in eval avaluation data (Candela et al., [009 Torralba & Efros 2011] AlBadawy et al. 2018
Taori et al. 2020; Tsipras et al. 2020), relative model performance- that is, model rankings tends
to transfer surprisingly well across classical benchmarks (Yadav & Bottou 2019. Recht et al. 2019
Miller et al. 2020). For instance, prior work (Kornblihh et al. 2019 Barbu et al. 2019) has shown
that model rankings ImageNet (Deng et al. 22009) also transfer to other image classification and
object recognition benchmarks. Moreover, Salaudeen & Hardt (2024) demonstrated that ImageNet rankings remain robust even under major dataset variations. This transferability of model rankings is highly desirable, as it indicates that progress on specific benchmarks reliably reflects broader scientific advancements (Liao et al. 202 Hardt 2025).
However, the emergence of foundation models has dramatically transformed the benchmarking landscape compared the ImageNet era (Liang et al. 2023 Srivastava et al. 2022
Weidinger et al.
2025). With huge training costs and much improved capabilities (Yang et al. 2025
Grattafiori et al.
2024; Ramesh et al. [222]] Gemini [023] OpenAl (023). practitioners now lean towards directly
evaluating LLMs across wide range of different benchmarks, in the hope of obtaining a more comprehensive assessment of their capabilities (Liang et al.) 2023 (Suzgun et al. 2022; Hendryccks) et al. 2020: |Beechngg et al.) 2023 Fourrier et al.] 2024). This shift introduces new challenges, as
model rankings across different tasks may vary significantly (Huan et al. 2025 Lourie et al., 2025b).
Zhang & Hardd (2024) draw an analogy between multi-task benchmarks and voling systems (Arrow
1951), revealing that a multi-task benchmarking approach with diverse rankings inherently lacks robustness to minor changes and thus cannot provide a stable unified ranking.
This lack of unified ranking is sometimes seen as a desirable feature within the community (Liang al. 2023). Some argue that variability reflects the multifaceted strengths and weaknesses of LLMS. suggesting that users should select the best model tailored to their specific needs (Gihosh et al.
2024 Zhang et al. 2024b;; (Shnitzer et al. 2023). For example, user who focuses on mathematical lasks could prioritize the math benchmark to choose the optimum model. However, there are two significant concerns regarding this approach: First, the user-driven selection strategy poses challenges for model developers. Given the resource- resource-intensive nature of LLM development (Guo et al., 2025). ), it ismpacctical release a different model for every potential use case. Moreover, developers typically aim to create a general-purpose purpose model (Yang et al. 2025 Grattafiori et al.) (024); however, such desideratum often difficult to reliably measure due to the inconsistent rankings observed across benchmarks. Second, we demonstrate in this paper that benchmarks within the same task category
can still exhibit substantial discrepancies in model rankings.
One potential reason for the observed inconsistencies in model rankings is that models vary substan- tially in their training data (Gadre et al., 2023 Albalak et al., 2025). In particular, Baminnuuz-Olmedd et al. (2024) show that models vary in their degree of preparedness for popular benchmarks, which confounds model evaluations. Inspired by this finding, we investigate a different question: if varying preparedness confounds evaluations, can equalizing preparedness harmonize contradictory model
3

---

Published as a conference paper at ICLR 2026
rankings across benchmarks? We therefore introduce the notion of vrii-beefore-ess wherein fine-tune each model on the corresponding training set so every model arrives well prepared. Through comprchensive evaluations across 24 benchmarks, demonstrate that train-before-test harmonizes otherwise contradictory model rankings, establishing it as a practical evaluation methodology.
While extensive literature exists investigating different fine-tuning strategies for LLMs Zhang et al. 2024a Zeng et al., [0255 Lester et al. [022], this lies outside the scope of our investigation. Instead, we apply standardized fine-tuning (Mangrulkar et al. (022) as evaluation tool to give all models equivalent preparation before testing. Another relevant area is the literature on scaling laws (Kaplan et al. 2020). Lin al. (2024) predict
performance of a single model from partial finetuning on one task using their rectified scaling law. Zhang et al. (2024a) study how different factors scale within individual tasks. These scaling law approaches model performance changes using model-specific and task-dependent parameters. In contrast, we investigate how standardized fine- tuning harmonizes model rankings across diverse benchmarks the wild, examining 61 models from six families across 24 tasks spanning multiple categories. Complementary to our focus, Heineman et al. 2025) analyzes the statistical reliability existing benchmarks, while Gu et al. (2024) proposes standardized evaluation protocol to reduce variability arising from formatting and scoring choices.
3 EXPERIMENTS
3.1 EXPERIMENT SETTING
Benchmark selection. We begin our pack- age (Gao et al., (023),, which offers a com- prehensive suite of language model bench- marks. To accommodate the -rforeeefore- test methodology which requires dedi- cated training set for fine-tuning, we first identify benchmarks that provide at least 1, 1,000 training examples. This yields a to- tal of 37 benchmarks, which we broadly categorize into 28 likelihood- based and 9 generation-based benchmarks.
Table I: We categorize benchmarks into language un-
study with the eon-eval-aaaee
derstanding (LU), commonsense reasoning (CR), ques- tion answering (QA), physic/biioooy/cceniity (PBC), math (Math), and medicine (Med).
Calegory
Benchmarks
LU
MNLL, ONLL, RTE, CoLe, 55T-2, MMPC, OUP, W.C, ANL. Minogrance, CommonanneeaA Bellaswag, Bocial-IgA
CR
QA
openBoskka, NQ-Open, Hoolg, ARC-Laayy Ahchhalleeeee
PBC
Sc10, PIQA
Math
MathQn,
Mod
Me-MCOA, HeadOA
Generation-based benchmarks are often computationally intensive evaluate, as base models typically generate text until reaching their maximum sequence length. These benchmarks are also over-challenging for smaller models with limited parameters, such as GPT-2 (Radford et al. 2019). Therefore, we select only NQ-Open and GSM8K from the generation-base benchmarks. Among the likelihood- based benchmarks, we further exclude six due to observed anomalies during fine-tuning, such as lack of performance improvement in over 20% of models. See Appendix A. for details.
Our final selection consists of 24 benchmarks covering diverse domains and task types. These benchmarks are primarily multiple-choice question answering benchmarks, with accuracy as the task metric. We categorize all benchmarks by their descriptions, see Table If a benchmark does not come with validation split, we randomly allocate 20% of the training data as the validation set. To save computational resources, we cap the number of training data at 50,000, validation data at 1,000, and testing data at 10,000.
Model selection. We consider 61 language models across six model families: LLAMA Grattationi et al. 2024), QWEN (Yang et al. 2025). GEMMA (Gemma et al. 2024), PYTIIAA (Biderman et al. 2023). GPT-2 Radlord al. 2019) and YI (Young al. 2024). Due to computational constraints, we select models with no more than 14B parameters. See Table 2 for the full list. We include both base and instruction-tur -uned models, and use the suffix -IT to denote instruction-tume -uned models.
Evaluation setup. We evaluate the 61 models across all 24 benchmarks using both direct evaluation library for evaluation. We evaluate
and train-before-ee before-test evaluation. We use the in-eval-harne models zero-shot (Brown et al. 22020). For direct evaluation, we simply evaluate the model as it is. For
main-before--est, fine-tune models for five epochs using learning rates in {le 5, 2e 5, 5e 5).
4

---

Published as a conference paper at ICLR 2026
Table 2: Models considered, categorized by model family.
Family
Model
Name Suflix
LLAMA-
3-8B, 3.1-8B, 3.2- 3.2-3B, 3-8
1.5-0.5B, 1.5-1 1,5-4B, 1.5-78, 1.5-148, 2-0.5B, 2-1.58, 2.5-,,25- 2.5-1.5H,
QWEN-
2.5-3B, 2.5-7B, 2.5-14.,, L.5-0.5BIIT, L.S- 1.5-4--1, 1. L.5-14--I..
2-
21--1-8-.1...5381111.......1 2.5-1BB-1
GEMMA-
2B. 7B..B...
2-2B. 2-98, 2B-IT. 2--1T
GPT2-
124M. 335M, 774M. 1.5B
PYTIIA-
70M. 160M, 410M, IB, 1.4B. 2.8,, 6.9B. 12B
6B, 9B, 6B-IT, 1.5-6B, 1.5-9B, 1.5-6B-IT, 1.5-9B-TT
Direct evaluation
train-beffree-est
Mean Kendall
0.8
0.6
0.4
NO-Open
NQ Op-Open
0.0
MRPC
WIC
QNLI
OPP
COLA
SeiQ
SST-2
GSMSK
MedMCQA
ANLI
RTE HeadQA
ARC-Eas Easy
MNLI
Winogr ande
Commonsenss se
PIOA
Social- Social-iQA IQA
ARC ARC-Cha Challenge Hella HellaSwart Swag MathQA OpenBookOA BoolQ
Figure 2: Mean ranking agreement between each benchmark and all others. We calculate Kendall's between each benchmark and every other benchmark, and then average the results. Compared to direct evaluation, train-bffore-test consistently improves ranking agreement. A detailed comparison of Kendall's r values for every benchmark pair is provided in Appendix [ On average, the overall average Kendall's is 0.52 for direct evaluation and 0.76 for train-before-test.
separately. The best performing checkpoint is then selected based on performance a separate validation set, yielding 61 x 24 1, 464 fine-tuned models in total. We use parameter-efficien fine-tuning (PEFT) (Hu et al. 2021 Mangrulkar et al. (022).. See more details in Appendix(A.2 Each fine-tuned model is then evaluated on the corresponding benchmark's 's test set. For each benchmark, rank models according their performance. We then measure the ranking correlation across pairs of benchmarks using Kendall's T (Kendall 1938).
3.2 DOWNSTREAM RANKING AGREEMENT
As depicted in Figure 2 direct evaluation shows only modest ranking agreement between the 24 benchmarks, with an average Kendall's of 0.52. This lack of agreement across benchmarks complicates model assessment and makes it challenging to aggregate results into a meaningful overall ranking (Zhang Hardt (2024). In contrast, the train-before-test methodology leads a substantial improvement in ranking agreement. Under this approach, 274 out of 276 benchmark pairs show higher Kendall's 5 scores, with the average rising from 0.52 0.76. This stronger consistency suggests that model potential ranking on one benchmark is likely to generalize to others, including practitioners' own cases, which simplifies model comparison and selection. Notably, benchmarks that appeared to be outliers under direct evaluation, such as NQ- and MRPC, demonstrate much greater ranking consistency under main-before-test.. NQ-Open and all other benchmarks improves from 0.23 to 0.74.
example, the average Kendall's r between
We further split all benchmarks into six categories (e.g., language understanding, math), see Table For each category pair, report in Figure 3 the intra-category average ranking correlations and inter-category average ranking correlations across all relevant benchmark pairs. Consistent with our previous findings, observe reasonably poor ranking agreements across categories under direct evaluation. While one might expect high intra-category agreement- after all, tasks within the same category tend to be relatively similar- direct evaluation results in low intra-category agreement in
5

---

Published as a conference paper at ICLR 2026
EU
0.54
0.51
0.37
0.33
0.52
0.41
1.0
0.75
0.75
0.76
0.71
0.72
0.72
1.0
CR
0:51
0.68
0.63
0.64
0.64
0.65
CR
0.75
0.76
0.81
0.78
0.72
0.76
0.8
QA
0:37
0.63
0.61
0.66
0.55
0.64
0.6
Kondail
QA
0.76
0.81
0.83
0.81
0.73
0.81
0.6
FBC
0.33
0.64
0.06
0.66
0.54
0.67
PBC
0.71
0.78
0.81
0.77
0.66
0.76
0.4
Math
0.52
0.55
0.54
0.55
0.59
Math
0.72
0.72
0.73
0.84
0.77
0.2
Med
0:41
0.65
0.64
0.67
0.59
0.62
Med
0.72
0.76
0.81
0.76
0.77
0.8
0.0
0.0
13
CR:
0A
PBC
Matth
Med
1
QA
PBC
Math
Med
(a) Direct evaluation.
(b) Train-before-test.
Figure 3: Cross-category ranking agreement direct evaluation (left) and train-before-test (right). We categorize benchmarks into language understanding (LU), commonsense reasoning (CR), question answering (QA), plysicshiology/chhciissr (PBC), math (Math), and medicine (Med), see Table [ Kendall's is averaged across all pairs of benchmarks that belong two given categories. The diagonal entries represent intra-category agreement and the others represent inter-category agreement. best-beefre-test improves both intra- and inter-category ranking agreement in all instances.
many cases. For example, the intra-category mean Kendall's J is 0.54 for language understanding and 0.55 for math. This further underscores the difficulty of selecting models based direct evaluation. Even if the goal is to choose model that excels within specific domain, the low intra-category agreement makes this decision challenging.
In contrast, train-before before-teste boosts both intra- and cnter-category consistency. For example. the intra-category mean Kendall's for language understanding raises from 0.52 to 0.75, as well as from 0.55 to 0.84 for the math category. Moreover, agreement between categories is often nearly high as agreement within categories. This suggests that models with higher potential in one domain tend to also perform well across other domains after adaptation.
3.3 PERPLEXITY AGREEMENT
We now study the agreement between downstream benchmark rankings and perplexity rankings general domain corpora. To do so, we collect three corpora from Wikipedia, StackExchange, and arXiv, retaining only contents from 2025. Because all models used were released before 2025, they could not have seen these texts during pretraining. Specifically, we collect 3,366 documents for Wiki, 6,001 for St StackExeae Exchannge and 44,384 for arXiv. These datasets are split into training, validation, and testing sets, in an 8:1:1 ratio.
We measure perplexity in bits per byte, using the 1m-eval Seerallhhane -harness library. We then compute models rankings based on the perplexity evaluations, and compare the rankings with those of the downstream benchmarks considered earlier sections. We exclude the four GEMMA models from these results, as Im-eval harness provides unreliable perplexity measurements for GEMMA models due its rolling window implementation. See Appendix B.2 for details.
The results are presented in Figure In contrast to downstream tasks, perplexity rankings demon- strate strong agreement both under direct evaluation and rrainbbefore-tes Specifically, the average Kendall's 7 between the perplexity rankings is 0.76 for direct evaluation and 0.78 for train-before-tee We hypothesize that this reasonably strong agreement arises due to the smooth relationship between perplexity evaluations [Brandfombrener et al. 2024 Mayilvahanan et al. 2025).
When comparing ranking agreement between perplexity evaluations and downstream benchmarks, we find that agreement is low under direct evaluation, with mean Kendall's 7 of 0.48. This lack of agreement is concerning, as it signals a disconnect between the language modelling pre-training objective and downstream benchmark performance. Fortunately, methodology improves ranking agreement substantially, with the mean Kendall's 7 ranking correlation between perplexity rankings and benchmark rankings rising 0.74. This finding is reassuring:
find that trainbbffore-te

---

Published as a conference paper at ICLR 2026
Direct evaluation.
Kendall
Wil
0.76
0.75
0.35
0.26
0:50
0.22
0.46
0.35
0.75
0.58
0.52
0.58
0.4
0.78
0.59
0:45
0.54
0.77
0.32
0.2
0.6
0.11
0.72
0:45
0.86 0.66
10
Stack
0.78
1
0.78
0.42
0.38
0.6
0.29
0.42
0.34
0.61
0.52
0.55
0.57
0:57
0.58
0.6
0.5
0.51
0.29
0.31
0.25
0.6
0.2
88
0.49
0.68
0.61
0.5
Abdy
0.78
0.41
0.58
0.3
0:38
0:50
0:58
0.5
0:34
0.32
0.59
0.26
0.50
0.58
0.0
Wili
Stack
Andr
MNIL
oree
QMUJ NQ Open
SST-2
Wineppando
Hellawaa
Soulak-Ce ADA
ANL
Commenssmannt Beai? CoLA
GSMIK
OpenBookC
Wic
MAsic HeadEzA
RTE ARC Easy ARC. Challingo
Train-before-ee test.
Kendall
Wili
0.79
0.75
0.76
0.74 0.75
0.74
0.85
0.67
0.20
0.89
0.79
0.71 0.73 0.28 0.78 0.67 0.81
0.72
0.66
0.71 0.81 0.00
0.87 0.08 0.84 3.84
1.0
0.8
0.79
0.57
0.8
0.75
0.75
0.79
0.7
0.00
0.85 0.83
0.8
Apdly
0.8
0.67
0.55
0.78
0.72 0.74 0.09 0.75
0.7
0.67
0.81
0.00
0.77
0.76 0.75
wiki
Stack
And
MAJI
oge MesMMcaA
No Copan
Sat-a
Winepaadde HellaSwag
Sosaligh
Matia
ANIL
PRAA
Commersanogg
sel
Bod
CoLA
GSBASK
OpeniaacKAA
WCC
NRPCC
HeadQA
ARC ARC-Eady Ecc-Eayy
Rte
ARC Cha-aaan
Figure 4: Ranking agreement between perplexity rankings and downstream benchmark rankings under direct evaluation (top) and train-before- (bottom). Perplexity rankings are consistent with each other under both evaluation schemes, with an average Kendall's r of 0.76 and 0.78, respectively. However, for direct evaluation, agreement between perplexity rankings and downstream rankings is low, with an average Kendall's r of just 0.48. Fortunately, train-before-tes. results in higher agreement between perplexity and downstream, increasing average Kendall's 0.74.
Base madel ranking agreement,
Kendall
WM
0.88
0.74
0.61 0.89 0.74
0.86
0.67
0.85
0.78
0.85
0.69
0.87
0.85
100
Stuck
0.83 0.75 0.87 0.72
0.79
0.67
0.84 0.ea 0.75 0.95
0.77
0.77
0.74
0.84
0.82
0.79
0.85
0.87
0.74
0.9
0.85
Apdi
0.76
0.75 0.72 0.87
0.71
0.77
0.78 (aea 0.78 0.76 0.71 0.74 (87 0.78
0.71
0.68
0.72 08
0.71
0.82
08
0.77
0.0
MNLI
QQP
MedMCQA
CHIJ HQ-Open
SST-2 Winegrtende HallaSwap
Socul-ic
ADA Maroa
And
piah
Commonoonsaah
sda
Book
CoLA GSMMK
OpenBleROAA
MRPC
HeadQR
RTE ARC-ES ARC Chalkenge
instruction-uueed model ranking agreement.
Kendall
Wild
0.57
0.
0.7
0.57
0.58
0:45
0.7
0.62
0.52
0.5
0.22
0.62
0.5
0:44
0.53 0.61
0.59
0.46
0.58
0.41
0.69
0.54
0.08
0.63
1.0
Stack
0:39
0:36
0.56
0.38
0.38
0.26
46
029
0.6
0:36
0.6
033
oar oar 0:45 0.49
0.34
0:37
0.44
051
0.43
as
Apdiy
6.52
0.6
0:47
0.34
0.62
0:45
0.46
0.57
0.65
0.
0.55
16
0.66
MNLJ
QCP
MedACOA
No Op-n
SST-2
Winegrande Halla
Socill HQA
MattQ
ANd
PIOA
Commarceranta G
sdg
Boelq
CoLA GSMMK
OpenBoook.
Wic
MRPC
HeadgA
ATE ARC Easyy
ARC Chateenge
0.0
Figure 5: Ranking agreement between perplexity rankings before fine-tuning (direct evaluation)
and downstream benchmark rankings after fine-tuning (train- seaii--efore-testt for base models (top) and instruction-tunae models (bottom). Unlike Figure 4 where both rankings in each comparison use the same evaluation scheme, here we test whether pre-fine-tuning perplexity can predict post- fine-tuning downstream performance. Base models show strong correlation (average Kendall's T = 0.78), suggesting perplexity a good predictor of model potential. This indicates that the ranking consistency we observe reflects inherent model potential rather than artifacts introduced by fine-tuning. Instruction-tunet models show much weaker correlation (average Kendall's 7 = 0.51).
light amount of fine-tuning on task data is sufficient to align the language modeling training objective with downstream performance. Moreover, we find that ranking agreement between perplexity and downstream evaluations is roughly similar to agreement across downstream evaluations. This suggests that, despite perplexity typically not being used for benchmarking purposes, it can be as effective a ranking metric as benchmark evaluations.
Drawing inspiration from prior work (Liu et al. 2023 Xia et al., 2023 Gadre et al. 2024 Du et al.
2024 Zhang et al. (024a), we further examine the correlation between model rankings according to average perplexity across the three text corpora and average downstream performance across the 24 benchmarks. [aadre et al. (2024) show that, when models are trained on the same pretraining data, perplexity is well-correlated with aggregate benchmark performance. Our setup differs in that we consider a diverse set of model families, each trained on different pretraining data. Under direct evaluation, we find that the ranking correlation is modest, with a Kendall's 's of only 0.55. We hypothesize that this relatively weak agreement is due to differences in pretraining data and instruction tuning, resulting in varying levels of exposure to benchmark tasks during training (Dominguez)
7

---

Published as a conference paper at ICLR 2026
Direct evaluation.
Train-before-test.
0.9
86%
Variance Explained Ratio
0.8
0.7
70%
Arannce Explained
0.6
0.6
0.4
0.4
0.3
0.3
0.2
0.2
13%
0.1
466
3%
2%
2%
1.6
19
0.0
PC1
PC2
PC3
PC4
PCS
PC1
PC2
PC3
PC1
PC5
Principal Components
Principal Components Ratio
Figure 6: Explained variance ratios of the top five principal components of the benchmark score matrix, under direct evaluation (left) and crain-before-test (right). train-before-tes! substantially increases the amount of variance explained by the first principal component, from 70% 86%. This indicates the model potential is dominated by one single latent factor.
Olmedo et al. 2024). Fortunately, when applying train-before-ter methodology, the ranking correlation between average perplexity and average downstream performance improves substantially. with Kendall's increasing from 0.55 0.84.
We additionally examine the agreement between perplexity prior to fine-tuning and downstream task performance after fine-tuning. That is. between direct evaluation perplexity rankings and rriinbeffore- test downstream performance rankings. We plot such ranking agreement in Figure dividing models into base models and instruction-tus strong indicator of model potential on downstream tasks, with an average Kendall's т of 0.78. This indicates that, for base models, direct evaluation of perplexity is already reasonably reliable metric for ranking models. Moreover, it indicates that the ranking consistency we observe reflects inherent model potential rather than artifacts introduced by fine-tuning.
models. For base models, perplexity prior to fine-tuning is
However, the same does not hold for instruction-tunee models (average Kendall's = 0.51). Instruction-tuning renders perplexity rankings unreliable, as ranking agreement low across the board. This is to be expected: instruction fine-tuning tends increase both benchmark performance (t) and perplexity (L) on general text corpora, thus clouding the relationship between perplexity and downstream evaluations. Fortunately, shown earlier, main-before--est restores high ranking agreement between perplexity evaluations and downstream performance.
3.4 LOW-RANKED MODEL SCORE MATRIX
So far, have shown that comparing model potential using the train-before befi-bbefore-tet yields consistent rankings across benchmarks. We now examine the implications of this finding by analyzing the resulting matrix of model scores, where each entry (i.j) corresponds the performance of model j on benchmark i. We use Principal Component Analysis (PCA) to examine the structure of the model score matrix.
Pytlic
Owon
5.0
2.5
score
-5.0
Figure O shows the explained variance ratios of the first five principal components. These results support previ- ous findings that the score matrix is of low rank (Ruan et al.) 2024). Under direct evaluation, the first five components account for 91 of the total variance. A similar trend is observed for train- wein-before-tes scores, where the first five components explain 97% of the vari- ance. Notably, under mein-before-test the first principal
7.5
-10.0
1023
10at
1022
1023
10a4
Pre-training compute (FLOPs)
Figure 7: PC1 scores under train- hain-hefore-test align with the pre-training compute.
8

---

Published as a conference paper at ICLR 2026
Direct evaluation.
Train-before-test
9.3%
0.9
Variance Explained Ratio
0.8
74%6
Vanance Explained Ratio
0.7
0.6
0.6
0.4
0.4
0.3
0.3
0.2
0.2
0.1
996
3%
3%
1%
1%
1%
0.0
PC1
PC2
PC3
PC4
PC5
PC1
PC2
PC3
PC1
PC5
Principal Components
Principal Components
Figure 8: Explained variance ratios of the top five principal components of the QWEN matrix. For
the explained variance ratio of PCI increases to 93%. Controlling the model family to QWEN has made the score matrix essentially rank
component (PC1) captures a much larger share of the variance: 86%, compared to 70% for direct evaluation.
Prior works interpret PCI scores under direct evaluation as an indication of general capability, with later principal components denoting Somain-specifee capabilities not captured by PCI (c.g., reasoning ability, coding ability) (Ruan et al. 2024, Burnell et al. 2023). Unlike out-of-the-bor performance, which is controlled by multiple factors (Ruan et al. 2004, Burnell et al.) (023). model potential is dominated by one single principal axis. This dramatic change indicates that train-before-test removes confounding factors, such differential exposure to benchmark-relan that create artificial diversity rankings. It is of no surprise that PC1 also positively correlates with pre-traine training compute, shown in Figure
data during pretraining,
i| which have been identified crucial to model performances (Kaplan et al. 2020 Hoffmann et al. 2022). See detailed PC1 scores in Figure 11
Case study for Qwen models. We repeat our PCA analysis on the score matrix containing only QWEN models, depicted in Figure 8 (see other models in Appendix
Remarkably, we find that PCI for train-before-tes explains 93% of the variance, roughly as much as the variance explained by the top five principal components under direct evaluation. That is, whereas for direct evaluation the score matrix is low-rank; trainbbefore-ter renders the score matrix essentially rank one.
DISCUSSION, LIMITATIONS, AND CONCLUSION
Train-before-test fundamentally reframes how we interpret model evaluation. Whereas direct evalua- tion yields benchmark sppecffcc rankings that often contradict one another, rain-before-test harmonizes rankings across a wide array of tasks and datasets. This shift from measuring out-of-the-boo perfor- mance to comparing achievable potential equips the community with more stable and externally valid evaluation methodology.
This emphasis on model potential is particularly valuable for scenarios involving model development and adaptation. Practitioners frequently need to make decisions during model development-selleee checkpoints mid-pre-training or choosing base model for further instruction tuning or domain- specific adaptation. In these scenarios, direct evaluation, while useful for assessing deployment readiness, is of limited relevance and utility. A model that performs poorly on direct evaluation might excel when adapted new tasks. meii--before-tss.. by contrast, shows that rankings any task will also generalize to others, offering more promising guidance for model selection.
One might argue that ranking consistency is unnecessary if to given downstream application. However, our findings highlight three challenges with that view. First, even benchmarks that purport measure the same skill (e.g. question answering) produce
simply choose benchmarks close
We only plot models whose number of training tokens is publicly available. See Table for details.

---

Published as a conference paper at ICLR 2026
contradictory rankings under direct evaluation. Second, benchmark perfectly captures the specifies of an application, making some degree of generalization unavoidable. Third, in real deployments, models are often adapted to varying degrees, making the potential the relevant signal for comparison.
Limitations. Train-before-test requires fine-tuning models on task-specific data before evaluation, which certainly increases the evaluation cost. However, this investment yields dividends through improved reliability. Our findings suggest that fewer benchmarks suffice under train-before-test, as rankings from one benchmark reliably transfer others. This reduction in required evaluations can offset the per-benchmark cost increase. Second, despite significant improvements in cross- boos-beecchark ranking consistency, the ranking agreements remain imperfect. The residual imperfect correlation may arise from incomplete adaptation with PEFT (Mangrulkar et al. (022) or irreducible measurement noise (Fisher & Sen, 1994, Heineman al. 2025). Third, unfortunately, many benchmarks no longer come with training data, making it more difficult to apply train-before-test In light of our findings, we recommend that future benchmarks provide fine-tuning data for the benchmark. A fourth limitation that some commercial model providers do not easily allow fine-tuning of their models. We contend that in this case the problem is with the model provider. There clearly scientific value creating an ecosystem of models that can be fine-tuned. Train- train-before-test evaluation creates additional incentives for making models easy to fine-tune.
Overall, train- before-test complements existing evaluation practices by distinguishing between performance and potential. Importantly, potential comparison is not intended to replace direct evaluation-bone serve distinct purposes. Direct evaluation remains useful for understanding immediate deployment readiness, while potential comparison provides insights into adaptability and development prospects. Together, they offer a more complete picture of model capabilities. We believe that adopting train-before-test as a standard alongside direct evaluation can significantly improve the reliability, interpretability, and practical utility of the model evaluation ecosystem.
Conclusion.
ACKNOWLEDGEMENT
We thank Yatong Chen, Florian Donnrr, Mina Remeli and Jiduan Wu for helpful discussions and feedback on draft versions of this work.
REFERENCES
Ehab A AlBadawy, Ashirbani Saha, and Maciej A Mazurowski. Deep learning for segmentation of brain tumors: Impact of cross- insstiintitt training and testing. Medical physics, 15(3):1150-11) 2018.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. Transactions on Machine Learning Research, 2025.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Concel-Kedziorskt Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms, 2019,
URL http::ttt///att
org/ abs/ 13319
Kenneth J. Arrow. Social Choice and Individual Values. Wiley, 1951. URL https: //api
Semanticscholi wttp:/oooo//DDDi1eeee.... 1449105113
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open I1m leaderboard. https co/ h/t.gg/iiifcccccccoo/ssseeeeggg/ http:///sppppc.eocggggggaaaa///a
leaderboard 2023.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The
fifth PASCAL recognizing textual entailment challenge. In Text Analysis Conference (TAC), 2009.
10

---

Published as a conference paper at ICLR 2026
Stella Biderman, Hailey Schoelkopf, Quentin G. Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. ArXiv, abs/2304.01 013733,, 2023. URL https
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URL
11 1
David Brandfonbrener, Nikhil Anand, Nikhil Vyas, Eran Malach, and Sham Kakade. Loss-to-loss prediction: Scaling laws for all datasets. arXiv preprint arXiv:2411.12925 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HeeberrtVooss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020. URL https: semant icscho ar. wwwg/oooropaoooooooo 211::1 8971783
Ryan Burnell, Hank Hao, Andrew R. A. Conway, and José Herúández Orallo. Revealing the structure of language model capabilities. ArXiv, abs/2306.10062, 2023. BtRpttts://app
J Quiñonero Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. The MIT Press, 1:5, 2009.
Christopher Clark, Kenton Lee, Ming-Wa Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL
httpss//arxxv..orrrab////// 10044
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.
/5ff:::8000055500 2018.
Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/21 10. 14168, 2021. URL https
do Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pp. 177-190. Springer, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing, 2005.
Ricardo Dominguez-Olmedo, Florian E Dorner, and Moritz Hardt. Training on the test task confounds evaluation and emergence. arXiv preprint arXiv:2407.0788 07890, 2024.
Zhengxiao Du, Aohan Zeng. Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Nicholas I. Fisher and Pranab Kumar Sen. Probability incqualities for sums of bounded random vari- ables. 1994. URL http::tttp/.o //t:://app api..... semant icchol ar www..a.oooooooooo.o....... 123205318

---

Published as a conference paper at ICLR 2026
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open Ilm leaderboard v2. httpp::/ Stattts:///uggingaaaeee huggingface..eccccooo..o....oace co/ http:/paaeecppplee im- 1mm-leaderbarrd/ open 11m Leaderboard 2024.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Muss- mann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander J. Ratner, Shuran Song. Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alexan- dros G. Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. ArXiv, abs/2304. 14108, 2023. URL http:::://ap...mmatitieceice
wwwpiio....rrroooooooooooo wttp//or/////oooo 1D:35552822
Samir Yitzhak Gadre, Georgios Smymis, Vaishaal Shankar, Suchin Gururangan, Mitchell Worts- man, Rulin Shao, Jean-Pierre Mercat, Alex Fang, Jeffrey Li, Sedrick Scott Keh, Rui Xin. Mar- ianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilhareo, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on downstream tasks. ArXiv, abs/2403.00 00.5400, 2024. URL httts::t/aaaat wtwpppppssaiisiiieoeeomom. www.seetiiissooom 268379614
lar wwwphhholarrrro//ooooooooooo
Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, al. Predictability and surprise in large generative models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1747-1764, 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac' h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. httpttsss:// //eeodo/ o org/ rttp///rooord/12555555
Gemini. Gemini: A family of highly capable multimodal models. arXiv, 2023.
Gemma, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma Improving open language models at practical size. arXiv preprint arXiv:2408.00 00118, 2024.
Adhiraj Ghosh, Sebastian Dziadzio, Ameya Prabhu, Vishaal Udandarao, Samuel Albanie, and Matthias Bethge. Onebench to test them all: Sample-level benchmarking over open-ended capabilities. arXiv preprint arXiv:2412.06775 2024.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop textual entailment and paraphrasing, pp. 1 1-9. Association for Computational Linguistics, 2007.
Aaron Grattafiori, Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407.22 17783 2024.
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hanna Hajishirzi. Olmes: A standard for language model evaluations. ArXiv, abs/2406.08444. 2024. URL https
/
wwwpcchoooaa...ooooooooupoe..uee 277399115
Etash Kumar Guha, Ryan Marten, Sedrick Scott Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean-Pierre Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Ben Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng- Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Fales Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran
12

---

Published as a conference paper at ICLR 2026
Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models. ArXiv, 0ass/2506004178, 2025. URL htttt:://ppppi wtw.aappissemannscchhooooo
htt//orppusD:2279155
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in llms reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Moritz Hardt. The emerging science of machine learning benchmarks. Online at https:// milenchmarks.com 2025. Manuscript.
Moritz Hardt and Benjamin Recht. Patterns, predictions, and actions: Foundations of machine learning. Princeton University Press, 2022.
David Heineman, Valentin Hofmann, Ian Magnusson, Yuling Gu, Noah A. Smith, Hanna Hajishirzi, Kyle Lo, and Jesse Dodge. Signal and noise: A framework for reducing uncertainty language model evaluation. ArXiv, /ttp/250..131445 2025. WtRLhttps: semanticschooo
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/a009.030000000 2020. URLpttps::/app. semant icscholar :ttp://oooous::::11100000
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. Training compute-optimal large language models. ArXiv, abs/2203. 15556, 2022. URL https //api. www.steaatttsae iommmatissccholaa
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. ArXiv, abs/2106.09685, 2021. URL
lattps://ant /app..p.../ www.stteetnnni.o.......
org/ Cttp//rrppuuicccceee 22::::23550000
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Pooven- dran, Graham Neubig, and Xiang Yue. Does math reasoning improve general Ilm capabilities? understanding transferability of Ilm reasoning. arXiv preprint 1xXi:22577.0432,, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar0i 000361, 2020.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
Maurice G Kendall. The treatment of ties in ranking problems. Biometrika, 33(3):239-25)) 1945.
Simon Kornblith, Jonathon Shiens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp.
2661-2671, 2019.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466 2019. doi: 10.1162/tacl a 00276. URL https: htap:s://acclatten wwwphhoooyy..ooo htg.lllooyyyoo//11111//00/
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing, 2021. URL https://apl. www.nteiesssteoo. ar.
Hector Levesque, Ernest Davis, and Leora Morgenstem. The Winograd schema challenge. AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, pp. 47, 2011.
13

---

Published as a conference paper at ICLR 2026
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian
Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Annals of the New York Academy of Sciences, 1525:140 146, 2023. URL https: //api sema nt www.cccchllarr...o desemantcccchoar.....
Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. Are we learning yet? a meta review of evaluation failures across machine learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Mark Liberman. Obituary: Fred jelinek.. Computational Linguistics, 36(4):595 -9995,59999999 2010.
Haowei Lin, Baizhou Huang, Haotian Ye, Qinyu Chen, Zihao Wang, Sujian Li, Jianzhu Ma, Xiaojun Wan, James Zou, and Yitao Liang. Selecting large language model fine-tune via rectified scaling law. ArXiv, abs/2402.02314, 2024. httthhtps://// www.nt.eaannn..m. www.htiscchc........ org/
CorpusID:26741171.8
Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. In International Conference on Machine Learning, pp. 22188-2221 22214. PMLR, 2023.
Nicholas Lourie, Michael Y. Hu, and Kyunghyun Cho. Scaling laws are unreliable for down- stream tasks: A reality check. ArXiv, abs/2507.00885, 2025a. URL https://api
semanticscholaxs http://ccccoaaa.oooooouousseseee ott///orpss11:1220111555
Nicholas Lourie, Michael Y Hu, and Kyunghyun Cho. Scaling laws are unreliable for downstream tasks: A reality check. arXiv preprint arXiv:2507.00883 2025b.
Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, A. Jha, Oyvind Tafjord, Dustin Schwenk, Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groeneveld, Iz Beltagy, Hanna Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit. ArXiv, abs/2312. 10523, 2023. URL hhttp:://aapt ott.pitteee/pppppooeoiiocoooooon semanticschol..rr...............
CorpusII::260 220ppsiD::66548885
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. htttps://qithu> com haam/huugiingffcc/pttt 2022.
Prasanna Mayilvahanan, Thaddäus Wiedemer, Sayak Mallick, Matthias Bethge, and Wieland Brendel. Lims on the line: Data determines loss-to-loss scaling laws. In Forty-second International Conference on Machine Learning, 2025.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2381-2391, Brussels, Belgium, URL https: 2018, Association for Computational Linguistics. doi: 1/L1853//1/D18111 //cclannhoooog..orr/1113
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect natural distribution shift on question answering models. In International conference machine learning, pp. 6905- 6916. PMLR, 2020.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding, 2020. URL httppss///arxx..orrrg abs/1 14599
OpenAl. Gpt-4 technical report. arXiv, 2023. URL http://aaxxv...rrra///300.0000.
David Owen. How predictable language model benchmark performanccee ArXiv, 4rxth,abb/2401.047757 2024. URL / http://aapp..aa.aa..amm/
semant. cscholar. caacchllaa..otgooooooooslo:::::.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale
multi-subjed multi- dataset for medical domain question answering, 2022. URL (thhttps
//arxiv.. wtw./bbb///aa. 14371
14

---

Published as a conference paper at ICLR 2026
Mohammad Taher Pilehvar and Jose Camacho-Collades Wie: the word-in- dataset for evaluating context-sensitive meaning representations. arXiv preprint arXiv: 000211 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. In OpenAl Technical Report, 2019. URL https:
//api. semant icscholar.
1100:553033333 OpenAl technical report.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions Association for
for machine comprehension of text. In Proceedings of EMNLP, pp- Computational Linguistics, 2016.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, 2021, URL
and Ila Sutskever. Zero-shot text-to-image generation. ArXiv, abs/2102. ID:000::332333 https: /tapss:/app.. http://....ssm...ii...emmmmm
semant www.entisschoo.........o ar wwtp:oog//oooo. http://...aatta.......o....oooaa
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers
generalize to imagenet?, 2019. URL http::: / wwwpxiiiorrg/aa/.. h/bgg//b///// 902 10811
Yangjun Ruan, Chris J. Maddison, and Tatsunori B. Hashimoto. Observational scaling laws and the predictability of language model performance. ArXiv, abs/2405. 10938, 2024. httptt
//api. semant cacho warboorrr.... wwwp//orpgoo...oe 669999995
Olawale Salaudeen and Moritz Hardt. Imagenot: A contrast with imagenet preserves model rankings. arXiv preprint arXiv:2404.02112 2024.
Olawale Salaudeen, Anka Reuel, Ahmed M. Ahmed, Suhana Bedi, Zachary Robertson, Sudharsan Sundar, Ben Domingue, Angelina Wang, and Oluwasanmi Koyejo. Measurement meaning: A validity- framework for ai evaluation. ArXiv, abs/2505, 2025.
URL https
//api. segpi..setattcsccho.sss. wttp:oorr///oooooeooooooe
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions, 2019. URL attps:
www.wwivvvooo..... http::/ww.w....///.....ooo htsp//bss1199 00.944.97288 09728
Tal Shnitzer, Anthony Ou, M'irian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets. ArXiv, abs/2300.115889 2023. htthttss:: www..iiccosaaiin.a.aaa.. wtt.oaarr.ro/ooooouDD....
263151991
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pp. 1631-1642 2013.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alona et al. Be- yond the imitation game: Quantifying and extrapolating the capabilities of language models. ArXiv, abs/2206.04615, 2022. URL https: hcce:::://pp.eeeeatiiccchhooo wwt.:iccccccc....oo.ooooooo 263625818
Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. In Annual Meeting of the Association for Com- putational Linguisties, 2022. URL https:://// / http://app..ssaaniii.acaa wew.nticscholiee.. wwwpollaaa.rr/ooopusoss.
252917648
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019. URL httttss://axx.v.ogg/
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt, Measuring robustness natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583-18599, 2020.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 1521-1528. IEEE, 2011.
15

---

Published as a conference paper at ICLR 2026
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. From imagenet to image classification: Contextualizing progress on benchmarks. In International Conference Machine Learning, 2020. URL https:// seapsssmmaniccchhlaaaee org/
Corpus I:: 21 6282858
David Vilares and Carlos Gómez-Rodriguez HEAD-QA QA: A healthcare dataset for complex reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 960-966, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19- 1092. URL https: www ac web. http/onthooooggeg ogy/
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint 1805. 12471, 2018. 2018.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
Laura Weidinger, Deborah Raji, Hanna Wallach, Margaret Mitchell, Angelina Wang, Olawale Salaudeen, Rishi Bommasani, Sayash Kapoor, Deep Ganguli, Sanmi Koyejo, and William Isaac. Toward an evaluation science for generative systems. ArXiv, abs/2503.0533 2025. URL lattps://apl www.seeettnnniiieeeeeee oww.:ar.ooooooorer ID 22: 276884497
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions, 2017. URL htttt:///aaattt /ttp://:///xvvvoo//a/a./oo www.iiii.go...ooooooooooooo abaaab//700 /wtp//177..0600
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HLT, 2018.
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Veselin Stoyanov. Training trajectories of language models across scales. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
Chhavi Yadav and Léon Bottou. Cold case: The lost mnist digits. In Neural Information Processing Systems, 2019. URL frttps:/a/ semant cschola ar. wwr..rroroooooooooo...oe.ooooere CCrrpssrrsss ID: 166227957
An Yang. Baosong Yang, Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: /wtpvvaivvvvvv////iv//aa.aa..av. o abs// 21t..2112..0 2111..111155
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang. Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai. ArXiv, abs/2400..0465, 2024. URL https: wrwp/orrg///ooooooo.ooooo 222::::26265555
//api. www.sssematiccccccccooomm
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL httttttts:/:aartto org/ abs 19011005888001
Xinyue Zeng, Haohui Wang, Junhong Lin, Jun Wu, Tyler Cody, and Dawei Zhou. Lensllm: Unveiling fine-tuning dynamics for I1m selection. ArXiv, abs/2505 03793, 2025. URL https::/api
semanticscholar.coo
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets Ilm finetuning:
The effect of data, model and finetuning method. ArXiv, abs/2402.171933, 2024a. URI. https
//api. seppiisemanticac. ics.mnaattcccccaaa.c......aaaaaa chwlar.ccrr....ooooooo wwwaaarorrggioee wtt.://........c.ooooooo/o/o///0
Guanhua Zhang and Moritz Hardt. Inherent trade-offs between diversity and stability in multi-task benchmarks. arXiv preprint 290.v.240..01719, 2024.
16

---

Published as a conference paper at ICLR 2026
Jieyu Zhang, Weikai Huang, Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, and Ranjay Krishna. Task me anything. ArXiv, abx/2406.11775 2024b. http:ttp:/// //api:://pp.p//
17

---

Published as a conference paper at ICLR 2026
A ADDITIONAL EXPERIMENT SETTING
A.1 BENCHMARK SELECTION
We begin our study with the Im-eva loval-hor -haannss package (Gao et al. 2023), which offers a com- prehensive suite of language model benchmarks. To accommodate the train -ean-beffoe-test methodology which requires a dedicated training set for fine-tuning, we first identify benchmarks that provide at least 1,000 training examples. This yields total of 37 benchmarks.
These benchmarks can be broadly categorized as 28 likelihood-basee and 9 generation-based bench- marks. Likelihood-based evaluations test for the likelihood of different completions given some input string: for example, different answer choices given a multiple-choice input question. Since the number of completions is usually small, likelihood-basee evaluations are generally compute-efficient.
Generation-base evaluations, in contrast, generate some output response given an input query. If responses tend to be long, then generation-baset evaluations naturally become compute-intensive This is particularly true for base models, which are usually not trained for instruction following and therefore continue generate tokens until hitting their maximum token limit. These generation-base benchmarks are also over-challenging for smaller models with limited parameters, such as GPT- 2 (Radford et al., 2019). Therefore, we exclude seven generation baare benchmarks, Drop, CoQa, ReCoRD, bAbi, WebOA, TriviaQA and Fid-De Fid-Default, fault. Nevertheless, we retain two widely used generation- benchmarks, GSM8K and NQ-Open, in our experiments.
We additionally excluded five benchmarks due to anomalies observed during fine-tuning: MedgA-40pt.inss, LogiQA, Mutual, Mela-EN, and Swag. For these benchmarks, more than 20% of models, most of which are small models with fewer than 0.5B parameters, showed no performance improvement after fine-tuning. We also excluded Paws-EN as its corresponding model ranking under direct evaluation negatively correlated (Kendall's less than zero) with 23 out of 24 other benchmarks. We attribute this anomaly to the unusual prompting template used by 1m-eval 1-11-hhe whaaaasssss...
Our final selection includes 24 benchmarks: MNLI
(Williams et al. 2018 QNLI Rajjuurar et al.
2016), RTE DDgan al.) [0066] Giampiccolo et al.
2007|
Bentivogli al. 2009), , CoLA (Warstadt
et al. 2018), SST-2 2 (Socher et al. 2013), MRPC
Dolan & Brockett 2005). QOP, Wic (illehvar
Camacho-Collados 2018). ANLI (Nie al. CommonsenseQA TTlmor et al. 2019), He 1elllas a awag
2020). Winogrande (Levesque al. 2011).
Zellers et al.) 2019), Socia al - -QA (Sap
al. 2019), OpenBookOA (Mihaylov et al.
2018).
NO-Open (Kwiatkowski et al.
2019).
Bool0 O (Clark et al. 2019), ARC-EasE Easy, ARC-Chaalenge
(Clark et al., 2018), Scio
(Webl
jet al. 2017). ) P PTQA Bisk et al. 2019). Mathoa (Amini et al. 2019). GSM8K (Cobbe al.
2021
MedMCQA Pal et al.
2022), HeadQA (Vilares & Gömme-RRodiiuee 2019).
A.2 EVALUATION SETUP
For our train-before-test evaluations, we fine-tune each model for five epochs and select the best- performing checkpoint based on evaluations on separate validation set. We use the AdamW optimizer with a weight decay of 0.01. For each model-benchmark combination, we perform hyperparameter search over three learning rates {le- - 5, 2e - 5, 5e 5} and select the optimal one based on validation performance. To reduce memory consumption, we employ parameter-efficien fine-tuning (PEFT) (Hu et al.) 2021]] Mangrulkar et al.] (022), We use LoRA configuration with rank 8, a = 32, and dropout O. Most of our experiments are conducted on Quadro RTX 6000, Tesla V100-SXM2-32GB and NVIDIA A A100-SXM4-80GB GPUs.
In cases where models show no performance improvement after fine-tuning, we report their pre- fine-tuning results. This scenario is rare and typically occur with smaller models (less than 500M parameters) that lack the capacity to perform certain tasks, resulting in near-random performance both before and after fine-tuning. Additionally, since all training datasets in our study are publicly available, some models may have encountered this data during pre-training, potentially limiting the benefits of additional fine-tuning.
For instruction tuned models, we evaluate performance both with and without chat templates, selecting the configuration that yields better results. Specifically, during direct evaluation, we assess model
18

---

Published as a conference paper at ICLR 2026
performance the validation set under both conditions and apply the better- setter-performing configuration to the test set. In the train-before before-test setting, we similarly fine-tune two variants: one with training data formatted using chat templates and one without. We then select the approach that achieves the best performance on the validation set for final evaluation.
19

---

Published as a conference paper at ICLR 2026
-10
0.61
0.59
2.0.00
0.31
0.4
0.53
0:45
0.31
0.26
0:39
0.51
0:51
0.41
0.38
9.29
0.35
NeaMOGA
0.53
0:35
0:31
0:47
0.06
0.73
0:45
0.53
9.30
633
0.04
0:36
KG-0pen
012.0.0410.11
6.004
3.06
0.44
0:18
0.24
0.34
0.11
6,67
0.54
0:32
6:27
-0.
082
-0.11
9.
0.
0.0000010
0.11
6:51
0.43
55R2
0.45
0.56
0.53
Winggrnnde
0.82
0.66
0.75
0.82
0:39
0.47
0.28
0.70
0.27
HellaSwag-
0.50
0.4
0.60
0.38
0.36
0.3
0.82
0.7
0.76
0.5
0.84
0.58
0.64
0.29
0.44
0.54
0.37
0.81
0.37
0.76
0.55
6.73
0.79
SocialigA
0.66
0.45
0.72
MehhGA
-0.67
0:46
0.75
0.64
0.75
0.26
0.74
0.72
0.25
0.46
0.55
0.79
0.75
0.66
0.67
ANU -0.63
0.5
0.54
0.5
0.13
0.6
0.54
0.6
0.64
0.6
0.53
0:38
0.67
0.66
0.56
0.54
0.51
0.63
0.41
0.68
0.49
0.58
Seiq 0.38
0.26
0.58
0.54
0.333
0.85
0:38
0.58
0.64
0.53
0.21
0.98
0.18
0.55
0.16
0.65
0.37
CommonsenseGA
00.72
0:12
0.6
8.55
0.64
0.72
0.7
87
0.71
0.62
0.64
0.54
0.63
73
0.57
BoolQ
ColA
0.63
0.59
0:45
0.6
0.00
0.554
0.119
0:44
0.52
0.45
0.54
0:36
0:21
0.62
0.53
0.57
0.52
0.48
0:79
0:15
0.63
0.12
0.41
GSMBK-DAA
0.47
0.51
0.57
0.52
0.5
0.55
0.94
0:28
0:37
0.48
0.39
0:39
0.55
OperBookGA -0.0A-0044 0.65
0.41
0.64
0.47
0:37
0.5
0.75
0.81
0.72 0.75 0.55
0.73
0.58
0.63
0.27
0:48
0.55
0.3M
0.32
0.75
0.55
0.72
0.79
HeadgA
0.47
0.29
0.49
0:38
0.76
0.76
0.82
0.66
0.64
0.35
0.64
0.27
0.75
0.83
RTE
0.78
0.6
0.54
0.63
0.11
0.62
0.48
0.56
0.65
0.62
0.66
0.46
0:37
0.71
0.62
0.61
0.62
0.51
0.63
0.41
0.45
0.45
0.53
ARC Challenge
0.55
36
0:67
0.41
0.43
0:48
0.83
0.79 0.71 0.78 0.58
0.8
0.7
0.57
0.73
0.41
0.52
0:31
0.73
0.25
0.83
0.53
0.86
ogP
MadMcCA
ONL
Opon
537-2
Wonogrande HetaSwag
Sedille upA
Masion
AMHL
Commons ero
5:0
Boota
CoLA
GBM
Opentiook@A
WdC
AMPP
HeadoQA
ARC- ABC-Easy
ATE
ARC. whw.chaeeegge
0.0
Figure 9: Cross benchmark ranking agreement under direct evaluation. Benchmarks are sorted based on the training dataset size. Kendall's is calculated for every benchmark pair.
B ADDITIONAL EXPERIMENT RESULTS
B. DOWNSTREAM RANKING AGREEMENT
We plot detailed pairwise ranking correlation agreement between benchmarks in Figure (direct evaluation) and I0
corresponding to Figure 2 in the main text.
20

---

Published as a conference paper at ICLR 2026
Kendoll's
00P-0.84
0.73
0.75
0.75
0.65
0.81
0:83
0.73
0.74
0.83 0.78 0.71 0.72
0.84
0.8
0.67
0.8
0.79
0.76
0.76
0.8
0.78
0.63
NeaMOqA 0.79 0.75
0.62
0.76 0.70
0.87
0.77 0.72
0.70
0.58
6.02
0.75 0.74
6.72
0.75
Open 0.74 0.75 0.74
0.76
0.68
0.84
0:84
0.78
0.68 0.72 0.82 0.74 0.64
0.77 0.72
0.62
0:68
0.78
0.65
0.81
0.67
0.79 0.79
551-2
0.68
0.58
0.67
Winggrande- -0.84
0.69
0.52 0.83
0.76
0.69
0.77
0.86
0.71
0.85
0.75
Hellefwng-- -0.87 0.83 0.78 0.8 0.80 0.0 0.92
0.81
0.77
0.83
0.9 0.78 0.72
0.8 0.78
0.09
0.76 0.87
0.72 0.06 0.75
c.nn 0.9
SocialiQA
0.83 0.81
0.68
0.61
0.69
0.77
MethQA (0.83 0.74 0.73 0.68 0.65
0.76 0.27
0.57
0.83 0.75
3.67
0.8
0.72
0.84
0.8
0.77
0.74
0.79 0.81
ANU -088 0.83 0.77
0.73 0.72 0.69
0.8 0.83
0.69 0.83
0.77 0.72
0.81
0.84
0.79
0.75
0.82
0.8
0.76
0.74
0.83
0.8 0.84
SciQ -0.74 0.71 0.67
0.7 0.74 0.6 0.76 0.78
0.77
0.67 0.72 0.77
0.62
0.77
0.68
0.59
0.69
0.76
0.6
0.79
0.67
0.78
Cammonsense@j .0.79
0.67
0.71 0.72 0.50
0.75
0.77
0.70
0.79
0.65
Boolo -0.99
0.87 0.5
0.78
0.86
0.74
CoLA 0.76 0.8 0.68 0.72 0.72 0.68 0.78 0.78 0.68 0.72 0.79 0.72 0:68
0.77
1
0.64
0.73
0.74
0.71
0.72
0.78 0.74 0.76
GSM 0.75
0.69 0.69 0.51
0.64
0.72
WCC
0.67
0.77 0.76 0.69
0.8
0.75
0.8
OpenBookgA- -0.82 0.79 0.74 0.76 6.78 0.05 0.86 0.87 0.77 0.79 0.8 0.84 0.76 0.73
0.86
0.74
0.68
0.75
1
0.71 0.83
0.71
0.86 0.87
0.71 0.72
0.71
HeedGA 0.79 0.76 0.77 0.63 0.85 0.86 0.77 0.74 0.74 8.84 0.79
0.72
0.85
0.7
0.83
0.65
0.7 0.88 0.86
RTE- -0.8 0.83 0.8 0.75
0.7 0.67 0.66 0.75 0.75 0.67 0.79 0.83 0.7 0.67 0.81 0.78 0.78 0.74
0.8
0.73
0.73
0.7
0.74 0.77
ARClasy
ARC-Challenge-e - 0.88 0.83 0.78 0.79 Diss 0.88 0.9 0.79 0.6 0.84 0.87 0.23 0.89 0.76 0.72 0.79 0.87 0.73 0.86 0.77
MHNL
ogP MedMCoA
ONL
open
597-2
Weeggandd HanaSwaag
Secialae lok
AMUL
Commersansec Bosia
ColA
GSMRK
OpentBookAA
AWpC
Heado4
RIE ABC- Basy
ABc. Aac-Challeegh
Figure 10: Cross benchmark ranking agreement under trainbbefore-tes. Benchmarks are sorted based on the training dataset size. Kendall's 's is calculated for every benchmark pair.
21

---

Published as a conference paper at ICLR 2026
Table Bits per byte (BPB) of eight excluded GEMMA models compared PYTHIA-410M across the three newly collected corpora. The GEMMA models exhibit abnormally high BPB values on Wike and Stack, likely due to the greater average sequence length these two datasets. Specifically, Arxiv v has an average of 163 words per document, compared to 250 for Stack and 1502 for Wi ki.
Arxiv
Wiki
Stack
GEMMA-2B
0.766
1.578
1.139
GEMMA-2B-IT
0.770
1.524
1.222
GEMMA-7B
1.013
4.780
4.053
GEMMA-7B- 1.053
18.711
20.958
GEMMA-2-21 0.730
1.784
1.340
GEMMA 2--B-A
0.705
1.191
0.997
GEMMA-2-9B 0.709
2.216
1,685
GEMMA-2-9B-IT
0.638
1.234
0.978
PYTHIA-410M
0.791
1.065
0.945
B.2 PERPLEXITY RANKING AGREEMENT
In this work, collect three corpora from Wiki iedia,, Stack Exchange, and arXiv. We only collect documents from 2025. More specifically, we collect 3,366 documents for Wiki, 6,001 for StackExchange and 44,384 documents for arXiv. These datasets split into training, validation, and testing sets, in an 8:1:1 ratio. For arXiv, we utilize only the paper abstracts, while for Sxchnnxc,,g,, we only the questions. Consequently, the average document length is 163 words for arXiv, 250 words for St StackExchange, and 1,502 words for Wik ipedia.
We exclude GEMMA models from our perplexity agreement experiments, as sir-eval-harneee provides unreliable perplexity measurements for GEMMA model We report the bits per byte (BPB) for the GEMMA models in Table While the BPB values for GEMMA on a ariiv (the dataset with the shortest average sequence length) are mostly reasonable, the performance on St StackExchang.g and Wik pedia notably worse, even compared to smaller models like PYTIIAA-1ОMM
This anomaly stems from how nni-vval-hareess handles long sequences via rolling window mechanism. Unlike other models, GEMMA requires every input sequence to begin with the BOS token. If this constraint is not met, perplexity degrades significantly. Consequently, when processing long sequences that are chunked into multiple windows, GEMMA'S 's performance degrades.
B.3 PC1 SCORE UNDER TRAIN-BEFORE- TEST
We plot the PC1 scores under wrain-before-test in Figure 11 We also provide the pre-training compute details for models with publicly available training token counts, as shown in Table 4
"See discussion
22

---

Published as a conference paper at ICLR 2026
Qwen2.5-148
Qwen2.5-14B-IT
Gemma-2-98
Gemma-2-96-IT
Gemma-7B
Lama-3-88
Llama-3-8--IT
Lama--31-88
Qwe41.5-14B
Llama-3.1-8B-1
Qwen2,5-78
Qwen2-7B-IT
Qwem1.5-14B-4
Qwen2-78
Qwen2.5-7B-IT
Yi-1.5-9B
Y--15-9B-IT
Yi-98
Qwen1.5-78
Yi-1.5-08-IT
Gemma-7B-IT
Y--155-BB
Owen1.5-7B-IT
Liama-3.2-38-1
Qwe32
Llama-3.2-33
Yi-68-TT
Yi-6B
Gemma-2-2B-IT
Qwen2,5-38-IT
Gemma-2-28
Qwen1.5-48
Owen1.554B-IT
Qwen225-1.5B
9wen2.5-1.68-IT
Qwen2-1.58
Qwen2-L.5BIIT
Gemma-2B
Pythia-12B
Pythia-6.95
Qwen1.5-1.88
@wen1 5-1.88-IT
Gemma-28-IIT
Llama-3.2-1BB
Llama-3.2-18
P-thia-2.88
Qwen2. 5-0.58
Owen2 5- 5-0058-IT
Qwen2-0.58-IT
Qwen2-0.5B
Pythia-1.4B
@w051.5-0.58
Qwen1 5-0.58-I-
Pythia-18
GPT2-1.58
GPT2-774M
Pythia-410M
GPT2-335M
Pythia-160M
GPT2-124M
Pythia-70M
-12.5
-10.0
-7.5
-5.0
-2.5
0.0
2.5
5.0
PC1 Score
Figure 11: PCI scores under train-hefore-test.
23

---

Published as a conference paper at ICLR 2026
Table 4: The models used in Figure 7
The number of training tokens of these models is publicly
available. We compute the number pre-training FLOPs #Parameters #Tokens.
Model
#Parameters (B)
#Tokens (B)
#FLOPs (10*18)
Llama-3-8B
8.03
15000.0
722700.00
Llama-3-882
8.03
15000.0
722700.00
Llama-3.1-8B
8.03
15000.0
722700.00
1amma-3.18B--1
8.03
15000.0
722700.00
1lama-3.2-3B
3.21
9000.0
173340.00
Llama-3.2
Jamm-332-3B-IT 3.2-3B-IT
3.21
9000.0
173340.00
Qwen .5-0.5B
0.62
2400.0
8928.00
Qwen1.5-1.8B 1,5-1.8B
1.84
2400.0
26496.00
Qwenl.5-44
3,95
2400.0
56880.00
Qwen
7.72
4000.0
185280.00
Qwenl.5-14p
14.20
4000.0
340800.00
Qwenl.5-0.5B-IT
0.62
2400.0
8928.00
Qwenl.5-1.8B.
1.84
2400.0
26496.00
Qwen1.5-4BBBl
3.95
2400.0
56880.00
15-7BI
7.72
4000.0
185280.00
14.20
4000.0
340800.0K
Gemma-7B
8.54
6000.0
307440.00
Gemma-7B-IT
8.54
6000.0
307440.00
Gemma-2-2B
2.61
2000.0
31320.00
Gemma-2-2B-IT
2.61
2000.0
31320.00
Gemma-2-9B
9.24
8000.0
443520.00
Gemma-2-91B-IT
9.24
8000.0
443520.00
Pythia-70M
0.07
300.0
126.00
Pythia-160M
0.16
300.0
288.00
Pythia-410M
0.41
300.0
738.00
Pythia-
1.00
300.0
1800.00
Pythia-
1.40
300.0
2520.00
2yth
2.80
300.0
5040.00
Pythia-6.9B
6.90
300.0
12420.00
Pythia-12B
12.00
300.0
21600.00
Yi-6B
6.06
3000.0
109080.00
Yi-6B-IT
6.06
3000.0
109080.00
Yi-9B
8.83
3800.0
201324.00
Yi-1.5-6B
6.06
3600.0
130896.00
Yi-1.5-6B-IT
6.06
3600.0
130896.00
Yi-1.5-9B
8.83
3600.0
190728.00 0.
Yi-1.5-9B-IT
8.83
3600.0
190728.00
24

---

Published as a conference paper at ICLR 2026
Table 5: We calculate Kendall's T between each benchmark and every other benchmark, and then
average the results.
Direct evaluation (0 shot)
Direct evaluation (5 shot)
Srrin-before-test
NQ-Open MRPC WiC
0.23
0,44
0.74
0.34
0.39
0.71
0.39
0.58
0.75
QNLI
0.43
0.58
0.74
QQP
0.43
0.62
0.77
CoLA
0.45
0.65
0.73
SciQ
0.47
0.49
0.71
SST-2
0.50
0.62
0.65
GSM8K
0.51
0.62
0.70
ANLI
0.54
0.56
0.78
MedMCQA
0.55
0.66
0.75
RTE
0.55
0.59
0.74
HeadQA
0.55
0.62
0.77
ARC-Easy
0.55
0.63
0.80
MNLI
0.56
0.63
0.80
PIQA
0.56
0.62
0.78
Winogrande
0.58
0.63
0,80
CommonsenseQA
0.58
0.65
0.72
Social-IQA
0.60
0.67
0.73
ARC-Challenge
0.61
0.70
0.81
HellaSwag
0.61
0,65
0.81
MathQA
0.61
0.67
0.76
OpenBookOA
0,61
0.64
0.78
Bool0
0.61
0.68
0.80
B.4 FEW-SHOT EVALUATION
We perform a 5-shot direct evaluation for all 61 models on benchmarks to examine its impact Gu et al. 2024) cross-benche boss-bennchark ranking agreement. The overall average Kendall's 5 is 0.52 for direct evaluation (O-shot), 0.61 for direct evaluation (5-shot), and 0.76 for train-before-test (0-shot). Train-before-test outperforms 5-shot direct evaluation 89% of benchmark pairs. We also present the mean ranking agreement between each benchmark and all others in Table Train-before achieves better ranking agreement across all benchmarks. We view in-context learning weaker form of task preparation compared to fine-tuning- both give models task preparation, but fine-tuning more thorough.
B.5 IMPACT OF TEST SET SIZE
We experiment only with benchmarks with more than 2,000 test samples understand how irreducible statistical noise (Fisher & Sen 1994 Heineman et al. 2025) in test sets affects ranking agreement. The remaining benchmarks include MNLI, OCF, MedMCQA, QNLI, NQ-Open, Hel laSwag, Mat BcolQ, HeadQA, and ARC Easy. The overall average Kendall's r across all benchmark pairs is 0.51 for direct evaluation (0-shot), 0.63 for direct evaluation (5-shot), and 0.80 for train-before-test. In other words, while the test set size impacts the statistical significance of the test scores, the cross-benchmark ranking agreement remains largely unchanged.
25

---

Published as a conference paper at ICLR 2026
Table 6: The overall average Kendall's r across all benchmark pairs for models in each size bin.
Model size (B)
Direct evaluation
Train-before-eest
0, 1)
0,38
0.65
1, 2)
0.53
0.56
2, 3)
0,45
0.70
6,
0.34
0.42
77, 8)
0.19
0.28
99 10)
0.15
0.43
All models
0.52
0.76
B.6 RANKING AGREEMENT FOR MODELS OF THE SAME SIZE
We group models into size bins, each containing at least 5 models, and compute the average Kendall's T across all benchmark pairs for each bin (Table 6). Train-before-test consistently enhances ranking consistency compared to direct evaluation in every bin. The lower consistency observed when controlling for model size (compared to 0.76 for all models) is expected. Model potential strongly correlates with pretraining compute, shown in Section 3.4 removing size variation reduces the primary signal that distinguishes models.
26

---

Published as a conference paper at ICLR 2026
Table 7: The overall average Kendall's across all benchmark pairs for each model family,
Model family
Direct evaluation
Train-beforr-testt
LLAMA
0.51
0.65
PYTHIA QWEN
0.25
0.88
0.57
0.79
GEMMA
0,43
0.77
Y1
0.25
0.30
All models
0.52
0.76
Table Explained variance ratios for top five principal components score matrix for each model family, under direct evaluation and train-before-tes respectively.
Direct Evaluation
Train-Before- Beith-BBeorreTtt Test
Model Family
PCI
PC2
PC3
PC4
PCS
PC1
PC2
PC3
PC4
PC5
LLAMA
69%
21%
4%
3%
2%
90%
4%
3%
1%
1%
PYTHIA
61%
12%
9%
8%
6%
89%
8%
2%
1%
1%
QWEN
74%
9%
5%
3%
3%
93%
2%
1%
GEMMA
58%
27%
6%
4%
2%
92%
3%
2%
2%
1%
YI
48%
21%
12%
10%
6%
67%
17%
7%
5%
3%
ALL MODELS
70%
13%
4%
3%
2%
86%
7%
2%
1%
1%
B.7 RANKING AGREEMENT FOR MODELS OF THE SAME FAMILY
We group models by the model family, each containing at least models, and compute the average Kendall's across all benchmark pairs for each family (Table (7). Train-before-tess consistently achieves higher ranking consistency than direct evaluation for every family. The only anomaly occurs in the YI models, where ranking consistency by train-before-test also quite low. This is possibly due to the pretraining compute for all considered Yi-models being very similar; see Table
We further conduct the PCA analysis in Section 3.4 for each model family, and report the explained variance ratios by the top five principal components. PCI explains over 89% of the variance in four out of five families under the train-before-tes, scenario, exceeding 86% for all models. This suggests that differences among model families, including architectural choices and training data composition, may account for the remaining variance captured higher-order principal components.
27

---

Published as a conference paper at ICLR 2026
C ACCOUNTING FOR STATISTICAL SIGNIHICANCE
C.I RANKING ALIGNMENT IN FIGURE
We plot the rankings of 61 language models on two question-answering benchmarks: Natural Questions Open and ARC Challenge in Figure We greedily align each ranking as much as possible without violating confidence intervals, thus revealing only those ranking changes that are statistically significant. See Algorithm for more details.
C.2 DOWNSTREAM RANKING AGREEMENT
We additionally supplement the experiments presented the main text by modifying the ranking correlation metric account for statistical significance in the benchmark evaluations. Specifically, we use Kendall's r- (Kendall, 1945), which adjusts for ties in rankings. We consider two models tied a given benchmark if their performance difference is not statistically significant at the 95% confidence level. We assess statistical significance using t-test based on the standard error of the mean performances.
We reproduce the ranking correlation figures of the main text using the modified Kendall's T which treats -tnsstaiitticllly significant performance differences ties. See Figure 12 and 13] as well as Figure [4] and Figure [5] for more detailed results. We observe that accounting for statistical significance in models' performance differences leads to slightly higher ranking correlations, as measured by Kendall's 's T-b. For direct evaluation, average agreement increases from 0.52 to 0.58. For train- before-test, average agreement increases from 0.76 to 0.77. Therefore, train-before-tee continues to lead to large improvements in raking agreement (from Kendall's 7- 0.58 to 0.77).
28

---

Published as a conference paper at ICLR 2026
Algorithm I: build_ pttplpaaallrdren(ssereeoeo stderrs)
Input: Model performance scores and standard errors
Output: Directed graph G representing significant model orderings
Initialize graph G with models as nodes foreach pair of distinct models (m1, m2) do m2 is significantly better than m2 then
L Add directed edge (m1 m2) to G
return G
Algorithm gorallei_rredyy_rale rannkmsselss,, G1, G2, score1, score2)
Input: List of models, two directed graphs G1, G2, and two score series Output: Two lists representing the parallel ranking order for each task Initialize vani lonnillaaen 1laaank,,, Initialize available; and availabley as models with zero in-degree in G₁ and G2 Initialize empty lists orderı, order2 for i to number of models do
rankdata(scorre), vani 11 aRanky +- rankdata(scoreg)
Initialize empty list pairs foreach my available, do
foreach in availables do
Compute cost for pair (m1, m2) based on:
(1) Placement of mi in order2 and m2 in orderi
(2) Whether my = m2 (prefer matching)
(3) Combined vanilla ranks: vanillaRankgImj + vani [aRanky[mg]
Append (cost, m1, m2) to pairs
Sort pairs by cost (ascending) Select (m,, m2) with minimal cost
Append mi order1, m2 orders
Remove my from G and update availabler
Remove m2 from G2 and update availableg
return orderj, order2
Algorithm 3: mank_models(ccorej, stderr,, scorez, stderry)
Input: Scores and standard errors for two tasks
Output: Parallel rankings for both tasks
+ Baa=Buullg__artaaaaoodeeoeooo, stderry)
stderr,)
(order),ordery)
htwpieee_gggeetr.raaaoooeeeeeeee rank(modess, G1, G2, score1, score.)
rankj[m] = position of m in order (starting from !)
rankg[m] position of m in ordera (starting from 1)
return rank1, rank2
29

---

Published as a conference paper at ICLR 2026
Direct evaluation
Train before-foore teste-ttstt
1.0
Mean Kendall 7
0.4
0.2
OQ-Open
0.0
MRPC
WIc
pop
ONLI
CoLA
Solo
GSMBK
SST-2 ARC Easy
ARC-Easy Head
MNLI MedMcQ
PIQA
ANLI
Commonsenee
RTE
AA
Social-IO HellaSwaa Swag IQA
ARC Challenge
OpenBook
BoolQ MathQA
Figure 12: Mean ranking agreement between each benchmark and all others, measured by Kendall's tau-b, with von-statistically significant performance differences being treated ties. We calculate Kendall's r-b between each benchmark and every other one, and then average. Compared to direct evaluation, wrain-before consistently improves ranking agreement- often by large margin. On average, the overall average Kendall's t is 0.58 for direct evaluation and 0.77 for train-before-test.
0.47
CR
0.57
0.76
0.69
0.7
0.7
0.73
0.8
CR.
0.77
0.82
0.86
0.73
0.79
0.83
Kendall
QA
Kendall
PBC
0.39
0.7
0.72
0.74
0.59
0.73
...4
PBC
0.68
0.73
0.72
0.64
0.65
0.72
0.65
Math
0.65
Med
0.47
0.73
0.7
0.73
0.65
0.69
Med
0.76
0.83
0.87
0.72
0.82
0.85
1
QA
PBC
Math
Med
c
QA
PBC
Math
Med
(a) Direct evaluation.
(b) tesie-tefore-test.
Figure 13: Cross-category ranking agreement for direct evaluation (left) and train train-before-test (right), measured by Kendall's taw-b, with nit-statistically significant performance differences being treated as ties. We consider language understanding (LU), commonsense reasoning (CR), question answering (QA), alysies/biooogy/chemistt (PBC), math (Math), and medicine (Med) categories. Kendall's 7-b is averaged across all pairs of benchmarks that belong to two given categories. The diagonal represents the intra-category agreement and the others represent the inter-category agreement. train- before-test improves both intra- and inter-category ranking agreement in all instances.
30

---

Published as a conference paper at ICLR 2026
00P-0.64
0.48
0.64
0.0g
0.34
0:41
0.57
0.5
0.57
0.34
0.29
0.68
0.49
0.68
0.53
0.55
0:45
0.41
0.2
0.64
0.3
0.38
NeaMOGA
0.75
0.79
0.71
0.22
0.53
5.7
Open
0.31
3.06
0.36
0.000
0.09
0.09
0:41
0.33
0.4
0.35
0.54
0.62
0.38
0.3
0.012
002
06:0
460.064
0.50
0.21
0.57
0.52
0.56
Winggrande
0.55
0.56
0.88
0.75
0.82
0:48
0.52
0:36
0.82
0.32
0.84
0.84
Hellswwng-
0.41
0:41
0.71
0.4
0.41
0.50
0.80
0.78
0.84
888
0.88
0.64
0.71
0.82
0.57 0.52
0.45
0.84
0:47
0.81
0.87
0.78
0.66
SocialiQA
0.75
0.78
0.52
MethGA
0.68
0.49
0.4
0.62
0.82
0.84
0.81
0.79
0.82
0.55
0.59
0.44
0.79
0.43
0.74
0.72
0.74
-0.66
0.57
0.64
0.65
0.19
0.73
0.64
0.69
0.71
0.69
0.59
0.77
0.76
0.74
0.65
0.62
0.67
0.53
0.59
0.79
0.67
SciQ
0.41
0.29
0.64
0.3
0.62
0.42
0.72
0.64
0.54
0.7
0.44
0.74
0.53
0.59
0.27
0.4
0.23
0.65
0.24
0.71
0.8
0.74
Cammonsenseej. 0.78
0.18
0.75
0.65
0.71
0.77 0.77
0.77
0.50
0.72
0.51
BoolQ
0.81
0.79
0.82
CoLA
0.72 0.68
0.53
0.72
3.012
0.6
3:48
0.52
0.51
0155
0.74
0:37
0.27
0.66
0.59
0.63
0.54
3.5
0:40
0:43
0.68
0.37
0.49
MB
0.52
0.55
0.58
WCC
0.0990.55
0:45
0.44
0.54
0.55
0:45
0.50
OpeeBookGA-DO
-0.71
0.45
0.7
0.55
0:00
0.00
0.82
0.84
0.8 0.79
0.67
0.81
0.66
0.72
6.81
8.5
0.58
0.45
1
0.4
0.79
0.71
0.77
0.8
0.32
0.42 0.45
0.56
HeedgA
0.5
0.64
0:35
0.55
0.16
0.84 0.81 0.71 0.74 0.58 0.88 0.71
0.43
0.47
0.32
0.79
0.31
RTE
0.85
0.66
0.21
0.76
0.65 0.67
0.7 0.71 0.79
0.59
0:49
0.81
0.74
0.66
0.68
8.56
0.73
0.46
0.54
0.64
ARC-Chalknne
0.74
0.46
0.52
0.56
0.88 0.86 0.79 0.85 0.67 0.81
0.74
0.65
0.81
0.49
0.55
0.36
0.8
0.35
0.86
0.64
0.9
MedMCcA
ONU
open
597-2
HenaSwagg Wesegrande
Secialln lok
AMMJ
Commens enneeg
900
Bosiq
CotA
GSMBK
OpenBBokOA
WMC
MNPC
HRad04
RTE
ARC- Basy
ABC- ABC-Challengp
Figure 14: Cross benchmark ranking agreement under direct evaluation, measured by Kendall's tau-b
with insignificant model comparisons treated ties.
31

---

Published as a conference paper at ICLR 2026
Kendoll's
00P-0.86
0.71
0.82
0.72
0.77
0.64
0.8
0.7 0.71 0.59 0.74
0.53
0.79
0:44
0.62
0.59
0.43
0.57
0.7
0.49
0.73
0.61
NeaMOqA 0.84 0.71
0.85 0.81 0.75
0.78
6.87
5.81
0.77
0.85
NQ Open 0.82 0.72
0.81
0.81
0.64
0.91
0:89
0.85
0.74 0.81 0.88
0.68
0.72
0.86
0.8 0.68 0.78
0.87
0.72
0.89
0.78
0.85 0.86
0.65
0.53
0.41
Winagrande-. -0.85
0:52
0.56 0.83
0.83
0.54
0.84
0.75
0.82
0.92
0.75
0.51
Hellefwng-- 00.8 0.8 0.81
0.85
0.89
0.65 0.96
0.86 0.82
0.89
0.92
0.72
0.79
0.93
0.66 0.73 0.83 0.92 0.76 0.91
0.83 6.93 0.94
SocialiQA
0.83
0.76
MethaA 0.86 0.71
0.88
0.74 0.74
0.85
0.82
0.75
0.89 0.76
0.59
0.91
0.85
0.8 0.88
0.82
0.85
0.84
ANU ANU-0.81 00.81 0.59
0.86
0.71
0.81
0:45
0.87
0:89
0.75
0.89
0.8
0:47
0.88
0.80
0.53
0.81
0.84 0.86 0.77 0.84
0.87
0.91
SciQ-00.7 -0.77
0.61
0.74
0.68
0.79
0.54
0.72
0.52
0.59
0.47
0.64
0.52
0.7
0.39 0.51
0.46 0.4 0.44 0.61
0.63
Cammonsenseaj. 0.84
0.52
0.79
0.79
0.771
0.8M
6.81
0.86
0.79
Boolo
0.91
0.53
0.86
0.85
0.87
0.78
CoLA 0.73
0.48
0.78
0.54
0.8
0:33
0.84
0.86
0.68
0.8 0.53
0.77
0.39
0.84
0.85
0.72 0.81 0.82
0.7 0.78 0.86 0.85
0.84
MB 0.77
0.75
0.23
0.57
0.77
W.C-0.85
0.82
0:83
0.70 0.82 84
0.78
0.82
OpenBookgA- -0.78
0.49
0.83
0.62
0.87
0.31
0.92 0.92
0.76 0.85 0.86 0.82
0.4
0.82
0.87
0.82
0.74
0.78
1
6.7
0.89
0.8
0.9 0.99
0.75
0.76
0.60
0.82
0.7
HeedGA 0.83
0.89
0.5
0.91
0.51 0.83 6.6 0.84 c.87
3.61
0.88
0.78
0.72 0.76 0.89 0.72
0.77 0.91
RTE 0.8 0.81
0.78
0:35
0.82
0:83
0.72 0.84
0.79
0.4
0.88 0.87
0.86 0.8 0.87
0.8 0.75 0.77
0.84 0.83
ARClasy
ARC-Challnng--o - 0.83 0.81 0.88 0.74
0.86
0.49
0.92
0.94 0.79 0.88 0.91 0.84
0.89
0.84 0.79 0.81 0.91 0.76
0.83 0.93
MHNL
MedMCcA
ONL
open
597-2
Weegrande HenaSSwag
Secialae lokk
AMH
Commensanseck egk
9000
Basia
ColA
GSMRK
OpentBokkA
Headoo
RIT
ABC Basy
ARC- ARc-Challeegg"
Figure 15: Cross benchmark ranking agreement under train-before-test measured by Kendall's tau-b
with insignificant model comparisons treated as ties.
32

---

Published as a conference paper at ICLR 2026
D BROADER IMPACTS
We do not anticipate any direct societal impacts from this work, such as potential malicious or unintended uses, nor do we foresee any significant concerns involving fairness, privacy, or security considerations, Additionally, we have not identified potential harms resulting from the application of this technology.
REPRODUCIBILITY STATEMENT
Detailed experimental settings are in Section and Appendix Code available https: com/ h/
1m harmony
F THE USE OF LARGE LANGUAGE MODELS
In this paper, we use large language models to aid and polish writing. Large language models are not used for retrieving related work or generating research ideas.
33


Reviews:
[{"review_id": "1YingaGdVE", "reviewer": "Reviewer_ZDg6", "summary": "Disclosure: I reviewed an earlier version of this paper for NeurIPS. I had significant concerns with that draft, but the revised version is substantially improved.\n\nThe paper addresses a common problem in LLM benchmarking: benchmarks often result in contradictory model rankings, even if they are supposed to measure the same underlying capability (e.g., reasoning). The authors offer a new perspective on this problem: they suggest to focus on model potential after finetuning rather than model performance under direction evaluation. Thus, they propose to add a step of task-specific finetuning before evaluation, an approach they call _train-before-test_. In their experiments, the authors show that train-before-test leads to more consistent model rankings across benchmarks, and even increases the correlation with perplexity-based model rankings.", "questions": "You write in lines 97 to 99: \"Perplexity benchmarks used to be popular, but fell out of fashion because of the apparent disconnect between perplexity and downstream task performance\" &mdash; can you clarify what specifically you base this claim on? In contemporary LLM development, perplexity remains a primary evaluation metric, despite all its limitations.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The problem addressed by the paper is important. The experiments conducted by the authors are very extensive, comprising a large set of LLMs and benchmarks. The paper is well written and easy to follow.\n\nCompared with the NeurIPS submission, the revised framing in terms of model potential is persuasive and clarifies the significance of the experiments and results. I also liked the added sections connecting the work to the scaling-laws literature.", "weaknesses": "- The method proposed by the authors only works in the narrow setting of LLM evaluation _where subsequent task-specific finetuning is guaranteed_. As a result, train-before-test provides little evidence about performance in typical deployment without such fine-tuning. The authors note this in the discussion, but because deployment without task-specific fine-tuning is far more common, this is a substantial limitation and should be stated upfront &mdash; in both the abstract and the introduction &mdash; to avoid readers over-generalizing the scope of the paper.\n\n- The central results of the paper are expected given established facts about the connection between (i) model size/pretraining compute and (ii) downstream performance after task-specific finetuning. The main empirical take-away (supported by the PCA analysis in section 3.4) is that larger/more pretrained models are consistently better across downstream tasks than smaller/less pretrained ones _after_ task-specific finetuning, but not _before_. However, this appears to follow directly from prior work:\n  - After task-specific finetuning, model size/pretraining compute (in combination with task-specific features) strongly predict downstream task performance (e.g., [Lin et al., 2024](https://arxiv.org/abs/2402.02314); [Zhang et al., 2024](https://arxiv.org/abs/2402.17193)).\n  - Under direct evaluation (i.e., without finetuning), model size/pretraining compute are much less predictive of downstream task performance (e.g., [Magnusson et al., 2024](https://arxiv.org/abs/2312.10523); [Lourie et al., 2025](https://arxiv.org/abs/2507.00885)).\n\n   The paper would be stronger if this connection were made explicit. Given that the authors already refer to the scaling-laws literature, adding this would be relatively easy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uQrid0wW5f", "reviewer": "Reviewer_EhcJ", "summary": "This work introduces a way compare model ranking based on performance of models after they are fine tuned on train split for a task. they say that this approach measures \"model potential\". The approach is dubbed \"train-before-test\". Through extensive experimental results, the paper shows that train-before-test stabilizes rankings of models: whereas before (under traditional protocol, which they dub \"direct evaluation\") rankings of models would change depending on benchmark, now rankings transfer between one model and the others. Overall, the paper represents an interesting contribution, and would benefit from being presented at ICLR.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is generally well motivated, easy to follow, and well written.\n- Experimental setup is mostly sound (see weaknesses below), and the paper studies models across many LM families. \n- I appreciate the framing of model potential as a mean to pick model to best adapt to a task. paper would benefit from stating the goal of train-before-test even more explicitly in the abstract: the presented technique is NOT an intrinsic evaluation of models as finished products, but as starting point for fine tuning.", "weaknesses": "- One limitation of this approach is that it does not provide an estimate of the magnitude of improvement. This could have been achieved by either proposing a way to average score, or use rankings to determine if any two models statistically different. Given the focus on practitioners using this method to choose models for downstream applications, providing a single, easy to interpret number is crucial. \n- The paper lacks other comparison with other techniques that could be used to improve rankings. For example, the paper evaluates all models using shots, which gives puts models, especially those that are not instruction tuned at a major disadvantage [\\(Gu et al., 2024\\)](https://arxiv.org/abs/2406.08446). The test split of some of these benchmarks is also very small, and ranking stability would likely improve by adding training samples to the test set (see Fig 9 of [Heineman et al., 2025](https://arxiv.org/abs/2508.13144)) . \n- I would caution against comparing with perplexity, as decontamination cannot be ensured for all models used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qdrAkHe7aA", "reviewer": "Reviewer_KmFP", "summary": "This work is follow-up to a previous ICLR 2025 paper proposing “training on the test set” as a better means of evaluating language models; this prior work showed that properties like “emergence” arise due to language models being tested on benchmarks that don’t take into account whether the model has been adapted to the downstream task; with “training on test set”, phenomena like emergence disappear. This work builds on this idea by showing further benefits of “training on test set”:\n\nThis work shows that after performing “training on test”, model rankings between similar benchmarks (e.g. benchmarks both testing QA) become more similar. Intuitively, one interprets this as – If you have multiple instantiations of benchmarks that all seem to test a common capability (e.g. medical knowledge vs math vs ..), then evaluating on those benchmarks in standard way can lead to very different evaluation results due to benchmark-specific properties (or if the models have trained in a manner arbitrarily to favor one benchmark but not the other), but with “train on test”, one is evaluating “potential” of the model on the underlying capability and not these other task-specific factors. In a similar analysis, this work shows that after performing “training on test”, model rankings between perplexity evaluations and downstream tasks are also more aligned.", "questions": "Even after this fine tuning, there is still not perfect correlation between (i) pairs of benchmarks measuring similar underlying capabilities, and (2) perplexity and downstream benchmarks. In fact, the best these rank correlations reach is somewhere around 0.7-0.8. That means there is something else irreducible that causes model rankings to be different. I would love to see authors provide a bit of discussion as to whether this is due to (a) intrinsic benchmark noise (an interesting reference would be Signal to Noise 2025 paper by Heineman et al though I understand it was published after this submission), (b) some differences in the underlying potential that these benchmark pairs are capturing, (c) some benchmark specific properties that even finetuning isn’t helping with, or something else? Just want to understand what is “left” between this work & maximal rank correlation. \n\nCan you explain more the finding in Sec 3.4? In short, it seems that running PCA before train-then-test shows a single main principal component that is driving most of the variation in benchmark scores; and that this component is highly representative of pretraining compute. After train-then-test, it looks like, while the PC1 explains more variation, the conclusion about pretraining compute being the primary driver is still the same, is it not? I’m not understanding what net new learning we are getting here or how to interpret the significance of increasing the amount of variation that is explained by PC1.\n\nSimilarly, I’m not sure what is the takeaway from Qwen PC1’s analysis.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is a good paper and merits publication at ICLR. While it is a follow-up to a previous ICLR paper, it adds net new contributions, notably showing how “train on test” fits into the existing language model benchmarking paradigm, which the previous work did not show explicitly. This work also shows interesting finding about how to unlock correlation between perplexity and downstream benchmark, which has long been a frustrating topic in language modeling research. Experiments are sensible, varied in scope of models used in experimentation (though would’ve preferred to have seen more modern fully open models like Olmo or SmolLM; Pythia is outdated & not reproducible because not all data is public).", "weaknesses": "I think the work is good, so not too many critiques. I would say first, the heat map figures for rank correlation are hard to read (Fig 3, 4, 5). Maybe use a different color scheme, consider annotating where we should pay attention, and make caption a bit more self contained so we can read the Figure + grasp the finding instead of having to cross reference it with the body text.  \n\nGiven the chosen models, I would’ve liked to have seen more findings that take advantage of: (1) comparing models of similar size across families, (2) comparing models within same family of different sizes. Aside from a Qwen analysis, this paper didn’t really dive into any of that, which is a shame.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "agJbgnUh5p", "reviewer": "Reviewer_WYQU", "summary": "This paper studies how LLM performance varies across tasks. It proposes that instead of directly evaluating LLMs on downstream tasks, LLMs should first be finetuned on the training set of the downstream task (a paradigm they call \"train-before-test\"). The paper shows via a large-scale empirical evaluation that model rankings after train-before-test are much more correlated with each other across benchmarks than the out-of-the box performances are.", "questions": "1. \"Model potential\" does not feel like the right term for finetuned performance. To me, the natural meaning of \"potential\" would apply *before* finetuning instead of afterwards. The phrase is currently used to mean performance after finetuning, so I'd suggest thinking of a different term.\n\nSmall comments:\n- Line 46: \"\\citet{Kaplan et al. (2020)}\" -> \"\\citep{Kaplan et al. (2020)}\"\n- Line 52: I'd avoid \"regret-free\", since it's a technical term in online learning for sublinear regret, which is not what's being referred to here.\n- Line 310: \"\\texttt{Wiki}pedia\" -> \"\\texttt{Wikipedia}\" and \"\\texttt{Stack}Exchange\" -> \"\\texttt{StackExchange}\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper provides a comprehensive empirical evaluation to show that a) model rankings change after finetuning and b) model ranks are much more correlated across benchmark tasks after finetuning.\n2. The paper shows that pre-finetuning perplexity model rankings correlate with post-finetuning benchmark performance model rankings for base models but not for instruction-tuned models.\n3. The paper shows that after fine-tuning, the model-benchmark score matrix becomes nearly rank 1.\n4. The paper is quite well-written.", "weaknesses": "I see no major weaknesses of this paper. My biggest concern is its difference from [1], which is currently mentioned a few times in the paper. Lines 154-157 state that the paper introduces the \"train-before-test\" paradigm, but [1] already proposes to finetune on task-relevant data before evaluation (Section 2.1 of [1]). Can you more thoroughly articulate the differences between your paper and [1]? As it stands, I think this paper provides valuable insights about the stability of model rankings across tasks after finetuning, but the magnitude of this paper's contribution is smaller in light of [1].\n\n[1]: \"Training on the Test Task Confounds Evaluation and Emergence\", Dominguez-Olmedo et al, ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)