Fluid Language Model Benchmarking
Authors: Valentin Hofmann
David Heineman
Ian Magnusson
Kyle Lo
Jesse Dodge
Maarten Sap
Pang Wei Koh
Chun Wang
Hannaneh Hajishirzi
Noah A. Smith
Venue: COLM 2025 (2025)
Keywords: language models
evaluation
item response theory
efficiency
robustness
TL;DR: Fluid Benchmarking improves evaluation by adapting items to the capability level of language models.

Abstract: Language model (LM) benchmarking faces several challenges: comprehensive evaluations are costly, benchmarks often fail to measure the intended capabilities, and evaluation quality can degrade due to labeling errors and benchmark saturation. Although various strategies have been proposed to mitigate these issues, they tend to address individual aspects in isolation, neglecting broader questions about overall evaluation quality. Here, we introduce Fluid Benchmarking, a new evaluation approach that advances LM benchmarking across multiple dimensions. Inspired by psychometrics, Fluid Benchmarking is based on the insight that the relative value of benchmark items depends on an LM's capability level, suggesting that evaluation should adapt to each LM. Methodologically, Fluid Benchmarking estimates an item response model based on existing LM evaluation results and uses the inferred quantities to select evaluation items dynamically, similar to computerized adaptive testing in education. In our experiments, we compare Fluid Benchmarking against the common practice of random item sampling as well as more sophisticated baselines, including alternative methods grounded in item response theory. We examine four dimensions&mdash;efficiency, validity, variance, and saturation&mdash;and find that Fluid Benchmarking achieves superior performance in all of them (e.g., higher validity and less variance on MMLU with fifty times fewer items). Our analysis shows that the two components of Fluid Benchmarking have distinct effects: item response theory, used to map performance into a latent ability space, increases validity, while dynamic item selection reduces variance. Overall, our results suggest that LM benchmarking can be substantially improved by moving beyond static evaluation.

Published as a conference paper at COLM 2025
Fluid Language Model Benchmarking
Valentin Hofmann" 00
David Heineman ?
Ian Magnusson 3
Kyle Lo
Jesse Dodge Maarten Sap*
Pang Wei Koh 00 Chun Wang v
Hannaneh Hajishirzi 0
Noah A. Smith 90
Allen Institute for Al
University of Washington
Carnegie Mellon University
Abstract
Language model (LM) benchmarking faces several challenges: comprehen- sive evaluations are costly, benchmarks often fail to measure the intended capabilities, and evaluation quality can degrade due to labeling errors and benchmark saturation. Although various strategies have been proposed to mitigate these issues, they tend to address individual aspects in isolation, neglecting broader questions about overall evaluation quality. Here, we in- troduce FLUID BENCHMARKING, new evaluation approach that advances LM benchmarking across multiple dimensions. Inspired by psychometrics, FLUID BENCHMARKING is based on the insight that the relative value of benchmark items depends on an LM's capability level, suggesting that evaluation should adapt each LMM Methodologically, FLUID BENCH- MARKING estimates an ifem response model based on existing LM evaluation results and uses the inferred quantities to select evaluation items dynamically,, similar to computerized adaptive testing in education. In our experiments, we compare FLUID BENCHMARKING against the common practice of ran- dom item sampling as well more sophisticated baselines, including alternative methods grounded in item response theory. We examine four dimensions efficieenc,, validity, variance, and saturation and find that FLUID BENCHMARKING achieves superior performance in all of them (e.g,, higher validity and less variance on MMLU with fifty times fewer items). Our analysis shows that the two components of FLUID BENCHMARKING have distinct effects: item response theory, used map performance into a latent ability space, increases validity, while dynamic item selection re- duces variance. Overall, our results suggest that LM benchmarking can be substantially improved by moving beyond static evaluation.
Code and Data github.com/alleee
1 Introduction
The field of language model (LM) evaluation is experiencing a moment of crisis. With new benchmarks being released by the day, it becomes increasingly difficult decide which benchmark(s) to pick for a certain evaluation goal (Ni et al., 2024; Perlitz et al., 2024b). At the same time, evaluating LMs on ever-growing sets of benchmarks leads to substantial computational-andd consequently, financial and invironmental-coe all while producing brittle results that fluctuate due evaluation noise (Madaan et al., 2024; Mizrahi et al., 2024). More alarmingly, it is often unclear whether specific benchmark in fact measures the capability that purports to evaluate (Liao et al., 2021; Saxon al., 2024), problem exacerbated by labeling errors (Northcutt et al., 2021; Gema et al., 2024; Vendrow et al., 2025) and benchmark saturation, when many LMs are scoring near the maximum on benchmark (Vania et al., 2021; Xia et al., 2024).
(Liang et al., 2023),
These challenges have spurred various efforts to improve benchmarking, by increasing efficiency (Perlitz et al., 2024a; Polo et al., 2024; Vivek et al., 2024; Kipnis et al., 2025), detecting and correcting mislabeled items (Gema et al., 2024; Vendrow et al., 2025), reducing evaluation variance (Madaan et al., 2024), and enhancing benchmark difficulty (Suzgun

---

Published as a conference paper at COLM 2025
211
1924
wndl
ABIL ABILITY (mar (n)
(BILITY Q2))
progsng
(c) Variance
c:
Frumt BESCHOARINGG
(a) Item response theory
(b) FLUID BENCHMARKING
(d) Validity
Figure 1: (a) Given a benchmark Q, we train an IRT model on publicly available LM evaluation results, providing useful information about individual items (specifically, about difficulty and discrimination). The figure illustrates this with results for four LMs and difficulty, symbolized by item darkness. In practice, we use more than hundred LMs. (b) FLUID BENCHMARKING leverages the IRT-enriched benchmark in two ways: it uses item difficulty and discrimination to (i) dynamically select an item subset Q' that matches given LM's capability
easier items are routed to the weaker LM May more difficult latent ability space rather than standard accuracy space. (c, d) Compared to baselines such as evaluating on random subset of items (RANDOM), FLUID BENCHMARKING improves benchmarking in various ways: it substantially decreases step-to-step evaluation variance, exemplified by training curves of Pythia-2.8B evaluated on ARC Challenge with 30 items (c), while at the same time increasing the external validity of evaluation, shown as the mean rank distance between an LM's predicted and true rank (d). See text for more details.
items to the stronger LM and (ii) represent a given LM's performance
et al., 2023; Gupta et al., 2024; Paech, 2024). However, most of these studies have addressed individual aspects of evaluation quality in isolation, sometimes with unintended negative consequences consequences-for for example, Madaan et al. (2024) showed that efficient benchmarking methods can increase evaluation variance between training runs with different random seeds, thus reducing benchmarks' practical utility.
In this paper, we propose FLUID BENCHMARKING, a new benchmarking method that improves evaluation across multiple relevant dimensions. FLUID BENCHMARKING is based on the insight that the relative value of benchmark items depends on an LM's capability level; for example, a hard question might be too difficult for a weak LM, but informative for strong LM. FLUID BENCHMARKING integrates item response theory (IRT; Lord, 1980; van der Linden & Hambleton, 1997; DeMars, 2010), which represents performance in latent ability space, with methods from computerized adaptive testing used in education (Meijer & Nering, 1999; Chang, 2015; Magis et al., 2017): IRT draws upon existing LM evaluation results to enrich benchmarks with information about item difficulty and discrimination, which is leveraged to dynamically select items that match an LM's capability level (Figure 1). This contrasts with the until now universal practice of what we call static benchmarking, which assumes globally optimal set of evaluation items for all LMs.
In our experiments, we investigate how different methods for improving evaluation affect the efficiency, validity, variance, and saturation of benchmarks. We specifically focus on LM evaluation during prefraining, a key application benchmarking. We evaluate six LMs on six benchmarks, comparing FLUID BENCHMARKING against broad set of methods proposed in prior work. We find that FLUID BENCHMARKING consistently outperforms all baselines across all dimensions of evaluation quality. For example, compared to the common practice of random item sampling, FLUID BENCHMARKING improves validity and lowers step-to- step variance MMLU using fifty times fewer items. Our analysis attributes these gains to the complementary effects of the two key components of FLUID BENCHMARKING: IRT enhances validity, while dynamic item selection reduces variance.
2

---

Published as a conference paper at COLM 2025
2 Preliminaries: Benchmark Refinement
In this paper, we introduce benchmark refinement as the problem of improving benchmark- ing by optimizing (i) the selection of evaluation items as well as (ii) the aggregation of their results into benchmark-level scores. We argue that many existing efforts in LM evaluation, previously considered in isolation, can be productively unified under this umbrella.
2.1 Evaluation is Selection, Scoring, and Aggregation
Let m; be an LM that is to be evaluated on benchmark Q. We refer to the elements Oj E as items. In general form, evaluating my on Q can be expressed as
Q
EVALUATE (M; Q) = AGGREGATE (SCORE(M;)))) q (ESELBCT(())
(1)
benchmark-level
item-level
aggregation
scoring
where SELECT is selection function that determines the set of evaluation items, SCORE is a scoring function that quantifies LM performance on each item in the evaluation set, and AGGREGATE an aggregation function applied over the item-level scores. For notational convenience, we denote the evaluation set as Q*, which may be a subset, superset, or identical Q. If SCORE {0,1} binary function, AGGREGATE returns the mean of the item-level scores, and (ELECT(Q) Q we recover the standard accuracy metric commonly used LM benchmarking, which we denote as ACCURACY Q).
With Equation 1, we can break down LM evaluation into two components: item-level scoring and benchmark-level nggregation. While evaluation quality can be improved at both levels, many studies take item-level scores given and focus improving benchmark-ler aggregation. The present line of work asks: how can we improve LM evaluation through the choice of both (i) the selection function SELECT and/ or (ii) the aggregation function AGGREGATE? We refer to this problem as benchmark refinement.
2.2 Dimensions of Evaluation Quality
What aspects of evaluation can be improved through benchmark refinement? In this paper, we focus on four dimensions, each motivated by prior work:
Efficiency. Evaluation can be made more efficient by selecting a small evaluation set Q*, with Q- [2] Prior work has explored random sampling (Perlitz et al., 2024a), item clustering (Polo et al., 2024; Vivek et al., 2024), heuristic filtering (Gupta et al., 2024), and information filtering (Kipnis al., 2025). Several studies have paired this with modifying AGGREGATE (Polo et al., 2024; Kipnis et al., 2025).
Validity. As a means of measuring an underlying capability in LMs, a benchmark should be predictive of LM behavior beyond the benchmark itself. Prior work has explored pr
different ways to increase benchmark validity via SELECT-for example, by removing and replacing items in Q that trivially fail to measure the intended capability, such as mislabeled items (Northcutt et al., 2021; Gema et al., 2024; Vendrow et al., 2025).
Variance. If evaluation results on benchmark fluctuate significantly due to evaluation noise (e.g., metric instability), the benchmark becomes less useful in many practical settings, such as tracking progress during training. While has been shown that removing items with low discriminative power from Q can reduce variance, attempts to modify AGGREGATE have so far proven less effective (Madaan et al., 2024).
Saturation. Given the rapid improvement in LM capabilities, frontier models often solve most items in benchmarks within a short time, limiting their practical value (Vania et al., 2021; Liang et al., 2023). This saturation has motivated the development of more challenging benchmark variants by choosing SELECT such ih focuses on more difficult items (Suzgun et al., 2023; Gupta et al., 2024; Paech, 2024).
In $4.2, we operationalize each of these dimensions into metrics.
3

---

Published as a conference paper at COLM 2025
3 Methodology: FLUID BENCHMARKING
We introduce FLUID BENCHMARKING, a new method for benchmark refinement that departs from prior work (i) by changing AGGREGATE such that LM performance is represented in a latent ability space rather than the standard accuracy space ($3.1), and (ii) by choosing SELECT dynamically adjust the subset of evaluation items to an LM (53.2).
3.1 Measuring Language Model Performance in Latent Ability Space
Over the past several decades, research in psychometrics has developed suite of methods to address the challenges discussed in the previous section, which arise in a similar form in human testing. We argue that psychometric methods can be fruitfully applied to the evaluation of LMs. In particular, we draw upon item response theory (IRT; Lord, 1980; van der Linden & Hambleton, 1997; DeMars, 2010), which represents test takers in latent ability space. The specific IRT model we use is two-parameter logistic (2PL) model (Lord, 1952; Birnbaum, 1968). Before providing formal definition, we begin with a quick overview of the advantages of IRT-based ability estimates over accuracy.
The key property that distinguishes IRT-based ability estimates from accuracy is that IRT takes item characleristics into account, whereas accuracy treats all items equally. In the 2PL model that we consider here, the two item characteristics are:
Item difficulty. Correctly answering an easy item has a different impact on ability esti- mates than correctly answering a difficult item.
Item discrimination. Items exhibit varying rates at which the likelihood of correct response increases with ability. Low-discrimination items are often problematic-for example, we find empirically that many of them are mislabeled (see $6).
These features might be beneficial for benchmark refinement. In terms of efficiency, item parameters provide a principled basis for selecting Q". Item discrimination potentially offers dual benefits: it could enhance validity by reducing the impact of mislabeled items, while simultaneously decreasing variance by placing less weight on items that inconsistently differentiate between similar LMs. Finally, the fact that difficult items affect ability estimates differently than easy items could delay saturation effects, as differences in performance among strong LMs on difficult items are better captured than by accuracy.
Formulation. Let M = {m....., } be set of LMs that have been evaluated on benchmark Q. Assuming items with two outcomes, the probability that an LM m; answers item 9j correctly can be modeled as Bernoulli random variable Uij, where Uij (success) iff the LM's answer is correct. The probability that Uij 1 is modeled as
puuj 1) = logistic (a; - bb;),,
(2)
where the parameter 9: corresponds to the ability of LM m;, and the item qj characterized by parameters aj > 0 (discrimination) and bj (difficulty). Equation 2 commonly visualized using so-called item characteristic curoes (see Appendix A for examples). For model esti- mation, we assume local independence and maximize the probability of the full response matrix U E (0, fkxl using Markov chain Monte Carlo (Junker et al., 2016), with hierarchical priors on all parameters as suggested by Natesan et al. (2016).
Given fitted 2PL model, the item parameters aj and bj can be used to estimate the ability 0, of a previously unevaluated LM m; by maximizing
0, max 8
logistic (a, b;)) Mij [1 logistic (0 b;)) mg.
Here, the item parameters aj and by are treated as fixed. We use maximum a posteriori estimation (Birnbaum, 1969) to determine 0,. Equation 3 defines a benchmark-level ag- gregation in Equation 1, with SCORE(M; Mij and AGGREGATE Q) = 0. We
4

---

Published as a conference paper at COLM 2025
denote this form of model evaluation ABILITY(m;, Q), which constitutes one of the two methodological pillars of FLUID BENCHMARKING.
So far we have only modified AGGREGATE, not SELECT, but IRT allows for principled way to dynamically adapt Q* to an LM. Next, we present method how to do so.
3.2 Dynamic Selection of Evaluation Items
Benchmarks are used to monitor performance during pretraining, when LMs are undergoing rapid development. Can the same Q* be optimal for both a near-random word predictor (early in training) and a highly capable model?
One way to approach this question is by examining the informativeness of items with respect to the ability estimate for a given LM, which can be formalized using Fisher information (Reckase, 2009). In the case of the 2PL model, this is given by
aj, = of logistic (a; - bj)) [1 - logistic (a; bj))]-
It can be shown that items with higher Fisher infor- mation yield more precise ability estimates (Reckase, 2009), and they should be prioritized in Q"
Fisher information 9g.
005
Hom ditioult b).
To analyze how the informativeness of items change as a function of LM ability, we examine HellaSwag (Zellers et al., 2019). We consider the scenario of pre- training mentioned above and simulate training run with 50 checkpoints.
01
00
Figure 2, we show how the Fisher information distributes over HellaSwag items as a function of training progress. The subset of items with the highest Fisher information substan- tially changes over the course of the training run, from very easy items at the beginning of training, to very difficult items at the end of training. These findings suggest that adapting Q* to the capability level of an LM could result in more precise ability estimates compared to using a static set of items.
60%
Fisher information (Equa- tion 4) of HellaSwag items as a func- tion of training progress. Lower panel: simulated trajectory of LM ability, which evolves linearly from
Figure 2:
Formulation. Inspired by these observations, we draw upon methods developed in the education- research context of computerized adaptive testing (Meijer & Nering, 1999; Chang, 2015; Magis et al., 2017) adapt Q" to the capability level of an LM mi. Specifically, we evaluate the LM by iteratively selecting the item from Q with the highest Fisher information given the current ability estimate,
Ø! = = Fisher information of HellaSwag items. The HellaSwag items with highest Fisher information change drastically during training (see Ap- pendix B for more details).
7 to 8 +7; upper panel:
(0) = Ø; (1(1) Q7 (t-1))
arg max T (ABILITY(MM, Q7 (1-)))), aj. bi) 4 (L(Q))(1
(5)
We repeat this procedure until the total number of administered items has reached the budgeted size for Q', at which point we let Q; Q, (t). Using Equation 5 for SELECT, we compute EVALUATE(M;, Q) ABILITY(M,, Q7) the final evaluation score.
Dynamically selecting items based on Fisher information is expected to reinforce the very properties that make IRT-based methods promising for LM evaluation to begin with. For example, given that 1 a? (Equation 4), discrisiriinnaooon items are unlikely to be included in Q* Similarly, because I is maximized when 0; by (where a² /4), dynamic selection naturally adapts to the capability level an LM, evaluating weaker LMs on easier items and stronger LMs more difficult ones.
5

---

Published as a conference paper at COLM 2025
4 Experiments
4.1 Experimental Setup
In this paper, we focus on LM evaluation during pretraining. While the four dimensions of benchmark refinement, introduced in $2, are relevant across evaluative settings, pretraining provides particularly suitable testbed, as it allows for straightforward quantification and measurement of each dimension (see $4.2).
More specifically, we examine the pretraining runs of six LMs with publicly available checkpoints. Our main focus lies on 7B LMs, for which we pick Amber-6.7B (Liu et al., 2023), OLMol-7B (Groeneveld et al., 2024), OLMo2-7B (OLMo et al., 2025), and Pythia-6.98 (Biderman et al., 2023). We also examine smaller LM, specifically Pythia-2.8B (Biderman et al., 2023), as well as a larger LM, specifically K2-65B (Liu et al., 2025). For each LM, we evenly select between 61 and 94 checkpoints (see Appendix C for more details).
In terms of benchmarks, we focus on the Open LLM Leaderboard (Beeching et al., 2023), which comprises ARC Challenge (Clark et al., 2018), GSMSK (Cobbe et al., 2021), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), TruthfulQA (Lin et al., 2022), and Wino- Grande (Sakaguchi et al., 2020). For the IRT models underlying FL.ID BENCHMARKING, we fit 2PL models to the evaluation results of LMs contained in the Open LLM Leaderboard. We exclude the six test LMS and related models (e.g., OLMo1-1B), well as posttrained models, since our experiments focus on evaluation during LM pretraining This results in final set of 102 LMs used for IRT model training (see Appendix D for the full inclusion criteria). We fit separate unidimensional IRT models for each benchmark; we initially experimented with multidimensional models as well as a single unidimensional model across all benchmarks, but these approaches yielded worse results (see Appendix E for details).
We then evaluate all checkpoints of the six selected LMs on the six benchmarks and vary the evaluation strategy (see $4.3). In total, we examine 2,802 theckpoint-berchhaatt combi- nations, resulting in over 13 million item-level evaluations.
4.2 Evaluation Measures
We operationalize the four dimensions of evaluation quality introduced in $2 as follows:
Efficiency. We measure efficiency by systematically varying the number of items used for evaluating on benchmark (i.e., the size of Q*). We explore range of subset sizes, varying from 10 to 500 items per benchmark.
Validity. We evaluate validity by testing how well estimated performance on one bench- mark predicts performance
different benchmark that targets the same capability. Specifically, we compute the distance between an LMs' predicted ranks on the two benchmarks. We always calculate the rank for the second benchmark based on accuracy. We examine ARC Challenge and MMLU, which assess knowledge and reasoning, and HellaSwag and WinoGrande, which assess commonsense reasoning.
Variance. We measure the step-to-step variance of the training curve for combination of LM and benchmark. Specifically, let x² = EVALUATE(MI, Q) represent the measured performance (e.g., accuracy) on benchmark Q for model m, at certain checkpoint t. We measure the normalized total variation,
(0)
TV(M; Q)
(6)
x
where a lower value means lower variance and hence better evaluation quality.
Saturation. To measure the saturation benchmark under a given evaluation strategy, we compute the monotonicity of the training curve, defined as the absolute Spearman rank correlation between the sequence of checkpoints and the predicted performance values (e.g., accuracies). More monotonic training curves indicate that increased pretrain- ing consistently yields better performance, suggesting that the benchmark has not yet saturated (at least for LMs within the considered capability range).

---

Published as a conference paper at COLM 2025
Baseline/teme per benchmark
Measure
Method
1o
A15
1B
MB143
SM460
MA1 1,888
Validity
BASELINE
20.0
15.2
9.8
8.7
15.9
14.5
Rank distunce +
FLUID BENCHMARKING
10.1
8.6
14.0
8.3
Variance
BASELINE
28.3
19.1
30.5
17.9
10.0
20.4
Total variittion
FLUID BENCHMARKING
10.7
5.5
2.8
4.8
Saturation
BASELINE
0.48
0.62
0.69
0.79
0.88
0.64
Rank correlation
FLUID BENCHMARKING
0.76
0.86
0.85
0.85
0.97
0.77
Table 1: Comparison against baseline methods. AP: ANCHOR POINTS (Vivek et al., 2024); TB: TINY (Polo et al., 2024); MB: METABENCH (Kipnis et al., 2025); SM: SMART (Gupta et al., 2024); MA: MAGI (Paech, 2024). The table shows the results averaged across six benchmarks, six LMs, and between 61 and 94 checkpoints per LM, totaling 2,802 values contributing to each mean. For METABENCH, the number of items is an average across benchmarks, and we exactly match the benchmark-level numbers for the comparison.
4.3 Baselines
We compare against several previous benchmark refinement methods. First, we examine ANCHOR POINTS (Vivek et al., 2024), a method for efficient evaluation based on item clustering. We use the Open LLM Leaderboard to cluster the benchmarks and consider two subset sizes in the range examined by the authors (10 and 50). We also examine two IRT-based methods, TINYBENCHMARKS (Polo al., 2024) and METABENCH (Kipnis al., 2025), and compare directly against their subsets and evaluation tools. In terms of methods for increasing difficulty, we include the hard versions of ARC Challenge and MMLU from SMART (Gupta et al., 2024) and MAGI (Paech, 2024), respectively.
FLUID BENCHMARKING differs from prior methods through its AGGREGATE (53.1) and its SELECT ($3.2). To disentangle these factors, we consider baseline in which we ablate SELECT and compute an ability estimate based on a random subset of items (RANDOM IRT). In addition, we consider baseline in which we ablate both SELECT and AGGREGATE, using a random subset of items to compute accuracy (RANDOM), a popular approach for efficient evaluation (Liang et al., 2023; Gu et al., 2024; Perlitz et al., 2024a).
5 Results
FLUID BENCHMARKING outperforms all baselines across all dimensions and sample sizes, often by a wide margin (see Appendix for breakdowns by benchmark and LM).
Validity. Table (top panel) shows that FLUID BENCHMARKING leads to smaller rank distances than all baselines. It outperforms ANCHOR POINTS, SMART, and MAGI by wide margins, almost halving the mean rank distance of ANCHOR POINTS. The IRT-based methods are better, but FLUID BENCHMARKING still outperforms them.
Table 2 (top panel) shows that ablating the dynamic selection of items (FLUID BENCH- MARKING RANDOM TRT) results in lowered validity, but the gap diminishes with more items. This is expected since (dynamic) Q; approximates (static) Q* as the number of items increases, resulting in converging ability estimates. Ablating the IRT-based ability estima- tion (RANDOM vs. RANDOM IRT) leads to a much bigger drop in validity (see Figure 1d), suggesting that the information provided by IRT is particularly beneficial for improving the predictiveness of performance estimates. This is also supported by the high validity of the two IRT-based baselines TINYBENCHMARKS and METABENCH.
Variance. Table (mid panel) shows that FLUID BENCHMARKING outperforms all base- lines in terms of step-to-step variance. This trend holds consistently across LMs, benchmarks, and subset sizes (see Appendix G for details). Figure 1e illustrates this with the evaluation of Pythia-2.8B on ARC Challenge, using 30 items. Interestingly, the gap between TINY-
7

---

Published as a conference paper at COLM 2025
Items per task
Measure
Method
10
50
100
500
Validity
RANDOM
20.0
15.2
16,9
9.1
Rank distance 4
RANDOM IRT
14.1
11.1
10.6
8.4
FLUID BENCHMARKING
10.1
8.8
8.3
Variance
RANDOM
29.0
19.1
19.8
10.2
Total veriation
RANDOM IRT
18.2
15.7
17.8
10,9
FLUID BENCHMARKING
10.7
6.5
6.1
4.9
Saturation
RANDOM
0.47
0.62
0.64
0.79
Rank correlation 1
RANDOM IRT
0.48
0.69
0.71
0.85
FLUID BENCHMARKING
0.76
0.86
0.85
0.88
Table 2: Comparison against ablated methods. See caption of Table 1 for more details.
BENCHMARKS and METABENCH on the one hand, and the remaining baselines on the other, is much less pronounced than for validity-the two methods even lead to worse results for variance (e.g., IINYBENCHMARKS 100 items: 30.5 vs. RANDOM/ 100 items: 19.8).
Table (mid panel) reflects this trend: the gap is smaller between RANDOM and RANDOM IRT than between RANDOM IRT and FLUID BENCHMARKING- on 500 items, RANDOM IRT even leads to a higher variance than RANDOM. This suggests that the key to FLUID BENCHMARKING'S low variance lies in its dynamic item selection, which is consistent with psychometric theory: since the variance of ability estimates is inversely proportional to test information (Lord, 1983), and since FL,ID BENCHMARKING selects highly informative items, the resulting measurement error is substantially reduced.
Saturation. Tables 1 and 2 show that FLUID BENCHMARKING consistently outperforms all baselines in terms saturation as well (see Appendix for details). SMART and MAGI perform better than some of the other baselines, suggesting that these methods partially mitigate the saturation problem, yet FLUID BENCHMARKING addresses it more effectively.
Efficiency. Taking global look at the results, we observe that FLUID BENCHMARKING leads to improvements across all subset sizes, but is especially effective for small sample sizes. For example, with 500 items FLUID BENCHMARKING improves the mean rank distance (validity) of RANDOM by 0.8, but with 10 items by 9.9.
In Appendix H, we show that FLUID BENCHMA RKING can improve evaluation quality even when efficiency is not concern, outperforming full-benchmark accuracy.
6 Analysis and Discussion
FLUID BENCHMARKING Avoids Mislabeled Items. To test whether FLUID BENCHMARK- ING indeed avoids problematic instances such as mislabeled questions, we leverage MMLU- Redux (Gema et al., 2024), recent effort that annotated MMLU questions for label errors. We compute the average number of mislabeled items in FLUID BENCHMARKING and RAN- DOM (IQ") 100) across all LMs and checkpoints, finding that it is nearly two orders of magnitude smaller in the former (0.01) than in the latter (0.75)-in other words, while it takes roughly 100 benchmarking sessions for a mislabeled item to appear with FLUID BENCH- MARKING, one occurs in nearly every session with RANDOM. This suggests that FLUID BENCHMARKING is highly effective at avoiding mislabeled items.
FLUID BENCHMARKING Adapts Items to Language Model Capability. To test whether item selection indeed dynamically adapts to the capability level of given LM, we analyze how item selection changes as an LM gets better over the course of pretraining. Figure visualizes the items selected for FLUID BENCHMARKING (IQ*l evaluated on HellaSwag. We observe a substantial shift in the selected items: initially, items very easy, but they get gradually more difficult the LM improves.
50) with OLMo1-7B
8

---

Published as a conference paper at COLM 2025
40
30
Dam
Chedkpoint
Figure 3: FLUID BENCHMARKING of OLMo1-7B (HellaSwag/5h items). The figure shows items (stacked along y-axis) selected for FLUID BENCHMARKING as a function of different checkpoints. Items are ordered by difficulty b,. Items selected for FLUID BENCHMARKING are colored by time of selection; brighter colors reflect earlier appearance during evaluation. The bright line close to O represents the first item, which is always the same. Depending on how the LM responds, the next item either easier (incorrect response, see first few checkpoints) more difficult (correct response, checkpoints after 11).
Setiaated asttty
27
0.P
2.6
0:11
0000%
15.5%
91.5
55.0%
1000%
7006
81.05
65.0
43.0%
95.05
10005.
progroo
(a) RANDOM
(b) FLUID BENCHMARKING
Figure 4: Training curves of OLMo2-7B (HellaSwag/ 500 items) with RANDOM (a) and FLUID BENCHMARKING (b). The figures plot the final 30% training. While performance in accuracy space shows meaningful improvement (a), performance in ability space continues to provide a clear learning signal through the end of training (b).
FLUID BENCHMARKING Delays Onset of Benchmark Saturation. To test whether FLUID BENCHMARKING indeed delays the onset of benchmark saturation, we focus on HellaSwag ((0") 500). Figure 4a shows OLMo2-7B's performance during the final 30% of the training run, measured with RANDOM. Performance is already high by the 70% mark and does not show a consistent upward trend thereafter, instead fluctuating around the same level. By contrast, with FLUID BENCHMARKING (see Figure 4b), performance continues to improve steadily through the end of training, suggesting that FLUID BENCHMARKING effectively mitigates early benchmark saturation. This difference is captured by our measure of saturation: for the entire training run, the monotonicity of the HellaSwag curve 0.91 for RANDOM, compared to 0.99 for ULD BENCHMARKING.
Dynamic Stopping. A further advantage of FLUID BENCHMARKING is its support for dynamic stopping.
In Figure 5, we demonstrate this with OLMo1-7B and HellaSwag, where we use the standard error of the ability estimate as the stopping criterion (Magis et al.,
Nivetvve of teme (0")
80-
2017). Specifically, we terminate the evaluation once the standard error falls below the average ability gap between two rank-adjacent LMs on the Open LLM Leaderboard. The number of items required to reach this precision varies substantially over training, from around 20 at the beginning to over 80 midway, indicating that the common practice of using a fixed number of evaluation items is suboptimal.
20
Chedipoint
Figure 5: FLUID BENCHMARKING with dynamic stopping on OLMol- 7B 7B/HellaSwag (see text for details).

---

Published as a conference paper at COLM 2025
The False False Promise of Item Response Theory. Madaan et al. (2024) criticized IRT- based benchmark refinement methods for increasing variance, speaking of "false promise of item response theory" for LMs. Our findings contextualize this in crucial ways. On the one hand, we confirm Madaan et al. (2024)'s observation that IRT-based methods (Polo et al., 2024; Kipnis et al., 2025) increase step-to-step variance. On the other hand, our results demonstrate that the issue is not intrinsic to IRT itself, but rather arises from the fact that prior IRT-based methods have not fully leveraged central strength of IRT: dynamically adapting items to the LM's capability. We find that exploiting this potential substantially reduces variance compared to accuracy-based evaluations.
Extension to Other Settings. While we focus LM evaluation during pretraining in this paper, where efficiency is especially critical due to high computational costs and the need for frequent in-loop evaluations, FLUID BENCHMARKING is not inherently limited to this phase and holds potential value for posttraining as well. Furthermore, FLUID BENCHMARKING is readily extendable to other languages and modalities, provided that evaluation results are available to fit an IRT model. For example, applying FLUID BENCHMARKING to vision- language models could leverage leaderboards such as VHELM (Lee et al., 2024).
Generalization Beyond Train Language Models. While IRT ability estimates are not inherently upper bounded by the abilities of the train LMs (i.e., the LMs used to estimate having stable and
item parameters), the utility of FLUID BENCHMARKING still depends up-to-date IRT models, especially given the rapid pace of LM development. Consider the subset of benchmark items that were not answered correctly by any train LM. These items are effectively assigned the same maximum difficulty. If we conduct FLUID BENCHMARKING with new LM that is better than any train LM, evaluation will quickly move to those most difficult items. However, fixed IRT model cannot distinguish finer levels of difficulty among them. Therefore, IRT models used for FLUID BENCHMARKING should be regularly updated with fresh evidence. We hope that the IRT models released as part of this paper can serve as a starting point for such an extensible reference standard.
7 Related Work
Our study adds to the growing body of work on benchmark refinement (see $2 for details). Besides providing formal definition of this emerging field, we introduce a method that improves benchmarking across multiple dimensions.
Prior work has used IRT models in natural language processing (Lalor et al., 2016; 2018; 2019; Lalor & Yu, 2020; Rodriguez et al., 2021; Vania et al., 2021; Rodriguez et al., 2022; Lalor et al., 2024). Recently, there have been several attempt to use IRT in the context of benchmark refinement, to improve efficiency (Polo et al., 2024; Kipnis et al., 2025) and mitigate benchmark saturation (Paech, 2024). Our work differs by considering wider set of criteria and focusing evaluation during pretraining; we also show that static benchmarks forego the full potential of IRT, which lies in the possibility of adaptive testing.
So far, uses of adaptive testing in natural language processing have been confined to improving the cold start problem (Rodriguez et al., 2021).
8 Conclusion
In this work, we unify disparate lines of research to introduce the general problem of benchmark refinement. We define four key dimensions along which benchmark refinement methods should be evaluated: efficiency, validity, variance, and saturation. We introduce FLUID BENCHMARKING, a new benchmarking method that combines item response theory with adaptive testing, improving over prior approaches along all dimensions. In a recent perspective, Zhuang et al. (2024) argued that adaptive testing "will become the new norm in Al model evaluation," but so far large-scale analysis of its potential as a general evaluation method has been missing. Our study is the first to provide this analysis and establishes a foundation for new, exciting research in AT evaluation methodology.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
This material is based upon work supported by the U.S. National Science Foundation (#2113530, #2313998). Any expressed opinions, findings, and conclusions or recommenda- tions are those of the author(s) and do not necessarily reflect the views of the U.S. National Science Foundation. IM was supported by the NSF CSGrad4US Fellowship. PWK was supported by the Singapore National Research Foundation and the National AI Group in the Singapore Ministry of Digital Development and Information under the AT Visiting Professorship Programme (#AIVP-2024-))
and the A12050 program at Schmidt Sciences. Our special thanks go to the members ALLe Oyvind Tafford, and Sarah Wiegreffe for insightful discussions, well as to the reviewers for their valuable feedback.
References
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leader- board. http:///wgggnggggeeee////eee///e 2023.
wwwpodd/pppollllloeee Ilm leaderboard
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Allan Bimbaum. Some latent trait models and their use in inferring an examinee's ability. In Frederic M. Lord and Melvin Novick (eds.), Statistical Theories of Mental Test Scores, pp. 392-479. Addison Addison-Wesley, Reading, MA, 1968.
Allan Birnbaum. Statistical theory for logistic mental test models with a prior distribution of ability. Journal of Mathematical Psychology, 6(2):258-277 1969.
Hua-Hua Chang. Psychometrics behind computerized adaptive testing. Psychometrika, 80 (1):1-20, 2015.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. arXiv:1803.05457 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. 1xii:22100.11666 2021.
Christine DeMars. Item Response Theory. Oxford University Press, Oxford, UK, 2010.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. Are we done with MMLU? arXiv:2406.04127, 2024.
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkin son, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muen- nighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan
Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
11

---

Published as a conference paper at COLM 2025
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. ÓLMES: A standard for language model evaluations. Aktu:22060084445 2024.
Vipul Gupta, Candace Ross, David Pantoja, Rebecca J. Passonneau, Megan Ung, and Adina Williams. Improving model evaluation using SMART filtering of benchmark datasets. ar arXiv:2410.20245 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceedings of the Ninth International Conference on Learning Representations, 2021.
Brian W. Junker, Richard J. Patz, and Nathan M. VanHoudnos. Markov chain Monte Carlo for item response models. In Wim J. van der Linden (ed.), Handbook of Item Response Theory, volume 2, pp. 271-312. CRC Press, Boca Raton, FL, 2016.
Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. metabench: A sparse benchmark of reasoning and knowledge in large language models. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025.
John P. Lalor and Hong Yu. Dynamic data selection for curriculum learning via ability 2020, 2020.
estimation. In Findings of the Association for Compulational Linguistics:
John P. Lalor, Hao Wu, and Hong Yu. Building an evaluation scale using item response theory. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016.
John P. Lalor, Hao Wu, Tsendsuren Munkhdalai, and Hong Yu. Understanding deep learning performance through an examination of test set difficulty: A psychometric case study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
John P. Lalor, Hao Wu, and Hong Yu. Learning latent parameters without human response patterns: Item response theory with artificial crowds. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019.
John P. Lalor, Pedro Rodriguez, João Sedoc, and Jose Hernandez-Orallo. Item response theory for natural language processing. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, 2024.
Tony Lee, Haoqin Tu, Chi H. Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin S. Roberts, Michíhiro Yasunaga, Huaxiu Yao, Cihang Xie, and Percy Liang. VHELM: A holistic evaluation of vision language models. In Proceedings of the 38th Conference on Neural Information Processing Systems, 2024.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Ya- sunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D. Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav San- thanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023.
Thomas I. Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. Are we learning yet? A meta-review of evaluation failures across machine learning. In Proceedings of the 35th Conference on Neural Information Processing Systems, 2021.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics, 2022.
12

---

Published as a conference paper at COLM 2025
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, and Eric P. Xing. LLM360: Towards fully transparent open-source LLMs. XXie:2312.065500 2023.
Zhengzhong Liu, Bowen Tan, Hongyi Wang, Willie Neiswanger, Tianhua Tao, Haonan Li, Fajri Koto, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Liqun Ma, Liping Tang, Nikhil Ranjan, Yonghao Zhuang, Guowei He, Renxi Wang, Mingkai Deng, Robin Algayres, Yuanzhi Li, Zhiqiang Shen, Preslav Nakov, and Eric Xing. LLM360 K2: Building 65B 360-open-source large language model from scratch. drXiv:2501.07124, 2025.
Frederic M. Lord. A Theory of Test Scores. Psychometric Corporation, Richmond, VA, 1952.
Frederic M. Lord. Applications of Item Response Theory to Practical Testing Problems. Lawrence Erlbaum Associates, Hillsdale, NJ, 1980.
Frederic M. Lord. Unbiased estimators of ability parameters, of their variance, and of their parallel-forms reliability. Psychometrika, 48(2):233-22 245, 1983.
Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation 2024.
benchmarks. ar arXi::2406.100
David Magis, Duanli Yan, and Alina A. von Davier. Computerized Adaptive and Multistage Testing with R. Springer, Cham, 2017.
Rob R. Meijer and Michael L. Nering. Computerized adaptive testing: Overview and introduction. Applied Psychological Measurement, 23(3):187-194, 1999.
Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of what art? A call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933-949, 2024.
Prathiba Natesan, Ratna Nandakumar, Tom Minka, and Jonathan D. Rubright. Bayesian prior choice in IRT estimation using MCMC and variational bayes. Frontiers in Psychology, 7, 2016.
Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. MixEval: Deriving wisdom of the crowd from LLM benchmark mixtures. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems, 2024.
Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the 35th Conference on Neural Information Processing Systems, 2021.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Mur- ray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 2 Furious. arXio:2501.00653 2025.
Sam Paech. Creating MAGI: A hard subset of MMLU and AGIEval. 2024.
http:///saamaacc..uusuae /pt httplo///////ee/eg//e/eeae/ haaiagg-----------uubr--eeeeeeee
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Fin-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuen. and Leshem Choshen. Efficient benchmarking (of language models). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024a.
13

---

Published as a conference paper at COLM 2025
Yotam Perlitz, Ariel Gera, Ofir Arviv, Asaf Yehudai, Elron Bandel, Eyal Shnarch, Michal Shmueli-Scheuer, and Leshem Choshen. Do these LLM benchmarks agree? Fixing benchmark evaluation with BenchBench. (rXii:2407.13699 2024b.
Felipe M. Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinyBenchmarks: Evaluating LLMs with fewer examples. In Proceedings of the 41st International Conference on Machine Learning, 2024.
Mark D. Reckase. Multidimensionall Item Response Theory. Springer, New York City, NY, 2009.
Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd-Graber Evaluation examples are not equally informative: How should that change NLP leaderboards? In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021.
Pedro Rodriguez, Phu Mon Htut, John Lalor, and João Sedoc. Clustering examples in multi-dataset benchmarks with item response theory. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, 2022.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Michael Saxon, Ari Holtzman, Peter West, William Yang Wang, and Naomi Saphra. Bench- marks as microscopes: A call for model metrology. In Proceedings of the First Conference on Language Modeling, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Chal- lenging BIG-Bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, 2023.
Wim van der Linden and Ronald K. Hambleton (eds.). Handbook of Modern Item Response Theory. Springer, New York City, NY, 1997.
Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, and Samuel R. Bowman. Comparing test sets with item response theory. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021.
Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do large language model benchmarks test reliability? ar xrXi::2502.03461, 2025.
Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. Anchor points: Benchmarking models with much fewer examples. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguisfics, 2024.
Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang, Top leaderboard ranking top cod- ing proficiency, always? EvoEval: Evolving coding benchmarks via LLM. In Proceedings of the First Conference on Language Modeling, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Zachary A. Pardos, Patrick C. Kyllonen, Jiyun Zu, Qingyang Mao, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Shijin Wang, and Enhong Chen. From static benchmarks to adaptive testing: Psychometrics in AI evaluation. ar arXio:2306.1055 2024.
14

---

Published as a conference paper at COLM 2025
da 00
Item 6,
Succes probainitt plbc
Item 10,
0.6
0.4
0.2
LM ability 9,
Figure 6: Example item characteristic curves. The x-axis shows the ability parameter 8;; the
greater 6, the higher the success probability p(u; = 1). The difficulty parameter bj indicates the value of 0; at which p(u; 1) = 0.5, reflected by the location of the curve (compare 41 vs. 93). The discrimination parameters indicates how sharply p(uij = 1) changes when 8 is close to bj. aj is proportional to the slope of the curve (compare 91 vs. 92). When the curve is flat (i.e., low a,), this implies that even some high-ability LMs failed on this item.
60=0
Fisher informaties ten 10 b.
Itam diserimiiation..
Irem difficulty by
Figure 7: Fisher information of HellaSwag items halfway through the simulated training run, when of = O. The figure corresponds to the distribution obtained by taking a vertical slice through Figure 2 at e = O. In line with Equation 4, Fisher information is highest for items whose difficulty b, is close to 0 It also increases with item discrimination a, effect that is particularly pronounced when by A ef. By contrast, when by far from A higher discrimination has an only modest effect on Fisher information.
A Item Characteristic Curves
We provide example item characteristic curves in Figure 6.
B Fisher Information of HellaSwag Items
For illustrative purposes, Figure shows the Fisher information of HellaSwag items halfway through the simulated training run, when 0f 0.
C Checkpoint Details
We provide details about the selected LM checkpoints. For Amber-6.7B, we select 73 check- points. For OLMo1-7B, we select 83 checkpoints. For OLMo2-7B, we select 94 checkpoints. For Pythia-6.9B, we select 78 checkpoints. For Pythia-2.8B, we select 78 checkpoints. For K2-65B, we select 61 checkpoints. For all LMs, checkpoints are selected to ensure even coverage throughout the entire training run.
15

---

Published as a conference paper at COLM 2025
Benchmark
Measure
Method
ARC
GSM
HS
MMLU
TQA
WG
Validity
RANDOM
21.9
12.9
20.5
12.4
Rank distance
RANDOM IRT
15.9
5.0
13.4
8.2
FLUID BENCHMARKING
14.5
4.5
10.7
4.9
Variance
RANDOM
10.2
22.2
3.8
49.7
18.1
14.6
Total periation
RANDOM IRT
28.9
12.0
20.8
15.1
22.1
FLUID BENCHMARKING
3.3
9.1
2.0
6.3
9.8
5.8
Saturation
RA NDOM
0.75
0.66
0.88
0.51
0.43
0.61
Rack correlation
RANDOM IRT
0.82
0.60
0.88
0.56
0.63
0.76
FLUID BENCHMARKING
0.95
0.86
0.98
0.67
0.71
0.93
Table 3: Comparison against baselines, split by benchmark. ARC: ARC Challenge; GSM:
GSM8K; HS: HellaSwag; TQA: TruthfulQA; WG: WinoGrande.
D Language Model Inclusion Criteria
We used the following criteria when selecting LMs for IRT model training:
We only included pretrained LMs. Finetuned, merged, fused, distilled, or continually pretrained LMs were excluded, as they can lead to clusters of highly similar models, potentially skewing the IRT model.
In the rare cases where an LM appears on the Open LLM Leaderboard with multiple checkpoints, used only the final checkpoint listed.
We excluded LMs trained solely on non- non-English data, but multilingual LMs were included as long as English data were part of their training corpus.
We removed any LMs from the same model family as the test LMs (e.g., OLMo1-1B).
E Item Response Model Details
In the main experiments, we fit separate unidimensional IRT models to each benchmark. Initially, we also experimented with two alternative setups:
We experimented with fitting a single unidimensional IRT model across all benchmarks, following prior work suggesting that one latent trait can capture overall model behavior (Kipnis et al., 2025). However, we found that this substantially reduced construct validity. For example, the performance of Amber-6.7B on TruthfulQA decreases during pretraining (Liu et al., 2023); by contrast, when we evaluated Amber-6.7B using a unidimensional IRT model trained across all benchmarks, the estimated ability increased- the IRT model effectively emphasized TruthfulQA items aligned with general trends, obscuring the fact that Amber-6.7B actually becomes less truthful during pretraining.
We experimented with fitting separate multidimensional IRT models (with two to five latent traits) to each benchmark. These models, however, did not yield consistent improvements in model fit compared the unidimensional IRT models.
Ultimately, fitting separate unidimensional IRT models to each benchmark offered the best trade-off in our experiments. That said, multidimensional IRT models may offer greater advantages in other settings (e.g., when evaluating multimodal models).
F Breakdown of Results by Benchmark and Language Model
Table 3 breaks the comparison against baselines down by benchmark. Table 4 breaks the comparison against baselines down by LM. We examine the ablated baselines here, fixing the number of items per benchmark to 100.
16

---

Published as a conference paper at COLM 2025
Language model
Measure
Method
A-7B
K-65B
O1-7B
O2-7B
P-3B
P-7B
Validity
RANDOM
25.5
5.1
10.7
7.1
23.1
28.1
Rank distance
RANDOM IRT
20.2
5.2
8.0
6.1
8.3
15.3
FLUID BENCHMARKING
19.3
2.1
6.7
8.1
Variance
RANDOM
21.8
14.7
10.2
15.4
35.7
20.9
Total variation 1
RANDOM IRT
16.2
27.0
11.4
12.4
26.1
13.7
FLUID BENCHMARKING
5,5
7.1
5.8
6.8
6.5
4.5
Saturation
R RANDOM
0.47
0.65
0.77
0.63
0.62
Rank correlation
RANDOM IRT
0.66
0.73
0.83
0.63
0.67
0.73
FLUID BENCHMARKING
0.82
0.89
0.91
0.80
0.81
0.87
Table 4: Comparison against baselines, split by LM. A-7B: Amber-7B; K-65B: K2-65B; O1-7B: OLMo1-7B; O2-7B: OLMo2-7B; P-3B: Pythia-2.8B; P-7B: Pythia-6.9B.
G Variance and Saturation Plots
Figure 8 provides more detailed comparison of FLUID BENCHMARKING and RANDOM in terms of variance (Figure Sa) and saturation (Figure 8b). FLUID BENCHMARKING improves RANDOM for almost all combinations of benchmark, subset size, and LM.
H Comparison Against Full-Benchmark Accuracy
We have shown that FLUID BENCHMARKING improves evaluation quality in terms of validity, variance, and saturation, compared against alternative evaluation methods using the same number of items. Do these advantages persist when evaluation cost not concern (i.e., when it is feasible to evaluate on the full set of benchmark items)? To test this, we compare FLUID BENCHMARKING (IQ"l = 500) with full-benchmark accuracy, using the same LMs and benchmarks in our main experiments (see $4).
We find that full-benchmark accuracy performs worse than FLUID BENCHMARKING across all three evaluation dimensions, despite using substantially more items. This holds for validity (9.1 vs. 8.3 for FLUID BENCHMARKING MARKING), and saturation (0.85 vs. 0.88 for FLUID BENCHMARKING). Notably, even FLUID BENCHMARKING with only 50 items outperforms full-benchmark accuracy on all three dimensions (cf. Table 2). These results suggest that FLUID BENCHMARKING can improve evaluation quality even in settings where efficiency is not limiting factor.
variance (23.8 vs. 4.9 for FLUID BENCH-
17

---

Published as a conference paper at COLM 2025
ARcCiss
ARCChalknge
COMBEE
colvee:
Hill-wang
064
NDALL
Trahhlag
WiraGorde
Rume BescweAANND
60-
86-
Arthme/
Arbone//
304
32438
Kh-sa
OLNe5575
01M6377
Pydfeezdt 2db
02
Pytheu--9 4-8
(a) Variance (lower is better)
Saturation (higher is better)
Figure 8: Variance and saturation results. The figure shows pairwise comparisons measuring the total variation (a) and monotonicity (b) of training curves based on RANDOM and FLUID BENCHMARKING For variance, lower total variation is better. For saturation, high monotonicity is better, as indicates that increased pretraining consistently yields better performance, suggesting that the benchmark has not yet saturated. Thus, for variance, points in the lower right triangle indicate that FLUID BENCHMA RRKING better than RANDOM, and for saturation, points in the upper left triangle indicate that FLUID BENCHMARKING is better than RANDOM. FLUID BENCHMARKING improves on RANDOM for almost all combinations of benchmark, subset size, and LM.
18


Reviews:
[{"review_id": "3YhmKw567k", "reviewer": "Reviewer_JcBQ", "summary": "FLUID BENCHMARKING​ introduces a novel approach to evaluating language models (LMs) by combining ​Item Response Theory (IRT)​​ and ​dynamic item selection​ to address key challenges in LM benchmarking:\n\n​High costs​ of comprehensive evaluations.\n​Poor validity​ due to misaligned benchmarks and labeling errors.\n​High variance​ in evaluation results.\n​Benchmark saturation​ (models scoring near maximum).\n\nThe method dynamically selects evaluation items based on an LM’s capability level, improving efficiency (fewer items needed), validity (better rank prediction across benchmarks), and stability (lower variance), while delaying saturation. Experiments on six benchmarks (e.g., MMLU, HellaSwag) and six LMs (e.g., Pythia, OLMo) show consistent improvements over static and IRT-based baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Addresses multiple benchmarking issues (efficiency, validity, variance, saturation) in a unified framework.\n- Outperforms baselines across all metrics\n- Leverages well-established IRT and adaptive testing principles, ensuring methodological rigor.", "reasons_to_reject": "- Effectiveness on non-English or multimodal benchmarks remains unverified.", "questions_to_authors": "- How does FLUID BENCHMARKING guard against inadvertently overfitting to the \"known\" capability trajectory of LMs during pretraining? Could dynamic item selection bias the evaluation toward reinforcing existing LM weaknesses/strengths rather than holistically measuring progress?\n- Traditional IRT assumes unidimensional ability (a single latent skill). How valid is this assumption for multimodal LMs or models with heterogeneous capabilities (e.g., strong in reasoning but weak in factual recall)? Could multidimensional IRT better capture LM capabilities?\n- FLUID BENCHMARKING relies on historical LM evaluation data to estimate item parameters. How would the method perform for evaluating novel LM architectures or tasks with no prior data? Is there a risk of perpetuating bias toward existing paradigms?", "extra_scores": {}}, {"review_id": "PEQRLmCPMO", "reviewer": "Reviewer_5T1x", "summary": "* The authors propose Fluid Benchmarking, a new LLM evaluation framework inspired by Item Response Theory (IRT).\n* The method combines a two-parameter logistic (2PL) IRT model—capturing item difficulty and discrimination—with a novel AGGREGATE function (latent ability estimation) and a SELECTION strategy (adaptive item selection) to improve benchmarking.\n* They demonstrate that Fluid Benchmarking outperforms both traditional random sampling and existing IRT-based methods across four key dimensions: efficiency, validity, variance, and saturation.\n* Notably, dynamic selection improves precision and reduces variance, while IRT-based scoring improves validity. For example, the approach can detect mislabeled via item-level discremenation.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed Fluid Benchmarking framework is grounded in a well-established theoretical foundation—Item Response Theory (IRT), specifically the 2-parameter logistic (2PL) model, which has decades of use in psychometrics.\n* It addresses several critical challenges in modern LLM benchmarking, including efficiency, validity, variance, and saturation, which are often treated separately in prior work.\n* Empirically, the framework outperforms both traditional baselines and recent IRT-based methods across all four dimensions on multiple benchmarks and model families.", "reasons_to_reject": "* The current scope of Fluid Benchmarking is limited to pretraining evaluation. While pretraining is a natural setting for measuring dynamic changes in model ability and efficiency is important, the method itself is not inherently restricted to this phase. Extending the framework to post-training settings (e.g., instruction tuning, domain adaptation, or RLHF) would broaden its applicability and strengthen its practical impact.\n* Since Fluid Benchmarking relies on latent ability estimation using IRT, the quality and stability of its outputs depend heavily on the LLMs and benchmarks used to fit the IRT model. Although the Open LLM Leaderboard is a good choice, the rapidly evolving LLM landscape means that item parameters and ability scores may become outdated quickly. To ensure wider adoption and consistency, the community would benefit from a standardized and regularly updated IRT reference model.", "questions_to_authors": "* Figure 2 (upper panel): While it’s clear that the figure illustrates how Fisher information varies over training progress, the distinction between Fisher information, item discrimination, and their roles in item selection may be confusing at first glance. A brief clarification in the caption or main text would help readers unfamiliar with IRT.\n* Dynamic stopping criterion: Currently, the number of evaluation items (i.e., benchmark size) must be specified ahead of time. It would be valuable to explore adaptive stopping rules—for example, stopping when the ability estimate converges or when additional items provide diminishing information gain.\n* Multi-trait benchmarks: Many real-world benchmarks implicitly test multiple capabilities (e.g., reasoning, commonsense, factual recall). Can Fluid Benchmarking be extended to multi-dimensional IRT (MIRT) models to disentangle and score these traits separately? Alternatively, could single benchmarks be decomposed into interpretable skill dimensions?", "extra_scores": {}}, {"review_id": "yuA45xDUbb", "reviewer": "Reviewer_22Eb", "summary": "The central idea in this paper is to use item response theory (IRT) and estimate model capability and item difficulty/discrimination at the same time, all of which are latent variables within the IRT model. The estimated latent capability is then used as the number representing model's performance rather than accuracy aggregated over correct/incorrect responses on a benchmark. The author also show that this method can be used to dynamically select items from a benchmark for evaluation, with applications to 1) track/visualize the model's performance over pre-training, 2) reduce the number of items to estimate a model's capability, and 3) avoid \"bad\" or not-so-informative items, such as mislabeled ones, in a benchmark.\n\nThe way it works is essentially 1) select a number of models, 2) select a benchmark, 3) run each model on this benchmark to get its binary response to each item, and 4) fit an IRT model on this response matrix. For dynamic item selection, the method works by selecting the most \"informative\" item at each step that maximizes the fisher information between the item and the current estimation of the model's capability.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a really nice application of a very well known concept in education to LLM benchmarking. The idea itself is intuitive and works well, with substantive empirical evidence to back it. Also nicely written and easy to follow. Overall a good paper.", "reasons_to_reject": "1. From my understanding, the cost of benchmarking is trivial compared to the cost of pre-training. It is not super clear to me what the benefits of benchmark refinement are.\n\n2. This work assumes the item parameters are fixed after learning. These parameters are estimated based on the capabilities of the \"students\" solving these problems, where each \"student\" is a model in this case and the problems are items in a benchmark. Therefore, my understanding (I could be wrong) is that the estimated model capability is upper bounded by the best model used to estimate these parameters. Therefore, if a better model is being evaluated, whose \"ability\" is \"better\" than all the models used to compute the item parameters, then we may not get a very good estimate of this model's capability. This is concerning because 1) the models used to compute the item parameters are small and not representative of the very best in the open-source community, and 2) models are always improving. As a result, when using this method to benchmark models that are better than those used to estimate IRT parameters, the estimation may not be very accurate.", "questions_to_authors": "1. What are the benefits of benchmark refinement if cost is not a concern (compared to the cost of pre-training) and we can easily afford to run the evaluation on all items to estimate the model's capability?\n\n2. What happens if you are allowed to run each model on all items of a benchmark? Do the proposed method and the baselines converge? How does the proposed method compares to if you were to represent the model's capability/rank in terms of accuracy, when running the evaluation on all items of a benchmark?\n\n3. What happens if you use the already estimated item parameters to benchmark much more powerful models, such as the latest models such as R1, Qwen 3, Llama 4, and perhaps closed-source ones? How does the capability estimation from the latent IRT parameter compared to if you were to use accuracy aggregated over all items on a given benchmark: e.g. do they correlate well?", "extra_scores": {}}]

Author rebuttal:

Meta-review: - **Summary**: This paper proposes Fluid Benchmarking that dynamically selects evaluation items based on Item Response Theory (IRT) to address current challenges in LLM Benchmarking: efficiency, validity, variance, and saturation. This paper empirically demonstrated by showing consistent improvements over static and IRT-based baselines.
- **Review summary**: All reviewers leaned toward acceptance, praising the critical problem and the empirical results.
- **Strengths**: 
(1) this paper addresses an important benchmarking issues;
(2) the idea is intuitive and grounded in theoretical foundation;
(3) it is empirically demonstrated; 
(4) the paper is well-written.
- **Weakness**:
(1) the validity for models with heterogeneous capabilities -- additional analysis and discussion is needed;
(2) lack of full-benchmark evaluation;
(3) lack of discussion on generalization beyond train models and no prior data settings.
- **Meta-review decision**:
I recommend acceptance, as the paper deals a critical problem with proposing empirical solid methods and the reviewers’ concerns are addressable.
Decision: Accept