Don’t Pass@k: A Bayesian Framework for Large Language Model Evaluation
Authors: Mohsen Hariri
Amirhossein Samandar
Michael Hinczewski
Vipin Chaudhary
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Evaluation Metrics
Bayesian Methods
Uncertainty Quantification
Model Ranking
Reasoning
Statistical Significance
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a Bayesian evaluation framework that replaces Pass@k with stable, uncertainty-aware metrics for reliable and compute efficient LLM evaluation.

Abstract: Pass@$k$ is widely used to report performance for LLM reasoning, but it often yields unstable, misleading rankings, especially when the number of trials (samples) is limited and compute is constrained. We present a principled Bayesian evaluation framework that replaces Pass@$k$ and average accuracy over $N$ trials (avg@$N$) with posterior estimates of a model's underlying success probability and credible intervals, yielding stable rankings and a transparent decision rule for differences. Evaluation outcomes are modeled as categorical (not just 0/1) with a Dirichlet prior, giving closed-form expressions for the posterior mean and uncertainty of any weighted rubric and enabling the use of prior evidence when appropriate. Theoretically, under a uniform prior, the Bayesian posterior mean is order-equivalent to average accuracy (Pass@$1$), explaining its empirical robustness while adding principled uncertainty. Empirically, in simulations with known ground-truth success rates and on AIME'24/'25, HMMT, and BrUMO, the Bayesian/avg procedure achieves faster convergence and greater rank stability than Pass@$k$ and recent variants, enabling reliable comparisons at far smaller sample counts. The framework clarifies when observed gaps are statistically meaningful (non-overlapping credible intervals) versus noise, and it naturally extends to graded, rubric-based evaluations. Together, these results recommend replacing Pass@$k$ for LLM evaluation and ranking with a posterior-based, compute-efficient protocol that unifies binary and non-binary evaluation while making uncertainty explicit. Source code is available at https://github.com/mohsenhariri/scorio.

Published as a conference paper at ICLR 2026
DON'T PASS@K: A BAYESIAN FRAMEWORK FOR LARGE LANGUAGE MODEL EVALUATION
Mohsen Hariri," Amirhossein Samandar; Michael Hinczewski," Vipin Chaudhary
"Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, OH, USA DDpartment of Physics, Case Western Reserve University, Cleveland, OH, USA (mohsen.c hariri, amir amirhossein, samandar, mxh605, vipin) @case.
ABSTRACT
Pass@k is widely used to report the reasoning performance of LLMs, but it often produces unsta- ble and potentially misleading rankings, especially when the number of trials (samples) is limited and computational resources are constrained. We present a principled Bayesian evaluation frame- work that replaces Pass@k and average accuracy over N trials (avg@NV with posterior estimates of a model's underlying success probability and credible intervals, yielding stable rankings and transparent decision rule for differences. Evaluation outcomes are modeled as categorical (not just 0/1) with a Dirichlet prior, giving closed-form expressions for the posterior mean and uncertainty of any weighted rubric and enabling the use of prior evidence when appropriate. Theoretically, un- der uniform prior, the Bayesian posterior mean is order-equivalent explaining its empirical robustness while adding principled uncertainty. Empirically, in simula- tions with known ground-truth success rates and on AIME' AIME'24/225 24'25,, HMMT'25, and BrUMO'25, the Bayesian/avg procedure achieves faster convergence and greater rank stability than Pass @k and re- cent variants, enabling reliable comparisons far smaller sample counts. The framework clarifies when observed gaps are statistically meaningful overlllerlg credible intervals) versus noise, and it naturally extends to graded, rubric-based evaluations. Together, these results recommend re- placing Pass @k: for LLM evaluation and ranking with posterior-based, compute-efficient protocol that unifies binary and non-binary evaluation while making uncertainty explicit. Source code
average accuracy (Pass @1), )
available at Scorio
1 INTRODUCTION
Large language models (LLMs) have moved rapidly from research artifacts to everyday infrastructure (I 2) Students use them for homework and exam preparation; developers rely on them for code synthesis and refactoring 3 analysts and clinicians use them for decision support; and agents built atop LLMs are increasingly embedded in workflows across industry and government. This demand has catalyzed unprecedented investment: specialized chips, datacenters, and startups dedicated to LLM training, serving, and tooling (4). As deployment accelerates, trust, oversight, and comparability become central: how we evaluate LLMs directly shapes which models are adopted, what progress is declared, and how resources are allocated (5 6 2 10.11) I]).
Evaluation, however, remains the weakest link in the LLM pipeline. Alongside advances in model efficiency and compression[12 13 14
17 18), training and fine-tuning (PEFT/LORA, Rl--from-human--eeehhaed
(19; 20:
I.. and inference/decoddig (sampling strategies, caching, efficient attention) (2I (2). the community still leans simple, yet flawed, success rates and Pass @k-style metrics to summarize capabilities (23). These practices are convenient but fragile. On small or costly benchmarks (e.g., math reasoning sets with only tens of problems such AIME) (24 25), Pass @ak or single-run accuracy often produce unstable rankings (26), are sensitive to decoding choices and seed effects (2 (27), and provide little guidance on whether observed gaps are meaningful or mere noise 29),.. Averaging across multiple runs ("avg@ helps but is compute-hungry (30), offers no unified way handle graded/rubric outcomes, and lacks a principled decision rule for significance (28) 32).
This paper takes different approach: treat evaluation itself posterior-based framework that replaces Pass@l and avg@N with estimates of model's underlying success prob- abilities and associated uncertainty (33). Outcomes are modeled as categorical (34) rather than purely binary: each item can yield correct, partially correct, formatting-error, refusal, or rubric-defined levels. A Dirichlet prior over these
statistical inference problem. We introduce
See Appendix G for API documentation.

---

Bayesian
pass @2
pass @4
pass@
Beakings ADMES5
Ovenc teeking
0.020
24ME
25
ggicen Mich
No Convergence
Crepetire
0.010
EXAONNE-40
CT Needed
0.2
0.005
0.000
0.0
No Convergence
0.150
HMMT225
BrUMO'25
Oweah Tiankep
0.125
-4.22
69.5
Operhaattrd
0.100
0.075
0,050
Openitantert
EXACNE-40
0.025
Neraaree
0.000
0.00
AM
20
40
60 80
20
Convergence@n
(a)
Convergence
(b)
Figure I: a) Probability mass functions (PMFs) of convergence@n,, the number of trials n above which ranking of LLM models consistently matches the ranking using Nmx = -80 trials. Eleven LLM models (listed on the right) and four math-reasoning datasets used-AIMER
AIME'25, HMMT and BrUMO' BrUMO'25-com 25-ccompriing Pass Pass@2/4/8 against (Bayes@N). Each PMF is estimated by bootstrapping with 10" samples over the Neax trials; vertical lines indicate the mean of each convergence distribution. On AIME the Pass family frequently fails to converge, whereas Bayes@N converges. On HMMT and BrUMO, Pass methods converge more slowly (mean required trials z 69.5 and = 48.5) than Bayes@N (= 44.2 and = 27.1). respectively. Right: Example competition-style ranking from single bootstrap replicate, highlighting the mean convergence for AIME'25 and BrUM0'25. Per task rankings, including worst-case replicates, are in Appendix (Figure T0
Bayesian posterior evaluation
categories yields closed-form posterior means and credible intervals for any weighted rubric, allowing the evaluator to report both point estimate and principled uncertainty with negligible overhead. In the binary special case under a uniform prior, its posterior mean is order-equivalent to average accuracy, explaining the empirical robustness of avg@1 N while making uncertainty explicit.
The framework addresses four persistent pain points. Convergence: as shown in Figure 1 we ideally want methods that can converge the true underlying ranking with the smallest number of trials, but different approaches can have significantly different convergence speeds. Credible intervals: simple, transparent rule-do not declare a winner when intervals overlap-reducee leaderboard churn and over-interpretation of tiny gaps by introducing a compute- efficient credible interval (CI). Updates are analytic; one can monitor interval widths online, and allocate additional trials only when needed (no Monte Carlo/bootstrapp simulations are required for CI estimation). @ Categorical evalu- ation: our approach unifies binary and non-binary evaluation. Graded rubrics are natural in this framework, so one can evaluate step-by-step reasoning, partial credit, or judge categories without ad hoc aggregation. Prior information: can incorporate prior evidence when appropriate (c.g., reuse of stable rubric distributions across closely related tasks or versions).
We validate the approach in two settings: In controlled simulations with known ground-truth success rates, the pos- terior procedure converges to correct rankings with fewer samples than Pass @k and recent variants, and flags when ties statistically unresolved. On real math-reasoning benchmarks (AIME'247 24/225 (24) 25). HMMT (35), and BrUMO'25 (36)-derived sets), we observe the same pattern: the posterior method achieves greater rank stability at far smaller sample counts than Pass @k, while clarifying when differences meaningful versus noise. Practically, this yields a computationally efficient protocol that is easy to implement and audit.
We summarize our contributions follows:
unified Bayesian evaluation framework. We model per-item outcomes as categorical with a Dirichlet prior, yielding closed-form posterior means and credible intervals for any weighted rubric, with binary eval- uation as a special case, This unifies 0/1 and graded evaluations and supports reuse of prior evidence when justified.

---

Published as a conference paper at ICLR 2026
A compute-efficient, interval-aware protocol. We provide a simple recipe: report posterior means with credible intervals; only declare differences when intervals do not overlap; adaptively allocate additional sam- ples until intervals meet pre-specified widths. This protocol naturally supports sequential/online evaluation. Empirical evidence on simulations and math benchmarks. On synthetic data with known ground truth and on 24M2'24/25, 25, HMMT and BrUMO'2s 25 datasets, method achieves faster convergence and greater rank stability than Pass@k and recent variants, enabling reliable comparisons with far fewer samples.
2 BAYESIAN FRAMEWORK FOR EVALUATING LLM PERFORMANCE
2.1 BACKGROUND: THE PASS @k: METRIC AND ITS LIMITATIONS
Evaluation metrics for LLMs aim to quantify performance on tasks like reasoning or programming. but they often struggle to provide reliable relative rankings across models. Pass @k for instance, estimates the probability of at least one correct answer within model attempts (see Appendix H for details). While convenient, this metric exhibits high variance (37), particularly when & approaches the total number of trials, NN, resulting in unstable rankings (23), Small fluctuations correctness can distort comparisons, particularly in benchmarks with few problems or limited computational resources, raising doubts about its suitability for differentiating model capabilities. If metric cannot consistently distinguish stronger models from weaker ones, its value as a benchmarking tool is undermined (26).
Estimating uncertainty in Pass @k scores is also challenging, it lacks closed-form expressions for variance, rely- ing instead on computationally intensive approximations like bootstrapping. A truly effective metric should yield reliable performance rankings with a minimal number of trials, prioritizing both accuracy and efficiency in resource- constrained environments. To address these limitations, we propose a Bayesian evaluation framework that provides more stable estimates of performance, incorporates uncertainty, and facilitates robust relative comparisons across models 33 38 39).
2.2 RESULTS MATRIX
Consider a results matrix R for an LLM evaluated on test set comprising M questions. Due to the stochastic nature of LLM sampling, responses may vary across independent trials, so we run the LLM N times per question. The outcomes are captured in the M N matrix R, where element Rai represents the score in the ith trial for the oth question. This score an integer ranging from a maximum value C, reflecting a rating system with C + categories. In the binary case (C = 1), 0 indicates an incorrect answer and 1 correct one, though we accommodate more nuanced rubrics generally.
2.3 WEIGHTED PERFORMANCE METRIC
For the oth question, a 1,.... M, there an underlying probability Tox that the LLM's answer falls the &th category. We denote n. as the (C' + 1)-dimensio vector with elements Taks k =, .,..,, C. If all л, were known, we could calculate a desired performance metric म as a weighted average over these probabilities:
M
W Wk
M
M
where w is a (C + 1) dimeensioall vector of constant weights. For example, if wy = k. then म represents the average category label. In the case where C 1, this average corresponds the mean probability of correct answer over the entire test set. However, we allow for general choice of w to accommodate a wide range of possible metrics.
2.4 BAYESIAN ESTIMATOR AND UNCERTAINTY FOR THE PERFORMANCE METRIC
In principle, we could estimate Ta by running an arbitrarily large number of trials with the LLM, yielding an accurate estimate of म. However, we are typically constrained to small N due to limited computational resources. Our goal is to develop Bayesian approach to estimate म and its associated uncertainty given a finite N. The first step is to construct P(na R.), the posterior probability of π. given the ath row of the matrix R, denoted R.- This posterior depends The
the data in R. and chosen prior distribution P(TT.) ) for the unknown underlying probability vector prior could be uniform (assuming no prior information) or incorporate previously gathered evidence about the L.LM's performance. The Bayesian framework focuses on two quantities: the first is the mean of over the joint posterior for

---

Published as a conference paper at ICLR 2026
all questions, which we denote as p RR. This is a Bayesian optimal estimator, minimizing the quadratic loss function (( BR. (# R)) - *) over all possible estimators (R), where the expectation value is over all possible To and realizations of R (40). The second quantity is the variance (f)), which quantifies the uncertainty of the
I estimate. Both (t(t) and (H) have exact closed-form expressions, derived in Appendix A and can be simply
calculated for any R using Algorithm
Algorithm 1 L.MM performance evaluation using the Bayes@N framework.
function EVALJUATEEPRRFORMMNNGGG
[R". w)
input: M x N matrix R of results, with each element Rai = -....., C weight vector w (Wo,.. ................................ wc) defining performance metric π optional input: M x D matrix Rt of results for prior; otherwise D
output: performance metric estimate and associated uncertainty o
T = 1 + D N for a = 1 to M do
b Tally results in R and R
for k = 0 to C do
Rak = Li=1 Ra.
na = 1 + Ei=l dg,
Vak nak + Pakk
end for
end for
f Wp
MT La=1 Dj= [wj(W]] -wo)
C
(2i= + (w: wo) ) 1/2
MI[++I 2o=1
M
2j=0 44 (wo wo)2
C
return 14, 0
end function
2.5 USING UNCERTAINTY ESTIMATES TO DECIDE SIGNIFICANCE OF PERFORMANCE DIFFERENCES
U(R) In general, the expressions for μ R) and (R) are valid for any M and N, and do not rely asymptotic arguments like the central limit theorem (CLT). However, there are useful simplifications that occur in specific limiting cases. For example as the size of the test set M becomes large, we can derive not just the moments of the posterior distribution for न,, but also its shape, which becomes approximately Gaussian: P( R) ~ (()(R), (R)). This allows us to assess whether two methods exhibit a statistically significant performance difference. Consider results matrices R and R' from two approaches, with corresponding means Jr, jı' and standard deviations a'. The distribution of the performance difference A = के N' is a convolution of the individual posteriors, yielding another normal distribution: P(AT|R, R') N(ū, where the mean of the difference is confidence in the ranking of the two methods, we need to determine the probability
μ n², and the standard deviation is ā
VO² (a')2. To determine that sign(Ar) sign( - n²). This can be done by calculating the absolute z-score, = |1 1 /02/ + (o')² The probability that the ranking based on μ and pí' is correct (the ranking confidence p) is given by p = (1/2)(1 + er)( For example 1.645 corresponds to p = 0.95.
2.6 EQUIVALENCE OF BAYESIAN AND AVERAGE RANKINGS FOR UNIFORM PRIOR
In the results below, will denote ranking based on the Bayesian estimator μ with uniform prior Bayes @N Because j is related to a naive weighted average accuracy via a positive affine transformation, it turns out the ranking based on the average, denoted avg @N, is identical to Bayes@N (for the detailed proof, see Appendix B). In the large-trial limit N - oo, the value of f approaches the average, as expected, but the ranking equivalence holds at all finite N. This relationship also extends to uncertainty quantification, where the standard deviation of the average relates to the Bayesian standard deviation a by a scaling factor, providing a concrete method to compute uncertainty in the average without relying on the Central Limit Theorem. This is particularly advantageous in small-sample regimes common in LLM evaluations, where CLT-based methods often underestimate uncertainty and produce invalid intervals (e.g., extending beyond [0.1] or collapsing to zero) 4I.. As highlighted by (41), Bayesian approaches with uniform priors (e.g., 1)t in the binary case) yield well-calibrated credible intervals even for datasets with fewer than few hundred datapoints, outperforming CLT approximations in coverage and handling complex structures like clustered data.
4

---

Published as a conference paper at ICLR 2026
2.7 GOLD STANDARD FOR RANKING
Strictly speaking, the underlying true ranking of LLMS for a particular performance metric स is unknown, because it would require determining the infinite trial limit, lim nt, for each L.M.. In practice, we have to settle for an approximation to 9, calculated at some large but finite value N = Nax (for example Nax = 80 in our LLM is the same as the ranking based on avg@ Nmax as our "gold standard" or reference ranking (42). In other words, rankings using smaller N will be compared to this gold standard to assess their accuracy.
experiments). Specifically, we use Bayes@Nmax-
For this comparison, we employ Kendall's T, nonparametric correlation coefficient that measures ordinal agreement between two rankings by comparing the number of concordant and discordant pairs of models. The coefficient ranges from -1 (perfect inversion) to +1 (perfect agreement), with 0 indicating no association. We specifically use the variant, which properly accounts for ties in the rankings (e.g., g., the intentional tie in our simulation below), ensuring that equivalences do not artificially inflate the correlation. See Appendix for further discussion and formal definitions.
To validate our claims about the gold standard as Bayes@N max. specifically to determine which evaluation meth- ods converge to the true ranking, we conduct a simulation using biased coins as metaphor for LLMs. In this setup, we already know the underlying performance distribution (the success probabilities Tya for each question), allowing to establish a known ground truth म. We generate 11 sets of these 30 probabilities, with मे values of (0.2332, 0.2545, 0.3604, 0.3642, 0.3642, 0.4466, 0.5418, 0.5276, 0.608, 0.6213, 0.7327 representing different LLMSs (note the tie at 0.3642 to test handling of equivalent performances). We run experiments for M 30 questions, where each LLM "answers" all the questions in each trial according to its success probabilities Ta1. Panel (a) of Figure 2 shows results without bootstrapping: we generate 1000 independent R matrices, each with 80 trials; for each step in the number of trials (from N - to 80), we compute scores using Pass @k (k = 2, k = 4. and k = 8 with an unbiased estimator Equation (2I,. Bayes@N, Pass^k (Equation (22). G-Pass K; (Equation (33 with F 0.5), and mG-
Pass @ @(Equation (24)) then derive rankings and compare them to the gold standard using Kendall's T as a measure of rank correlation (where T = 1 indicates perfect alignment with the gold standard), and report the average r over the 1000 R matrices. Note that we do not explicitly show average accuracy avg @ N because it is equivalent to Bayes@N, as discussed in Section 2.6] In practice, we are computationally limited to a small number of trials per question. To examine what happens with only N = 80 trials, we apply two methods of bootstrapping with replacement to the R matrix, allowing us to estimate how results differ from the ideal case with a large number of independent R matrices (panel a). For both methods, a single R matrix. Panels (b) and (c) of Figure 2 Jillustrate this using two bootstrapping schemes. In the first scheme (panel b, column-wise bootstrapping), we resample trial indices; in the second (panel c, row-wise bootstrapping), we resample answers independently for each question. In both cases, the resulting bootstrap replicates are used to recom- pute evaluation scores, rankings, and values, which are then averaged to produce smoothed convergence curves. The two bootstrapping approaches yield nearly identical behavior, and both closely match the baseline in panel (a). This demonstrates that the r convergence behavior is robust and not sensitive to the ordering of answers in either the rows or columns of R. Though in our L.LM mimic simulations, we do not have to use bootstrapping (since we can easily generate an arbitrarily large number of It matrices), in actual LLM experiments, we have limited trial data, and these results show that bootstrapping provides viable way of estimating statistical properties like convergence.
generate 10,000 bootstrap replicates for each of the N 1 to 80 trials, derived from
As seen in Figure 2 Bayes@N begins with relatively high agreement with the gold standard and converges much faster to r = 1 than Pass @ i and its variants, which suffer from greater variance and bias at small N. All methods eventually converge to the same ranking, but their rates of convergence differ substantially. This makes the convergence rate a crucial factor when choosing between different LLM evaluation methods. While we focus here uniform priors, even faster convergence can be achieved by incorporating information from correlated models (e.g.. base, older, or quantized versions) through non-uniform priors. We provide a preliminary demonstration of this potential using synthetic data in Appendix C
2.7.1 RANKING WITH UNCERTAINTY
In Section 2.5 we described how uncertainty estimates from the Bayesian approach can be used to evaluate the relative performance of two models. Here, we extend these ideas to incorporate uncertainty into the ranking of multiple models. We do this via our biased-coin LLM mimics, which we denote LLM& 3 for = 1,..., 11, described in the previous section. To incorporate a chosen credible interval in the ranking, we order their μ values from highest to lowest, choose
the appropriate z threshold (for example = = 1.645 for 95% CI in the ranking), and assign two consecutive methods the same ranking if the absolute z-score falls below this threshold.
The first row of Table shows the underlying gold standard ranking for all the LLM mimics, since in this case we know the true स. values. Note the tie between L.L., and LLM,, because their मं = 0.3642 is the same. The second
5

---

Published as a conference paper at ICLR 2026
(a)
Kendall's T
0.8
0.3
0.7
No Bootstrap
0.7
Column-wise Bootstrap
10
40
50
60
10
30
40
30
Trials
Trrils
(c)
Bayes@N
G-pass@@2o.5
Kendall's T
pass
G-pass@40
0.8
pass@1
G-pass@/8o.s
pass 8
mG-passs222
Pass
0.7
Row-wise Bootstrap
mG-pass@44
Pass"4
mG-pass@@8
Pass
N Trials
Figure 2: Kendall's 7 rank correlation for various evaluation methods compared to the true ranking of 11 sets of biased coins (LLM mimics) with known mean success probabilities 0.5418, 0.5276, 0.608 0.6213, 0.7327. The simulation evaluates methods including Pass @k (k 2, 4, 8), Bayes@N, Pass G-Pass k; (f 0.5), and mG-Pass@k across to 80 trials. Panel a) shows T results without bootstrapping. while panels b) and c) use two different bootstrapping approaches with 10 samples.
0.2332, 0.2545, 0.3604, 0.3642, 0.3642, 0.4466,
row shows the Bayes@80 ranking without credible interval (CT) and the third row shows Bayes @80 incorporating the 95% CI. The Bayes@8) @80 ranking without CI aligns with the gold standard, except for two differences: the order of 1L and LLMg is swapped, and the tic between LLM,, and LLM, is not captured, which is expected since this ranking relies solely μ estimates without accounting for uncertainty In contrast, the third row, which incorporates the CI, reveals multiple ties across several models. Interestingly. LLM and LLMg are now indistinguishable at the 95% CI. Despite the fact that N 80 would be atypically large number of trials for an actual LLM evaluation, it insufficient to confidently distinguish the small performance difference (Ā = 0.608 vs. 0.6213) between the two models. In Appendix D we show that would actually require increasing N by factor of achieve 95% CI, highlighting the difficulties of reliably ranking models with similar performance.
Table 1: Comparison of biased-coin LLM mimic rankings based on the gold standard, Bayes@80 without credible interval (CI), and Bayes@NO with CI.
LLM mimic
LLM
LLM10
LLM,
LLMg
LLM, LLMg LLM;
LLM,
LLMg
LLMg
LLM
Gold Standard
2
3
5
7
8
9
10
Bayes@8a (w/o CI)
1
3
2
5
4
7
8
9
10
11
Bayes @80 (w/ CI)
2
2
3
5
5
3 EXPERIMENTS
In this section, we empirically validate proposed evaluation methods using real-world datasets, focusing on ranking LLMs for mathematical reasoning tasks. We employ bootstrapping to compute the expected value of each evaluation at given N. First, we present rankings of L.MSs the AIME'24, AIME'25 25, BrUMO'25, and HMMT'25 datasets without accounting for variance, based solely on evaluation scores (with ties occurring when scores are iden- tical). Subsequently, we demonstrate how incorporating uncertainty in these scores can alter rankings across different datasets. Building on the discussion in Section 2.7 we adopt the ranking derived from avg @80 (Pass or Bayes @80 (uniform prior Bayesian estimator) at 80 (the total number of trials conducted per dataset) as our gold standard for comparing current L.Ms,, noting their equivalence in rankings (as proven in Section 2.6). For each N from to (with Pass @h and similar methods starting from = l to avoid computation with insufficient samples), we com- pare the rankings produced by various evaluation methods against this gold standard, reporting the average Kendall's
6

---

Published as conference paper at ICLR 2026
over 104 bootstrapped resamples to estimate the expected rank correlation at each step (assuming independence among questions and trials).
3.1
CONVERGENCE TO GOLD STANDARD
1.00
1.00
(b)
Kendall's T
0.90
0.50
Bayes@N
0.85
pass@2
passiğ4
pass(g8
0.75
0.75
A1ME25
AIME'24
Pass
Pass"4
Pass"8
G-pass@:2a.s
G 4pp
(c)
(d)
6.95
0.55
G-paas@88a.5
Kendall's T
mG-pass082
mG-pass@4
0.85
0355
mGG-pass@8
HMMT25
BrUMO'25
6.70
19
20
30
10 20 30
40
50
60
N Trials
Trials
Figure 3: Average Kendall's r correlation between rankings produced by various evaluation methods and the gold standard (derived from Bayes @80, or equivalently avg@80), as function of the number trials N. Results are averaged over 10s bootstrapped resamples for each dataset: (a) AIME'25, (b) AIME'24, (c) HMMT'25, and (d) BrUMO'25 Methods include Bayesian estimation Bayes@N Pass@k (k 2, 8), PassAA, G-Pass k2 (7 0.5). and mG-Pass@k.
To assess the ability of different evaluation methods to compare the performance of different LLMs, we plot the average Kendall's т against the gold standard as function of the number of trials N in Figure 3 combining results from AIME'25 (panel a), AIME'2 24 (panel b), 25Mann2 c), and BrUMO'25 25(panel d). Across all datasets, the Bayes@N and avg@N curves overlap completely (so we only plot Bayes @.N) and demonstrate to high values, indicating robust alignment with the gold standard even in low-sample regimes. In all four datasets, Bayes @N reaches > 0.90 by N 10 and approaches т z 1 at zo 80. The only exception is AIME'25 25, where 0.90 is achieved by N 10, but the curve converges to 0.95 at N 80.
fastest convergence
In contrast, Pass@k variants (k: 2, 4, 8) and their variations (e.g., Pass G-Pass@k; with = 0.5, mG-Pass @k) start with lower Kendall's compared Bayes@N and converge Bayes@N consistently shows faster convergence and higher agreement with the gold standard. These findings align with our biased-coin simulations in Section 2.7 demonstrating that the Bayesian method best satisfies the gold-standard criteria-low uncertainty, minimal ties, and rapid convergence-across diverse mathematical reason- ing benchmarks.
slowly in all four datasets. At every N,
3.2 RANKINGS WITH CREDIBLE INTERVALS
Following the methodology of Section 2.7.1 we compare model rankings across four datasets (AIME'25, AIME'24,
25 and BrUMO'25) using Bayes (80 the gold standard (see Figure Table summarizes these compar- isons by reporting, for each dataset, two versions of the ranking: the rank with 95% CI and the rank without CL. The "w/ CI" rank accounts for uncertainty in the Bayes 80 scores and therefore allows models with overlapping CIs to share the same rank; the "w/o CI" rank is the strict ordering determined by the point estimates of Bayes @80 for that dataset.
Table 2 indicates that point-estimate rankings diverge from those accounting for credible intervals. &> Qwen3-30B- SSB-Thinking 2507 and &
2507 consistently secure the top positions across all four datasets;
7

---

Published as a conference paper at ICLR 2026
specifically, the dominance of the 30B model is statistically distinguishable at the 95% CI level in every case. Con- versely, the relative ordering of the remaining models varies by dataset.
When incorporating 95% Cls, we observe that while all four datasets exhibit five tied groups, the extent of ambiguity varies significantly. AIME'25 yields the fewest distinct ranks (up to 11), followed by AIME'24 (up 13), and both 25 and BrUMO'2. 25 (up 14). This compression of ranks indicates greater uncertainty in the Bayes @80 gold standard for AIME'25 (due to more extensive ties) compared to the others under our current trial budget. Intuitively, this higher uncertainty in AIME'25's gold-standard scores implies that more additional trials would be required for that dataset to empirically produce statistically stable ranking; conversely, we can more confident in the estimated gold standards for 14, HMMT and BrUMO' 25 given the current number of trials. This distinction also explains why AIME' 25 reaches a Kendall's 7 of 0.95 at N = 80, whereas the other three datasets converge to ~ 1 at the same sample size in Figure
Table 2: Rankings for four datasets. Models are listed in the order of their gold-standard ranking (Bayes @80 point estimates, i.e., without uncertainty) for AIME'25. Each dataset column gives the rank with a 95% CI (left) and the rank without CI (right).
AIME'25
'5
AIME'24
H5MT'25
BrUMO'25
Model
w/ CI
w/o
CI
w/
CI
w/o CI
w/
CI
w/o CI
w/
CI
w/o Cl
Qwen3-30B-A3B -iniiiinnn---iiing-------
1
1
Qwen3-4B-Thin
2een3-44--hiinnin-25500
2
2
2
2
gpt-oss-20b-high
3
3
5
4
gpt-oss-20b-medium
3
3
3
3
Phi-4-reasoning-plus
3
3
4
5
AccReasonnNemottron-...11
4
5
9
6
Phi-4-reasoning
5
5
10
8
4
7
gpt-oss-20b-low
5
8
6
12
11
17
11
17
OpenThinker2-32B
5
4
8
7
2
3
Light-R1-14B-DS
5
10
4
6
11
4
8
5
11
4
7
9
3
NVIDIAA-Nemotron-Nane.
12
10
LIMO-v2
6
13
13
12
EXA0NE-4.0-1.2B
7
14
14
13
10
15
OpenRI-Distilll-
7
15
15
10
16
8
OpenThhinker3-1.1
8
16
10
16
14
14
OpenReasoning-
8
17
11
17
15
10
16
9
18
12
19
12
18
13
19
Sky-TI-32B-Flash
10
19
12
18
13
19
12
18
Bespoke-Stratos-77
11
20
13
20
14
20
14
20
3.3 CONVERGENCE
In this section, investigate the convergence of model rankings, building the showcase figure (Figure I}. We define Sonvergence@1 as the smallest trial n at which the ranking induced by the first trials matches the gold standard ranking from all 80 trials (without bootstrapping) and remains unchanged thereafter.
Lower convergence @ values indicate that fewer trials are sufficient to achieve stable rankings. As detailed in the caption of Figure [ the figure displays the probability mass functions (PMFs) of convergence@1 @ for each method across the datasets. These PMFs are empirically estimated by generating 10a column-wise bootstrap replicates through resampling the N₂ Nmax trials, then for each replicate, cumulatively evaluating the ranking at every N (from to 80) and identifying the minimal n where the ranking stabilizes to the gold standard. This process captures the distribution of convergence points under repeated sampling, reflecting the inherent uncertainty finite-sample rankings due to stochastic trial outcomes.
This bootstrapping approach provides a distribution over possible convergence points (n), offering insights into the variability and reliability of each evaluation method: Pass@k (for k 2, 4, 8) versus our Bayes@N. A lower
8

---

Published as a conference paper at ICLR 2026
24
Convergence @n Mean
AIME'25
BrUMO'25
60
25
40
A4ME'24
24ME
30
AIME'25
20
BrUMO'25
BrUMO'25
10
H5MT25
H5MT25
Figure 4: Convergence @ without CL. Mean convergence@1 across model combinations for AIME 24, AIME'25,
HMMT'25, and BrUM0'25. Top: 50 combinations of models. Bottom-left: 20 combinations of 10 models. Bottom-right: 20 combinations of 15 models. Color indicates the mean convergence@n over 10 bootstrap replicates (green: fast convergence; red: slow convergence).
mean convergence @n signifies more cost-effective convergence, while failure converge within 80 trials (as seen in AIME'25) indicates more trials are needed to confidently rank L.MSs or we must include CI for reliable ranking.
The key takeaways from Figure [ as summarized in its caption, underscore the superiority of the Bayes@N: it converges reliably on all datasets except AIME'25, often with fewer trials than Pass @k. For instance, on 25MT'25 and BrUMO'25,, Bayes @N achieves mean convergence at approximately 44.2 and 27.1 trials, respectively, compared to around 69.5 and 48.5 for the best scores of Pass@k family. The right panel of the figure further illustrates this through an example ranking from a bootstrap replicate, emphasizing differences in convergence for AIME and BrUMO'25. See Appendix Figure for the corresponding cumulative distribution functions (CDFs).
Worst-case scenarios To further distinguish the Bayes@N framework from avg@N, we analyze the worst-case bootstrap replicates, i.c., those that either require the maximum number of trials to stabilize the rankings or fail to converge. For 11 LLMs, Figure T0 shows these trajectories as competition rankings, with each line tracing model's rank as trials accumulate; convergence is defined as the point at which the ranking order remains unchanged for subsequent trials. In AIME'24 (panel a) the ranking converges at trial 75, in BrUMO'25 (panel c) at trial 68, and in 25 (panel d) at trial 78, whereas in AIME'25 (panel b) convergence is observed within 80 trials, underscoring persistent instability and the need for additional trials or Bayes@N credible intervals. When a ranking does not converge within the trial budget (as for AIME'25 25 in Figure 1 only Bayes @N be used to quantify uncertainty and estimate the minimum required for a reliable ranking (see Section 2.7.1 2.7.1
This situation becomes consider pool of 20 LLMs (Table 77) and construct 50 subsets of 5 models (Table ( 20
more severe more models included. As shown in Figure when the number of models is increased to L = 20, none of the datasets exhibit convergence. To examine convergence as a function of
L more systematically, subsets of 10 models (Table 10). and 20 subsets of 15 models (Table 11). For each subset, we generate 10 bootstrap replicates estimate convergence nn. Figure 4 reports the resulting convergence@n values across all subsets and replicates, showing that as the number of models increases, evaluation methods such as avg@N and the Pass @k family become unreliable for estimating model abilities and producing stable rankings.
3.4 RUBRIC-AWARE CATEGORICAL EVALUATION
While evaluation often reduced to binary correctness, this simplification discards valuable signals that capture other aspects of model behavior. For instance, L.M outputs be assessed not only correctness but also whether they are well-structured, coherent, or exhibit step-by-step reasoning in mathematical tasks. In practice, evaluators routinely record richer dimensions such format compliance, calibration of confidence, degenerate outputs, out-of-diittibuttin (OOD) behavior, and verifier scores. This limitation is especially important for reasoning models, where overthinking (43) inflates token usage without corresponding gains in reliability. Bayes@N provides a principled way capture these richer outcomes. By treating per-item results as categorical rather than binary, the approach aligns more closely with actual goals while preserving statistical rigor and transparency. This method enables nuanced understanding of model performance across diverse dimensions, offering insights into trade-offs between correctness, efficiency, and robustness. For a comprehensive discussion of the categorical Bayesian evaluation framework, including base signals, schema definitions, and their impact on model rankings, see Appendix
9

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of the Bayesian framework and other evaluation methods.
Methods (N trials)
Convergence
Credible interval
Prior knowledge
Categorical
Pass@k and alternatives avg@N
X
X Limited (via bootstrap/binomial Cls)
X
X V (Sec.
Bayes@N
(Sec.
3.3
Figs.
and
(Fig. 6 Table 12
A(App.
3.4
4 RELATED WORK
Functional-coorrecteess evaluation with Pass @k became standard in code generation with HumanEval (OpenAl Codex): generate & samples, a task is solved if any sample passes unit tests, and estimate the overall rate with an unbiased estimator that requires producing n k samples per task (23). Although Pass@k was initially introduced in the context of coding, it later became the de facto choice to evaluate LLMs not only on math reasoning tasks (44) 45
46 47
48
49
50
51
26
52
but also on safety evaluations spanning agent red red-teaming, jailbreaks, and backdoor analyses (53)
54
55
56
57, 58). For a broader review of these metrics and their variants, see Appendix H Beyond standard
Pass @hi, pass*k quantifies reliability across l i.i.d. trials for agents, while the generalized G-pass@k. k, continuum (and
its area-under-7 summary mG-Pass) jointly assess potential and stability in reasoning outputs (59, (6)
Efforts like HELM advance holistic, transparent evaluation across scenarios and metrics (5) while practice guidelines distill reproducibility pitfalls and prescribe multi-run, uncertainty-awwe reporting with fixed prompts, decoding, and dataset/version control (60). The LM Evaluation Harness offers standardized, reproducible frameworks to implement these recommendations (60). It addresses the need for calibrated uncertainty in small-sample reasoning by employing exact methods for error quantification in binomial settings.
The last category of related work focuses measuring uncertainty in LLM evaluation. These works converge interval-aware saalll-amppl-vallll reporting rather than CLT/Wald error bars. Bowyer et al. show that CLT-based intervals miscalibrate on small benchmarks and advocate -mplo-n-appropriite frequentist Bayesian intervals for re- liable comparisons (41). A Bayesian alternative models capability as a latent success probability and reports posterior uncertainty that remains informative with limited trials, yielding more stable rankings (33). In judge-based settings, Judging LLMs on a Simplex places model and judge behavior on the probability simplex, enabling uncertainty-awan comparisons and highlighting how distributional structure matters for evaluation (61). Beyond bespoke LLM metrics, prediction powered inference supplies general procedures for valid credible intervals that leverage model predictions to reduce -ample requirements (62). Finally, in adjacent retrieval evaluation with -.M--genrrate assess- ments, Oosterhuis et al. construct reliable credible intervals and demonstrate that calibrated uncertainty, rather than point estimates, should guide decisions, reinforcing this shift for L.M evaluation more broadly (63).
5 CONCLUSION: STRENGTHS, LIMITATIONS & FUTURE DIRECTIONS
The overall benefits of the Bayesian framework are summarized in Table 3 it provides fast convergence, analytical uncertainty estimates, and the incorporation of prior knowledge and categorical results. However, it is worth noting that our approach quantifies statistical uncertainty from finite samples; it does not fix dataset bias, distribution shift, rubric misspecification. Results therefore depend on the chosen benchmark, prompts, and inference settings (hard- ware). Although we have validated our approach with biased-com LLM mimic simulations, together with experiments using actual LLMs (up to Nmax 80 trials across four tasks and 20 models), more extensive evaluations may be constrained by computing and academic budgets.
The focus of the current work the simplest version of the Bayesian approach, using uniform prior, which provides a conservative and reproducible starting point. But the theory allows for more complex, informative priors, and this opens up rich vein of future directions that should be systematically explored: for example priors from past runs, domain- or task-conditioned priors, and expert-elicited priors. These have the potential of accelerating convergence even further, but must be chosen and reported carefully. Clear guidance and tools for prior elicitation will hopefully ensure that gains in sample efficiency do not come at the cost of hidden bias.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This research relies only on publicly available, non-personal benchmarks; no human subjects, user data, PII are involved. Potential misuse includes cherry-picking priors, rubrics, or samples to exaggerate performance. To pre- vent this, use of Bayes@N with user-defined priors requires clear documentation and reporting of posterior credible intervals.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, detailed implementation instructions are provided in Appendix
ACKNOWLEDGMENTS
This research was supported in part by NSF awards 2117439, 21122600 2112606, and 2320952.
REFERENCES
[]] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
URL /ttpp:::/axxxii.....am
abs/ httpaaaas///..///// 03766
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, et al. Language models
few-shot leamnes. In Advances in Neural Information
Processing Systems, 2020. URL lattps://proceedingg wwwrine..r.ccmm hww.:/ppp.cc/pppee/////////////e
[3] StackOvertlow. Stack Overflow Developer Survey 2025: Al and Developer Tools, 2025. URL https://
Accessed: 2025-09-24.
[4] Nestor Maslej, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Njenga Kariuki, Emily Cap- stick, Anka Reuel, Erik Brynjolfsson, John Etchemendy, et al. Artificial intelligence index report 2025. arXiv preprint arXiv:2504.07133) 2025. URL
[5] Percy Liang, Rishi Bommasani, et al. Holistic evaluation of language models. arXiv:2211.09110, 2022. URL https://axxiv.orot http:/bb///21111
[6] Dan Hendrycks, Collin Burns, Steven Basart, al. Measuring massive multitask language understanding, In International Conference on Learning Representations (ICLR), 2021. URL https://arxiv.orrt abs/
2009. 03300
[7] Aarohi Srivastava, Abhinav Rastogi, et al. Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models (BIG-bench), arXiv:2206.046115 2022. URL/attps://arxis org/ abs/ 2206.044 04615
[8] Jared Kaplan, Sam McCandlish, Tom Henighan, et al. Scaling laws for neural language models. arX1v22001.0836,, 2020.
08361
[9] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, al. Training compute-optimal large language models. arXiv:2203. /5556, 2022. htt.httsss//ww...../../o////////
15556
[10] Jason Wei, Xuezhi Wang, Dale Schuurmans, et Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 2022. URL /ttp:ttttt::/oeeeevieeee net/
Training language models to follow instruc- 00050033333111110333330000000rr0
[11] Long Ouyang, Jeff Wu, Xu Jiang, et al. tions with human feedback. 2022.
In Advances Neural Information Processing Systems, URL https://proceedigs .nw.reeeedddnnn.n.nnissss.s.. www..pscccpppppprrrrrrrr...ee files/pape:/002/Eiee
[12] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke L.ke L.LMintt0:: 8-bit Matrix Multiplication for Transformers at Scale, 2022.
wbtp:////rrrbb////..//////// 07339

---

Published as a conference paper at ICLR 2026
[13] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers, 2022. URL
17323
[14] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In NeurlPS, 2015. URL https::/ wwtp:/iippppp.pppp.....pppppppae
5784-learma leabneignnng---- saatrniingooohh---e weights and-aaddddeeeee cow--cooonnnnnnnoo for for-efficient neura network,
[15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. URL https///xxxxv....//boo.
02531
[16] Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611-626, 2023. URL https:
7 //axxiv.org abs/ /abs//399.. 2309. 06180
[17] Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. Kv cache is bit per channel: Efficient large language model inference with coupled quantization. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan. U. Paquet, J. Tomezak, and C. Zhang, editors, Ad- vances in Neural Information Processing Systems, volume 37, pages 3304-3331. Curran Associates, files/ selle//aper//224/E11e
Inc., 2024. URL
[18] Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. Pqcache: Product quantization- kvcache for long context IIm inference. Proc. ACM Manag. Data, 3 (3), June 2025.
doi: 10. 00.1145/3725338. Utt.htts:://do: 1tt.///....001111111//1111111111 1145/3725338
[19] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09683, 2021. URL
https: fetts://arriv... wrg/ 2106. 09685
[20] F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurlPS, 2017.
03741
[21] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2020. httpp/ss//peeeeeeeeeeeeeeeeeerep
1rXiv:1904.09751 (2019).
[22] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
[23] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprimt arXiv:2107.03374 2021. RRLLhhtps::/ pestps://arxiie org/ 2ab0/2107.03
[24] Mathematical Association of America.
American invitational mathematics examination (aime).
compptiiionns 2024. URL hhtpps:/ hamps:://ma.... org
maa- ltwr--iiitttiioll-cmmmeeeeeeeeee 2024).
Official MAA page for the AIME competition (covers AIME
[25] Mathematical Association of America, American invitational mathematics examination (aime).
wtw.:ttataaaoooooooooottteetteee compet competitions// 2025. URL htttps://maa.rg: haa--nvttattiona--cooeettttooo.t 2025).
Official MAA page for the AIME competition (covers AIME
[26] Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, and Kai Chen. Are your lims capable of stable reasoning? arXiv preprint arXiv:2412.131477 Decem- ber 2024. doi: 2144 131477 URL htt.Lo////AAA....o///////222.... camera-ready version.
ACL 2025
[27] Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. arXiv preprint arX0v:2504.07066 2025.
07086
[28] Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The Hitchhiker's Guide to Testing Statistical Significance in NLP. In ACI., pages 1383-1392, 2018. http:htttt///tclttt ooy.lol...o....o................ wttp:/111/11111/
12

---

Published as a conference paper at ICLR 2026
[29] Alexander Yeh. More accurate tests for the statistical significance of result differences. In COLING, 2000. URL https://acl hn wogy. org/
[30] Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Im- proved reporting of experimental results. In EMNLP-IJCNLP, 2019. https::///ccaaaaaaaeee
org/
019-12247
[31] Lianmin Zheng, Wei-Lin Chiang, Yingbo Sheng, and et al. Judging LLM-as---Judge with MT-Benchh and Chatbot Arena. arXiv preprint arXiv:2306.0566 2023. URL (tLnttts:/// arxiv. wttp:iwww..r//bb//2//////// 200.b///6800060665553 05685,
[32] Guande Chen, Kai Shen, Saurav Shah, and et al. Humans or LLMs as the Judge? A Study on Judgement Bias. In EMNLP, 2024. URL
www.entthoooo..ooooooooooo logy. 2024. emn somnlp-main. 474. pdf
[33] Xiao Xiao, Yu Su, Sijing Zhang, Zhang Chen, Yadong Chen, and Tian Liu. Confidence in large language model evaluation: A bayesian approach to limited-sample challenges. arXiv preprint arXiv:2504.2 2303, 2025.
[34] Dustin Hayden and Thomas Armitage. Straightforward bayesian a/b testing with dirichlet posteriors. arXiv preprint arXiv:2508.0m 08077, 2025. URL httpp:://aaxiiiiaa org att///b///2500/ 08077
[35] Harvard-Ma MIT Mathematics Tournamet. Hmmt february 2025 archive (problems and solutions). https://
www. hmmt. . org/ http://wwwa//hhecc 282) 2025. URL httpp::////w/t hnmmt wwtp://////wwwww////// http://wwww...tg//////////////// Official HMMT archive page for February 2025 competition.
[36] Brown University Math Olympiad Organizers. Brown university math olympiad (brumo). https:/ /ww::/ brumo. . wwt./oounaamet.....o.o -nfoo 2025. URL httts://www. http:://wwwwwbbr brumo. org//oournamett-nnnn Oi-
ficial BrUMO website with tournament information (Apr 4-5, 2025).
[37] Uri Dalal, Meirav Segal, Zvika Ben-Haim, Dan Lahav, and Omer Nevo. Leveraging to Boost @ass@ k Performance. arXiv preprint arXiv:2505. 2025. URL httpps:/erxx...orr/ab////55
12938
[38] Brendan Leigh Ross, NoAGI Vouitsis, Atiyeh Ashari Ghomi, Rasa Hosseinzadeh, Ji Xin, Zhaoyan Liu, Yi Sui, Shiyi Hou, Kin Kwan Leung, Gabriel Loaiza-Ganem, et al. Textual Bayes: Quantifying Uncertainty in LLM- Based Systems. arXiv preprint 25066 10060, 2025. htLtttts:://aatt org. abs http://bb///555... 10060
[39] Roman Vashurin, Maiya Goloburda, Albina Ilina, Aleksandr Rubashevskii, Preslav Nakov, Artem Shelmanov, and Maxim Panov. Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency, arXiv preprint arXiv:2502.04900 2025. CRLhttts:///azxt www.wawwwwwa.aaaaaaaoooooaoooeoe abs 2502. 04964
[40] Edwin T Jaynes. Probability theory: The logic of science. Cambridge university press, 2003.
[41] Sam Bowyer, Laurence Aitchison, and Desi R Ivanova. Position: Don't Use the CLT in LLM Evals With Fewer Than Few Hundred Datapoints. arXiv preprint 2rX1v:2503.01777, 2025. URL https: orw.
abs/2503.011 01747]
[42] Mohsen Hariri, Michael Hinczewski, Jing Ma, and Vipin Chaudhary. Ranking reasoning Ilms under test-time scaling, 2026. URL htttttttt/s:iiiii org/ http://b////..111///00
[43] Xingyu Chen, Jiahao Xu, Tian Liang. Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-like Ilms. arXiv preprint
2024. URL stttpa:/aatt //arxiv...iiivivaa org/ abs 21187
[44] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arxiv:2501. 12948, 2025.
/anps//2501.0 12948
[45] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. meepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint
2024. URL. httptt:////attt //tp///axvvvv////////aa///al/aaa v. org/ abs/ 2402 03300
[46] Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty-wwan rejection tuning for mathematical problem-solving. In A. Globerson, Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tom- czak, and C. Zhang, editors, Advances in Neurul Information Processing Systems, volume 37, pages 7821-7846. Curran Associates, Inc., 2024. URL /ggp///////tsss/rrrreeeeeeeeeee
13

---

Published as a conference paper at ICLR 2026
[47] Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward. and Yi Zhang. TinyGSM: Achieving 80% on GSM8K with small language models, 2023. URL https: 7/arxiv. wtt..orr/abbb///////
[48] Hyeongdon Hwang et al. Self-explore: Enhancing mathematical reasoning in large language models by finding the first pit. In Findings of EMNLP. 2024. URL
Anddings-emn 787
[49] Yan Yang et al. Weak-to-strong reasoning. In Findings of EMNLP, 2024. UtL/tttpss//aaaaanooooog 4907
org/2024. Findings-diiingnn
[50]
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang. Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL https:
/ V/rxxvv.org wttp:/orgggbb///1...2222
[51] Feng Chen, Allan Raventós, Nan Cheng, Surya Ganguli, and Shaul Druckmann. Rethinking fine-tuning when scaling test-time compute: Limiting confidence improves mathematical reasoning, 2025. URL https://
[52]
Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Yemuk Choi, Seokhee Hong, Junwon Hwang, Hyojin Jeon, Kijeong Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung. Hyosang Kim, Joonkee Kim, Seongh- wan Kim, Soyeon Kim, Sunkyoung Kim, Yireun Kim, Yongil Kim, Youchul Kim, Edward Hwayoung Lee, Haeju Lee, Honglak Lee, Jinsik Lee, Kyungmin Lee, Sangha Park, Yongmin Park, Sihoon Yang. Heuiyeen Yeen, Sihyuk Yi, and Hyeongu Yun. EXAONE deep: Reasoning enhanced language models. arXiv preprint arwiv:2503....... /2524, URL https: //arivvaaiiv////aaa otg/ absp//03./2252 12524
March 2025. doi: 10.48550/arXiv250000 15522
[53] Itay Nakash, George Kour, Koren Lazar, Matan Vetzler, Guy Uziel, and Ateret Anaby Tavor. Effective red- teaming of adhicy-a agents. In Proceedings the 2025 Conference on Empirical Methods in Natural Lan- guage Processing. pages 2250-2268, Suzhou, China. November 2025. Association for Computational Linguis- tics. ISBN 979-8-89176-3388 doi: 10.18653/1122255..o 114/]
main. 114. URL https:://clanttolll
[54] Hojjat Aghakhani. Wei Dai, Andre Manoel, Xavier Fernandes, Anant Kharkar, Christopher Kruegel, Giovanni Vigna, David Evans, Ben Zorn, and Robert Sim. TrojanPuzzle: Covertly poisoning code-suggestion models. In IEEE Symposium on Security and Privacy, SP 2024, San Francisco, CA, USA, May 19 23, 2024, pages 1122- 10 1109 10100005/S555333
1140. IEEE, 2024. doi: 10.1 N0.1109//P54263.20244444 2024.0014. URL
2024. 00140
[55] Hongyi Liu, Shaochen Zhong, Xintong Sun, Minghao Tian, Mohsen Hariri, Zirui Liu, Ruixiang Tang, Zhimeng Jiang, Jiayi Yuan, Yu-Neng Chuang, et al. LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Piae Ecosystem. arXiv preprint Stt1:24033001000 2024. URL (RLLhttss: //apxiv org/ wwwp/rrg///.....oo.............t 2403. 00108
[56] Shenao Yan, Shen Wang, Yue Duan, Hanbin Hong. Kiho Lee, Doowon Kim, and Yuan Hong. An LLs--ssitted -arigeo-Trigger backdoor attack code completion models: Injecting disguised vulnerabilities against strong detection. In 33rd USENIX Security Symposium (USENIX Security 24), pages 1795-1812, Philadelphia, PA. August 2024. USENIX Association. ISBN 978- 199111-99933-44411111
doi: 0155555/6688900036990000 URL https:: //www. usenix x, wwtp:/oggeoooeeeeee http:/ooffeeeeccoo /usenuuunnnne x http:///ewwaccaaaaaaaa/aa//eaaaa
[57] Lakshmi Likhitha Mankali, Jitendra Bhandari, Manaar Alam, Ramesh Karri, Michail Maniatakos, Ozgur Sinanoglu, and Johann Knechtel. RTL-Breaker: Assessing the security of LLMs against backdoor attacks on HDL. code generation. arXiv preprint 1.I.X::24111.15666 November 2024. doi: 10.48550/rrXi..4111.777 UiRL/atps://axxix/ org/ awws////111. 17569 Accepted at DATE 2025.
[58] Rylan Schaeffer, Joshua Kazdan, John Hughes, Jordan Juravsky, Sara Price, Aengus Lynch, Erik Jones, Robert Kirk, Azalia Mirhoseini, and Sanmi Koyejo. How do large language monkeys get their power (Laws)? In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien Felix Berkenkamp. Tegan Maharaj. Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 53132 53176. PMIR, 13-19 Jul 2025. doi: 10.48550/arXiv. 2502.17578. URLhhtts.: http:attss::/rrreediiie
mlr, wwwpsppwww..epppppppppppp hwtp//////ccacceee f2a2 html Oral.
[59] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent- user interaction in real-world domains. arXiv preprint, 2024. doi: 10.48855//rxxv.2406. 12045. URL //doi.org/10 wrog 10e..485550 48550 (8550/arxivv 2406. 12045. Introduces the pass* metric.
14

---

Published as a conference paper at ICLR 2026
[60] Stella Biderman, Hailey Schoelkopf. Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji. Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica 7osa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li,, Charles I Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang. Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang. Genta Indra Winata, François Yvon, and Andy Zou. Lessons from the trenches reproducible evaluation of language models. arXiv preprint 240.. 2024. URL
14782
[61] Patrick Vossler, Fan Xia, Yifan Mai, and Jean Feng. Judging Ilms on a simplex. arXiv preprint arX1v:2505.21992. 2025, httpp://///a: org/ abw.bbss.555 21972
[62] Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael Jordan, and Tijana Zrnie Prediction- powered inference. Science, (126)))6)6699 2023.
[63] Harrie Oosterhuis, Rolf Jagerman, Zhen Qin. Xuanhui Wang, and Michael Bendersky. Reliable confidence intervals for information retrieval evaluation using generative ai. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2307-2317, 2024.
[64] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv preprint arXiv: 2019. URL httptttts/////tt http://awwvva.ra..aaaaoooo org. hbtp://////a abs//asc.....///.//...//./eeeeee 11111.111111 03771
[65] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024. /OLLEEtSS///xxxx../////////
21787
[66] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. July 2002. Association for Computational Linguistics. doi: (0.3115/007303311773333 URL https: //aclanthology
[67] Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambro- sio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.1029 2020. URL/attps://axxiv.ooom absggg//b/////
10297
[68] Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang Spoc: bearch-based pseudocode code. Advances in Neural Information Processing Systems, 32, 2019. URL
04908
[69] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,
2021. httt::::: http://wwxxvv..raaaaa// abs C3. 0111733877733333111 03874
[70] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers solve math word problems. arXiv preprint arXiv:2 10. 14168, 2021. URL. htt.thttt:///m http://aaivvvvv.va
http://////wri.oooobb/////////// www./ogg//b////////o 2110. 14168
[71] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Selfscoonistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203./117// 2022. http:ttsss/s.....orrrt.t
11171
[72] Aitor Lewkowyez. Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Am- brose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sold et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857,, 2022.
[73] M. G. KENDALL. A new measure of rank correlation. Biometrika, 301-2)28113301121_1_91999 06 1938. ISSN 0006-3444, doi: 10. 10.1003fbiomeet30..1.2.11
[74] NovaSky Team. Think less, achieve more: Cut reasoning costs htt.a//iiiin.iiiiii///////aa/a/a overthinking, 2025. /tt.t//tt/////eearr....////////t cessed: 2025-01-23.
Ac-
15

---

Published as a conference paper at ICLR 2026
[75] Qwen Team. Qwen3 technical report, 2025. htt.ttts///axvv..////////553
[76] OpenAl. gpt-oss- 120b & gpt-oss- 20b model card, 2025. URL
10925
[77] Yixin Ye. Zhen Huang, Yang Xiao, Ethan Chem, Shijie Xia, and Pengfei Liu. LIMO: Less More for Reasoning, 2025, URL/nttss: //araiv.///vvv.a org/ absa 03387
[78] LG AI Research. Exaone 4.0: Unified large language models integrating non-eaaonnng and reasoning modes, arXiv preprint arXiv:2507.11440 2025. URL https::/ttssstttoe wwtpp/wwwww...//////.....// 11407
[79] Shubham Toshniwal, Ivan Sorokin, Aleksander Ficek, Ivan Moshkov, and Igor Gitman. GenSelect: A Generative Approach Best-of-N, In 2nd A/ for Math Workshop @ ICML 2025, 2025. URL httgs://openzeviee.
net d-8.-=-mnnmmmbbbbbb
[80] Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. AIMO-2 Winning Solution: Building State-of-the-Ar Mathematical Reasoning Models with OpenMathiReasoning dataset, 2025. httpt:tt/a/xxii........ wwtp:///rgggo/bo//o/oo.oo./ 2504 16891
[81] Wasi Uddin Ahmad, Somshubra Majumdar, Aleksander Ficek, Sean Narenthiran, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Vahid Noroozi, and Boris Ginsburg. OpenCodeReusooingg II: A Simple Test Time Scaling Approach via Self-Critique, 2025. URLhhtps:: aaaaaiia www..oooo/abb//55000000000 09075
[82] Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficek, Siddhartha Jain, Jocelyn Huang, Vahid Noroozi, and Boris Ginsburg. OpenCodeReusoning: Advancing Data Distillation for Compet- itive Coding, arXiv preprini arXiv:2504.01 2025. URL http:/ppss///rxx...rr//////ee/ee
22000 0tt.///5044009.9.... 01943.
[83] Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smymiss Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang. Shreyas Pimpal- gaonkar. Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng. Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chimmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexan- dros G. Dimakis, and Ludwig Schmidt. Open Thoughto Data Recipes for Reasoning Models, 2025. URL
https: http:://aaxivv.og/aab/.. wttp://gg/.../..ooooo 04178
[84] Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gus- tavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. Phi-4-reasoning technical report. arXiv preprint arXiv:2504.2 12318 2025. URL https: httttttt/ttxiiii/ie //tax/aii.rggaaaa//// http:/bbs/25000/ .21188
[85] Hugging Face. Open R1: A fully open reproduction of DeepSeek-RI, 2025. URL http:::ttttttteet com/ http:/hhggggggggggoggo www.gggaaaccecooooo open-rl
[86] Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, and Xiaojun Quan. Fusechat: Knowledge fusion of chat models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 21 21629-21653, 2025.
[87] Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Tanglifu Tanglifu, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 118-327,22 2025.
[88] Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shocybi, Bryan Catanzaro, and Wei Ping, Accreason-nemmotron 1.1: Advancing math and code reasoning through sft and rl synergy. arXiv preprint arXiv:2506. / /3244, 2025. URLpttts:://ast httpaaaxvvvv.rgg/eaa abs 22066256666 13284
[89] Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinav Khattar, Adithya Renduchintala, Aditya Malte, Akhiad Bercovich, Akshay Hazare, Alejandra Rico, Aleksander Ficek, et al. Nvidia nemotron nano 2: An ac- curate and efficient hybrid mamba-transformes reasoning model. arXiv preprint arXiv:2508. 240. 2025. URL.
nttps://arxiv httgg/bbs/500000 14444
[90] Bespoke Labs. Bespoke-stratos:
The unreasonable effectiveness of reasoning distillation. www.hespokelabs.dd hwwhhhhhhsppeeeetttt..eeeeeeeeee Accessed: 2025-01-22.
distillatton
2025.
16

---

Published as conference paper at ICLR 2026
[91] Shudong Liu, Hongwei Liu, Junnan Liu, Linchen Xiao, Songyang Gao, Chengqi Lyu, Yuzhe Gu, Wenwei Zhang, Derek Wong, Songyang Zhang, and Kai Chen. Compassverífier: A unified and robust verifier for ilms eval- uation and outcome reward. arXiv preprint arXiv22508.03666, 2025. URI. https:: /arp://////vvvo/r////// abs/
2508.03688
[92] Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Mare Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable., 2022. URL.tttss: //g
[93] Tianyi Zhang, Mohsen Hariri, Shaochen Zhong, Vipin Chaudhary, Yang Sui, Xia Hu, and Anshumali Shrivastava. 70% size, 100% accuracy: Lossless Ilm compression for efficient gpu inference via dynamic-length float.
Advances in Neural Information Processing Systems (NeurIPS 2025), 2025. URL https: /ttp://aavvvooo//o orgg/ abs 0aas/2504..o 11651
17

---

Published as conference paper at ICLR 2026
CONTENTS
Introduction
1
Bayesian Framework for Evaluating LLM Performance
3
2.1
Background: The Pass Metric and Its Limitations
3
2.2
Results Matrix
3
2.3
Weighted Performance Metric
3
2.4
Bayesian Estimator and Uncertainty the Performance Metric
3
2.5
Using Uncertainty Estimates to Decide Significance of Performance Difterences
2.6
Equivalence of Bayesian and Average Rankings for Uniform Prior
2.7
Gold Standard for Ranking
5
2.7.1
Ranking with Uncertainty
5
Experiments
3.1
Convergence to Gold Standard
7
3.2
Rankings With Credible Intervals
7
3.3
Convergence
8
3.4
Rubric-Aware Categorical Evaluation
Related Work
Conclusion: Strengths, Limitations & Future Directions
10
Appendix
Derivation of Bayesian Estimator and Uncertainty
19
B Proof of Equivalence of Bayesian and Average Rankings for Uniform Prior
20
C Potential benefits of non-uniform priors
21
D Model Distinguishability and Sample Size
22
Runtime
Categorical Evaluation
23
F.1
Rubric-aware Bayes@) N Evaluation of Reasoning Models
23
F22 Domain-agnosie rubric-aware Bayes@N
25
Scorio
27
H
Extended Related Work
29
Experiment Setup and Reproducibility
30
Metrics
30
2
Models and Datasets
30
3 Reproducibility
32
Convergence
34
18

---

Published as conference paper at ICLR 2026
A DERIVATION OF BAYESIAN ESTIMATOR AND UNCERTAINTY
As described in the main text, the Bayesian framework is built on two quantities. The first is j(R), the average of ă over the joint posterior for all the questions:
/ dT S......) JA dTM M II P(T. R.). a=1
M
AATTM
RR))
where the integration region A is the probability simplex defined the of all possible (C + 1)-dimensional vectors p such that =k=0 = The second is the variance (2) associated with our Bayesian estimator,
(² =
/ d S
ddT ( ((R))² II P(Ta|Ra).
M
(3)
Our derivation of closed-form expressions for μ and builds on the generalized (C > 1) and original (C 1) Laplace
nule of succession theory from (40), recovering those results in the special case of single question (M = 1). We start with Bayes' rule for each row of R:
P(T. R..) = P(Ra(TT)) (P(TO)
(4)
P(Ra)
The likelihood (( is a (C+1) multinomial distribution over N trials, with the probability distribution function:
N!
(.. = n nac
II (TaR) Park
(5)
k=0
where Tlckk = Em Ök.R... na is the vector with elements Nade and ōG,5 is the Kronecker delta.
The prior P(T chosen as the conjugate prior of the multinomial, a Dirichlet distribution P(TT.) ~ Dir(n.). (n.) with concentration parameter vector n. = (n-00--.. n.c)) (34) A uniform prior (no prior knowledge) sets n for all k. Prior information from an earlier x D matrix R (with Ra as the category for the ith trial of the ath question) can be incorporated as:
nak 14
Σ
6 6
(6)
i=1
The Dirichlet prior is:
P(T.))
II
(Tak)
(oat))
(7)
II ((mea)) k=0
where Σi=o aa C
The normalization constant ((R..) is:
P P(R.)= dpu(Ruup)P(p)))
(8)
and since the Dirichlet is the conjugate prior, the posterior is P( Ra) Dir(va), with va = na + no. The posterior distribution is:
I(T)
P(Ta R.)
(TrR) Vak-1
(Tak)
(9)
I I(vak)
k=
where T= D Vat = N.
The moment generating function ((t) = (oxp(#t)) is:
M II
(t) = dr ... /
di M exp (tT)
P(T Ra)
a=l
M
II
dr exp
t
P( Ra)
(10)
M
a=l
km0
f10g
dra exp
8/: Tak
P(Tra|
k=1
19

---

Published as conference paper at ICLR 2026
where 83 (wy, - /M, and Tao = Ex= Tak-
Each integral is the moment-generatin function for a Dirichlet distribution, expressed via the confluent Lauricella hypergeometric function y/c1:
M
(t) = e
Awo II
v cl
:at: tsc)
(11)
a=1
where
20 Σ
(Val)m ... (. (o (181) ... (Isc)
( (Val,.... MaC;
Maac, T; t81..........................1 (sc) =
(12)
We =
mc=0
and (z),, is the Pochhammer symbol.
The moments are:
# == ('(),,
0² === 00) (0)))))²
(13)
Expanding yICl to O(P²):
C
t2
C
yIc
1+
Sjj +
Σ
Vog (Vs + 1)s3
T i=l
2T(T +
j=l
(14)
t
Σ
aas + Ott))
T(T
m=1m=
Substituting into equation| and computing derivatives yields:
M
1
we +
Vaj (wy(w, - wo).
MT
1=0
(15)
M
0²
1
Voj (w, wy)) Vaj (w. wo)
M2(T +1)
The algorithm summarizing this caleulation is shown in Algorithm 1 in the main text.
B PROOF OF EQUIVALENCE OF BAYESIAN AND AVERAGE RANKINGS FOR UNIFORM PRIOR
For Bayesian estimators using uniform prior (where 0, - Von = 1 1 noR), the expression for the mean from equation 15 simplifies as:
/ = Wo + M(1 + N)
Σ
Σ
+ maj) wp)
a=1
(16)
1
A+ M(1C++N)
W Tajj
where the constant A is given by
A=
Σ
W/j
(17)
I+C+N
==
and ES Paj N. Here, μ relates to a naive weighted average accuracy a over the number of answers in each category,
WyDaj.
(18)
MN
1=0
20

---

Published as a conference paper at ICLR 2026
via
N
# = +
1 1+ a.
(19)
Note that in the binary case where C = 1, we = 0, wn = 1, the value of a is just regular average accuracy avg@N For categorical cases, it is just a weighted generalization of avg@N.
Since A is constant across models and the prefactor I+C+N is positive, we see that if > u², the corresponding values of and a' from the two methods must always give the same ranking, a > a'. Additionally, in the limit of a
.V
large number of trials, N + oo, we see that A - 0 and μ 2 a, as expected.
This equivalence extends to uncertainty quantification. The relationship between the standard deviation of the average (vaggā)) and the Bayesian standard deviation B from equation T5) is
C+C+N N
avgüu.
Bayesu.N
(20)
The Bayesian expression for B is valid for all M and N. providing a reliable method compute uncertainty in avg@ @ N without relying on the Central Limit Theorem (CLT).
C POTENTIAL BENEFITS OF NON-UNIFORM PRIORS
While the convergence results in the main text demonstrate that Bayes@N with a uniform prior outperforms alterna- tives like Pass @k ranking models, there are scenarios where non-uniform priors can achieve even faster convergence, This is the case when we have data from models that are related or closely correlated to the ones we are ultimately interested in ranking. Potential examples include: i) results from an older version of a model used as prior for ranking newer version; ii) a non-quantized version (where running trials is computationally expensive) used to provide prior data for quantized version (where achieving large N cheaper); iii) a base model used to provide prior data for a fine-tuned one. Though a full exploration of these kinds of priors will be left to a future work, in this section we will show the potential benefits through our synthetic biased-coin LLM models, introduced in Section 2.7
We start with a set of eight "original" models with C- = 1, labeled by i 1,...,&, 8. Each model consists of set of M = 30 success probabilites Tai drawn from a distribution Beta(i + 3, 12 - i). We fix these probabilities for all the numerical experiments described below, and their averages for the eight models are: म [0.3021, 0.3166, 0.4144, 0.4985, 0,5351, 0.5759, 0.6679, 0.7487]. Hence for the original models higher i corresponds to higher overall accuracy. We now imagine an "update" of model i that mimics some kind of revision, fine-tuning, or other modifica- tion. Because the performance of the updated model should be correlated with the original, we model the update as stochastic perturbation the Beta distribution from which success probabílities are drawn: for updated model the Vai values are drawn from Beta(i + 3 + a, 12 - + o'), where a = +1 and o' = +1 are random integers of unit magnitude. For the updated models the value of may not strietly increase with i, so the ranking of models could be different than the original. Figure 5]aa) shows a histogram of the Kendall's values comparing the original model set (described above) and 50k possible updated sets drawn using this stochastic procedure. A T value of corresponds to exactly the same ranking, and we see that the mean over the 50k realizations is 0.88. Hence there is some correlation between the original and updated rankings, but in the vast majority of cases (about 86% of the updates) the ranking has changed for the updated models.
The question we would like to ask is whether we can use the results from the original models as priors to help speed up convergence when ranking the updated models. To employ non-uniform prior for a given model, we follow the procedure described in Appendix A and incorporate the prior via the the M x D results matrix R corresponding to D trial results over M questions using the original model. Combined with N trial results from the updated model, we get the Bayes@M accuracy estimate y for the updated model. These estimates are then used to rank the 8 updated models. Because know the values for this set, we know the true ranking, and can compare the estimated and true rankings via Kendall's T.
For each choice of N and D we run 50k replicates, with each replicate consisting of a set of stochastic updates of the original models. The mean 7 values over all these replicates are shown in Figure () as function of N for several different D. As expected, the curves increase with N, since the ranking becomes more certain with more trials, but the convergence properties vary. The dashed line is the case of a uniform prior (D 0), while the solid lines represent five different non-uniform prior scenarios, with D 1, 2. 4, 8. and 16. For small N and small D < we see a clear benefit of the non-uniform prior: already at N = 1 the value of starts higher than the uniform case, and remains so until the latter catches up for N > 5. Thus when we have prior data available, we can extract more accurate rankings with just small number of trials of the updated model, relative to the uniform case. However there is possibility to
21

---

Published as a conference paper at ICLR 2026
1.00
0.10
17.5
(4)
(b)
[c)
non-uniform prior D=1
15.0
0.95
non-uniform ptior D-2
non-uniform prior D-4
Probability density
0.06
cnn-unnform prior
12.5
0.90
non-uniform prior D 16
Kendall
0.04
10.0
0.85
uniform prior
0.80
non-uniforn prior D=1
0.00
non-uniform prior D D-2 non-uniform prier
-0.02
0.75
non uniform prior D -
non-uniform
0.04
0.0
1.70
0.70
0.6
0.7
0.8
0.9
1.0
Kendall r
Number of trials N
Number of trials N
Figure 5: (a) Histogram of Kendall 7 values comparing original ranking of synthetic LLM models and 50k replicates of updated models. (b) Mean Kendall 7 between the estimated and true ranking for the updated models (50k replicates) as a function of N, the number of trials. The dashed line corresponds estimates using Bayes @N with a uniform prior (D = 0), while the solid lines are Bayes@N with a non-uniform prior and different choices of D. The non-uniform prior is based results from D trials of the original models. (c) Same panel (b), except showing the difference Ar between the non- non-uninee uniform prior curves and the uniform curve.
over-emphasize the prior: when D = 8 and 16, the benefit for small N turns into disadvantage at larger N. The r curves dip beneath the D = 0 result, indicating that the prior has impeded accurate ranking. Figure 5(c) shows these trends more clearly by plotting Ar, the difference between the for each D and the uniform T with D 0. So we see that priors have to be used judiciously, with large enough D to nudge the ranking in the correct direction, but not too large outweigh the results from the updated models. One of the goals of our future work will be establish practical guidelines for D in different real-world use cases.
D MODEL. DISTINGUISHABILITT AND SAMPLE SIZE
To quantify the trials needed to reliably separate models with closely matched performance, we simulated the prob- ability of correctly ranking LLM above LLMg as a function of the number of trials N, shown in the left panel of Figure 6 At N = 80, the probability of obtaining the correct ranking is 83.7%. The right panel plots the absolute z-score versus N; at N 80, the 1.14, corresponding to approximately 87% CI (though the plots exhibit some noise due to simulation variability). These values closely align with the empirical probabilities in the left panels.
We also determined the minimum sample size N needed to achieve z-scores of 1.645 and 1.96, corresponding CI of approximately 95% and 97.5%, respectively, for distinguishing between models. These thresholds occur at about N 199 and N 285. At these values, the simulated probability of correctly ranking the models is 94.7% and 96.9%, respectively, which is closely consistent with expectations given the inherent noise in the results. These results underscore the computational cost of distinguishing models whose true performance metrics differ only slightly. In our biased-coin setup, the underlying success probabilities were Пy = 0.608 and T10 = 0.6213, yet reliably establishing this distinction requires nearly 200 trials. Such large sample requirements highlight the importance of considering both uncertainty and convergence rates when interpreting ranking-based evaluations.
E RUNTIME
To see the asymptotic runtime and memory scaling let:
M = number of problems (rows),
N number of trials per problem (columns R),
D number of prior outcomes per problem (columns in Ro. which may be 0),
+ = number of categories.
From Algorithm 1, the work is:
22

---

Published as conference paper at ICLR 2026
1.00
194.7%
99.9%
Probability
83.7%
1.96
Z-score
1.64
0.50
1.14
0.25
(a)
(b)
0.00
trials
N trials
Figure 6: (a) Probability of correctly ranking LLM10 above LLMg using Bayes @N in the biased-coin simulations, shown as function of trial count N. The probability is 83.7% N 80, increases 94.7% at N 199, and reaches 96.9% at N = 285. (b) Corresponding absolute 2-scores as function of N, with values of ~ 1.14 at N = 80, 1.645 N 199 (95% CT), and 1.96 at N 285 (97.5% CI).
Two row-wise histograms: O(MN) for R + O(MD) for Ro:
Posterior mean and variance on Mx((()+)); O(M(C 1))
So the overall time complexity is:
OM +D+C)
i.e., linear in the number of entries in the result matrices and linear the number categories.
The memory footprint is likewise linear:
Store R and (optionally) Ro O(MN MD),
Store per-row category counts and derived arrays (v, v/T): (CM(++1))
Note that the evaluation consists of tallying counts and then plugging them into closed-form expressions for μ and a; no iterative optimization Monte Carlo sampling required.
F CATEGORICAL EVALUATION
F.1 RUBRIC-AWARE BAYES@N NN EVALUATION REASONING MODELS
As we discussed in Section 2.3 and Section 3.4 for each question a E 1,...,M, M, every attempt yields base signals such as has_box, box, ississs_oo__ correct, tcken ratio, prompt_ bpp,, completion bpt,, and verifier probabilities compasss_coml wattossssootteee____e thresholds and Boolean criteria, each attempt is mapped into one C' categories under a chosen schema (e.g.. Format Aware, Conf-Wrong Penalty, Efficiency-Adjuster Table 5 We instantiate categorical schemata and update posterior means via Dirichlet-multinmmial inference, yielding metrics that preserve correctness while explicitly re- flecting formatting, calibration, and efficiency.
wttpppsss__ooeeel____e cowppsssssssssssss cataasss__nneee______ for correct, wrong, and invalid/off-task. Using
Base signals All signals are directly obtainable from common LLM inference stacks such as Hugging Face trans- formers (64) and vLLM (16), via per-step scores/log-probs and termination metadata, and require model-specific instrumentation; the verifier probabilities compass_context__ context A, campass_context B, compass caaolssooocttteeeeeeeeeeeeeeeeee C are defined
in F.1
has box: if final boxed answer is present; else 0.
is correct: if the answer is correct; else 0.
23

---

Published as a conference paper at ICLR 2026
token ratio: completion tokens normalized by 32,768.
repeated_pattens pattern: 0 if finisiii sh_reason is stop; else 1 (degenerate output),
prompt_bpp bpt: negative average prompt log-prob in bits/token (lower is better).
completion_bee bpt: negative average completion log-prob in bits/token (lower is better).
campass_content A: verifier contextual probability of correct.
compass_context B: verifier contextual probability of wrong-
compass_context C: verifier contextual probability of irrelevantoof-task.
Reward models evaluation. While reward models are most familiar from fine-tuning (e.g., RLHF), we use one as a lightweight verifier to supply per per-attempt label probabilities for
comppsss_cotert___e A, compass_contert B, cowpppssscootttttttooooooo = {correct, wrong, invalid/off-task]
produce probabilities and then apply contextual calibration to obtain more robust, prompt-stable label distribution: we evaluate next-token scores for the candidate labels at a fixed answer slot, subtract a content-free baseline logit by from the task logit 8y for each label y, and apply temperature scaling to yield calibrated probabilities
in evaluation. Concretely, we employ OpenCompass CompassVerifier-
Syy by
ply 2) = softmax
T
This helps us mitigate saturation and the entanglement formatting and confidence seen with last-token probabilities, and improves probability calibration for downstream rubric scoring.
Selected categorical schema. We define 12 schemata (Table using the rubric variables (Table 4 derived from the base signals; here are two illustrative definitions (the others follow analogously):
Format Aware:
0 invalid
1 wrong A unboxed
cat
2 wrong A boxed
3 correct A unboxed
4 correct A boxed
Conff- Conf-Wrong Penalty:
0 invalid
1 wrongtigh_com
cat
2 wrong A low_conf
3 correct
Rubric weights w are chosen to reflect evaluation preferences. For example, Formar Aware might use 0. 0, 2, to mildly reward formatting when correct and slightly penalize confidently wrong (via schema choice); Efficiency- Adjusted can downweight verbose outputs among both correct and wrong categories.
Exact Match
Correctness only; ignores formatting, confidence, and length.
Format Aware
Rewards boxed, well-formatted answers; distinguishes boxed/unboxed even when wrong.
Conf-Calibrated Penalizes confidently wrong; grades correct answers by confidence (low/mid/high).
OOD Robustness
Separates in-distribution vs. OOD prompts; checks correctness under both.
Striet Compliance Requires boxed final answers; unboxed-correct is treated as non-compliant.
Conf- WoongWrong Penalty Heavier penalty for wrong answers high confidence; lighter when uncertain.
Verifier-Only Uses verifier signals alone to rank; model-agnostic probe of the verifier.
ConfateConfidence Balanced composite over (boxed/unboxed) x (low/high confidence) for both wrong and correct; emphasizes boxed, high-confidence correctness and penalizes confidently wrong.
Length-Robust Isolates correctness irrespective of verbosity; does not penalize length.
24

---

Published as a conference paper at ICLR 2026
Verifier Prob
Probes agreement with the verifier: flags wrong with high verifier A as inconsistent and distinguishes undef/over-confddence on correct.
Afficiency-Adjusted Rewards short, correct completions; penalizes verbose outputs (especially when wrong),
Concise High-Conf Prefers concise, high-confidence correct answers; downweights verbose correctness.
Table 4: Rubric variables, decision formulas, and brief descriptions used to map each model attempt into discrete categories. Thresholds (Thigh _ow_wrong: observed bits-per-token and token-ratio statistics. Category 0 is reserved for invalid outputs (degenerate repetition or high verifier compass context compass_conteat___ C"), and compass compass_contert contert_ calibrated verifier probabilities for correct, wrong, and off-task, respectively.
wrong 'promp)) and length quantiles (len_p33, len ren_p66) are computed per dataset from
composs_contert B, compass compass_conteres context_ C' denote
Rubric variables
Formula
Description
invalid
prpeaated_pattenne = 1) V (compass_conter'" C 2 0.50)
Category 0 reserved for invalid.
correct
(is (is_correct 2 0.5)
Boolean mask of correctness.
wrong
(is (is_correct < 0.5)
Complement of correct.
high_conf
(completion_pp S Thigh
Confidence proxy Complement of high_ conf. Penalize confidently wrong.
low_cont
(completion_bpe ppptiiionnnpootoo > Thigh
wrong _
wrong (completion bpt S Tlow (ow_wrong)
ood
(prompt_bp) 2 Tprompt/
prompt. In-distribution prompt. Short completions.
ind
(prompt_ bpt < 'prompt)
economical
(token_ ratio < (le__333)
moderate
(Ien (len_p33 < coken_ratio- < len cn__666)
Medium-length completions.
verbose
(token_rath) ratiratioo len_p66)
Long completions.
boxed
(has_box 2 0.5)
Answer is boxed.
unboxed
(has_box 0.5)
Answer is not boxed.
A high
(compass catrssss__ttt____ A 0.6)
Verifier confidence high.
Thigh
40th percentile of completion_by hpt
Tow wrong
60th percentile of completion_ bpp among wrong items
T "prompt
90th percentile of prompt_bpt
len len_p33, len p66
33rd and 66th percentiles of token ratio
corr_p33, corr 166
33rd and 66th percentiles of completion ppt correct items
Figure 7 summarizes aggregated results across tasks. The leader & Qwen3-30B-A3H -onn----oB--D---iiniinn ranks first under all selected schema, but the margin to rank depends on the rubric (largest under Verifier-Only. Mid-pack reorderings are rubric) sensitive: under Verifier Prob, @ OpenThinker2-322 edges 3 gpt- oss-20b oss-20b_meduum; under calibration-hean category (e.g.. Conf-Calibbrated overtakes @ OpennThinke22332B: OOD Robustness narrows the gap between ranks and Several categories (Format Aware, Length-Rooee Robust. Strict Compliance) agree closely, indicating that once correctness is accounted for, formatting and length rarely flip top ranks. contrast, cilbbrattonffouuseee categories emphasize and penalize confidently wrong behavior, and efficiency-orient
Penalty, smallest under
Carmat+Confidentt S gpt-oss-200 high
categories favor concision. The lower tier stable across categories ( EXAONE- 4.0-1.2B, @ OpenThinker3-11.55 = WomnReesoninn-Nemorrrr--- Qwen-1.5B), suggesting rubric choice primarily reshuffles the middle while preserving extremes. Overall, the cat- egorical schema surfaces complementary facets-format compliance, calibration, efficiency, OOD robustness, and verifier alignment- moainnnnttmakinggggg making jubric-dependen differences explicit and enabling compute-efficient uncertainty-aware comparisons aligned with evaluation goals.
Sky-T1-32B-Fllsh ar PrepSeek-Rl-Disiile
F.2 DOMAIN-AGNOSTIC RUBRIC-AWARE BAYES @
The Bayesian construction is intentionally domain- agnostic: it applies whenever model outputs can be mapped into finite set of categories equipped with a rubric. The evaluator specifies
1. a mapping from raw outputs (and any side information) to categorical labels Rai E (o...., C). and
2. weight vector that encodes how those categories valued.
Given these choices, Bayes@N N returns the posterior mean p(R) as rubric-aware point estimate, and (R) as an uncertainty estimate, for any such categorical evaluation.
25

---

Published as a conference paper at ICLR 2026
Table 5: Definitions of the twelve categorical evaluation schemata used in our Dirichlet-multnomii framework. Each schema specifies decision rules over correctness, formatting (boxed/unboxed), confidence (via completion_bp prompt distribution (in-distribution vs. OOD), output economy (via noken_ratio), and verifier signals (A, B, C). These rules map every attempt into C+1 discrete categories, enabling posterior means and credible intervals for any chosen weight vector w.
Categorical Schema
Rubric
Exact Match
invalid; wrong; 2 correct
Format Aware
0 invalid; wrong A unboxed; 2 wrong A boxed; correct A unboxed; 4 correct A boxed wrong__ighh_cmm 3 correct A low_conf; 4 correct A
Conf-Calibrated
invalid; wrong low low_conf;
mid; 5 correct A high_conf
OOD Robustness
invalid: ood A wrong; ind A wrong: ood A correct; 4 ind A correct
Strict Compliance
0 invalid: wrong V (correct A unboxed); 2 correct A boxed
Konf-Wrong Penalty
invalid;
Laig__hih__come
wrong A low_conf; correct
Verifier-Onl Only
0 invalid; high C; 2 high B; 3 A 4_high
Format+Confidence
0 invalid; 1 wrong A unboxed; 2 wrong A boxed A low_conf; 3 wrong A boxed A
high_conf, 4 correct A unboxed low_conf; correct unboxed A high_conf; 6 cor- rect A boxed A low_conf; correct A boxed A high_conf
Length-Robust
invalid; wrong; 2 correct
Verifier Probe
0 invalid; 1 wrong A A high; 2 wrong A - A A_gh; correct A - A Aigh; 4 correct A
A high
Efficiency-Adjusted
O invalid; 1 wrong A economical; 2 wrong A moderate; 3 wrong A verbose; 4 correct A
economical; 5 correct moderate; 6 correct verbose
Congision-High-Conf
0 invalid; 1 wrong; 2 correct A verbose; 3 correct A moderate; 4 correct A economical; 5 correct economical high_conf
This viewpoint naturally covers subjective tasks. For instance:
In summarization, each response could be rated {bad, okay, good, excellent) or by multi-criteria scores such faithfulness, coverage, style, and harmful content. Each discrete level becomes category index k. and wi. reflects the importance of that level or criterion.
In dialogue safety, categories might distinguish {unsafe, borderline, safe), or finer Amer-grained notions such as policy violations vs. merely over-cautious refusals.
Once the labels are available (from humans or an LLM-as-a-judge), Bayes @ N provides Bayesian estimates and cred- ible intervals for any chosen rubric-based score, reusing the same closed-fomm posterior in the binary case.
Two aspects are particularly promising for future work in such subjective domains:
1. Preference-based evaluation with rubrics. When model comparisons driven by preferences (either from human experts or LLM judges), each comparison can be converted into categorical labels over rubric dimensions (e.g., faithfulness, verbosity, harmfulness). A downstream weight vector w can then fold these dimensions into single scalar score that reflects application-specific trade-offs.
2. Transferring prior evidence across related tasks. The optional prior matrix Rg in Algorithm lets us encode earlier outcome frequencies as a Dirichlet prior. For example, if summarization system has been evaluated on news dataset, the empirical category counts that dataset can serve as prior counts when evaluating a closely related dataset. This allows stable rubric distributions to reused across adjacent tasks or benchmark revisions, while still updating with new data.
An important limitation in subjective settings is that Bayes@N N does not resolve disagreement or bias in the rubric or labeling process itself. The framework assumes a labeling scheme (from humans an L.LM-based judge) and weight vector w are given; it then provides a statistically principled way to aggregate those labels and quantify uncertainty. Designing good rubrics and calibrating judges remain separate modeling decisions.
26

---

Published as conference paper at ICLR 2026
Ranking Across Categorical Schema
Qwen3-Thinking
gpt-oss-medium
3
OpenThinker2
4
gpt-oss-high
Competition Rank
5
LIMO-v2
6
EXAONE-4.0
7
gpt-oss-low
8
OpenThinker3
9
OR-Nemotron
10
DS-RI-Qwen
11
Sky-T1-Flash
Exact Match
Format Aware
Conf- Conf-Calibrate Calibrated
OOD Robustness
Strict Compliance
Conf: Wrong Penalty
Verifier-Only
Confidecn
Rongth-Robust
Verifier Prob
Efficiency-Adjuee Adjusted
Concise High-Conf
Figure 7: Competition ranks by model across selected categorical schema. Each column is combination of base signals; lines indicate how a model's 's relative position shifts when the rubric changes.
G SCORIO
Alongside this paper, we release Scorio o, open-source Python package that implements the evaluation framework presented in this work. Scorio provides a simple, unified API for computing Bayes@N, @N,,,, Pass @k, and their credible intervals, enabling researchers adopt principled Bayesian evaluation with minimal effort. The package is available on PyPI and its documentation hosted at whhttps: scorro.........// heaocs io
Installation. Scorio o can be installed via:
pip install scorio
Basic usage. All evaluation functions operate results matrix R (O....,CJMM). problems, N is the number of trials per problem, and C is the number outcome categories. An optional weight vector of length C + 1 maps each category score. Listing shows binary evaluation using both the Bayesian estimator and Pass @k:
MLM where M is the number of
Listing 1: Binary evaluation with Scorio.
import numpy as np
2 from scorio import eval
Binary outcomes: M=2 problems, N=5 trials each
27

---

Published as conference paper at ICLR 2026
R (r 1, 2, 0, 11,
11, 1, C, 1111
7
8 Bayesian evaluation (binary: w defaults to (0, 1
9 mu, sigma eva.. (R)
print (f"Bayose5: mu=(mu:.4f) sigma=(sigma: 4f)
12 Average accuracy
sigma_a = eval. (R)
print (f"avge5: (f)(a: sigma=(sigma 411"))
16 Passék
13 print (f"Pass01 [eval. pas-ppasss_ap k=1): :. :44f)")
18 print (f"Pass02 = [eval.pass_ pass__t_3 (R, k-2) :)
Credible intervals. Each estimator has companion _ci function that returns the posterior mean, standard devia- tion, and credible interval, as shown in Listing 2
Listing 2: Computing credible intervals.
958 credible interval for Bayes@N 2 mu, sigma, 1o, hi eval.bayes_ci. (R, confidence=0. 95) print (f"*Aaeee5: [f;: [(1o:.4f), (hi:.4f)1")
958 credible interval for Pass9k mu, sigma, 1o, hi eval.pass at ci (R, k=1)
print (F"Passêl: (mu:.4f) [flo:.45), (hi:.4f)1")
Categorical (rubric-based) evaluation. For graded outcomes with C > 1 categories, weight vector w specifies the score associated with each category. Listing iillustrates evaluation under three-level rubric (C 2) with partial credit.
Listing 3: Categorical evaluation with a weighted rubric.
1 Graded outcomes: O=incorrect, 1spartial, 2=correct
2 R mp.array (I (1[O, 2, 1, 0, 21, [2, 1, 1, 2, 111)
3
Weight vector: incorrect=0, partial=0.5, correct=1
6 w sp.array(10.0, (10.0, 0.5, 1.01)
7
mu, sigma
(R, w)
print (f"Bayese5 (graded) mu=(mu:.4f), sigma= (asgma=ssigmal sigm:: .44))
mu, sigma, 1o, hi eval. heal.bayes_ce (R, w, confidence=0. 95)
12 print (E"958 CrI: [1lo:.4fl, (hi:.4f))
Incorporating prior evidence. When prior evaluation data available (e.g., from previous benchmark pilot study), they can be passed as prior matrix Rº to inform the posterior, as described Section C
Listing 4: Using prior evidence.
Prior outcomes from pilot study (M=2, D=3 trials)
2 R0 np.array (111, 11, [0, 1, 011)
3
sigma eval, bayes (R, w=None, RO=RO) print (f"Bayes05 (with prior): mu= (mu: 4f), (agma=(sigmm) 4))")
Table Sollmasssmmrzzes the main Scorio API.
28

---

Published as a conference paper at ICLR 2026
Table Summary of the Scorio evaluation API. All functions accept a results matrix R {o,.... CANNN. Functions with the _ci suffix additionally return credible interval.
Function
Returns
Description
bayes (R, H, R0)
(1,07
Bayesian posterior mean uncertainty
bayes_ci (R, w, RO)
(1,0,,o, Io, hi)
+ credible interval
avg (R, w)
(a,
Weighted average and uncertainty
avg_ci (R, w)
(a, Ga, lo, hi)
+ credible interval
pass_at_k (R, k)
P
Pass@k estimate
pass_at_k_ci (R, k)
(u, 0, Io, hi)
+ credible interval
pass_hat_k(( (R, k)
Pass"k
g_pass_at aatas_atk__ta_t tau (R, k, tau)
p
G-Pass @ k;
mg pass k (R, k)
p
mG-Pass@k
H EXTENDED RELATED WORK
The evaluation of LLMs in generative reasoning tasks, under test-time scaling (e.g., via repeated 66mpling(65), has evolved to address the stochastic nature of inference and the need for robust measures of functional correctness. Early approaches relied on syntactic similarity metrics like BLEU (66) and CodeBLEU (67), which compare generated answers against reference solutions. However, these metries often fail capture semantic correctness in reasoning tasks, motivating metrics based on execution-validation or test-based validation (68 67). This limitation has shifted focus toward functional evaluation, where the generated solution assessed via a ground truth verify correctness(68) 69). In this section, we review key functional metrics, focusing on those that leverage multiple samples to scale performance at inference time. These metrics form the basis to assess LLM capabilities but often overlook probabilistic uncertainty or consistency across samples, motivating our novel Bayesian framework.
The Pass & metric, originally introduced by (68 (33 for evaluating LLMs trained on code. It measures the probabil- ity that at least one of k independently generated samples for given problem passes all associated unit tests (i.e., by matching ground-truth answers or satisfying logical constraints), offering a practical estimate of a model's 's potential performance in solving variety of complex tasks and problems. The unbiased estimator of Pass k is computed as:
Pass @ k Eproblems
(21)
where is the total number of generated samples and is the total number of correct solutions within the n trials. This estimator has smaller uncertainty in the limit of n >> k, ensuring reliable approximations. However, due to computational costs, & is often comparable to in practice, which can increase variance and weaken evaluation stability. The Pass k metric has been adapted beyond code to evaluate LLMs in various tasks requiring verifiable correctness, such as math, logic, and general reasoning (69) 77 72).
Pass^ k, introduced in (59), extends the @ass@k k metric to capture both the potential performance and the consistency of LLMss in reasoning tasks, where evaluating the reliability and stability of generated solutions is crucial. Pass is defined as the probability that all l trials are correct:
c kk
Pass*k
Eproblems
(22)
where and n retain the same meanings in Pass@k. This metric assumes that all the trials are independent and uniformly distributed, approximating the binomial distribution with a hypergeometric distribution to account for sam- pling without replacement. By requiring all & samples to be correct, Pass*k provides stringent measure of model consistency and stability.
To introduce flexibility, Liu et (66 proposed G-Pass @ kr, which incorporates tolerance threshold F E (0.0, 1.0]:
G-Pass@k: kG Eprohlems
Σ
(23)
k=
k
29

---

Published as a conference paper at ICLR 2026
where [T L] is the smallest integer greater than or equal T k. This formulation allows up to k - [T k] incorrect solutions, balancing the assessment of potential with consistency. As special case, Pass k corresponds to G- Pass@k; in the limit r - 0.
Furthermore, Liu et al. 26 introduced mG-Pass@R, k an interpolated metric that integrates G-Pass@ hi over e [0.5, 1.0]:
[ 1.0 o..
2
k
mG-Pass@k = G-Pass@k,
G-Pass@ki/k
(24)
k
-1
providing a more comprehensive measure that jointly reflects performance potential and reasoning stability,
These extended metrics have been applied to mathematical reasoning benchmarks such as LiveMathBench, MATH, and AIME, where they reveal substantial performance degradation LLMs under stricter stability requirements.
EXPERIMENT SETUP AND REPRODUCIBILITY
I.1 METRICS
Kendall's tau (r) 73) is nonparametrie rank correlation cocfficient that quantifies the ordinal relationship between two ranked sets by evaluating the consistency in their orderings. For two rankings of n items, it examines all unique pairs (i,j) where i < j:
Kendall's Tau:
A pair is concordant if the relative ordering of items i and is the same in both rankings (both place before
j or vice versa).
pair discordant if the relative ordering different.
Pairs with ties in either ranking neither concordant nor discordant.
Define n, the number of concordant pairs, n, the number of discordant pairs, and no n(n 1)/2 the total number of unique pairs. Let nu represent the number of tied pairs in the first ranking, and m2 similarly for the second ranking. The two common variants are the following:
Tau-a:
Ta
(no adjustment for ties),
(25)
Ro
Tee
Tau-b:
Th
(adjusts for ties in both rankings).
(26)
(no n1 (no - 12)
Tau-a assumes ties and may underestimate correlation when ties occur. Tau-b, which corrects for ties, is better suited for datasets with equivalent rankings.
use scipy. stats s.tass.eedaalee www.kkenalllaaa with its default variant='b', which computes ты effi- ciently and handles ties appropriately. The coefficient ranges from -1 (perfect disagreement) to +1 (perfect agree- ment), with indicating no association. This metric provides robust, distribution free measure for comparing model performance rankings, particularly when ties reflect meaningful equivalences.
In our implementation,
Convergence@n. For given bootstrap replicate, we measure convergence in terms an exact ranking match. each step ,1...., NMma compute the ranking induced by the first trials and compare it to gold-standard ranking (obtained from all Nmax trials). We then define
8 min < Nmax -1
the ranking after trials matches the gold- standsta ranking, and remains unchanged after every subsequent trial
and refer s* as the convergence@1 value for that replicate. If such s* < Nmax exists, we declare that replicate to exhibit no convergence.
I.2 MODELS AND DATASETS
Datasets. We evaluate on four math-reasoning test sets: 24ME (24), AIME'25 (5),, BrUMO'25 (36), and 25 (35). AIME is administered by the Mathematical Association of America and consists of two sets of
30

---

Published as a conference paper at ICLR 2026
ID
Model
Short name
1
DeepSeck-RR Baobeck-R1--n R-I-Disill--wen.
DS US-RI-Qwae R1-Qwen
2
LIMO-v2
LIMO-v2
3
OpenThinner2-32B
Open 0hinkein
4
OpenThinker3-115
OpenThinkens
5
Qwen3-30B-44
Qwen3-
6
Sky-TI-32B-Flan
Sky-TI-Flash
7
gpt-oss-20b_ b_ighh
gpt-oss-high
8
ppt-oss-20bllow
gpt-oss-low
9
gpt-oss-20b_ 20s 200_mmmmmmm
gpt-oss-medium
10
EXAONE-4.0-1.2B
EXAONE-4.0
11
OpeuRkessoring-eeeottoo---L3-
OR-Nemotron
12
Phi-4-reasoning
Phi-4
13
Phi-4-raasoning-plu
Phi-4-plus
14
T I
ORI-Distill
15
SascO--DS-Qw--Syye
16
Light-RI-14B-DS
Light-RI-DS
17
AccReawom-NNioorron----117
AR-Nemotron
18
IVIDIA-Nemmotro--Nnn-99---
NVIDIA-Ncmotron
19
Owen3-BB-Thinning-25777
Qwen3-4B
20
Bespoke
Table 7: Mapping between model IDs, full model names, and the shortened names used in figures and legends. Corresponding subsets are listed in Tables 10 and
15 integer-answer problems; we the 2024 and 2025 problem sets. For HMMT'25, we use the officially posted February 2025 contest set (algebra, geometry, number theory, and combinatorics). For BrUMO'25, we use the pub- lished 2025 problem sets from the tournament archive.
Models. Unless noted otherwise, we run each generator with the provider-recommended chat template (DeepSeek/Qwen style when unspecified) and identical decoding settings (below) to minimize template-induced variance. The base model cohort includes models (8 distinct models + 3 modes (low, medium, and high) of gpt-oss) as follows: Sky-T1-32B-Fie
"flash" variant tied to werthinkinngreedeeee (Qwen3 series, reasoning variant), or DeepSeek RR -I-----ii----------- Distill-Qwen 1.5B (44 (distilled reasoning model), c gpt-oss-20b (76) (OpenAI open-weight reasoning model;
(74) geasoning-optimiin
work), by Qwen3-30B A3B-Ain-Ti 2507
the default quantization, MXFP4, and, for prompting, rely on OpenAl Harmony, which defines three levels of reasoning ef- fort), LIMO-v2 (77) (data-efficient reasoning fine-tuned on curated traces),
EXAONE 4.0-1.2B (78) (hybrid
non-reasoning/rrasoning modes), Openneessoinng-Nemorrnn-- ..B
779
80
81
82
(open-weight small reasoning
model), @ OpenThiiker2-32BB
833 and OpenThinker3-1.55
(83 (trained on OpenThoughts2/3 data recipes).
investigate the effect of the number of models required to reach a stable ranking with and without credible in- tervals, in addition the 11 above-mentioned models, extend the evaluation 20 models in total (17 3): Phi-4-reasoning and Phi-4-reasoning-plle (84) (14B small language models with supervised "teachable" reason- ing traces and an RL-enhanced variant), OpenRI-Distille (85) (an open 7B distillation of DeepSeck-Ri using fully public data), FuseOl-DeepSeckR fusion of DeepSeek-R1, QwQ, and Sky-TI- T---33--FFassi of-thought model further improved with GRPO-style reinforcement learning). AceReason- Naansonn-Nmmtto-11111177 (88) (7B NVIDIA Nemotron math/code model trained on OpenMathReasoning www.soninnnnppeeeooeeeaannnnnnnn data). = NVIDIA- Transf--Trnnnmmerr "Nano 2" model with controllable reasoning mode), & @wee3-4B--o 4h-hniniig- 2507 (75) (4B "thinking" variant of Qwen3 with scaled reasoning depth), and B Bespoke-
(86) (System-lI "long-short" reasoning Light-RI-14B-DD DS (7) Qwen2.5-based long-chain-
CectotronNaao-999
(89) (a hybrid
Stratos-7B 900 (Qwen2.5-7B student obtained via DeepSeek- ---Seckk----bsedd Rb-based reasoning distillation on Bespoke-Stratos- 17k).
For verification, we additionally use CompassVerifier-3B (T), lightweight answer verifier suitable for outcome reward and equivalence checking.
Prompting. For most models, follow the provmeeereeommmedee DeepSeck/Owen-sttee prompt: "Please reason step by step, and put your final answer within \boxedf J."
For apt-oss-2000 in-
31

---

Published as conference paper at ICLR 2026
stead use the OpenAI Harmony prompt template, which provides three levels of reasoning effort. For OpenReasoni ng-Nemot ron-l. 5B, we adopt the task-specific prompt: "Solve the following math problem. Make sure to put the answer (and only the answer) inside \boxed )."
I.3 REPRODUCIBILITY
Sampling setup. All trials use top-p sampling with temperature 0.6, 0.95, batch size 1, and seeds 1234-1313. We perform = 80 trials per dataset x model.
Verifier. We use CompassVerifeer 3B as a reward model. During evaluation, we leverage the model's scores prompts generated by other models to create categorical schemas. We rely on the Transformer (64) and Accelerate (92 libraries. To maximize throughput, we enable FlashAttention kernels (22 and adopt the
DFloat11 format
(93)
Serving stack. Token generation served with vLLM (PagedAttention) (66), and models are loaded in bf16 6 unless the release requires MXFP4 (e.g.. gpt-oss). We record og-probabillttes for both the input prompt and generated tokens, and cap max_tokent at 32,768.
Hardware. All runs execute clusters with 8x NVIDIA H200 (141GB).
COMPUTATIONAL COST AND TOKEN STATISTICS
Across all tasks, we evaluated 20 models with 80 trials per model and 30 questions per benchmark, yielding total of 192,000 independent inference runs. This required 7,445 GPU-hours (~310 GPU-days) and generated 2.96B tokens 1.9663311,,160 total (see Figure for details).
Task
Inference Time (hours)
Completion Tokens (M)
AIME'24
1,699.4
680.0
AIME'25
1,878.4
728.3
HMMT'25
2,216.5
851.2
BrUMO'25
1,650.9
666.9
TOTAL
7,445.2
2,926.4
Table 8: Task-level computational cost aggregated over 20 models, 80 trials, 4 tasks, and 30 questions per task. Token counts correspond to completion tokens only.
Task-level computational cost. 25s the most expensive benchmark in terms of GPU time (2,217 GPU- hours), while BrUMO'25 25is the least expensive (1,651 GPU-hours). Figure provides of these patterns, showing inference time and completion-toke usage across models and tasks.
provides a complementary visualization
Token breakdown. Aggregating across all tasks and models, the total number of tokens (prompt completion) is 2.96B. The breakdown is:
Prompt tokens: 37M (1.2%)
Completion tokens: 2.93B (98.8%)
Average per query: 15,434 tokens
GPU-hours by model efficiency.
The 20 model configurations varied substantially in computational efficiency:
Most efficient: gpt-cop sopt-oss- 20b-low (48.4 GPU-hours for 9,600 queries)
Least efficient: LIMO-V2 (894.3 GPU-hours for 9,600 queries)
Average per query over all models: 139.6 seconds (~2.33 minutes)
32

---

Published as a conference paper at ICLR 2026
@wen3-Thinking
52566.7
Qwen3-48-
85.7
50129.3
58846.5
49098.9
Phit-reason-plus
120.6
152.2
112.1
35299
39203
5.009'
305.16.00
60000
4. gpt-oss-medium
27.2
32.8
38.2
23.5
4
22568.6 6
27128.4
31571.7
19537.6
5. hit-ass-high
35.5
42.1
74.0
56.4
5
29344.4
3666334
61128.1
40.193.6
6. AR-Nemotron
114.5
131.9
149.2
107.4
-200
34381. 9
39432.7
44446.0
32244.5
550000
7. QhenThinker2
246.2
100.7
189.0
139.3
7
26010. 3
29090.4 4
33444.
24794.5
Phi4-reason
169.8
40000
FO1 SkyT1
-150
K Tokens
Light DS
109.9
Hours
11. NViDiA-Nemoroon
159.0
189.0
131.0
38713.7
38643.2
330000
12. LIMO-V2
217.4
233.6
249.3
194.1
12:
381.36.
40066. 8
43359.0
341.39.4
13. gpt-oss-low.
18.7
21.4
4.4
3.9
13
15619.4
17849.5
3752.3
3308.9
14. 4.00
68.1
70.8
60.0
66.1
45852. 8
47259 8
53680-2
44433.
20000
15. ORI-Disull
101.0
118.3
93.6
30534.5 5
32727.3
35899.1
20452.1
Open Thinker3
50.0
47775.2
100000
DS-R1-Qwen
40.6
40357.8
38967.4
44741.9
19. Sky-T1 Flash
46.7
35.0
53.6
42.9
0278.7
6208.1
9442.8
7596.4
20. Bespoke-
74.1
55.3
73.2
62.1
43746 4
32710.7
43447.7
36924.6
AIME'24
AIME'25
HMMT'25
BrUMO'25
AIME'24
AIMÉ'25
BrUMO'25
Figure 8: Computational cost analysis. (Left) Total inference time in hours aggregated over trials and 30 questions per benchmark (2,4 22,400 inference runs per cell). (Right) Total number of completion tokens (in thousands) generated across the same runs. Models are ordered by overall performance (best to worst, top to bottom).
33

---

Published as conference paper at ICLR 2026
CONVERGENCE
While Figure shows the PMF of convergence@n,, FigureSshows the corresponding cumulative distribution functions (CDFs). For Pass @1 and Pass @8, there is no convergence, as the figure shows CDFs associated with them. The computed
CDFs are computed using the same bootstrap replicates as in Figure I The distribution of convergence using the result matrices R from the first 11 models (Table Among the 108 replications, Figure shows the worst- case scenarios in which convergence@1 n. attains its maximum value. As discussed in Section 3.3 convergence@1 depends the number of models I: I increases, convergence@1 grows. When extend the pool of LLMs from 11 to 20 models, convergence n reaches no convergence for all datasets (see Figure IT..
Bayesian
pass @ 2
pass @4
pass @ 8
1.0
1.0
AIME'25
AIME'24
0.8
0.8
No Convergence
0.6
0.6
0.4
CI Needed
0.2
0.2
0.0
0.0
1.0
1.0
BrUMO'25
25MT
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
10
20
30
40
50
60
70
80
10
20
30
40
50
60
70
80
Convergence@n
Convergence
@
Figure 9: CDF of convergence@n.. Complementing the PMFs in Figure 1 these CDFs plot P(k: S n) for the convergence threshold across AIME'24, AIME'25, HMMT'25, and BrUMO'22 25. Steeper and earlier rises indicate faster convergence. Bayes@N accumulates mass with fewer trials than Pass @ 2/4/8, and on AIME' 24/'25 the Pass curves do not reach by N Nmax Greater convergence suggests that credible intervals should reported for the evaluation tasks.
To complement the worst-case trajectories discussed in Section 3.3 and shown in Figures 10 and I1 we provide additional details on the construction of the model subsets and the resulting convergence behavior. Table 7 lists the pool of 20 LLMSs used in this analysis, together with the shortened identifiers that appear throughout the figures and tables. From this pool we construct 50 subsets of models, 20 subsets of 10 models, and 20 subsets of 15 models, as summarized Tables 9 to IT Each row in these tables corresponds to one subset, indicating which models are included and reporting, under each task, the convergence @n metric computed without credible interval; each entry is the mean over 10S bootstrap replicates. Thus, the tables make explicit how convergence @n depends not only on the task but also the particular mixture of models being compared. Aggregating across all subsets and replicates, Figure [ then visualizes the distribution of convergence @n n as a function the number of models L, confirming the trend anticipated in the main text: as L grows from 15 and ultimately the full set of 20 LLMss, the required number of trials increases and non-convergence becomes common, indicating that rank-based evaluation methods such as avg @N and the Pass @k family become increasingly unreliable without an accompanying Bayesian uncertainty quantification such as Bayes@N.
34

---

Published as conference paper at ICLR 2026
AIME'24 Convergence @75
AIME'25 No Convergence
11
10
9
Competition Rank
8
6
X0
5
MI
X
4
3
2
HMMT'2 Convergence@71 @78
BrUM0'25 Convergence@@5
10
9
Competition Rank
8
MV
W/W
7
5
4
MV
3
2
1
11
21
31
41
71
80
11
21
51
61
71
80
Trials
Trials
DS-RI-Qwen
OpenThinker2
Qwen3-Thinken
gpt-oss-high
spt-oss-medium
OR-Nemotron
LIMO-v2
OpenThinker3
Sky-TI-Flash
gpt-oss-low
EXAONE-4.0
Figure 10: Worst-case bootstrap rank trajectories. Each line shows the ranking of model as trials are added (11 models in total). Convergence is defined as the minimal N after which the ranking remains unchanged. (a) AIME M2: converges N 75. (b) AIME'25: no convergence observed within 80 trials. (c) H5 converges at N 78.
(d) BrUMO'25. 25: converges at N = 68.
35

---

Published as conference paper at ICLR 2026
AIME'24 No Convergence
AIME'25 No Convergence
20
17
Compettion Rank
14
WM
11
NNWAGO
25MT No Convergence
BrUMO'25 No Convergence
18
Competition Rank
15
51
41
Trials
Trials
DS-RI-Qwen
Qwen3-Thinking
gpt-oss-medium
Phi-4-plus
AR-Nemotron
LIMO-v2
Sky-T1-
EXAONE-4.0
ORI-Distll
NVIDIA-Nemotroa
OpenThinker2
apt-oss-high
Qwen3-4B
OpenThinker3
gpt-oss-low
Phi-4
Light-RI-DS
Bespoke
Worst-case bootstrap rank trajectories. Each line shows the ranking of a model as trials are added (20 models in total). Convergence is defined the minimal N after which the ranking remains unchanged. There is at least one no convergence replicate among the 10a bootstrapped replications.
Figure 11:
36

---

Published as conference paper at ICLR 2026
Table 9: 5-model combinations. Matrix showing model presence across the 50 evaluated combinations. Values under each task report the convergenceãnn metric computed without a credible interval; each value is the mean 100K bootstrapped samples. Model identifiers are listed in Table 7
Comb.
2
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
AIME'24 AIME'25 HMMT'25 BrUM0'25
13
14
18
19
13.5
69.0
11.9
8.8
2
1
2
14
18
20
3.0
38,4
13.3
59.3
3
1
2
13
20
1.5
3.1
2.7
2.7
10.9
12.6
5.0
20
5.5
50.2
6
10
14
17
18
11.4
15.3
7.9
3.1
7
10
12
13
19
11.0
3.3
12.5
11.1
8
10
17
18
5.9
13.1
9.8
6.3
20
44.3
2.8
9.4
10
2
13
17
18
6.8
38.8
14.6
68.4
11
10
11
14
17
19
10.4
66.9
5.3
16.6
12
11
13
14
17
20
3.7
65.1
5.4
50.3
13
10
9.2
72.0
17.2
17.7
14
12
20
2.9
4.4
5.8
11.5
15
11
12
14
20
12.9
79.0
15.0
8.4
16
2
10
11
18
19
4.7
41.1
27.4
60.3
17
2
17
20
1.8
3.2
3.1
2.3
18
38.3
14.3
59.4
19
2
12
17
19
44.3
4.2
8.1
11.1
20
10
11
18
20
1.8
13.1
7.8
15.0
21
10
12
18
20
5.4
4.1
16.4
3.1
22
12
14
17
20
44.3
4.1
7.9
9.8
23
17
14.7
71.2
19.7
50.9
24
2
4
14
18
8.4
72.3
14.0
59.8
25
4
10
12
18
20
6.3
13.6
17.1
7.0
26
18
19
20
1.4
2.9
2.0
1.8
27
10
20
15.3
2.5
5.2
28
17
18
19
20
5.4
1.6
3.3
4.5
29
10
12
17
18
20
45.1
4.1
17.5
8.4
30
2
10
11
17
18
7.3
41.1
27.5
60.3
31
7
16
39.6
73.6
38.1
13.1
32
15
16
48.0
32.9
18.6
33
5
7
16
19.5
39.5
1.6
10.4
34
5
16
23,9
67.8
6.5
3.3
35
7
16
35.7
73.1
36.9
17.1
36
7
10.2
61.1
2.3
10.2
37
7
15
16
47.3
74.7
47.4
16.6
38
5
7
29.6
75.4
37.1
12.1
39
5
6
15
16
35,6
47.0
28.6
10.1
40
16
23,9
67.8
6.4
11.0
15
36.0
10.9
13.9
42
7
15
16
40.5
59.1
28.6
15.8
43
5
6
15
16
47.8
60.1
32.9
14.7
44
7
15
43.0
72.5
39.1
16.0
45
7
15
31.3
72.6
36.9
12.1
46
16
19.8
11.0
47
5
15
33,0
64.3
10.3
13.9
48
6
7
9
16
39,6
73.6
38.1
13.1
49
5
15
16
39.9
47.0
28.6
10.5
50
16
29,8
11.2
37

---

Published as a conference paper at ICLR 2026
Table 10: 10-model combinations. Matrix showing model presence across the 20 evaluated combinations. Values under each task report the convergence@n metric computed without a credible interval; each value is the mean of 10 bootstrapped samples. Model identifiers are listed in Table 7
Comb.
2
4
7
9
10
11
12
13
14
15
16
17
18
19
20
AIME'24 AIME'25 HMMT'25 BrUMO'25
1
2
4
5
6
14
16
19
20
28.2
71.9
23.3
35.4
2
2
4
11
14
16
17
18
20
16.4
79.0
43.2
60.4
7
15
16
68.4
71.3
73.5
4
2
7
11
12
13
16
17
18
76.5
79.0
70.3
74.8
5
2
10
11
14
17
18
19
20
18.1
79.0
30.6
60.9
6
2
4
11
12
13
16
17
19
20
47.9
72.8
30.5
70.0
7
7
12
14
15
16
56.6
78.3
68.7
73.5
7
15
46.3
49.4
38.6
9
2
4
10
11
12
13
16
18
19
20.8
74.4
48.1
73.0
10
7 8 9
11
12
13
17
18
19
76,5
78.9
70.1
53.6
11
11
12
13
14
16
17
19
20
47.6
67.5
27.2
70.0
12
15
16
51.3
40.6
71.9
13
7
12
16
20
75.9
79.3
71.5
63.6
14
2
4
10
11
12
13
16
19
29.0
76.4
50.8
66.0
15
2
4
6
7
15
19
45.0
75.8
49.6
38.8
16
2
4
11
12
14
17
18
20
46.4
79.0
26.3
60.4
17
7
10
70.3
51.8
61.9
18
7
9
10
12
13
16
18
75.9
78.9
70.1
63.9
19
5
6
10
11
12
13
15
16
18
49.2
68.4
72.8
73.5
20
2
5
7
11
13
16
19
66.6
75.6
70.2
38.2
15-model combinations. Matrix showing model presence across the 20 evaluated combinations. Values under each task report the convergence@nn metric computed without a credible interval; each value is the mean of 1 bootstrapped samples. Model identifiers are listed in Table
Table 11:
Comb.
1
2
4
7
9
10
11
12
13
14
15
16
17
18
19
20
AIME'24
ALME'25
HMMT'25
BrUMO'25
1
2
4
5
6
12
13
14
15
16
17
18
19
20
59.3
76.1
72.9
77.1
2
2
4
5
10 11
14
15
16
17
18
19
20
51.3
79.1
71.7
67.6
7
15
16
20
76.6
76.4
77.1
7
12
14
15
16
17
20
76.7
79.5
76.5
77.1
5
4
5
6
7
9
12
13
14
15
16
19
20
68.5
79.6
71.7
73.9
6
4
5
6
7
11
14
15
16
17
18
19
73.1
79.7
72.7
53.2
7
2
4
7
10
12
13
14
17
18
20
76.5
79.9
70.3
69.0
7
10
20
76.5
79.6
70.3
68.9
9
2
7
9
10
11
12
13
17
18
19
20
76.5
79.3
71.5
69.9
10
2
4
7
10
11
12
13
17
18
19
20
76.5
79.7
71.5
69.9
11
4
7
11
12
13
14
15
16
17
18
76.7
79,9
76.5
75.8
12
7
15
16
20
76.6
76.4
75.8
13
2
4
5
10
14
15
16
19
20
54.4
78.7
55.8
40.3
14
2
4
5
10
11
12
13
14
15
16
19
20
50.9
79.2
55.6
73.9
15
2
4
6
11
12
13
14
15
16
17
18
19
59.4
79,2
73.0
77.1
16
10
12
15
16
20
59.3
77.0
73.0
77.1
10
16
20
52.8
57.0
74.9
18
2
4
10
11
12
13
14
15
17
18
19
20
55.9
79.2
71.4
75.3
19
2
4
5
6
7
9
10
14
15
16
17
18
73.1
78.8
72.7
66.5
20
4
5
6
7
14
15
16
17
18
19
20
73.1
78.7
72.7
53.2
38


Reviews:
[{"review_id": "i8708uYRBy", "reviewer": "Reviewer_ma3N", "summary": "The paper proposes a unified Bayesian evaluation framework that models LLM results as categorical outcomes with a Dirichlet prior, yielding closed-form posteriors and credible intervals for both binary and complex, rubric-based metrics. Empirical evidence shows that the proposed approach chieves faster convergence and greater rank stability than Pass@k and recent variants, enabling reliable comparisons at smaller sample counts.", "questions": "- How does runtime scale with large M and N (e.g., thousands of problems and multiple categories)?\n\n- How well does the proposed framework handle tasks where correctness is subjective (e.g., summarization, dialogue safety)?\n\n- How sensitive are rankings and credible intervals to poorly chosen priors? The authors use an informative prior but what happens if you change that?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally well written and easy to follow.\n\n- It offers a principled alternative to Pass@k.\n\n- The ability to save compute and provide statistically sound model comparisons is highly valuable.", "weaknesses": "- The paper only focuses on math benchmarks and functional correction domain. It would be nice to see application is diverse domains and tasks.\n\n- There is a lack of discussion on prioi choice. Can an informative prior reduce the number of trials? If so, how would that go?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kGEz0dnQfn", "reviewer": "Reviewer_4Ua4", "summary": "In this paper, the authors present a bayesian framework for LLM evaluation, especially on the reasoning tasks. The main target is on Pass@k metric, which was claimed to be with high variance, slow convergence. It considers per-item outcomes as categorical with a Dirichlet prior. Under a uniform prior, they aimed for a principled uncertainty. It shows faster convergence and greater rank stability than Pass@k and variants in simulations on real math benchmarks.", "questions": "1. As a particular interesting finding in the work, the i.i.d samples appear to be the focus. However, many modern LLM evaluation strategies, such as chain-of-thought prompting or self-consistency sampling, introduce correlations between trials that violate the i.i.d. assumption. Given that these methods are prevalent (e.g., in math reasoning tasks like AIME), how would this work handle these scenarios?\n\n2. In this paper, the experiments are primarily confined to math benchmarks, but LLMs are widely used in diverse domains (e.g., HumanEval for coding, or safety benchmarks like TruthfulQA). Testing on these could reveal domain-specific limitations, such as handling partial correctness in code or nuanced harm categories in safety, which might require different rubric designs.\n\n3. I wonder for the informative priors, what are the strategies for the authors to ensure bias is not introduced in the process? In general, it risks cherry-picking or overfitting. Especially, when the test was done with roughly 30 questions, what will be the outcome in real-world applications covering hundreds of questions or complex rubrics. \n\n4. For the experiment, it lacks details. For example, what are the details on setting up categorical rubrics and verifiers? It may also need to consider how the framework perform under noisy or adversarial data conditions? Moreover, what is the minimum sample size N required for reliable rankings across all tested datasets? How do you justify the potential overfitting to the gold standard?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "> The paper contributes on an interesting problem, which is about the unstable rankings from limited samples.\n\n> It provides a straightforward derivation linking to Bayesian approach, and has provided basic test on a few math datasets. \n\n> The interval-based decision rule is easy to understand and could help reduce some ranking noise.", "weaknesses": "- The focus is narrow. It has only covered math reasoning tasks with a small set of models, leaving its broader applicability untested.\n\n- I don't quite understand the strong claim with the i.i.d trial assumption. It may not hold for common sampling methods, potentially skewing results. In particularly, the paper mentions informative priors but doesn’t explore their practical use or risks, sticking mostly to uniform ones.\n\n- This work doesn't provided thorough comparisions with other uncertainty methods or diverse rubrics. In this case, the implementing categorical rubrics might add complexity, and scalability isn’t well addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5fuXQVVXNd", "reviewer": "Reviewer_68QZ", "summary": "This paper presents a principled Bayesian evaluation method called Bayes@N that replaces Pass@k and avg@N. The proposed method is to use posterior estimation using a Dirichlet prior. This method also produces confidence intervals; when the difference in scores between two models is within the confidence interval, the comparison should be considered as a tie. In the special case of a uniform prior, the method is equivalent to avg@N.", "questions": "- What was the approximate cost in tokens or GPU hours for running the experiments? Please consider including this information in Section F.3 Reproducibility.\n- Is there an example of a real-world use case where incorporating a non-uniform prior would be useful?\n- Could you elaborate on the real-world usefulness of the categorical schemata method that discussed in Appendix D?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Bayes@N is well-principled.\n- The method allows computing confidence intervals, which allows determining when a pairwise comparison should be considered as a tie.\n- The paper demonstrates that Pass@K leads to ranking instability, and that the ranking fails to converge.\n- The proposed Bayes@N approach is more stable and converges quickly.\n- The experiments use a range of real-world models and datasets, and show that the method works well in practice.", "weaknesses": "- The proposed method is cost-prohibitive in practice. In particular, Bayes@N requires the number of trials N = 10 to achieve Tau > 0.95, which is more trials than is usually done in practice. For instance, HELM (Liang et al., 2022) used N = 3. In the paper, the method is applied to mathematical reasoning datasets, which increases the cost further because mathematical reasoning typically requires long reasoning outputs.\n- Although the paper mentions that priors can be incorporated into the method, it does not present a use case where a prior would be useful in practice.\n- Appendix D Categorical has an interesting discussion of incorporating categorical features. However, this was not well-incorporated into the main narrative, and the argument for this technique's usefulness is not that strong.\n- Nit: In Appendix D, when `finish_reason` is `stop`, the instance is annotated as `repeated_pattern`, but this can occur for reasons other than a repeating pattern (e.g. the model continues reasoning for too long because it fails to arrive at the right answer).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NWIAmG4zl9", "reviewer": "Reviewer_4SJp", "summary": "This paper proposes replacing the widely-used Pass@k metric for LLM evaluation with a Bayesian framework that models evaluation outcomes as categorical distributions under a Dirichlet prior. The approach yields closed-form posterior means and credible intervals, enabling principled uncertainty quantification and stable model rankings. The authors demonstrate that under a uniform prior, their Bayesian estimator is order-equivalent to avg accuracy, which helps explain avg@N's empirical robustness while adding explicit uncertainty. Through simulations with known ground-truth and experiments on math reasoning benchmarks (AIME'24/'25, HMMT'25, BrUMO'25) with 11 LLM models, they show their method converges faster to stable rankings than Pass@k variants and naturally extends to graded, rubric-based evaluation.", "questions": "You mention incorporating prior evidence from past runs or domain knowledge (Section 2.4, Conclusion), but this is never demonstrated. Can you provide examples showing how informative priors affect convergence speed? What safeguards prevent gaming the system through carefully chosen priors?\n\nWhat principled approach do you recommend for choosing among the 12 categorical schemas? Should practitioners report rankings under multiple schemas? How sensitive are conclusions to schema choice?\n\nYou claim \"negligible overhead\" but provide no timing experiments. Can you quantify the computational cost relative to simply computing Pass@k or avg@N? For a benchmark with M=100 questions and N=80 trials, what is the actual runtime difference?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses a genuine pain point in LLM evaluation: Pass@k's instability with limited samples creates unreliable rankings and makes it difficult to determine when performance differences are meaningful versus noise. The principled Bayesian treatment is a natural and well-motivated solution.\nThe equivalence proof showing that Bayes@N with uniform prior produces rankings identical to avg@N (Appendix B) is reasonable and explains why average accuracy has been empirically robust. The extension to categorical outcomes via Dirichlet-multinomial conjugacy provides a unified framework for both binary and graded evaluation.\nThe framework provides actionable guidance through credible intervals: only declare ranking differences when intervals don't overlap. This \"interval-aware protocol\" reduces leaderboard churn and over-interpretation of small gaps.", "weaknesses": "While the Bayesian framework is principled, the key finding that Bayes@N with uniform prior is order-equivalent to avg@N means the main contribution is essentially formalizing average accuracy with better uncertainty quantification. The paper would be strengthened by deeper exploration of informative priors, which are mentioned but not systematically investigated. When and how should practitioners incorporate prior knowledge? What are the risks of prior misspecification?\n\nAll experiments cap at N=80 trials. Given computational costs are claimed to be \"negligible,\" why not demonstrate convergence at N=200 or N=500 to better understand asymptotic behavior? Is N=80 a result from cherry-picking? The paper acknowledges this: \"more extensive evaluations may be constrained by computing and academic budgets\", but at least one large-N experiment would strengthen claims about efficiency gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)