When LLMs get significantly worse: A statistical approach to detect model degradations
Authors: Jonas M. Kübler
Kailash Budhathoki
Matthäus Kleindessner
Xiong Zhou
Junming Yin
Ashish Khetan
George Karypis
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Benchmarking
Statistics
Accuracy
Primary area: generative models
TL;DR: LLM optimizations like quantization cause subtle accuracy drops, but even "lossless" changes introduce evaluation noise. We propose a hypothesis testing framework based  on McNemar's test that detects real degradations as small as 0.3%.

Abstract: Minimizing the inference cost and latency of foundation models has become a crucial area of research. Optimization approaches include theoretically lossless methods and
others without accuracy guarantees like quantization. In all of these cases it is crucial to ensure that the model quality has not degraded. However, even at temperature
zero, model generations are not necessarily robust even to theoretically lossless model optimizations due to numerical errors. We thus require statistical tools to 
decide whether a finite-sample accuracy deviation is an evidence of a model's degradation or whether it can be attributed to (harmless) noise in the evaluation. We 
propose a statistically sound hypothesis testing framework based on McNemar's test allowing to efficiently detect model degradations, while guaranteeing a controlled rate
of false positives. The crucial insight is that we have to confront the model scores on each sample, rather than aggregated on the task level. Furthermore, we propose 
three approaches to aggregate accuracy estimates across multiple benchmarks into a single decision. We provide an implementation on top of the largely adopted open source
LM Evaluation Harness and provide a case study illustrating that the method correctly flags degraded models, while not flagging model optimizations that are provably 
lossless. We find that with our tests even empirical accuracy degradations of 0.3% can be confidently attributed to actual degradations rather than noise.

Code: https://github.com/amazon-science/LLM-Accuracy-Stats

Published as a conference paper at ICLR 2026
WHEN LLMS GET SIGNIFICANTLY WORSE: A STATIS- TICAL APPROACH TO DETECT MODEL DEGRADATIONS
Jonas Kübler, Kailash Budhathoki, Matthäus Kleindessner, Xiong Zhou, Junming Yin. Ashish Khetan, George Karypis
Amazon
(kuebj, j, ka bud, matkle, xi xiongzho, junmingy, khetan, gkarypis] @amazon, com
ABSTRACT
Minimizing the inference cost and latency of foundation models has become a crucial area of research. Optimization approaches include theoretically lossless methods and others without accuracy guarantees like quantization. all of these cases it is crucial to ensure that the model quality has not degraded. However, even at temperature zero, model generations are not necessarily robust even to theoretically lossless model optimizations due to numerical errors. We thus re- quire statistical tools to decide whether a finite-sample accuracy deviation is an evidence of a model's degradation noise in the evaluation. We propose a statistically sound hypothesis testing frame- work based on McNemar's test allowing to efficiently detect model degradations, while guaranteeing controlled rate of false positives. The crucial insight is that we have to confront the model scores on each sample, rather than aggre- gated on the task level. Furthermore, gate accuracy estimates across multiple benchmarks into a single decision. We provide implementation uation Harness and provide a case study illustrating that the method correctly flags degraded models, while not flagging model optimizations that prov- ably lossless. We find that with our tests even empirical accuracy degradations of 0.3% can be confidently attributed actual degradations rather than noise. https: http///gttu...t... com amazon science/ LLM accuracy-Stare Stats
whether can be attributed to (harmless)
propose three approaches to aggre-
top of the largely adopted open-source LM Eval-
INTRODUCTION
The widespread adoption of Large Language Models (LLMs) has led to a surge in new approaches accelerate their inference and reduce their usage cost. The spectrum of such optimizations is wide (Zhou et al., 2024| Park et al., 2024), ranging from optimizations that leave the computed function untouched like efficient software stacks (vLLM Contributors 2024) and efficient kernels (Dao al.) 2022) over optimizations that change the function but provably maintain the output distribution like speculative decoding (Chen al. 2023), all the way optimizations that change (potentially very slightly) the model's output distribution like quantization (Kurtic et al. 2025
Frantar et al., 2023
or sparsity (LeCun al. 1989 Hubara et al., [02T]] Frantar & Alistarh
2023 Sun et al. 2024).
On paper, the distinction between theoretically lossless methods and lossy methods seems clear, however, modern L.MMs are usually served at maximum in 16-bit precision for activations, some are already trained with 8-bit matrix weights (Liu et al., (024) and the recent gpt-oss is even na- tively quantized in a mixed-precision 4-bit datatype (Agarwal et al. 2025). In this regime the non- associative nature of floating-point arithmetic, where, e.g., (a b) + + (b c). accumulates emrors throughout the computational pipeline. In practice, this manifests in the observation that the same LLM can generate different responses depending the batch in which request is run. This happens even when the sampling temperature is set to zero, where, again paper, would expect them to be fully deterministic, see Yuan al. (2055 and
the hardware, the framework, and even

---

Px. i.e., I1 ...,,NN NN Lid. Px. From the finite sample we estimate the empirical accuracy as Ý EN L (M+)) /N. Conceptually, we are estimating the mean of a Bernoulli random variable, the Published as a conference paper at ICLR 2026
recent blog of thinkingmachines.ai This suddenly blurs the line between theoretically lossless and lossy optimizations.
The purpose of this paper is to develop a rigorous statistical framework to quantify the uncertainty of such deviations and to decide whether an observed deviation is based on statistical fluctuations or whether the model got actually worse. We rigorously define the problem of detecting accuracy degradation against a baseline model (and inference stack) and describe the common pitfalls in quan- tifying their uncertainty estimates. We identify that McNemar (947)) introduced a test for a similar situation; we adapt and refine this test to provide higher test power for our setting. Our theoreti- cal analysis of the asymptotic test power provides a recommendation on how to compress existing datasets for efficient degradation detection. We then introduce three variants to aggregate tests when evaluating on multiple datasets and illustrate their pros and cons through synthetic experiments. We provide a script to run the proposed statistical tests on top of the widely adopted LM Evaluation Harness and provide extensive experiments on LLMs showing that our testing framework succeeds in detecting even empirical accuracy degradations of 0.3% as significant. To facilitate the reading, we provide an overview of our notation and the introduced algorithms in Appendix A
BACKGROUND AND RELATED WORK
2.1 ACCURACY ESTIMATION
Let X' and y be some input/output domain, in our case texts of varying length. In benchmarking
settings of LLMs commonly input E X' is fed to LLM M X Y' which produces an output E y The benchmarking task defines a scoring criterion L : y + {0, 1] that classifies the generation of the model as correct or incorrect, where for simplicity we focus on binary accuracy criteria and consider 1 as success and O as failure to solve the problem æ. We present generalization to non-binary scores in Appendix [D] Together with a probability distribution Px over the input domain X, we can define the accuracy of model M as
y = L(M()))
Now in practice, we do not have full access to the distribution Pxx. Instead we have a set of E N examples that are potentially hand curated. For the statistical treatment, model this situation by assuming that the examples are independently and identically distributed and all drawn from
estimate y has variance Varlf] >(1-2), and the number of successes (N4) follows binomial distribution. In this work we are assessing whether an optimized model M, for which we define the population accuracy as 3 and its finite sample estimate as B, has degraded accuracy. We thus want to test the null hypothesis Ho B y against the alternative hypothesis HA : < Ty On first sight this might seem as trivial as deciding whether two coins have the same probability of heads or not. However, in LLM henchmarking there is one crucial difficulty:
using the same examples T1 ......., NN to estimate & and B. Thus our accuracy estimates are not statistically independent, the same problem as in before-after tests on the same set of patients in medical treatment testing. If the accuracy estimates were independent, the variance of the difference f - 3 would simply be the sum of the two variances. However, due to the dependence it is not just a function of the separate variances alone and in particular
Varli - B) + (1-2) 3)1-3)
(2)
N
N
For our further analysis we define the bivariate random variable
I ( (X)= = (X)), (x))) where ~ Px,
(3)
ie., the distribution of LA, is a forh-forward distribution of Px. has values in 10, and we define its probabilities as P. = P(O,0), P. = P(1,0), P. P(0, 1), Pa 1)1,
httpas//thinne now.acchhcntctnncchcccccc nes nes..a a.t.////... /b) /e/ feat i nonden ing-nnttnn armin' sm- in- 1m- inferenereee
The output generation of the LLM can be probabilistic itself, for example when the generation temperature
non-zero or due to rounding errors. However, we could absorb any additional randomness formally by redefining X. Therefore, to keep the notation simple, we treat M as deterministic function.
2

---

Published as a conference paper at ICLR 2026
2.2 STATISTICS AND HYPOTHESIS TESTING
Our goal is to test null hypothesis Ho B 7 against the alternative hypothesis HA : B 7. To do this we define test statistic, which we compute on our observed data. We will derive the distribution of this test statistic under the null hypothesis and reject the null hypothesis if the observed test statistic is unusually large. We want to control two types of errors (Lchmann & Romano 2005). If the null hypothesis is true, we want to ensure that the probability
false rejection is controlled at E (0, 1). Although there is no deeper meaning to 5% this is the default choice in many papers and we will follow that. If we falsely reject Ho we call this a type-I error. Correctly characterizing the null distribution of the test statistic and rejecting observations if and only if they are above Romano
the 1 1- a percentile, guarantees that the type-I error rate is controlled at a (Lehmann 2005). We consider correct type-I eπrоr control as necessary condition for a useful test. On the other hand, the null hypothesis possible, to minimize the rate of type-ll emror.. In order define reliable tests, it is crucial to correctly characterize the null distribution and in particular its variance.
false, we want our test to reject the null hypothesis as often
In 1947 the statistician McNemar (McNemar 1947) realized the
Table 2x2 Contingency Table M
same problem as in Equation (2): "There are many situations in which the sampling variance the difference between two pro- portions (or percentages) must take into account the fact that the two proportions are not based on independent samples." Instead of treating the accuracy estimates independent, we must assess how the two models agree or differ on the individual examples x; in our benchmark. To facilitate this we collect the outcomes of the N examples of the ran- where a counts the examples where both models failed, b the examples where the baseline model M succeeded and the optimized model M failed etc., such that N = a + b b+e c = d. Furthermore, the accuracy estimates of the models are f = (b + d)/N and = (c d)/N. While the actual values of the accuracy estimates depend on and a, McNemarr (1947) noted that to decide whether two proportions (in our case accuracies and 7) are the same, only the examples where the models disagree are relevant, i.e., b and Mc-
1
M
0
a
b
1
d
dom variable M,, Ay in a contingency table, see Table [
Nemar 1947) introduced the test statistic (b-o)² and derived that under the null hypothesis (and for fixed + e) this follows approximately a chi-square distribution with one degree of freedom. This
b+t-
provided reliable test to assess equivalence of proportions in non-independent examples.
2.3 LLM ACCURACY EVALUATION AND OPTIMIZATION
In the field of machine learning, McNemar's original test was previously used to assess which of either two models performs better single task (Meshbane & Morris 1996; Rainio et al. (024). Yang et al. (2025b) generalized McNemar's 's test to be used in repeated cross- cross-validation settings to decide which algorithms performs better. However, their test derives loose bounds on the null hypothesis and is thus overly conservative.
For inference optimization of LLMs tect accuracy degradation. For a general introduction on LLM evaluation, we refer to Chang et al.) (2024). Kurtic et al. (2025) provide a comprehensive analysis of the accuracy and performance im- pact of quantizing LLMs with one of their main finding that "FP8 (W8A8-FP) is effectively lossless across all model scales." They do extensive accuracy evaluations and simply aggregate the accu- racy numbers into a mean and state the difference of the means. Most publications investigating model compression techniques follow the same practice and simply provide the accuracy estimates (Frantar et al.] 2023, Frantar & Alistarh, 2023 Sun et al. 2024 Kübler et al. 2025). While this provides some level of relative guidance it does not quantify the certainty of these estimates, which is crucial because of aforementioned correlation and even more obviously as the used benchmark datasets have very different sample sizes.
not aware of work that used McNemar's test to de-
Libraries such as the Language Model Evaluation Harness (Im-eval) (Gao et al. 2024) actually pro- vide estimates of the standard error for each task individually and they also provide a per-sample score. However, using those standard errors independent when comparing two models, overesti- mates the error of the accuracy difference, see Figure [ and Equation (2). The Open LLM Leader- board (Fourrier al. 2024) provides comprehensive set of accuracy benchmarks, that can all be
3

---

Published as a conference paper at ICLR 2026
Individual Accuracies
Opinied Noazll
Probuiiity Daulty
405
41.5
42.0
-2.5
45.3
432
-15
-1.
-0.5
0.0
0.5
15
042
0.44
0.40
0,4
0.58
0.22
0.56
0.50
0.s
Empiica Accuracy (56)
Eeepirical Aucuracy Differance (5)
Empitiall Degracation Probucility
Detecting accuracy degradation Llama-3.1 8B Instruct based on empirical estimates and their uncertainties. The optimized model M quantizes the KV cache and the attention and results in an empirical accuracy drop of 0.79% over different datasets (Section treating the accuracy estimates independent, the sampling error of the difference is overestimated, hindering the detection of actual accuracy differences (middle). Instead, we advocate to compare the ser-sample scores. Our proposed exact one-sided McNemar's test focuses the degradation probability. In cases where there is no accuracy difference, the degradation probability is 0.5 and we exactly characterize its empirical distribution. This allows detect small accuracy degradations as statistically significant, here with a p-value of 1.69e-05 (right).
Figure 1:
When naively
within the Im-eval package. Having accuracy estimates from multiple benchmarks provides the additional difficulty of deciding how to aggregate those numbers.
In recent NeurlPS paper, Dutta et al. (2024) criticize that "all [considered] quantization schemes have negligible difference in accuracy (0- 2%) compared to the 16-bit version," " yet the models suitable means to detect model degradation. However, the 2% is an arbitrary choice theirs, without taking esti- mation errors into account. We will see that when the statistics are properly taken into account such deviations are often enough to conclude a significant degradation. Instead of accuracy, Dutta et al. flip in scoring of the answer. While they show that
changed in behavior. Dutta al. (2024) thus conclude that accuracies
(2024) propose to investigate the probability this correlates well with model degradation, they miss the point that such flips also often occur even without optimizing the model, see Section 4 Furthermore, their approach lacks quantification of the statistical relevance. Yuan et al.] (2025) illustrate that for reasoning models the accuracy devia- tions in lossless cases are even larger than for non-reasoning models, reaching up to 9% accuracy deviation. While their focus is characterizing the severity of such deviations and their root causes, their work further motivates the need for a proper statistical analysis such deviations.
THEORY
3.1 EXACT ONE-SIDED McNEMAR'S TEST
The random variable Las defined in Equation links the two model evaluations of M and M. Instead of thinking of two separate estimations, can think of it as a joint accuracy experiment. We can define two probabilities:
Pt= = (x]4(M(X)) + L(M(X))] = P.
(flip),
(4)
4. Px(L(M(X))
L(M(X)) + L(M(X)) = Po/Pt (conditional degradation).
(5)
The flip probability quantifies how often the two models' scores disagree, i.e., that the flipped Dutta et al. 2024). Given that the two model predictions disagree, the (conditional) degradation
probability quantifies how often the baseline model M scored and the optimized model M scored 0. For conciseness we will drop "conditional". Furthermore, we have the following fact that we will use formulate a crisp null hypothesis.
Fact 1 (Model Degradation). The accuracy 3 of an optimized model is worse than the accuracy ? of the original model and only if the degradation probability qi is larger than 1/2.
4

---

Published as a conference paper at ICLR 2026
The test statistic we propose is the empirical estimate of the degradation probability:
b
á b
(6)
Since b is the count of degradations out of b + flips. the distribution of ÿ. is directly linked to binomial distribution, i.e., qi (b++) = b ~~ (b qu). By Fact under the null hypothesis 21 = 1/2. To be more sensitive, we compute one-sided y-values, which is justified since we only test for degradations of the optimized model. For this we can use standard libraries like sc scipy Virtanen & SciPy 1.0 Contributors 2020 to compute exact p-values of observation dj. In effect, our test is just a standard binomial test (Algorithm D. To illustrate the relationship of our test to McNemar's original test, we note that b + c is treated as constant both in and McNemar's original test and obtain
(b (b-c)² b+c
26 b+e
b+ (b (2 (b)
Thus McNemar's ' original test statistic is purely the scaled square of affine transformation of the degradation probability. Since our test is one-sided and allows for the exact computation of p-values, we call it exact one-sided McNemar test
3.2 NORMAL APPROXIMATION AND TEST POWER
Our test treats b c as constant and only relies on the exact binomial distribution. However, to have data- and cost-efficient test also matters that out of N overall examples not unnecessarily many are discarded. To build this understanding, in this subsection we analyze the accuracy degradation 5 : y - / = Pr(2q. - 1). which is directly linked the degradation probability but also takes the flip probability into account. We will derive our insights based on an asymptotic analysis, which is Section 10.3.2)
a standard practice (Gretton et al., 2012), (Casella & Berger, 2002,
For the accuracy difference 8 we can define the function
if L(M(2)) (()
D(x) =
Lf and L(M(x) = 0,
(8)
L(M(r) = and ((M()) =
Such that Ex=Px [D(X] = = P. - P. =2p(91 -1/2). Thus is the sample mean of ((z)
b-c 8= = Σ D(x,) N
(9)
The variance of the random variable D is Var (D Ps P P2)2, see Appendix B Under the null hypothesis for which q 1/2, we have Ps = P. and, quite curiously, the variance simplifies to the flip probability Var (X)] P, P. = Pt- For the accuracy difference based on N samples have Vār Pt/N 4 Assuming pt > 0 and since 8 is an empirical mean, we can use a standard central limit theorem and Slutsky's theorem (Serfling 1980) to derive the asymptotic
distribution under the null hypothesis (8 =0, Varl = P
N/
- 1), N
(10)
We not aware of any published scientific work that defines one-sided McNemar test explicitly, but do not claim is novel, see also: https:://stats.
wot.ee/ecoosssssos///8118111111 wedooe-sdeed--eeeee menemars-tess 'If' we were to ignore that the models are evaluated on the same data and hence correlated, the variance of would be the sum of the the variances of the individual accuracy estimates 2y(1->), where we used that under the null hypothesis
N
& (see also discussion around Equation 2 For optimized LLMs, is usually around 50% and Pe 10%, see Section 4 and Dutta al. 2024). For these example numbers would have Var(8) 8) (VVar Varš). This implies that in relevant scenarios for LLMs naive approach that ignores the correlation would overestimate the uncertainty of the accuracy degradation by around v5 x 2.2 and thus wrongly attrihute actual degradations statistical fluctuations, see Figure]
5

---

Published as a conference paper at ICLR 2026
Thus, could use / /N/p 5 as test statistic and an asymptotic one-sided threshold would be ta a- a)), where T denotes the CDF of standard normal, Φ-1 its inverse. This is exactly the common 2-test (Casella & Berger] 2002 Section 10.3.2). Now given a certain model degradation qu 1/2 the test power can also be derived in closed form and is (Gretton et al. 2012. Section 3)
1 + ta VN/pz6).
(11)
So the test power is solely determined by the signal to noise ratio SNR :- VN/p: 6. Now let us assume that we modify our dataset to contain only the examples where the models disagree (we can emulate this by setting
0 and d = 0 post-hoc). Then we have å - 8' L N + N' Nyt.
p.
Pc - pr' = 1, i.e., the accuracy difference increases proportionally. the effective sample size gets reduced and the new flip probability is 1. For the updated SNR, however, we have
S/P: Ne
SNR' =- N'
VN = SNR,
(12)
Pp
i... the SNR remains unchanged even if we remove examples where both models agree. Since evaluations are costly and take time, this theoretical insight leads to the following: Recommendation 1. Examples that are unlikely to flip should be removed from datasets for degra- dation analysis.
If one is to interpret actual accuracy numbers, those examples matter too. But when assessing whether model has significantly degraded or not, they add cost, but no signal. Dutta et al. (2024) Section 5) have analyzed which examples are likely to flip in classification tasks based on the margin of the highest probability token. In Section 4.1 we elaborate more on this and generalize this to generative tasks. We evaluate an approach that allows us to reduce the sample size of MMLU Pro by almost 50% while keeping most of the flipping examples that are relevant to detect degradation.
Our previous discussion leads to the recommendation to redefine dataset in order to increase their flip probability (and thus also scale the empirical accuracy degradation). This underlines once more that a degradation analysis based on fixed accuracy thresholds or flip probabilities cannot be robust across models and datasets.
As user, one might care more about the actual accuracy drop ö rather than the degradation probabil- ity qu, so one would ideally like to answer questions like "How large do need to choose N such that I can detect 0.2% accuracy drop with probability 95%?" However, the test power Equation [I] is not only a function of N and but also of the flip probability Pt. Thus, we cannot answer such ques- tions in general. In fact, for fixed accuracy degradation 5 the probability of detecting it decreases as Pt grows. While this might seem counterintuitive at first sight, using ó = 2p(1. - 1/2) we get SNR = VN- vPF 2//Npp (q) - 1/2), which grows the degradation probability increases. We illustrate this in Figure 4
3.3 AGGREGATION ACROSS TASKS
When evaluating models across multiple tasks T E N, the question arises how aggregate the observations into a single p-value. Effectively, we want to test the null hypothesis Hy % = 3, Vi e
[T] against the alternative hypothesis HA 3i E [T] : % > B, As in Section 2.2 we collect contingency tables for each task i, yielding counts bj... b......by statistical tests that aggregate these individual task results (see Appendix(A):
bb and cI...., cr. We propose three
We define b- = Σ, by and e = E, cs then apply the exact one-sided McNemar test as if all tasks formed single large dataset. This approach naturally weights tasks by their sample sizes, providing high sensitivity when degradation affects multiple tasks homogeneously. On the other hand having tasks with large sample counts that do not provide signal, i.c., for which by R Ci will harm this test. One big advantage of the pooled test is that it is robust subgrouping of tasks, because every sample will always have the same weight. The following two tests instead will lead to varying p- values depending on how subtasks are defined and whether they are grouped together.
Pooled Test:
Max Drop Test: The purpose of this test is to identify whether the most significant observed degra- dation is significant overall. This is particularly useful if we suspect that an optimization might only

---

Published as a conference paper at ICLR 2026
affect single of our benchmarks. For each task i, we therefore caleulate the standardized degrada- tion statistic 2; = -.50.5 where SE((I,) = (0, + cj) is the standard error for a given task and takes into account that the task might have different overall counts numbers of flips. Follow- ing Section 3.2 this is an estimate of the SNR for each task. By our test power consideration, we then select the task, which seems most promising for detecting an accuracy drop. Our test statistic is thus 2max = max; 2,. Since the null distribution is complex, we use Monte Carlo simulation: for each round, we generate 6,th Bi Binomia a (6; + C; 0.5), compute the corresponding 2ith values, and calculate "max- The p-value is the proportion of simulated statistics S our observed value.
SElei,J
Since we assume that all tasks are based on independent samples, we can apply the standard Fisher's method (Fisher T932). We first compute p-values p3 for each task using the exact one-sided McNemar test. Fisher's method combines these p-values using the test statistic x² = -2 - In(p:). Under the null hypothesis, this statistic follows a chi-squared distribution with 27 degrees of freedom, allowing exact p-value computation. This approach provides balanced sensitivity between the pooled test and max drop test.
Fisher's Method:
We have proposed three variants to obtain p-values to test the null hypothesis of accuracy degradation. Since those p-values are statistically dependent (as opposed to the individ- ual task p-values we aggregated via Fisher's method), we cannot tightly combine them into a single p-value. Instead we propose to consider the simple decision rule to flag accuracy degradation when any of the three tests rejects at a. Formally, via the argument of Bonferroni correction. this controls type-I error at 3xx however, as we will see from our experiments in Figure 3 the effective type-I error control usually happens even tighter than 3a.
Combined Decision:
In Appendix 3 we include synthetic experiments generating scenarios for null hypothesis (q. 1/2 for all task for varying T E N) and alternative hypothesis (q), > 1/2 for all or a single out of varying number of T' E NN tasks) and demonstrate our theoretical insights and the pros and cons of the different aggregation strategies. In particular, we find that the pooled test is most powerful in scenarios where the degradation happens across tasks, whereas the max drop test is best when only a single task is affected. Fisher's s method strikes good balance between these both extremes.
4 LLM EXPERIMENTS
We apply our proposed tests on different configurations of Llama-3.1 8B Instruct and Llama-3.3 70B Instruct (Dubey et al., 2024 as well as Mistral Small 3.1. Furthermore, we also evaluate a pruned and distilled variant of the non-instruct Llama-3.1 8B model. We run the Leaderboard v2 (Fourrier) al. (0224) benchmark suite, which contains Big Bench Hard (BBH) (Suzgun et al., 2022). ) GPQA (Rein et al. 2023), IFEval (Zhou et al. 2023), MATH hard (Lewkowycz et al.) 2022] Hendrycks cet al. 2021). ). MMLU-pro (Wang et al. 2024). and Musr (Sprague et al. 2024). The sample sizes quite different,
top of Table 5 which necessitates proper statistical treatment. The total number of examples in this suite is 25,282. For the 70B model we ignore the MATH task, as even the baseline scores close to zero due to some parsing mismatch, reducing the overall sample size to 20,282 for the 70B model.
v0.10 10.0 and Im-eval v0.4.8. Our baseline is the official BF16 model served with tensor parallelism on 8 H100 GPUs. The results of our statistical anal- ysis are shown in Table 2 and in the appendix we include the full accuracy numbers (Table 5) and contingency tables (Table 6. Table 7 Table 8 Table D. While we do not want to recommend 5% as a general significance level used practice, we use it here classify "statistically significant" degradations when either of the three p-values is below this threshold.
We use vLLM (VLLM Contributors 2024
Baselines: We are not aware of any explicitly stated rules to detect statistically significant accuracy deviation. We use the two loosely defined rules by Dutta et al. (2024) Section 1) i) "difference in aggregate accuracy metric [...] is negligible [...] (S 2%)" that is ő > 2% means degradation and ii) "the actual percentage change [flip probability] the answer can be significant (2 5%)" that is p > 5% indicates degradation.
s hhtg:://missrralaaal///sssaaaaaa
7

---

Published as a conference paper at ICLR 2026
Table 2:
Statistical analysis of accuracy degradations Llama-3.1 8B Instruct, Llama-3.3 70B Instruct, 1ommal-3.1224B-Imsteee pooled accuracy estimate over all tasks from Table 9, the accuracy difference against the baseline ò and its standard error SE( Further, the three p-values we propose and the probability of flips.
and Llama-3. 8B (non-instruct). We report the
Model
?
SEŠ
Ppool
Pmax drop
Prisher
Pt
Baseline 3.1 8B
42.43%
Baseline (rerun)
42.45%
-0.02%
0.07%
6.29e-01
8.04e-01
8.68e-01
1.31%
transformers
42.51%
0.079%
0.10%
7.87e-01
7.144-01
8.66e-01
2.76%
TPI
42.45%
-0.01%
0. 10%
5.64e-01
7.58e-01
8.88e-01
2.40%
A100
42.58%
-0.14%
0. 1.00%
9.21e-01
9.93e-01
9.81e-01
2.73%
w4a16
40.70%
1.73%
0.22%
4.80e-15
0.00e+00
2.84e-15
12.63%
FP8
42.48%
-0.04%
0.19%
6.01e-01
2.82e-01
5.49e-01
8.71%
w8al6
42.32%
0.11%
0.13%
2.11e-01
1.36e-02
1.33e-02
4.45%
FP8-dynamic
42.39%
0.05%
0.17%
4.01e-01
4.29c-01
5.19e-01
7.58%
w8a8
42.41%
0.02%
0.17%
4.63e-01
2.04e-01
2.74e-01
7.46%
KV-FP8
41.65%
0.79%
0.19%
1.69e-05
9.28e-04
4.44e-04
9.03%
Baseline 3.3 70B
57.15%
w4al6
17.70%
39.46%
0.43%
0.00c+000
0.00e+000
0.00c+00
53.07%
FP8-dynamic
56.96%
0.20%
0.13%
6.33e-02
1.13e-01
6.06e-02
3.21%
w8a8
56.55%
0.61%
0.14%
1.34e-05
4.00e-05
1.83e-04
4.18%
KV-FP8
56.87%
0.29%
0.14%
2.51e-02
4.62e-02
7.57e-02
4.18%
Baseline Mistral
56.74%
FP8-dynamic w4al6
56.65%
0.09%
0.16%
3.07e-01
5.69e-01
6.24e-01
6.83%
55.87%
0.87%
0.21% %
1.30e-05
8.04e-03
2.84e-05
10.64%
Baseline 3.1B 8B
32.72%
2:4 Sparse
30. 10%
2.59%
0.29%
1.09e-19
0.00e+00
1.89e-35
20.99%
Lossless variants: For the Llama 3.1 8B Instruct model, we first run four theoretically lossless variants. For one, we simply rerun the exact same command for a second time, next we serve the model with the transformers engine (Wolf et al., (020). then we serve the model on only single GPU with tensor parallelism 1 (TP1). Lastly, (A (0100) again using TP8. It is very important to note that even those, in theory, lossless changes lead to a notable number of flips of 1.3% to 2.8%. This fact is not noticed by Dutta al. (2024) although is known that changes the computation graph can change the generation outputs We also observe that some tasks the accuracy deviation can be over 0.5%, e.g., A100 MUSR is 0.53% better than the baseline. A too tightly chosen hard threshold rule might already flag such deviations, though the two baseline rules (6 > 2% and pt 2 5%) we consider do not flag any of those checkpoints. Similarly, our statistical framework clearly indicates that none of the models had significant degradation, in fact they all improve within expected statistical variations.
serve the model on different hardware
Lossy variants: To assess whether quantized models lead statistically significant degradations we use series of quantized model variants provided by RedHatAl (Kurtic et al. 2025). see Table These checkpoints can readily be served with vLLM without any further flags. Furthermore, the last variant KV-FP3 uses the original BF16 checkpoint with kv_cache .ateeddyye www.pptpepppp.... porpe=fpp fp8, which uses FlashAutention3 in FP8 precision (Shah et al. 2024).
Our tests clearly flag all three INT4 checkpoints w4a16 6 with extremely small p-values, indicating strong evidence that the model degraded significantly. In fact, on the w4a16 variant of the 70B model evaluation indicates bug in vLLM. We spot-checked the checkpoint in transformes ct al. 2020), where the accuracy drop is much less severe. On the 8B model also the XV-FP8 variant
(Wolf
vyLLM for example does not guarantee stahility of outcomes, see "Q: Can the output of prompt vary across runs v.L.M? http:://tssssssiiiiisoss wttp://wwww....aooooooo////na/na
8

---

Published as a conference paper at ICLR 2026
gets clearly flagged by all tests. This is particularly interesting case. Here, purely looking at the accuracy numbers without proper statistical tools would make decision hard. On a few tasks, the model did not show degradation, while on MUSR it even improved slightly. However, properly take sample size difference into account and the accuracy drops on the large datasets BBH, MATH, MMIoUprr
tests
strong signal of overall accuracy degradation. This is despite the pooled accuracy difference being only 0.78%, hence the baseline criterion 8 > 2% would not detect it. We On the 70B model, we find that KV-FP8 8 has a drop of 0.3%
further illustrate this case in Figure [] and is still detected with statistical significance, showing the power of our approach, while here both baseline criteria fail to detect the accuracy.
We also note that within the Llama 8B Instruct experiments the two models that get clearly flagged have the highest flip probability, which seemingly aligns with the findings of [Dutta et al. (2024). However, our analysis shows that the 8B FP8 has no accuracy concerns at all, while the flip proba- bility (8.71%) is almost the same as for 8B KV XV-FP& (9.03%) and thus would be flagged by the hard flip criterion. Based on the accuracy results, this might indeed be a false positive of the hard flip the 70B models the flip probability is generally lower and already quite close
rule. Furthermore, to the flip probabilities of the even theoretically lossless variants of the 8B model. We emphasize that on the 70B model none of the checkpoints would have been flagged by the baseline criteria. This underlines that focus on a proper statistical treatment of accuracy is more appropriate than an investigation of the flip probability itself.
While some quantization approaches are indeed lossless, we also investigate a pruned and distilled variant of the Llama-3.1 SB Base model. This model has large 2.6% pooled accuracy drop and p-values that are effectively zero leaving no doubts about the statistical relevancy.
4.1 REDUCING DATASET SIZES FOR FASTER DEGRADATION TESTING
Recommendation motivates that for more effective degradation evaluation, we could con- sider trimming datasets to only include examples that are likely to flip. Dutta et al. (2024) Section 5) provided analysis for classification tasks showing that examples are likely to flip if the probabilities of the top two answers are close,
1100
2300
Numbar Oscuments
2262
whereas we here aim to identify examples that flip in generative tasks. We use Llama-3.1 8B Instruct and the generative variant of MMLU- Pro (Wang et al. 2024) as an example because it is a very large dataset (12,032 examples) so here a reduction is most relevant to reduce the overall evaluation runtime of compressed models.
310
Number of Successsss (out 10 scots)
Figure 2: Success histogram for MMLU-Pro.
The intuition of our approach is that examples whose score flips when adding a bit of noise will also likely be the flips in actual degradation tests. On the other hand examples that are always correctly or always incorrectly solved even with noise are unlikely to flip in degradation tests. We thus run multiple iterations of the base model at finite temperature (10 iterations, Temp=0.3 chosen to lead to & 20% flips each time), whereas the task is usually evaluated at temperature 0. For each document in the dataset we then count how often it is solved correctly out of the 10 runs and plot this in histogram, see Figure It quite striking that there are 5,604 examples in the dataset that never flip. We propose to remove those from the dataset when testing many different optimization approaches and only keep the other examples that are more likely to flip and thus provide relevant information for degradation testing. To verify how relevant the identified examples from our simulation are for the real optimization degradation tests, we run KV-FP8 and w4a16 6 variants against the original model. Using all examples we obtain 2,472 12,032 (20.55%) flips for w4a1e and 1,864 12,032 (15.4 (15.49%) for KV- FP8. After removing the 5,604 examples we identified to never flip in Figure 2 we obtain 2,1 / 6,807 (31.09%) for w4a16 6 and 1,662 / 6,807 (24.42%) for KV-FP 8. Our approach thus allows us almost halve the dataset size, while keeping most of the actually relevant examples in it. We believe that this is a promising direction to create compressed datasets in the future that are specifically designed for an efficient statistical evaluation of model degradation.
9

---

Published as a conference paper at ICLR 2026
5 INTERPRETATION, EXTENSIONS AND DISCUSSION
Statistical hypothesis testing is a tool to facilitate binary decisions based on noisy observations. There will always be cases of type-I and type-ll errors and we can only aim to minimize their rates. If the requirement is that an optimization is lossless and our tests flag it, we have clear and quantifiable statistical evidence that there is some accuracy degradation. In simplified terms, the p-value provides the probability of observing such an extreme empirical accuracy degradation if in fact the models are equally good. However, as is evident from the theory even
Interpretation of test outcomes:
and also illustrated in Figure [S] there is no magical discontinuity at the test threshold, checkpoint that barely passed might still have some (small) accuracy degradation. To provide a more comprehensive view, derived the variance formula of the accuracy difference and report those in our code package and in Table 2 Furthermore, the usefulness of an optimized model is not solely dependent on its accuracy but also its expected performance gains. Thus, a checkpoint can still be useful despite having some accuracy loss.
Extensions of our tests: Some benchmarks use metries that are non-binary, for example text sum- marization (Nallapati et al. 2016). For simplicity, we focused our main discussion on the binary case because of the special form of Equation (8). In Appendix [ we generalize our tests to non-binary metrics via a permutation approach. There we illustrate that the permutation-based test results in al- most exactly the same p-values for binary metrics. On non-binary metries, we could use the binary tests by thresholding scores, or adapting the contingency table use b and win rates. However, this loses relevant information and our experiments in Appendix D.3 and Appendix show that our permutation approach is more effective. For non-binary scores we thus always recommend to use the tests of Appendix D
Our tests being one-sided could also be considered limitation, however, it is deliberate choice. Since the binomial distribution is symmetric for our null hypothesis (g,- = 1/2), we obtain 2x smaller p-values whenever there is empirical accuracy degradation. This allows to be more sensitive when detecting degradation. If one were interested in general deviations, an adaption to a two-sided variant straight-forward
For simplicity, we focused our investigation on optimized model checkpoints. However, an equally important use-case for our proposed testing framework is integration testing in inference engines. For example, when developing a new kernel optimization or implementing an algorithm for specu- lative decoding, one can directly use the proposed tests to reliably assess the null hypothesis that the model did not accidentally degrade and thus to detect bugs in the implementation.
Optimizing the performance of large LLMs is crucial and with that the assessment of the accuracy implications. Prior work has considered the empirical accuracy drop to qualitatively assess which quantization schemes are more favorable than others (Kurtic et al.] (025). Recent work by Dutta et al. (2024) has proposed to use the flip probability to assess whether model is degraded or not. However, both approaches lack a proper treatment of the estimation uncertainty. In this work, we advocate that detecting model degradation should rely the degradation probability rather than the flip probability alone. The flip probability rather indirectly affects our criteria by increasing the effective sample size (b c) that tests can and in Section 4.1 we demonstrated how adapt datasets to relatively increase the flips. Furthermore, as we demonstrate, flips happen naturally also in theoretically lossless optimizations like changes of the hardware or the serving configuration. The root cause is that modern LLMs and serving frameworks rely on 16-bit data types for activations, which can cause harmless numerical errors. Thus, there is no principled way to formulate null hypothesis about flips, whereas for the degradation probability it is given by Fact [
Discussion:
The exact one-sided McNemar test is grounded in a statistical foundation and we demonstrated its usefulness for detecting model degradations in LLMs. However, such methods have yet to receive sufficient attention within the L.MM research community that evaluates optimized models. Further- more, the problem of aggregating outcomes from different benchmarks is not trivial and does not have a unique solution. Our three tests pooled, max drop, and Fisher are well motivated and cover different degradation scenarios as we illustrate in our synthetic experiments. Furthermore, on rel- evant LLMs they are powerful enough to detect even empirical accuracy degradations of 0.3% as statistically significant. We provide script for the existing Im-e package (Gao et al.; 2024) to generate the statistical insights and analysis presented in this work. We anticipate that this will facilitate the future assessment of accuracy degradations both in inuustrial and academic contexts.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applehaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss- 120b & gpt-oss-20b model card. arXiv:2508. 10925, 2025.
George Casella and Roger Berger. Statistical inference 2nd edition. Duxbury, 2002.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang. Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang. and Xing Xie. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol, 15(3), March 2024.
Charlic Chen, Sebastian Borgeaud, Geoffrey Irving, Yean-Baptiste Lespiaul, Laurent Sifrel, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv 2302.01318, 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv 2407.21783, 2024.
Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, and Ramachandran Ramjee. Accuracy is not all you need. In NeurlPS, 2024.
R. A. Fisher. Statistical methods for research workers (4th ed.). Oliver & Boyd, 1932.
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open Ilm leaderboard v2. https: open 11m Leaderboard, 2024.
http://acc...o...pccs///eeeeeee. 11m- wtw.aeeeeeennneaaeeeeeeeeeeeeeee leaderboard/
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training
quantization for generative pre-trained transformers. ICLR, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. BRRLhttps::/ /ttp://eeeooooooooo org http/recorrdsee 112608626662
Arthur Gretton, Dino Sejdinovic. Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. Advances in neural information processing systems, 25, 2012.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in llms through reinforce- ment learning. Nature,
2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accel- erated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in neural information processing systems, 2021.
Jonas M. Kübler, Yu- Yu-Xiang Wang, Shoham Sabach, Navid Ansari, Matthäus Kleindessner, Kailash Budhathoki, Volkan Cevher, and George Karypis. A proximal operator for inducing 2:4-sparsity. Transactions Machine Learning Research, 2025.

---

Published as a conference paper at ICLR 2026
Eldar Kurtic, Alexandre Noll Marques, Shubhra Pandit, Mark Kurtz, and Dan Alistarh. "give BF16 or give me death"? securacy-pefforanne trade-offs in LLM quantization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, Vienna, Austria, July 2025. Association for Computational Linguistics.
H Oliver Lancaster. Significance tests in discrete distributions. Journal of the American Statistical Association, 56(294):223-234) 1961.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 1989.
Erich Leo Lehmann and Joseph P Romano. Testing statistical hypotheses. Springer, 2005.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dye, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Mísra. Solving quantitative reasoning problems with lan- guage models. arXiv:2206. /4858, 2022.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguisties.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv:2412.19437, 2024.
Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153-157, 1947.
Alice Meshbane and John Morris. Predictive discriminant analysis versus logistic regression in two-group classification problems. Annual Meeting of the American Educational Research Association (New York, NY, April 8-12, 1996)., 1996.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv: 1602.060000 2016.
Youngsuk Park, Kailash Budhathoki, Liangfu Chen, Jonas M. Kübler, Jiaji Huang, Matthäus Klein- dessner, Jun Huan, Volkan Cevher, Yida Wang, and George Karypis. Inference optimization of foundation models ai accelerators. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2024.
Belinda Phipson and Gordon K Smyth. Permutation p-values should never be zero: Calculating exact p-values when permutations are randomly drawn. Statistical Applications in Genetics & Molecular Biology, 91), 2010.
Oona Rainio, Jarmo Teuho, and Riku Klén. Evaluation metrics and statistical tests for machine learning. Scientific Reports, 14(1):6086, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv:2311.12000 2023.
Robert Serfling. Approximation theorems of mathematical statistics. John Wiley & Sons, 1980.
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention--:: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37:68658-6866 2024.
Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of chain-of- oh-touht with multistep soft reasoning. arXiv:2310. 16049, 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. ICI.R, 2024.
12

---

Published as a conference paper at ICLR 2026
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou. and Jason Wei. Challenging big- bench tasks and whether chain-of-thoughh can solve them. arXhv:2210.092611 2022.
Pauli Virtanen and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Com- puting in Python. Nature Methods, 17:261-272, 2020.
vLLM Contributors. https//ggiiubbcooo//lllm-ppoeeep
Mar 2024.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiiv2406.01574. 2024.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. wwXv:25005.09388, 2025a.
Jing Yang, Ruibo Wang, Yijun Song, and Jihong Li. Block-regularized 5 x2 cross-validated mene- mar's test for comparing classification algorithms. UrXIv:2304.03990. 2025b.
Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. Understanding and mitigating numerical sources of nondeterminism in LLM inference. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instructionnfollooiing evaluation for large language models. arXiv:2311.09911 2023.
Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey on efficient inference for large language models. arXiv:2404. 14294, 2024.
13

---

Published as a conference paper at ICLR 2026
A NOTATION AND ALGORITHMS
In Table we summarize the notation that we use throughout the paper and Algorithms describe our (aggregation) algorithms.
Table Notation Summary
Symbol
Description
Formula
M, M
Baseline and optimized models
N
Total sample size Number tasks
T
a, b, c, d
Contingency table counts
Pt
Flip probability (models disagree) (Conditional) degradation probability
Pt Ebb+d/M
qu
P Po/Pa Pt
d.
Empirical degradation probability
4. b/(b+c)
7.
Baseline and optimized model accuracies Accuracy difference
8
8 3
8
Empirical accuracy difference
8 (b c)/N
Po. Po
Expected proportions
P. Ptq., Pe = Po=pr(1 -VL)
((x)
Difference indicator function
See Equation 8
Significance level
+
Standard normal CDF
h-1
Standard normal quantile function Signal-to-nois ratio
SNR
see Section 3.2
diips
Number of disagreements
Piiips 6
b, C
Task-specific counts
For task E {1,..., T} & (Ā); 0.5)/SE(gL,) SE V0.25/ + +
&
Standardized degradation statistic
SE(()))
Standard error degradation estimate
Algorithm McNemar: Exact One-Sided McNemar Test
Require: Contingency table with counts 6, c. Significance level
1: flips + b bbc
2: if fflips then
3: return "No disagreements Fail to reject Hp"
4: Reficpe
5: return peal <- binomtest( = b, = filppss P = 0.5, "errnativ="greaattrr)paa
Algorithm Pooled Test
Require: Lists brists Clist for T tasks. Significance level a.
1: Σx brsst
2: c c D Crisi [d]
3: return McNemar(6 - b,
B VARIANCE OF THE ACCURACY DIFFERENCE INDICATOR
We here derive the variance of the accuracy difference indicator function defined if L(M(x)) = (x)).
((x)= J
if L(M(())
and ((M(x) = 0, and ((M())
(13)
-1 if (f(())))
Denoting the population probabilities with P., Ph. Pc, Pa we have
E[D(X)] Pb Po.
(14)
E (X)) Ps + P..
(15)
14

---

Published as a conference paper at ICLR 2026
Algorithm 3 Fisher Aggregation Test
Require: Lists bliats Clist for I tasks. Significance level
1: for = 1 to T do
2: ps f- McNemarrb blist [i], = Ciist ())
3: Xstat + 2- i=1 In(p:)
4: return peal < chi2 2. s(aata) 2T)
Algorithm Max Drop Test
Require: Lists blists Clist for T tasks. Simulation steps S.
Wist [bisst i] + Crist [i]]?
2: for to S do
3:
for 1-1t0 To
4: bysm [s, 4 Aannomial(npaa 0.5)
5:
Ísim is
b.cm [s.4]
0.5
[
6:
a(s, i| + V
0.25
Misure [+]
7:
Zsim [s] f- max;
[esm
[[s.t]
8: Goda [r] f
biian (+)
0.5, Tobs [1] V
U.25 mesut [
Micas ())
gaks is]
9: Zabs max; doba
10: return peal= + E. 1 1z [s] 2 2obel
And obtain
Var[D(X)] EED [x)
(16)
P. + Pc -P. P
(17)
C FURTHER EXPERIMENTS AND DETAILS
C.1 LLM EXPERIMENTS
As detailed in the main paper, our default for running the experiments was to use tensor parallelism of degree 8. However, for the A100 and the KV8 experiments, had MATH a single device, i.e., TPI as otherwise we encountered a bug in the collectives in vLLM.
In Figure we additionally illustrate how the p-values of the pooled test depends on the overall sample size N, the empirical accuracy degradation as well as the flip probability Po-
C.2 SYNTHETIC EXPERIMENTS
To gain further insights into the properties of the proposed aggregation strategies, we evaluate the statistical tests across three scenarios with varying numbers of tasks T (1, For each task, simulate N samples where N ~ Uniform(500, 10000), with flip probability 0.1 across all scenarios. We simulate 1000 experiments per configuration and test at significance level = 0.05.
Scenario 1 (Type-I Error):
Under the null hypothesis, we set degradation probability q, = 0.5 for all tasks to measure false positive rates.
Scenario 2 (Balanced Degradation): Under the alternative hypothesis, we set q = 0.52 uniformly across all tasks to simulate consistent model degradation.
We set = 0.58 for the first task only and q = 0.5 for all remaining tasks to simulate degradation for single specific task.
Scenario 3 (Single Task Degradation):
For each scenario, we report rejection rates at the 5% significance level Figure B type-I error rates for Scenario 1, and test power (1 ttype-ll error rate) for Scenarios 2 and 3. First observe
15

---

Published as a conference paper at ICLR 2026
Table 4: Full model checkpoint specifiers used in the experiments. We refer the HuggingFace
model cards for full information about their configurations.
ID
HuggingFace Model Repository
Llama-3.1 8B
metta-llamaaLluma33---88--.
w4a16
http:///eeeeeaaaaaa-----------n- 64
FP8
w8al6
http://www....aaaaaa----a--aana- w8a16
FP8-dynamic
w8a8
SecClaaAJlMMaaaaaammmmslll------ w888
KV-FP8
mette/lamm/LLamm3331188----t-u vLLM flag wtl.kkkc.cceeeeee kv.cachh.dt.ye=-fpp
Llama 3.3 70B Instruct
metaallana//llanne333--0----
w4al6
-ns-tnnttttuuunnizedd.. w4a16
FP8-dynamic w8a8
KV-FP8
mCta:llaa//Llamm3333770----nnnee vLLMM flag kv-cache dt
Llama 3.1B 8B 2:4 Sparse
meta-llamaa/llaaa--3--1811
Mistral
FP8-dynamic
RellFatkAUWMisrra-SSaall1312-2 I-ntt--nntttuttt--eee----------- Instruct-2503 2503-FP8--0- FP8-dyyaamicc
w4al6
gpt-oss 20B KV-FP8
openai/ggttoos--00
openai/gpt-oss-200 20b vLLM flag kv.cache.dtype--pp
Hyprback
Rebrecdd Begndstion
Siyle Tack Deppstrin
Mos Dhy
Mos 2oe
0.0
Ya
6.2
mowe
Teat Dovee
6:00
Mae Dre Toe Meprreprpiin
Figure 3: Rejection rates the proposed aggregation schemes dependent on the number of tasks Error bars are left implicit and are given by the variance formula Var[p] p(1 p)/ 1)/1000, since estimate each probability from 1000 synthetic experiments.
that all our tests correctly control (up to finite sample deviations) the type-I error or below the significance level a = 5%. For larger T the Fisher aggregation is slightly more conservative, which can be explained by the fact that the binomial distribution discrete. Thus the individual p-values not exactly uniformly distributed but slightly more conservative, effect that disappears when each task has a large number of flips (Lancaster [961).
For the test power scenarios, expected, the pooled test yields the highest rejection rate when all tasks are equally affected by degradations, whereas the Max Drop works best on single task degradation. For single task degradation, the experiment also nicely illustrates the downsides of adding data(sets) that do only add noise but no signal. both simulated scenarios Fisher strikes good balance in between the two extremes.
16

---

Published as a conference paper at ICLR 2026
Table 5: Accuracy results of optimized Llama-3.1 8B Instruct, 3.3 70B Instruct, Mistral Small, and
Llama-3.1B 8B (non-instruct) variants.
Model
BBH (5761)
GPQA (1192)
IFEVAL (541)
MATH (5000)
MMLU-Pro (12032)
MUSR (756)
# Samples +
Baseline 3.1 8B
50.53%
28.61% %
74.86%
45.22%
37.56%
38.49%
Baseline (rerun)
50.53%
28.61%
75.05%
45.30%
37.56%
38.49%
transformers
50.69%
28.52%
73.94%
45.18%
37.77%
38.49%
TP1
50.62%
28.69%
74.49%
44.86%
37.68%
38.76%
A A100
50.62%
28.61 %
74.86%
45.26%
37.77%
39.02%
w4a16
50.10%
28.02%
70.98%
41.18%
36.06%
38.10%
FP8
50.79%
29.78%
73.94%
44.28%
37.86%
38.23%
w8al6
50.60%
27.52%
74.12%
44.56%
37.77%
37.57%
FP8-dynamic
50.06%
28.61%
76.16%
44.72%
37.84%
38.36%
w8a8
50.84%
28.36%
75.97%
44.36%
37.78%
37.17%
KV-FP8
50.25%
28.61% %
74.86%
43.20%
36.84%
39.02%
Baseline 3.3 70B
69.02%
31.38%
89.09%
53.41 %
44.18%
w4a16
27.58%
26.51%
12.75%
11.17%
35.85%
FP8-dynamic
68.91%
30.62%
88.54%
53.26%
43.65%
w8a8
68.63%
30.96%
89.09%
52.66%
43.39%
KV-FP8
69.05%
30.70%
89.65%
52.94%
44.31%
Baseline Mistral
66.98%
39.85%
63.96%
52.80%
55.45%
46.56%
FP8-dynamic w4al6
67.14%
39.77%
63.40%
52.62%
55.26%
47.22%
66.50%
38.26%
58.41% %
51.34%
54.85%
46.96%
Baseline B. 8B
46.42%
29.61 % 28.52%
7.95%
19.02%
32.92%
38.49%
2:4 Sparse
46.50%
13.31 %
12.70%
29.41 %
46.56%
0.2
0.55
1.4
-14
0.8
122
0.5
-12
0.6
Aecutay Dogi
Accussy Degndaoon (%)
Accuacy Degpadatis (%)
-04
-02
0.0
26p Probabikkty
Fin Probubilcy
Figure 4: Asymptotic test power for N 25, 282 and = 0.05 as function of flip probability pt and degradation probability d1-
We also additionally show the rejection rates of an aggregated test that rejects any of the three tests rejects at 5%. This test of course does not control correctly at the significance level, however, captures all potential scenarios.
17

---

Published as a conference paper at ICLR 2026
Table 6: Full contingency table for Llama-3.1 8B Instruct experiments. b, d are as defined in
Table I and pt denotes the flip probability (cf. Section 3)
Model
BBH
GPQA
IFEVAL
MATH
MMLU
MUSR
Total
Baseline (rerun)
0
1
162
163
Baseline (rerun)
0
2
166
168
Baseline (rerun)
2911
341
404
2099
4519
291
10565
Baseline (rerun)
Pt
0.00%
0.00%
0.55%
6.56%
0.00%
0.00%
1.31%
TPI
a
2812
847
119
2579
7430
462
14249
TPI
b
33
3
19
178
68
302
TP1
38
4
17
160
83
305
TP1
d
2878
338
386
2083
4451
290
10426
TPI
Pt
1.23%
0.59%
6.65%
6.76%
1 1.25%
0.53%
2.40%
A100
2810
838
119
2554
7409
461
14191
A100
13
17
183
79
327
A100
40
13
17
185
104
363
A100
2876
328
388
2078
4440
291
10401
A100
Pt
1.30%
2.18%
6.28%
7.36%
1.52%
0.53%
2.73%
w4a16
2480
809
107
2368
6981
431
13176
w4a16
395
49
50
573
712
37
1816
w4a16
370
42
29
371
532
34
1378
w4al6
2516
292
355
1688
3807
254
8912
w4a16
Pt
13.28%
7.63%
14.60%
18.88%
10.34%
9.39%
12.63%
FP8
a 2646
812
109
2362
7070
449
13448
FP8
b
189
25
32
424
407
18
1095
FP8
c
204
39
27
377
443
16
1106
FP8
2722
316
373
1837
4112
273
9633
FP8
Pt
6.82%
5.37%
10.91%
16.02%
7.06%
4.50%
8.71%
w8a16
2767
847
110
2469
7350
462
14005
w8a16
30
303
138
10
577
w8al6
83
4
26
270
163
549
w8al6
2832
324
375
1958
4381
281
10151
w8a16
Pt
2.81%
1.76%
10.35%
11.46%
2.50%
1.72%
4.45%
FP8-dynamic FP8-dynamic
2670
830
106
2385
7157
454
13602
b 207
21
23
379
322
12
964
FP8-dynamic
180
21
30
354
356
952
FP8-dynamic
d 2704
320
382
1882
4197
279
9764
FP8-dynamic
Pt 6.72%
3.52%
9.80%
14.66%
5.63%
3.04%
7.58%
w8a8 a 2674
827
104
2382
7171
455
13613
w8a8
158
27
26
400
315
20
946
w8a8
176
24
32
357
342
10
941
w8a8
2753
314
379
1861
4204
271
9782
w8a8
Pt
5.80%
4.28%
10.72%
15.14%
5.46%
3.97%
7.46%
KV-FP8
2616
826
113
2392
7120
445
13512
KV-FP8
250
25
23
448
479
16
1241
KV-FP8
234
25
23
347
393
20
1042
KV-FP8
2661
316
382
1813
4040
275
9487
KV-FP8
Pt
8.40%
4.19%
8.50%
15.90%
7.25%
4.76%
9.03%
18

---

Published as a conference paper at ICLR 2026
Table 7: Full contingency table for L.ama-3.3 70B Instruct experiments. a, b, c, are as defined in
Table
and pt denotes the flip probability (ef. Section 3).
Model
BBH
GPQA
IFEVAL
MMLU
MUSR
Total
w4al6
1342
595
53
5002
318
7310
w4al6
b
2830
281
419
5686
167
9383
w4al6
443
223
6
604
104
1380
w4a16
1146
93
63
740
167
2209
w4a16
Pt
56.81 %
42.28%
78.56%
52.28%
35.85%
53.07%
FP8-dynamic
1708
813
51
5394
418
8384
FP8-dynamic
11
230
346
FP8-dynamic
77
5
8
212
306
FP8-dynamic
3893
360
471
6196
326
11246
FP8-dynamic
Pt
2.78%
1.59%
3.51% %
3.67%
1.59%
3.21%
w8a8
1656
810
48
5396
418
8328
w8a8
151
13
11
300
10
485
w8a8
129
8
11
210
362
w8a8
3825
361
471
6126
324
11107
w8a8
Pt
4.86%
1.76%
4.07%
4.24%
1.85%
4.18%
KV-FP8
1665
812
51
5352
415
8295
KV-FP8
118
14
5
310
453
KV-FP8
120
6
8
254
395
KV-FP8
3858
360
477
6116
328
11139
KV-FP8
Pt 4.13%
1.68%
2.40%
4.69%
1.72%
4.18%
Table 8: Full contingency table for Mistral Small experiments. a, d are as defined in Table and pt denotes the flip probability (ef. Section 3).
Model
BBH
GPQA
IFEVAL
MATH
MMLU
MUSR
Total
FP8-dynamic
1793
670
163
1912
5164
383
10085
FP8-dynamic
b
100
48
35
457
219
16
875
FP8-dynamic
109
47
32
448
196
21
853
FP8-dynamic
d
3759
427
311
2183
6453
336
13469
FP8-dynamic
3.63%
7.97%
12.38%
18.10%
3.45%
4.89%
6.83%
w4a16
1711
668
161
1856
4942
365
9703
w4a16
219
68
64
577
490
36
1454
w4a16
191
49
34
504
418
39
1235
w4a16
3640
407
282
2063
6182
316
12890
w4a16
Pt
7.12%
9.82%
18.11%
21.62%
7.55%
9.92%
10.64%
Table 9: Full contingency table for Llama-3.1 8B Base experiments. a, a, d are as defined in
Table and Pt denotes the flip probability (cf. Section 3).
Model
BBH
GPQA
IFEVAL
MATH
MMLU
MUSR
Total
2:4 Sparse
a
2446
657
442
3733
7065
341
14684
2:4 Sparse
b
636
195
27
632
1428
63
2981
2:4 Sparse
641
182
56
316
1006
124
2325
2:4 Sparse
2038
158
16
319
2533
228
5292
2:4 Sparse
Pt
22.17%
31.63%
15.34%
18.96%
20.23%
24.74%
20.99%
19

---

Published as a conference paper at ICLR 2026
Probebilyy
Empertel Artrmyy Degidaaiie 1%
64
633
2.06
6.2
6.1-
10
1000
1900
20006
25000
30000
30900
40000
45400
50000
10000
19900
20000
25600
30000
35000
40000
45003
5000
Semple Size
Sumple Size (N)
Figure 5: p-values of the pooled test as a function of sample size N, empirical accuracy degrada- tion 8 and different flip probabilities. Observations above the dotted line are rejected as significant degradations at 5%.
20

---

Published as a conference paper at ICLR 2026
D GENERALIZATION TO NON-BINARY LOSS
For ease of presentation in the main paper we focused on binary scores, where we can model ev- erything via binomial distributions. A naive way to accommodate non-binary losses is either threshold them to binary scores or to simply populate the contingency table by counting how often the M scores higher than . and vice-versa, and just ignoring cases where they have the same score.
However, for any reasonable scoring criterion, we should assume that a score of 0.9 versus 0.1 carries more information about degradation than 0.6 versus 0.59, for example. In the following we motivate that this is also the right approach to take when evaluating models multiple times on the same input with finite temperature (Appendix D.I) and then introduce a generalized test based on permutations in Appendix D.2
D.I MULTIPLE RUNS OF THE SAME BENCHMARK
Many recent LLMs are reasoning models and use longer token sequences to reason about a task be- fore providing final answer (Guo et al., 2025 Agarwal et al. 2025 Yang et al., 2025a). To achieve highest accuracy, such models are usually used at non-zero temperature. This brings additional ran- domness into the evaluation process, as an input example will now generally have certain chance of being correct or false. To reduce this additional noise such models are often evaluated multiple times, especially for tasks with small datasets. For example, as of fall 2025, the Artificial Analysis Intelligence Index runs the AIME25 task, which only has 30 datapoints, ten times. Whereas they evaluate MMLU Pro, which has over 12,000 examples, only a single time Similarly, the GPT-OSS evals package, by default runs benchmarks multiple times [S)
If we were to use our binary framework and just treat repetitions of the same samples independent runs we would obtain unreliable p-values. To see this, consider the limiting case where the reruns are all done at temperature 0, or an arbitrary small finite temperature. And let us therefore assume that all R. E N reruns lead to the same contingency table values b and c. Then adding more reruns clearly does not increase the statistical significance of the results. Yet, if b > c, the p-value of the binomial test can get arbitrarily small when R + co.
Instead, we need to aggregate the score on the sample level, which we do by generalizing our scoring criterion to continuous values. For an example 24, the score is now the expectation over infinite repetitions, i.e., (x.) limg-+oo 7. Σ/ 12
L(M (I;, r)), where enumerates the repetitions and we leave the temperature settings implicit. For R E N, for a given task we thus in the end have a list of finite-rpettttonn estimates
LM (I)) where LM (x:) = -R Em= ((M(),;)))
D.2 PERMUTATION-BASE CRITERIA FOR NON-BINARY SCORES
For continuous scores, we cannot use tests based on the binomial distribution anymore. For large datasets, we could use tests based the asymptotic normality discussed in Section 3.2 However, since some datasets are small. this would require additional care to check whether the asymptotic approximations are usable. We instead rely on a permutation-basse tests that guarantees type-I error control for any sample size. Under the null hypothesis, both models should perform equally well and thus any test statistic we compute based on [LM (æ1),.... Lis (2:) )] and [L (x1), )..., L (x should not be systematically larger than we computed it on two lists where we randomly permute the i-th entry of the list for all i. We generally draw m E N permutations (with replacement) and count with E N the cases where the simulated test statistic is larger than the original one. Then the p-value p m+1
b+1 always correctly controls type-I errors (Phipson & Smyth 2010).
We implement three permutation-basee variants of our tests. The Permutation Pooted Test (Algo- r directly extends the pooled approach by computing the mean difference across all examples and comparing against null distribution generated by random sign flips. The Permulation Fisher Test (Algorithm 6) applics permutation tests to each task individually, then combines p-values using Fisher's method. The Permutation Max Drop Test (Algorithm 7 standardizes per-task differences
httppe///aatiiiiaaaaaaaaaaaaaaaa nnte hhgp://ceeeooooooooeoooooooooeeo benchaarkingg
21

---

Published as a conference paper at ICLR 2026
by their standard errors and uses the maximum standardized difference the test statistic, analogous to our binary max drop test but adapted for continuous scores.
Algorithm 5 Permutation Pooled Test
Require: Score lists LM [ZI( (IN())), Li - (oxr( L (xN)) for tasks combined. Permutations m.,
1: A + (x;)
(z;)) L (r))
2: dobs N D A|i
3: 0+0
4: for 1 to do
5: signs t- random_choice((-11 size=N)
6: dsim t Liz signs(i] A
7: if daim 2 dobs then
8: b+ b ++
9: return peal <- b+1
Algorithm Permutation Fisher Aggregation Test
Require: Score lists LMM LM, for T tasks. Permutations m.
1: for t = 1 to T do
2:
Pt < wew.mmtttiooooooleea.....
M. m)
3: Xstat + -2 In(pt)
4: return pval + chi2. f 21)
Algorithm Permutation Max Drop Test
Require: Score lists LM Ligt, for T tasks. Permutations m. Epsilon 10-10
1: for = 1 to T do
2: A, t [M,,(x;;)
M. (x:)l
3: data [t] f 1
Ni EE Ar
alt] + std(,, @dof=1) +
4:
VN.
5: Zobs f max, dotalt)
6: 1--0 0
7: for = to m do
8:
for f= 1 T do
9:
signs: + random.choice(((-., Niz=)) size=N,)
10:
doim it] + N, Σ -=1 signs, A.
No
11:
Zoim max, daim
if' Zaim Zobs then
13:
b+b+
14: return peal= + m+1
b+1
D.3 SYNTHETIC EXPERIMENTS
We claimed that the permutation-baset test should a/ give very similar p-values when used on bi- nary scores and should better detect small differences when used with continuous scores. We first illustrate this with two synthetic examples. As we show in Table the p-values are nearly identical (maximum absolute difference of 0.000919), confirming that permutation tests provide equivalent results to the exact binomial approach on binary data.
For the second test, generate two tasks with small accuracy degradation and continuous scores. To evaluate the binary tests here, we populate the contingency table based on whether the score between the models is larger or less. The alternative would be threshold, but this is even worse.
22

---

Published as a conference paper at ICLR 2026
Table 10: Comparison of p-values:
Binary tests Permutation tests on binary data
Test Method
Pooled
Max Drop
Fisher
Binary (McNemar-basel)
0.220127
0.474152
0.318846
Permutation
0.219208
0,474105
0.318749
Absolute Difference
0.000919
0.000047
0.000097
As shown in Table 11 the permutation-basel tests substantially more sensitive, detecting the degradation with p-values that are 2-10 times smaller than the binary approach. This demonstrates that thresholding continuous scores binary loses important statistical power for detecting subtle performance differences.
Table I1: Comparison of p-values: Binary tests vs Permutation tests on continuous data with accu- racy degradation
Test Method
Pooled
Max Drop
Fisher
Binary (thresholded)
0.073483
0.056188 0.024320
0.070337 0.011628
Permutation
0.007230
Ratio (Binary/Permutation)
10.16
2.31
6.05
D.4 LLM EXPERIMENTS
Llama 3.1 8B Instruct on TruthfulQA To strengthen our claim that a reduction to a binary test sacrifices test power, evaluate two lossy 1lama-3.1 8B Instruct variants the task TruthfulQA (Lin et al. 2022). We run the generative variant of Im-eval Gao et al. (2024) and use the metric RougeL Max, which measures the RougeL score of the LLMS generation with all options for cor- rect answers and then considers the maximum as final score. This metric is inherently continuous. We then compute the (single task) pooled p-value with a/ the permutation-bant mended), b/ our binary script where we threshold the scores to 0 or one at 0.5, and c/ where we consider just the win/loss rates for the contingency table.
approach (recom-
The results in Table 12 confirm that the permutation- test is the most sensitive test and should always be used for non-binary scores.
Table 12: Comparison of valuees: Binary tests vs Permutation tests on continuous data from Truth- fulQA on two variants of Llama-3.1 8B Instruct.
Model
Average Rougel.Max
Permutation
Threshold
Win-Rate
Baseline
59.22%
w4al6 w816
55.26%
1.00e-5 0.0014
0.042 0.047
0.023
58.21% %
0.0018
GPT-OSS For GPT-OSS models (Agarwal et al. 2025). OpenAI released their own eval- uation suite Ø We can use this to evaluate the the 20B model on GPQA (Rein et al. 2023] and A Since the GPT-OSS models already come with their MoE modules in MX-FP4 precision by default, we could not find meaningful models that are further quan- tized. We therefore compare the 20B model against a rerun and against a version with FP8 KV cache. We also include a pruned variant which only has 7 experts per MoE layer gpt-oss-66 6. Ob-speci aiiizddall-prrnee cooeendd-oo-eee -y---expertsff However, this turns out to be way too aggressive. Furthermore, to have model with more gradual degradation,
23

---

Published as a conference paper at ICLR 2026
also include run for which "optimization" constrain the models context to 32,000 instead of 128,000 tokens.
The peculiarity of the GPT-OSS eval suite is that each example of the dataset is evaluated with three different levels of reasoning effort and each of them is run eight times with finite temperature. Thus, although each individual run results in binary score, when aggregating each examples is evaluated on, we get a more fine-grained score. Thus, we also need to apply the permutation-based tests obtaín the strictest p-values as outlined above. The results of the tests are presented in Table 13 While the rerun and FP8 KV cache are effectively lossless, the pruned model and the model that is artificially constrained to 32,000 context length show clear accuracy drops that are detected as statistically significant. Notice that for our experiments we use m 10 permutations and thus the smallest possible p- value for the pooled test and the max drop tests are 1/(10" + 1) 1. 1.00e-05. If one wishes to obtain even more sensitive p- p-valuss, one can further increase the number permutations, but we do not foresee practical relevance for smaller p-values.
the 24 runs
Table 13: Evaluation of variants of GPT-OSS 20B with the OpenAl evaluation suite.
model
Ppool
Prshhr
Pmax drop
AIME25
GPQA
Baseline 20B
65.59%
64.44%
65.76%
Baseline (rerun)
66.06%
-0.47%
7.59e-01
8.38e-01
9.07e-01
65.41 %
66.16%
KV-FP8
65.42%
0.16%
3.87e-01
5.54e-01
5.75e-01
64.99%
65.49%
pruned exp
21.91%
43.7%
1.00e-05
2.40e-09
1.00e-05
0.14%
25.21%
max32k
58.63%
6.96%
1.00e-05
1.12e-08
1.00e-05
52.92%
59.49%
24


Reviews:
[{"review_id": "fLGYFDQUg5", "reviewer": "Reviewer_Uko7", "summary": "Authors argue that simply comparing aggregated accuracy scores of benchmarks are statistically flawed because it ignores the fact that two model evaluations are performed on the same set. Authors propose the testing framework based on the exact one-sided McNemar test. And further show that the testing framework can confidently flag model degradation as small as 0.3% to be statistically significant.", "questions": "1. Different inference engine backend (hf transformers, vllm, sglang) sometimes show different results on benchmarks. It would be helpful to test on different engines and tell if different engines may yield different results.\n2. The method may be useful for measuring forgetting of language models after finetuning/continual learning. I encourage authors to explore other scenarios where degradation is an important topic (pruning, finetuning, etc.)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Authors challenge a fundamental approach in LLM evaluation benchmark where accuracies may not necessarily show the degradation of LLM capability. Currently the general practice is to consider difference within a certain threshold as statistically insignificant. Authors provide a more principled way to identify the flaws of the convention.", "weaknesses": "1. As a work studying variance/randomness, in the main experiment, it might be important for authors to report results across three runs.\n2. Similarly, authors may want to reproduce an empirically lossless variant to rule out the hardware level randomness. (for example, use hf transformers backend and throw in the deterministic flag) Since we don’t necessarily know how the hardware-level randomness interact with the model-level randomness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LjVZH4PUIX", "reviewer": "Reviewer_NK8t", "summary": "This paper proposes a statistical framework to decide whether an optimized LLM has truly degraded relative too a baseline, rather than exhibiting harmless eval noise. The core is an exact one-sided McNemar test over per-item success/failure pairs fro the baseline vs optimized. This controls type-1 error and increase power by focusing only on disagreements (flips) between models. They derive theoretical analysis of test power, propose three aggregation methods for multi-task eval (pooled test, max drop test, Fisher's method), and provide implementation for the LM Evaluation Harness. Experiments on Llama-3.1 8B and Llama-3.3 70B with various quantization schemes demonstrate method can detect accuracy degradations as small as 0.3% while correctly not flagging theoretically lossless optimizations.", "questions": "-  Given the three aggregation methods with different trade-offs, how should a practitioner choose?\n- other than McNemar's test to answer if optimized model is statistically significantly worse than baseline, could you also use bootstrap confidence intervals for accuracy differences? or maybe permutation tests or bayesian model comparison (Bayes factors)? How do these compare in terms of power?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- well motivated as it addresses a real and understudied problem of rigorous statistical testing for LLM optimization \n- first to apply McNemar's test to LLM degradation detection\n- I like the practical contribution for LM eval harness which enables immediate adoption\n- very interesting that even a 0.3% degradation can be detected with statistical confidence", "weaknesses": "- limited novelty: this is more about an application and implementation of statistical tests than methodological innovation\n- I think some of the assumptions may not hold for many of the benchmarks today. they make an i.i.d assumption but real benchmarks (like MMLU) often stratify by difficulty, topic, or other factors which may violate independence. Additionally, the binary metric assumption is limiting as plenty of important metrics are continuous. \n- I believe there is a limited scope of evaluation. the authors only evaluate on llama models (3.1, 3.3). no other model families are tested. they also only test quantization optimizations, but pruning, distillation, etc is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hHuQpGNjRD", "reviewer": "Reviewer_9xbY", "summary": "The paper introduces a statistical framework for detecting significant performance degradations in Large Language Models (LLMs) after optimization techniques such as quantization. The paper adapts a one-sided exact McNemar test and offers three ways to aggregate over multiple benchmarks (pooled, max-drop with Monte-Carlo, Fisher) to statistically determine whether observed accuracy drops are due to true degradation or random noise. The result shows that even small drops (≈0.3%) can be detected as statistically significant and evaluates this on Llama-3.x variants run via lm-eval and vLLM.", "questions": "1.\tThe paper needs to replicate the main findings on at least one other family (e.g., Qwen, Mixtral, DeepSeek) and another serving stack, demonstrating that 0.3–1.0% effects are robust.\n2.\tBeyond adopting the exact one-sided McNemar/binomial test, what is the new statistical contribution? Could the paper include a non-inferiority (TOST) module or sequential design that adapts sample size for power? How robust is the one-sided McNemar test when model scores are not perfectly binary (e.g., tasks with multiple correct answers or graded scoring)?\n3.\tThe paper may run a prospective study showing how flip-focused trimming affects type-I/II error and bias across content strata, and propose guardrails. This directly addresses the validity of Recommendation 1.\n4.\tFor non-binary metrics, please add a summarization or QA experiment (e.g., ROUGE or F1) and explicitly show the reduction to pairwise wins/ties/losses (or thresholds). The paper may also report one-sided p-values, effect sizes, and calibration to validate the procedure and demonstrate its generalizability and broader applicability.\n5.\tCould the paper specify and discuss the exact decision rule for flagging degradation? Does the framework flag when any one-sided aggregated test rejects at α (e.g., 0.05), when (\\hat{\\delta}>2%) or (\\hat{p}*{\\updownarrow}\\ge 5%) (per Dutta et al., 2024), or only when both statistical and practical criteria are met? In the 70B KV-FP8 and 70B w8a8 cases where (\\hat{p}*{\\updownarrow}<5%), the framework still detects a performance degradation. \n6.\tThe manuscript contains several language issues that impede readability (e.g., apostrophe misuse “its/it’s,” misspellings such as “assess”/“further,” incorrect conjunction “correct or false,” article usage “a similar situation,” verb form “build,” spacing/typography around symbols like “X, Y,” a duplicated footnote/URL, and inconsistent dialect choices like “while/whilst”). Could the paper (a) perform a thorough language edit to correct these issues, (b) standardize on a single dialect, and (c) resubmit with tracked edits or a brief editorial checklist summarizing the changes?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper tackles an important gap in LLM evaluation, quantitatively distinguishing real degradations from noise, which is critical for optimization research. The framework is theoretically justified using McNemar’s test, with mathematical backing and variance derivations. The integration into the popular LM Evaluation Harness makes it useful for both academic and industrial research. The experiments cover different Llama models and optimization types (INT4, FP8, KV-FP8). The paper also gives a useful rule-of-thumb for compressing evaluation sets by removing examples that never flip in pilot runs, reducing evaluation cost while preserving signal.", "weaknesses": "•\tThe technical tool (McNemar 1947) and Fisher’s method are classical. The paper’s contribution is primarily in adapting these to LLM regression testing, making them one-sided, and packaging aggregation strategies. This is still valuable, but conceptual novelty is limited.\n\n•\tMain empirical claims revolve around Llama-3.x variants and specific serving stacks (vLLM; KV-FP8 issues noted). It is unclear how robust the conclusions are across other model families, in-house evals, or non-binary metrics. Without replication on other models/toolchains, significance to the broader community is limited.\n\n•\tWhile the paper sketches a generalization for non-binary scores, the actual implementation and experiments remain binary. Given how common non-binary metrics are for LLMs, a small worked example (pairwise wins/ties/losses with thresholds) would increase practical utility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LhCYVYGL45", "reviewer": "Reviewer_Fq3r", "summary": "This paper investigates an important problem of testing whether Large Language Models (LLMs) degrade, e.g., after quantization or different serving environments. It proposes to leverage the principled McNemar's test to detect statistically significant performance degradation. On evaluation of open-source datasets, the proposed method is shown to be highly sensitive in detecting model degradation.", "questions": "This is an interesting paper targeting an important problem of testing LLM degradation. The theoretical analysis with McNemar's test is highly encouraging, as it provides a principled way to detect statistically significant degradation. This is a new and useful contribution to the field. However, I am not an expert in this field, especially regarding the theoretical aspects. I would be interested in the following questions:\n\n- The evaluations are primarily on LLaMA models. How well does the proposed method generalize to other open-source LLMs?\n- Will this method be applicable to small models as well, or is it specifically designed for large models?\n- The \"faster regression testing\" part seems interesting. Could you elaborate more on the actionable advice on how to effectively use this method for degradation testing in practice?\n- If we have access to model internals, can we use this method to pinpoint the specific components causing degradation?", "limitations": "", "rating": 8, "confidence": 1, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Targets an important and practical problem of testing LLM degradation.\n- Theoretically grounded approach using McNemar's test for degradation detection.\n- Interesting empirical observations about LLM degradation with the proposed method.", "weaknesses": "- Evaluations can be expanded to more models other than LLaMA.\n- Practical implications of the findings regarding regression testing could be further elaborated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)