Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers
Authors: Thomas Klein
Sascha Meyen
Wieland Brendel
Felix A. Wichmann
Kristof Meding
Venue: NeurIPS 2025 poster ()
Keywords: Error Consistency
Behavioral Alignment
Vision
Primary area: neuroscience_and_cognitive_science
TL;DR: We present a method for calculating confidence intervals around error consistency values and revisit earlier work using this metric, showing that many differences found between DNN models of vision are statistically insignificant.

Abstract: Benchmarking models is a key factor for the rapid progress in machine learning (ML) research. Thus, further progress depends on improving benchmarking metrics. A standard metric to measure the behavioral alignment between ML models and human observers is error consistency (EC). EC allows for more fine-grained comparisons of behavior than other metrics such as accuracy, and has been used in the influential Brain-Score benchmark to rank different DNNs by their behavioral consistency with humans.  Previously, EC values have been reported without confidence intervals. However, empirically measured EC values are typically noisy - thus, without confidence intervals, valid benchmarking conclusions are problematic. Here we improve on standard EC in two ways: First, we show how to obtain confidence intervals for EC using a bootstrapping technique, allowing us to derive significance tests for EC. Second, we propose a new computational model relating the EC between two classifiers to the implicit probability that one of them copies responses from the other. This view of EC allows us to give practical guidance to scientists regarding the number of trials required for sufficiently powerful, conclusive experiments.
Finally, we use our methodology to revisit popular NeuroAI-results. We find that while the general trend of behavioral differences between humans and machines holds up to scrutiny, many reported differences between deep vision models are statistically insignificant. Our methodology enables researchers to design adequately powered experiments that can reliably detect behavioral differences between models, providing a foundation for more rigorous benchmarking of behavioral alignment.

Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers
Thomas Klein 1,2,3,4+
Sascha Meyen
Wieland Brende
Felix A. Wichmann
Kristof Meding
Abstract
Benchmarking models key factor for the rapid progress in machine learning (ML) research. Thus, further progress depends on improving benchmarking met- rics. A standard metric to measure the behavioral alignment between ML models and human observers is error consistency (EC). EC allows for more fine-grained comparisons of behavior than other metrics such as accuracy, and has been used in the influential Brain-Score benchmark to rank different DNNs by their behav- ioral consistency with humans. Previously, EC values have been reported without confidence intervals. However, empirically measured EC values typically noisy-thus, without confidence intervals, valid benchmarking conclusions are problematic. Here improve standard EC in two ways: First, we show how to obtain confidence intervals for EC using a bootstrapping technique, allowing propose new computational
derive significance tests for EC. Second, model relating the EC between two classifiers to the implicit probability that one of them copies responses from the other. This view of EC allows us give practical guidance to scientists regarding the number of trials required for sufficiently pow- erful, conclusive experiments. Finally, we use our methodology to revisit popular NeuroAI-reste We find that while the general trend of behavioral differences between humans and machines holds up to scrutiny, many reported differences between deep vision models are statistically insignificant. Our methodology en- ables researchers design adequately powered experiments that can reliably detect behavioral differences between models, providing a foundation for more rigorous benchmarking of behavioral alignment.
Introduction
Consider the following problem: Given two classifiers operating on the same domain, how should we quantify their similarity? This abstract problem has become highly practically relevant in the context of cognitive science and beyond. In cognitive science, deep neural networks (DNNs) have been proposed as models of the human visual system [Doerig et al. 2023 Kriegeskorte 2015 Cichy and Kaiser 2019 Kietzmann et al. 2017 Yamins et al. [014]] and thus the question arises how similarly other domains, e.g. law or medicine, is also crucial to know whether DNNs interpret and judge information similarly humans. Thus, metric is needed that can reliably quantify the degree of behavioral similarity between DNNs and human decision makers.
they behave to human observers.
The standard method proposed for this purpose is error consistency (EC) Geirhos et al. 2020 which has seen wide application the context of human coman-machine comparisons, e.g. in Geirhos et al. 2021
Max Planck Institute for Intelligent Systems, Tübingen, Germany 2 University of Tübingen. 3 Tübingen Al Center ELLIS Institute Tübingen. 5 CSZ Institute for Artificial Intelligence and Law. i Joint supervision. Correspondence to: f.klein @ uni -uin-tubbinge.de Code: gitggg..thhmnnn ftthhb..om//iichmaa lab/eroo
lab/error_consistennyy consistency
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Previous Work
Our Work
F
0.4
Insignificont differences
EClkappo]
Hunin-tachine
uncert ain gop
0.3
Gop
I
T
02-
00-
Model Ranking
0.0
humans
C
humans
Classifier
Classifier
Figure 1: Schematic: Error Consistency is noisy. The gist of our findings is that by calculating confidence intervals for empirical EC values, show that while the human-machine gap probably real, the measure is not stable enough to resolve differences between models.
Ollikka et al. [2024), Li et al. l Parthasarathy et al. [2023]. Similarity metrics based on EC
have also seen application in the behavioral similarity section of the vision-branch of Brain-Score [Schrimpf et al. 2018). More recently, a variation of EC-sharing many of the relevant properties, so
that considerations apply-has has been proposed metric compare Large Language Models
(LLMs) Goel et al. 2025].
EC builds on Cohen's [Cohen 1960]. which considers the trial-level agreement observed for pair of classifiers to compute a scalar similarity score, which is the two models are behaviorally indistinguishable and -1 if they
different they could possibly be. A value of 0 indicates independence. In practice, EC is reported as point estimate of a unit-less scalar value, based on classification decisions obtained relies on trial-by-trial similarity, the measure is inherently noisy, as individual trials can have large impact on the final score, especially when the total number of trials is small-as is often the when conducting experiments with humans.
single set of test stimuli, like in InFiguur 2h. However, since EC
Our contributions are as follows:
We explain how confidence intervals for empirical EC values can be estimated on any dataset on which EC may calculated using bootstrapping [Éfron and Tibshirani] 1994].
We propose novel method of modeling classifier consistency, allowing us to simulate classifiers with specific consistency and thus to plan sufficiently powerful, conclusive experiments.
We apply our methods to two influential benchmarks (Model-vs-Humaal [Geirhos et al., [021] and Brain-Score [Schrimpf al. 2018)), showing that the currently available amount human reference data is not sufficient to draw conclusions about model differences in many cases, illustrated in Figure
We provide a python package containing utility functions for obtaining confidence intervals around error consistency values, calculating p-values, planning experiments, and testing results for significance.
We will begin by providing brief introduction to BC in Section then explain how confi- dence intervals can be calculated in /Section 3 A detailed Related Work section is provided in Appendix A] since we discuss related work on measuring behavioral alignment, Error Consistency, and benchmarking.

---

2 Error Consistency
First, we provide brief introduction to error consistency and introduce our notation; see Geirhos et al.] [2020) for an in-depth description. EC is measured by applying Cohen's 's x to evaluate whether two classifiers' responses are jointly correct or incorrect-that is, whether they are consistent about when they make errors. The experimental setting in which EC can be calculated is the following: Two classifiers categorize each of samples (e.g.. natural images) into one of K classes (e.g.. the 1,000 ImageNet classes). Each sample, Ty has a ground-truth label y; E {I,..., K} and receives classifications, 2/57) E {I,..., K}. with j E 2}. Whether these classifications are correct or not is coded as r! E {0, 1} where r, 1 if and only if the j-th classifier responded with the true label, 0/ = y/- This renders EC agnostic to the numher of classes of the classification task because it only considers agreement which samples are jointly classified correctly and incorrectly.
In this setting, EC is measuring the agreement between ((1) and (2(2) using Cohen's &. (which is typically applied to g(2) and ÿ(2)):
K Pobs
Peup
Peap
where Pobs is the observed agreement, that is. the proportion of trials which are jointly classified correctly or incorrectly. This quantity is normalized by the amount of agreement expected from independent classifiers, Pezp- This normalization accounts for the fact that classifiers with high accuracies are priori expected to agree more often in their classifications than classifiers with low accuracies.
However, the normalization does not render EC fully orthogonal to the accuracies of the classifiers, because Cohen's ' x depends on the marginal distributions of classifier responses [Falotico and Quatto 2014 Grant et al. 2016]. This is easy to see in the limiting case where two classifiers have perfect accuracies (P1 P2 1): Division by zero renders EC undefined because, intuitively, error consistency can not be evaluated in the absence of errors. Moreover, if exactly one classifier has perfect accuracy, EC will be irrespective of the accuracy of the other classifier. This phenomenon, which we prove in Appendix C. may be counterintuitive and can lead to estimation instabilities, see below.
EC is bounded by the mismatch of the two classifiers' accuracies [Girrhos et al. 2020J. We provide an instructive visualization of these asymmetric bounds in Figure 2b. Only when the accuracies of both classifiers are equal can all values in [-1, 1] be achieved. This implies that EC can never be high between two classifiers if their accuracies are substantially different. In practice, one can therefore not take EC at face value: EC has to be interpreted with the accuracies of the two classifiers in mind, and is, unfortunately, strongly susceptible to both floor- and ceiling-effects. The precondition for straightforward interpretation of EC is thus good match of classifiers' accuracies.
However, previous work has evaluated EC in conditions in which accuracies of the two classifiers were vastly different, see Figuue 2p. Even without considering their joint probabilities, EC between classifiers must be low in these cases because their accuracy mismatch imposes an upper bound on the observable EC values.
To formalize this dependence of EC on classifiers' accuracy match, we present two propositions. First, when the accuracies of the two classifiers are equal, EC can best be understood as the proportion of copied responses: In model in which the second classifier copies the responses of the first with probability Peopy and responds independently at random otherwise, EC is related to that copy probability. In Proposition 1. we present this result general property of Cohen's н. (for relating ()) and (()) which immediately translates to the application EC (relating (1) and (())), where equal marginal distributions correspond to equal accuracies.
Proposition 1. Let a classifier give responses according to some marginal distribution g()) ~ P()) If another classifier copies that response jز²2 5( with probability Poopy and otherwise draws independently from a marginal distribution j(2) ~ p(2) that is the same as the first P(1) = P(2), then
Cohen's 's K. Peopy
3

---

10
0_8
Bounds on Error Consistency EC
Accuracy of the Second Classiffor
0.0
02
00
00
6
Mac EC
1
Acturazy of the Sipurd Classifer
02
5.
Accuracy of the First Classifier
Accuracy of the First Classifier
Figure 2: Theoretical issues with EC. (a) The bounds on error consistency (EC) depend on the mismatch between the accuracies of the two compared classifiers. (b) Similar to (a) but with accuracies on the x and y axes. Here, each orange dot corresponds to one Moodel-vs-uumal comparison and each black dot one numan-vs-human comparison in the analysis of Geirhos et al. [2021] Red dots correspond to conditions that were sampled but excluded for the analysis by the original authors. fluman-vs-uuman comparisons have lower accuracy mismatches, EC values expected to be higher a priori.
Proof. The underlying distributions translate into the observed distributions p(1) p(1) and p(2) Pcopy p()) (1 Peopy) ((2) P(2), With ((1) ((2) have p(l) p(2) p(1) p(2).
For the joint distribution, we have p(.2) 3) p()) (Poopy + (1 Peopy (2( because the second classifier makes the same prediction as the first not only in all copy cases (Peopy). but also in those cases in which it does not copy but coincidentally makes the same prediction (1 Peopy) ) P(2) (y)). With that, x simplifies to Peopy-
k Pobs PeEp
Pezp
ΣK p(12) (v.9) 1 (1)
1- E (1)
DK (p()(y)(Peopy (1 Peopy) P2 (y)) (y) (())
Σ K E p(1) P(2))
Σ/ (p())y) + (1 Propy (()) (v)) - (y)
Σx (())
P()))) (v)
Σx (y) p()) p(1) (v) p()) pp) Poopy (())
Σx ((U
> p()) >K p()) (1))
2y=1
-Peopy
1 EØ
ΣK (y)
PPopp
Ex=1 (1)
PPopp
Second, in the general case when the two classifiers have different accuracies, EC measures two aspects of similarity between the two classifiers:

---

(1) the proportion of copied responses (Poopy) and
(2) the (mis-)match between their accuracies (factor f in Proposition 2).
Both are intuitively valid determinants of the similarity between two classifiers. We formalize the interplay between these two aspects in Propositionn 2] In contrast, Safak [2020] tackled the same problem but proposed a linear scaling whereas our result reflects a more interpretable multiplicative scaling. Again, this result holds for Cohen's S in general and, therefore, also when it is applied to measure EC (for which y would be replaced by r and different marginals correspond to different accuracies).
Proposition 2. Let classifier give correct responses according to some marginal distribution g(l)) p(), If another classifier copies that response j( /( with probability Paopy and otherwise draws independently from its (potentially different) marginal distribution g(2) -((2,, then
1-E, (n) 1- E.
Cohen's 's K Peopy
factor I due to mismatching marginals
with observed marginal distributions p(1) = p(1) and P!2) Peopy P(1) + (1 Propy)
) P(2).
See Appendix B for the proof. Crucially, this view of Cohen's 's A holds with generality as long as K = 2, without limiting assumptions about the marginal distributions. Note that the relation between EC and Pcopy is not symmetric: Whether classifier is considered to copy from classifier or vice versa results in different mismatch factors f. But note also that this factor vanishes (ʃ 1) if the reference classifier, from which responses are assumed to be copied, has a uniform marginal distribution.
This inspires a strong recommendation for applications of EC: When comparing multiple DNNs with different accuracies to humans, ensure that the human responses have uniform marginal distributions (i.e., have equally many correct as incorrect responses). In these cases, EC becomes interpretable as the proportion of responses that the DNNs copied from the human responses, EC Peopy, regardless of the overall accuracy of the different DNNs. Without a uniform marginal distribution, DNNs with a higher-than-hume accuracy may be evaluated less similar to humans because their classification process is more accurate, not because it is functionally different. To be clear, we still believe that it is meaningful interpret differences in accuracies between classifiers. But, ideally, EC would offer additional, orthogonal information beyond that accuracy mismatch-whieh is ensured by keeping the marginal distribution of the reference classifier (humans) uniform.
Beyond this contribution to interpreting EC, the generative model in Proposition contributes in another way: It allows simulating data to predict the stability of real data. We will heavily use the latter to give guidance to practitioners planning how many samples they should present to real human participants in their experiments.
Calculating confidence intervals for EC values
Whenever an aggregate measure of empirical data is reported, it is best practice to also quantify the degree of uncertainty about the final value. Historically, this has not always been done for EC, e.g. none Geirhos et al. (2018) Ollikka et al. [2024] et al. 2025 report confidence intervals. While [Geirhos et al. 2020 provide a basic variant of confidence intervals, these were computed in a suboptimal way: They were based on the expected agreement Peap rather than the actual accuracies of the two classifiers, and p2. This creates an ambiguity because, for example, (p) = 0.5, = 0.5) as well as (p2 0.5, P1 0.99) both produce the same expected agreement of Pexp 0.5 but result in different confidence interval widths. Moreover, these confidence intervals were necessarily centered around O, because they were constructed only for independent observers. Our approach enables calculating confidence intervals for two dependent observers by making use of the observed accuracies. We begin by outlining how confidence intervals can be obtained for existing empirical data, using straight-forward bootstrapping approach. After that, we make use of the generative model implied by PProposiiion 2 to derive confidence intervals ahead of time to plan sufficiently powerful experiments.
5

---

3.1 Using existing data for bootstrapping
Assuming that we already have access to two sequences of binary responses, r() and y(2) of length N, we can obtain a measure of uncertainty simply by bootstrapping (Efron and Tibshirani 1994]. To do so, we sample N elements with uniform probability and replacement from the sequences,
E {I,...,N) N),, and re-calculate EC based on the sampled trials. This procedure established in other fields using consistency measures [MeKenzie et al., 1996 Vanbelle and Albert 2008]. Doing this repeatedly yields an empirical distribution of EC values, for which we can report a 95% confidence interval by taking the central 95% of the posterior mass, i.e. the interval 025. 90.975| where 9p denotes the p-th quantile of the posterior distribution. Alternatively one could report the Highest Posterior Density Interval (HDPI), which is the narrowest interval containing a certain amount (e.g. 95%) of the posterior mass. The only parameter of this procedure is the number of bootstraps, M, which we typically set to M = 10, 000, see Figure for a justification of this choice. For readers unfamiliar with the bootstrap approach, we provide more detailed description in Appendix F and an explicit algorithm in ilAloorihhm
((r
We apply this method of estimating CIs via bootstrapping to the data from Geirhos et al. 202 in [SSecttion 4 In their work, several human observers (typically four six) and 52 models were evaluated on corrupted natural images, which had to be classified into sixteen classes. There are 17 different corruptions, most of which are parameterized by scalar intensity level, leading to multiple conditions within each corruption. Each condition contains 160 to 640 images. For every model, its EC to human observers calculated a hierarchical fashion: First, within each condition, the ECs to all human observers are calculated and then averaged. Then, the conditions of each corruption are averaged again, before calculating the final nhuman-machinn error consistency" as an average over corruptions. To accurately reflect this procedure, we bootstrap through the entire calculation, obtaining a posterior over the final average itself and plot 95% percentile intervals of these averages in Figure 4
3.2 Using Proposition 2 for simulations
But what about the scenario in which no data is available yet, e.g. when planning an experiment? Without data, we first need a stochastic generative model that can produce trial sequences resulting in a target EC, given two classifiers characterized by their marginals, i.e. their accuracies.
Proposition 2 provides such a model in which the underlying marginals and copy probability can be specified to generate data. Note that, depending the copy probability, the observed marginals will deviate from the underlying marginal: The more responses the second classifier copies from the first, the more its marginal distribution adapts. To simulate data with specific marginal distributions p(1) and p(2) as well as EC, one can simulate data with copy probability and underlying marginals
K 1= (()) ((2)
Peopy = EC
1 (g)
p()(y) = p())(y, and
p() ((2)
p() (()
Peopy
1 Peopy
Equipped with this generative model, we can estimate confidence intervals for pair of classifiers as follows: We first sample N trials for the first classifier by drawing from Bernoulli distribution (1)) P(1), Next, we copy Peopy N responses from the first classifier to the second. For the remaining (1 Peopy)-A NN responses, we sample independently from Bernoulli distribution from the underlying marginal n,(2) ~ p(2), This yields one data set based on which we estimate EC. Repeating this process then yields estimate for the variability of these EC values. We provide algorithmic description of this approach in Alggorithhm 2
We plot the resulting confidence intervals in Figure and include sample sizes used in the literature
(Geirhos et al. 2021 Ollikka et al. 2024 Li et al. 2025 Wiles et al. 2024) as reference points. We focus on the most severe case where 0.5, but discuss the effect of on CI size in Appendixx E Note that the x-axis on logarithmic scale and reaches values that are very difficult to meet in experiments with human observers. Even so, the width of the confidence intervals (especially that

---

Accuracy
0.8
0.05
0.25
EmpiricalEc
0.6
0.5
0.75
0.4
0.95
0.2
0.0
-0.2
100
200
400
800
1600
Number Trials
Reference Works
Geirhos 2021
2025
Wiles 2024
Ollikka 2024
Figure 95% Confidence Interval sizes as function of trial number. We simulate data demonstrating how the size of EC confidence intervals depends on both the accuracy of the classifiers and the number of trials. The ground truth EC is 0.5, and bootstrap 1,000 times for each pair of observers, which are set to the same accuracy for simplicity. We also include the trial numbers employed in the literature for reference (symbols on the x-axis).
of classifiers close to floor / ceiling performance) is very large and shrinks only slowly, covering 0.2 units of even at N 400 trials, which is roughly the size of the difference found between humans and machines by Geirhos performance close to floor / ceiling, the bootstrapped ECs are biased and undershoot the true value. This happens for simple reason: one classifier is always correct (or always wrong) the EC will be zero irrespective of the accuracy of the other one. In the critical performance regimes and at low number of trials, sub-sampling trial sequences for which these edge cases occur becomes likely enough to bias results.
[2021]. Especially noteworthy the fact that at lower and
Limitations. A core limitation of our work is that we do not present an analytic solution for confidence intervals, but rely on bootstrapping instead. This also limits the numerical accuracy the significance tests, which cannot resolve p-value differences < T. While the analytic solutions (along
the lines Giwet (2166, Fleiss et al. 1969], and Vanbelle al. (2044) could improve this slightly, we believe that for the discussed purpose here, our methodology is the pragmatic choice that comes
with fewer restrictions (and certainly much better than not reporting confidence intervals all).
3.3 Significance Tests for Error Consistency
Now that we can quantify the degree of uncertainty, we can also derive significance tests for error consistency. Significance tests are invaluable tool for empirical scientists because they provide a standardized way of checking if results are statistically reliable. Given two sequences of binary trials of length N which lead to EC of probability observing an event at least as extreme as the observed one under the null hypothesis of two independent binomial observers, i.e. 0.
want calculate p-value for s. The p-value is the
Basic Significance Test. To obtain the distribution of EC values obtained under the null hypothesis Ho x O, we first need to model the independent binomial observers that constitute the null hypothesis. They fully characterized by their individual accuracies. of which the empirically observed accuracies provide an estimate. The reliability this estimate depends on the number of trials. If we assume uniform priors over these accuracies, the posteriors beta-distributions characterized by the number of (in)correctly solved trials, Beta(N Σ.r EEr )())
We can now sample the two individual accuracies from their respective posteriors, sample N trials from independent binomial distributions and calculate the resulting ECC Repeating this procedure M times leads to distribution of EC values expected under the null hypothesis. From this distribution,
7

---

we then calculate how many times the absolute value of the simulated EC exceeded the absolute value of the empirically measured EC, for a two-sided significance test. We demonstrate this method by applying it to data from Model-vs-Humaal in Appendix H to obtain p-values of the error consistency between a human subject and the berr-perf model, CLIP--rained ViT [Radford et al. 2021
Advanced Significance Test. The empirical scenario which we investigate in Section 4 demands a slightly different significance test: Given fixed reference observer (e.g. human) and two candidate observers (e.g. two DNNs), is the difference of ECs to the reference observer found between the two candidates statistically signifieant?
Such a test is also possible. Let Kez Riey be the error consistencies of the two candidates. We consider the null hypothesis of the candidates having the same error consistency to the reference classifier, Ho: Koy = Kieg We first obtain a distribution of error consistency differences via bootstrapping as above: First, we randomly sample N trial indices with replacement and uniform probability. We
select the corresponding response-corree cornensecrrreeeess---ee triplets (r-(re)), ,((e)) (((a)) E {0, 1)3 and re-calculate EC values and their difference. To obtain the distribution of differences one would expect under Ho. we simply exchange r y(()), (((a)) with probability p p= 0.5, as if there had been only one underlying model. We can then perform a t-test between the distributions of differences to compute p-alues.
4 Revisiting earlier work
4.1 Model-vs-Human
Next, we re-analyze well-known literature results by calculating confidence intervals around their empirically measured values. At the focus of our attention the work by Geirhos et al. [202T), where cohort of human observers was asked to classify corrupted natural images. The same images were also shown to 52 DNNs, to evaluate their alignment to human behavior. EC values were calculated per condition and averaged as described in Seectinn 3 From our analysis in FFigure 3 we already know that some of the error consistencies were calculated in a low-data regime that will result in large confidence intervals around the individual EC values, but now compute confidence intervals specifically for their empirical data. We do this by bootstrapping through the entire calculation, to arrive at an empirical distribution of final scores, which we plot for humans and every model in Figure 4] The best model is a CLIP-trained ViT RRdford et al. [022]] which performs much better on the distribution shifts like sketches and stylized images, achieving an EC humans of 0.279. The second-best model, BiTMMResnet 1 01x1, achieves an ECC of 0.251, resulting in a difference of 0.028, which, as a two-sided t-test confirms. is significantly different from (p < 1 x 10 10-19). But note how model 24 (VGG-19 as per than 70% of all models. It is conceivable that if these experiments were done again, VGG-19 might jump up to rank 8, or down to rank 45. It just so happens that the very best and very worst models broaden the range we have to plot, thus rendering this effect less obvious than in Figuue 1 Testing the differences for all neighboring models for significance, we find no other significant result, even without correcting for multiple comparisons.
perpTTba Table 2) has overlapping CIs with all models from 8 to 45, more
For each bootstrap, we also obtain a ranking of models. Given that the CIs are fairly wide (for example, no model's 's confidence interval is separated from that of its neighbors). we wonder whether different bootstraps imply different model rankings. To check this, we caleulate Kendall's T between the original model ranking and the ranking implied by each of the 10k bootstraps. T a statistical measure of the rank correlation between two ordered sequences, similar to Spearman's rank correlation, but with the benefit of an intuitive interpretation: 7 measures the difference between the probability that a pair of observations is concordant and the probability that it is discordant. The average 0.88, which means that two randomly sampled pairs have an 88% chance of being concordant. Evidently, the main conclusion drawn by Geirhos et al. [2021) clearly still holds: Humans are more consistent among themselves than models are to humans. Nevertheless, adding confidence intervals to the reported EC values better quantifies the degree of uncertainty about the measurements for individual models, and the 95% Cls do indeed overlap, meaning that differences between models are not resolved.
8

---

0.5
Error Consistency Humans [Kappa]
0.4
0.3
0.2
0.1
Models
Figure 4: Models from MvH ranked by their EC to humans. Inter-human EC plotted in red. We plot the 95% percentile interval for the mean EC obtained via bootstrapping, using the standard exclusion criteria of the benchmark (orange and black points in Figure 2p). For the mapping of model indices to names, see Table Note that while the gap to humans is large, confidence intervals of many models overlap.
4.2 Brain-Score
One of the most widely-known applications of EC is the influential Brain-Score benchmark [Schrimpf et al. 2018). which evaluates arbitrary DNNs by how much they resemble the brain. One dimension of this evaluation is the behavioral similarity to humans, including error consistency as measured via the data by Geirhos et al. [202T]. Here we assess how reliable the ranking of models implied by their EC to humans really is.
In Brain-Score, the EC for each condition is first divided by the human-to-human EC for this condition, to express values relative to ceiling. We begin by reverting this operation obtain raw EC values. To properly bootstrap EC values for these models, we would need access to the complete sequences of trials generated by the models, but this data is stored anywhere, and re-running the models (not all of which are publicly available anyway) seems excessive. Hence, we instead make mild assumptions estimate conservative CIs around the empirical values, demonstrating the utility of our copy model.. (See Appendix I for details on this procedure.) We obtain the result depicted in Figure for the top-30 models (by EC humans). Again, the best model is CLIP-trained VIT. For the other models, the CIs are so large that any of the top ten models could be the second-bes one, meaning that the true rank order between models remains unclear.
We believe that the issue we raise here generalizes beyond just EC and Brain-Sccre:: Any metric that is used to derive rankings of models in a benchmark should come with a method for calculating confidence intervals. Likewise, any benchmark that provides a ranking should quantify the degree of uncertainty surrounding individual values and the stability of the resulting ranking.
5 Discussion
In this work, we have built on the error consistency metric proposed by Geirhos et al. 2020 demon- strating how confidence intervals and significance tests for empirical measurements of EC can be calculated using bootstrapping techniques. Additionally, we have presented new computational model EC for planning sufficiently powerful, conclusive experiments. We have used our methodol- ogy to revisit influential Neuro-AI results Schrimpf et al. [2018), finding that many results are not statistically reliable, because previous studies were underpowered. In the following, we recommend best practices for future benchmarking of behavioral alignment, and facilitate their implementation by making our python package publicly available. core contribution of our work is improved understanding of error consistency through a new computational model: EC is broken down into
httpp:://ww/ brain- ocore.org

---

Emro Consistency Humans [Kappa]
0.34
0.32
0.30
0.28
0.26
0.24
0.22
0.20
Models
Figure 5: Confidence Intervals for Brain-Score ECs. Like in Figure 4 we plot the 95% confidence interval of the EC of the top 30 Brain-Score models. Note that the confidence intervals of many mid-range models overlap.
two interpretable components, copy probability Peopy and marginal mismatch f. As an analogy, this is similar to breaking down accuracy into true positive and false positive rates-a necessary shift in perspective that will improve interpretations in future studies measuring similarities between classifiers.
5.1 Practical Recommendations: How to reliably benchmark behavioral alignment
From analysis, a sufficiently high number of trials is required to obtain small confidence intervals around EC values. As rule of thumb, we suggest to collect least 1000 trials per classifier balance this requirement against practical constraints. In experiments with human observers, achieving very high resolution of EC differences could become prohibitively expensive, underlining the need for a good selection of stimuli, which probe differences efficiently. Since the size of Cls also depends on the accuracy mismatch between classifiers, we suggest to aim for accuracy around 75%, but more than 90% to avoid ceiling-effectts, effects, which can be severe. The same holds for floor-effects This is in agreement with standard best practices from psychophysics, where one aims for similar accuracy level in human observers to sustain motivation whilst avoiding ceiling effects. Finally, future work should always report confidence intervals and check results for statistical significance, which can easily be done with the python package we provide.
derive concrete recommendations for practitioners. As shown in Figure 3
5.2 Conclusion
The field of ML research relies heavily on the powerful machinery of benchmarking (Hardt 2025 Hence, the correctness and reliability of benchmarks crucial for progress, since ill-calibrated benchmarks will lead us astray. The field is also notorious for not quantifying uncertainty properly
[Pineau al. 2021 Miller 2024 Lehmann and Paromau 2025 Bouthillier al. 2021], which is especially important the context of benchmarks because of their large influence on other developments. Our work raises general questions about how benchmarks should aggregate results. For benchmarks that explicitly compute rankings, we give the concrete recommendations of (a) providing confidence intervals around individual estimates and (b) quantifying the stability of the resulting rankings as we did with error consistency.
In this work, we have revisited results by Geirhos et al. [222] which influence the Brain- Brain-Score benchmark Schriimp et al.] [2018), revealing that many differences reported between deep vision models are not statistically reliable. The error consistency metric particularly susceptible to such issues, because it is inherently noisy, so the requirements for stable measurements are hard to meet in practice. We envision that our work will help the field overcome these issues, by providing tools to plan sufficiently powerful experiments.
10

---

Acknowledgments and Disclosure of Funding
Funded, in part, by the Collaborative Research Centre (CRC) "Robust Vision Inference Principles and Neural Mechanisms" of the German Research Foundation (DFG; SFB 1233), project number 276693517. FAW acknowledges funding by the BBVA Foundation Programme Grant "Harnessing Vision Science to Overcome the Critical Limitations of Artificial Neural Networks". This work was additionally supported by the German Federal Ministry of Education and Research (BMBF): Tübingen Al Center, FKZ: Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 and via the Open Philanthropy Foundation funded by the Good Ventures Foundation. WB, FAW and K.M are members of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. KM is supported by the Carl Zeiss Foundation through the CZS Institute for Artificial Intelligence and Law. This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS -I) for supporting Thomas Klein.
WB acknowledges financial support via Emmy Noether
We would like to thank Robert Geirhos for helpful discussions, and for providing us with his raw data for Model-vs-Human. We also thank all anonymous reviewers, who provided valuable feedback during the rebuttal period, which improved the final manuscript.

---

References
Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaic, Yousef' Almushaygih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq. al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. In Proceedings of the 62nd Annual Meeting of the Association for Computaaiional Linguistics (Volume 1: Long Papers), pages 13787-13805, 2024.
Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems, 3:747-769, 2021.
Jeffrey S. Bowers, Gaurav Malhotra, Marin Dujmović, Milton Llera Montero, Christian Tsvetkov, Va- lerio Biscione, Guillermo Puebla, Federico Adolfi, John E. Hummel, Rachel F. Heaton, Benjamin D. Evans, Jeffrey Mitchell, and Ryan Blything. Deep problems with neural network models of human vision. Behavioral and Brain Sciences, 46, 2022. doi: https:/doi..rr// 100110177014455552000033
Radoslaw M Cichy and Daniel Kaiser. Deep neural networks scientific models. Trends in cognitive sciences, 3355) 2019.
Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46, 1960.
Adrien Doerig, Rowan P Sommers, Katja Seeliger, Blake Richards, Jenann Ismael, Grace W Lindsay, Konrad Kording, Talia Konkle, Marcel AJ Van Gerven, Nikolaus Kriegeskorte, et al. The neuroconnectionist research programme. Nature Reviews Neuroscience, 24(7):431-450, 2023.
Bradley Efron and Robert Tibshirani. An introduction to the bootstrap. Chapman and Hall/CRC, 1994.
Rosa Falotico and Piero Quatto. Fleiss' kappa statistic without paradoxes. Quality & Quantity, 49 (2):463-470, 2014. doi:
URL http::///d/ doi. org/10........ 1007/
811135-014-0003.
Chaz Firestone. Performance competence in
comparisons. Proceedings of the
National Academy of Sciences, 117(4)):25662-663) 2020.
Joseph L. Fleiss, Jacob Cohen, and B. S. Everitt. Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72(5):323-327, 1969. doi: 10. 10.1037/h0028106 URL http://dx. http:///xx.doi.or////// owg/oor///1 1037/00028101
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018.
Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. Advances in Neural Information Processing Systems, 33:13890-13902, 2020.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. In Advances in Neural Information Processing Systems 34, 2021.
Shashwat Goel, Joschka Struber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. Great models think alike and this undermines ai oversight. arXiv preprint arXiv:2502.04313. 2025.
Malcolm Grant, Cathryn M. Button, and Brent Snook. An evaluation of interrater reliability measures on binary tasks using d-prime. Applied Psychological Measurement, 41(4):264-27 2016. doi:
David Green. Consistency of auditory-detection judgments. The Journal of the Acoustical Society of America, 35(5_ Supplemen)):78877855 1963.
12

---

Kilem L. Gwet. Testing the difference of correlated agreement coefficients for statistical signif-
icance. Educational and Psychological Measurement, 76(4):609- 637, 2016. doi: 10.1177/ 0013164415594400 htlp:ttp://dxx
Moritz Hardt. The emerging science of machine learning benchmarks.
Online
https
//mlbenchmarks. org 2025. Manuscript.
Tim Kietzmann, Patrick McClure, and Nikolaus Kriegeskorte. Deep neural networks in computa- tional neuroscience. BioRxiv, page 133504, 2017.
Bernard J Koch and David Peterson. From protoscience epistemic monoculture: How benchmark- ing set the stage for the deep learning revolution. arXiv preprint arXiv:2404.06647, 2024.
Nikolaus Kriegeskorte. Deep neural networks: new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj, Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, et al. Brain-like object recognition with hegh-performing shallow recurrent anns. Advances in neural information processing systems, 32, 2019.
Christoph Lehmann and Yahor Paromau. Quantifying uncertainty and variability in machine leam- ing: Confidence intervals for quantiles in performance metric distributions. arXiv preprint arXiv:2501. 169311, 2025.
Fanfei Li, Thomas Klein, Wieland Brendel, Robert Geirhos, and Roland Zimmermann. Laion-c: An out-of-distribution benchmark for web-scale vision models. In ICLR 2025 Workshop Spurious Correlation and Shortcut Learning: Foundations and Solutions, 2025.
Ben Lonngvist, Alban Bornet, Adrien Doerig, and Michael H Herzog. A comparative biology approach to dnn modeling of vision: A focus on differences, not similarities. Journal of vision, 21 110):17-17, 2021.
Tim Maniquet, Hans Op de Beeck, and Andrea Ivan Costantino, Recurrent issues with deep neural network models of visual recognition. bioRxiv, pages 2024-04, 2024.
A. Martín Andrés and M. Álvarez Hernández. Estimators of various kappa coefficients based on the unbiased estimator of the expected index of agreements. Advances in Data Analysis and Classificattin, (91):177-207, 2024. doi: 10.1007/111634-024-00000 htg//0.1007//50 111111110000/111144-00-----00000
URL httt:://dsa doi
Dean P. McKenzie, Andrew J. Mackinnon, Normand Péladeau, Patrick Onghena, Peter C. Bruce, David M. Clarke, Susan Harrigan, and Patrick D. McGorry. Comparing correlated kappas by resampling: Is one level of agreement significantly different from another? Journal of Psychiatric Research, 483 492, 1996. doi: 10.1016/s0022- 000001660022-955600000333 org/ 10. 101.1/0011/00222 3956 (96) 00033-7
URL http:///t// doi
Evan Miller. Adding error bars to evals: statistical approach to language model evaluations. arXiv preprint 1XXi:2241.0006440 2024.
Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael Dusenberry, Sebastian Farquhar, Qixuan Feng, Angelos Filos, Marton Havasi, Rodolphe Jenatton, et al. Uncertainty baselines: Benchmarks for uncertainty & robustness in deep learning. arXiv preprint arXiv:2 04015 2021.
Netta Ollikka, Amro Abbas, Andrea Perin, Markku Kilpeläinen, and Stéphane Deny. Humans beat deep networks at recognizing objects in unusual poses, given enough time. arXiv preprint arXiv:2402.03973. 2024.
Will Orr and Edward B Kang. Ai as sport: On the competitive epistemologies of benchmarking. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1885,1884, 2024.
13

---

Nikhil Parthasarathy, SM Eslami, Joao Carreira, and Olivier Henaff. Self-supervised video pretraining yields robust and more human-aligna visual representations. Advances in Neural Information Processing Systems, 36:65743-65766 2023.
Joelle Pineau, Philippe Vincent-Lamarg Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). Journal of machine learning research, 22(164):1122 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-522, 2015.
Veli Safak. Min-mid max scaling, limits of agreement, and agreement score, 2020. URL https 12904
//arxiv.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007,
David Sculley, Jasper Snoek, Alex Wiltschko, and Ali Rahimi. Winner's curse? on pace, progress, and empirical rigor. 2018.
Thomas Serre. Deep learning: the good, the bad, and the ugly. Annual review of vision science, 5: 399,426, 2019.
Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D' Douza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah Smith, et al. The leaderboard illusion. arXiv preprint (iXx:25544.2879, 2025.
Jeyan Thiyagalingam, Mallikarjun Shankar, Geoffrey Fox, and Tony Hey. Scientific machine learning benchmarks. Nature Reviews Physics, 4(6)) 2022.
Rachel Thomas and David Uminsky. The problem with metrics is a fundamental problem for ai. arXiv preprint 2xXiv:2002.08512, 2020.
S. Vanbelle and A. Albert. A bootstrap method for comparing correlated kappa coefficients. Journal of Statistical Computation and Simulation, 78(11):1009-1015, 2008 doi: 10.1080/ 09949650701410249, URL http://////d doi. org 110.0000000000999900000004444440
Sophie Vanbelle, ChristinaA Hernandez Engelhart, and Ellen Blix. A comprehensive guide to study the agreement and reliability of multi-obserrerr ordinal data. BMC Medical Research
Methodology, 24(1), 2024. doi: 10. 10.186661287744244044444 URL http://dxx doi org/10 1186/ 024 024-02431-
Felix A Wichmann and Robert Geirhos. Are deep neural networks adequate behavioral models of human visual perception? Annual Review of Vision Science, 9, 2023.
Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kajić, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Pinelopi Papalampidi, Ira Ktena, Chris Knutsen, al. Revisiting text-to-image evaluation with gecko: On metrics, prompts, and human ratings. arXiv preprint arXiv:2404. 168200 2024.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimizze hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 11 (((3)):8619 8624, 2014.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LKK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 2180)):2011496111 18, 2021.
14

---

A Related Work
DNNs as vision models. Within the last decade, Deep Neural Networks (DNNss) have been proposed as models of the human visual system (Doerig et al., 2023 Kriegeskorte| 2015 Cichy and Kaiser 2019 Kietzmann et al. [017], not only because they form the only class of comge-co models that achieves human-level performance on benchmark tasks like ImageNet [Russakovsky et al., 2015)) but also because performance-optimized DNNs are the best predictors of biological neural activations
Yamins et al.) 2014 Kubilius et al., 2019 Zhuang et al.. 202T]. However, the suitability of DNNs
as models of vision is debated [Bowers ct 2022 Wichmann and Geirhos [0233] Maniquet ct al.
|024|Seerre| |Serre| 2019), as is the question of how comparisons should best be conducted Firestonee 2020
Lonnqvist et al. 202T Wichmann and Geirhos 2023
Error Consisteney. The underlying idea motivating error consistency that of "molecular psy- chophysies" [Green] 1963.. which is to go beyond aggregate measures like accuracy, and instead compare behavior on trial-by-trial basis. EC promises to achieve this by building on Cohen's
Cohenn 1960 Martín Andrés and Álvarez Hernández, 2024]. which considers the triel-level
agreement observed for pair of classifiers. Geirhos et al. [2020] proposed EC in the context of
human-machine comparison, but in principle, the method applies to the general setting of comparing
arbitrary classifiers. Martín Andrés and Álvarez Hernández [024 observe that is biased estimator
and should be corrected, which we support in our implementation. Safak (2200] also observe the
problem of k's dependency on the marginals, which we outline SSSecction 2 They propose to scale
between its theoretical limits in an attempt to obtain measure of error consistency that is orthogonal
to accuracy. [ool al. (2025) propose apply Cohen's i. directly to the responses given by both classifiers i! rather than the correctness values Ti, and adapt the marginal probabilities of independent classifiers to reflect their accuracy. We still focus on the standard error consistency proposed by Geirhos et al.] [2020) because it has seen the widest application. In principle, our bootstrapping methodology also works for these other metrics.
Uncertainties in Benchmarking. Benchmarking has a long history and is central to modern machine learning (Koch and Peterson, [0024] (Singh et al. [225 Hardt 2025 Orr and Kang [024]. For example, the popular Papers with Code database" listts approximately 14,000 different benchmarks. With the emergence of modern benchmarks and leaderboards, issues such as overfitting to benchmarks (Singh et al.] 2025], the limitations of metrics [Thomas and Uminsky 2020). and various sources of uncertainty have been identified and challenged [Lehmann and Paromau 2025 Bouthillier et al., 2021]. In general, it has been criticized that benchmarks should not be the primary goal of scientific machine learning research [Alzahrani et al., [024]. At the same time, it has been pointed out that benchmarking itself requires a scientific approach and some kind of standard methodology Thiyagalingam al. 2022, (Sculley et al.) 2018 Hardt 2025). Similar to work, Nado al. 2021] argued that uncertainty is an important aspect to consider for benchmarks. In contrast to previous work, focus uncertainty in the behavioral comparison of classifiers using error consistency.
B Proof of Propositiin 2
The underlying distributions translate into the observed distributions P1 P1 and p(2) Peopy P(1) (1 Peopy) p(2), Inverting the marginal p(2), we get p(2) (y) p(()(()) Peopy- p(l) (v)
Proof.
T-Peopv
For the joint distribution, we have P(y, y) = p()) (y) (Peopy + (1 Peopy) ) P2 (2()) because the second classifier makes the same prediction as the first in all copy cases (Poopy) and in those cases, in which it does not copy but coincidentally makes the same prediction ((1 Pcopy) P(2) (y)).
http:://paprrsstthcode.ooom
15

---

Pobs
Pexp
K
Peap
ΣØ=1 ((1,2)(y,)) E pit) (()
1-E ((2)
2-=11 (p) + Peopy) P2(U)) -E ((3(())
E p(t) (y) ((2) (g)
Σy=1
+ (y)) Peopy p(l) p(u - E (())
(()) P()
1 1 Σ
Σx (p()(y) (y) p()
= Peopy
1 ΣK 2-=1 (()(())
1 Eg p1) (1))
Peopy"
2 (y) ((2)
C Proof of Ceiling Effects
claim that as sequence of trials, the EC of the pair will be O irrespective of the accuracy of the other one. Here, proof this claim, which is easiest to derive from a confusion matrix shown in Table The observed agreement is found on the diagonal, poba = a + d, while the expected agreement is what would be on the diagonal if observers were independent (Pezp (a + b) c) + db d)). For the proof, consider without loss of generality that classifier 2 was perfect, i.e. a --= 1.
In Section 2,
classifier achieves perfect accuracy the investigated
(2)
(2) b
0
(1)
a
a+ b
(1)
d
c d
a a
b+ d
1.0
Table Error consistency matrix.
Proof. Let a + then + d 0, so Pesp = (a b(a)) d) d) = (a+ + b). Since b b2 (1 and d 2 0, b
d dd:0 0.
Pohs Pezp
Peup
(a d) (a b)
(a + b)
d-b
1 (a + b)
By symmetry, the same argument holds for the case where one classifier gives incorrect responses on all trials.
16

---

D Lower bound on EC
Figure 2 shows how the mismatch in accuracies between two classifiers imposes upper bounds on EC. In extension, Figure 6 shows that there are also non-trivial lower bounds (subplot b). These lower bounds are not a simple function of the upper bounds. Instead, they are inversely related such that when accuracies are inverted (i.e., Pl attained. But perhaps counterintuitively, even with inverted accuracies, EC -1 can not always be reached. This is in contrast to the upper bound, where aligned accuracies (i.e., p1 p2) always allows EC
P2 along the dfagona))) lower values can be
1.8
a
Accuracy of the Second Classifier
Accuracy the Second Classifier
0.8
08
88
00
Max EC
Min EC
$D.11
2 --.1
50.
-0.2
02
0.2
5..6
2 00.8
0.8
2-0.88
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Accuracy of the First Classifier
Accuracy of the First Classifier
Figure 6: Lower bound of EC in comparison to upper bounds. The left subplot (a) is the same as 2p, showing regions in which EC is upper bounded by the mismatch in accuracies of two classifiers (x and axes). Additionally, the right subplot (b) shows the lower bounds. Again, each point represents Model-vs-Human comparison in the analysis Geirhos et al. [22T]]
E Effect of EC on CI size
For Figure 3]
we have chosen ground-truth EC of 0.5 to demonstrate how the EC between two classifiers depends on their accuracy and the number of trials on which EC is measured. However, the choice of the ground-trun truth EC influences the CI size as well, which we demonstrate in Figure We again generate data using our copy-model, and plot the ground-truth EC (input of the model) against the distribution of empirical ECs (calculated on model outputs). To better show the changes in CI size, we plot the delta between empirical and ground-truth EC rather than the absolute EC. Note how the CI bounds are not symmetric and largest a ground-truth EC of 0.5.
F
Bootstrapping
For readers unfamiliar with bootstrapping, we provide basic introduction here, but refer to Efron and Tibshirani (1944) for details. The idea the following: We want estimate the uncertainty in summary statistic (e.g., the mean), which is computed for dataset X of size N. If we had access the underlying data-generating process, we could simply generate M more datasets (each of size N) and compute the summary statistic for each. This would give us distribution of values of the statistic, from which we could then compute a measure of uncertainty (e.g. a percentile interval distribution of means). Without access to the true data-generating process, we build the best possible parameter-free model: To generate a data point, randomly select one the data points x; with uniform probability. To generate an entire dataset, draw N data points with uniform probability, with replacement. In the context of error consistency between two observers, one data point corresponds to one trial, i.e. pair of responses = (r(") r
(2) E 1) The full algorithm to obtain confidence interval for two sequences of responses ( and (2) can thus be summarized in Algorithm 1
17

---

Delta between empirical and true EC
0,4
Trials
100
0.2
200
400
800
1600
0.0
-0.2
0.0
0.2
0.4
0.8
1.0
Figure 7: Ground-Truth EC against CI sizes. We simulate data with different ground-truth EC values and plot the size of 95% confidence intervals. Evidently, one obtains the largest CIs at ground- truth EC 0.5.
Algorithm Bootstrapping EC confidence intervals
Require: Arrays y(11, (2) E 1)x, number of bootstrap iterations M, sample size N Ensure: 95% confidence interval for enror consistency 1: L +-0 0
> Initialize list of EC values
2: for 1 to M do
3:
I + sample N indices from [I....,,N] uniformly with replacement
4: ec + (C(r (2) [Z]
>> Compute emror consistency Add to list
5: L CU {ec}
6: end for
7: [lower, upper] +- Percentileasg (C)
Compute 95% interval
8: return [lower, upper]
Similarly, the algorithm for estimating confidence intervals without access to real trials is described Algorithm 2
G Validating the copy model
As proven by Proposition 2 our copy model is correct in the limit of trials. Additionally, empirically validate the uncertainty estimates we derive from this model, by first generating some ground-truth data with fixed
fixed accuracy of both classifiers. We then bootstrap this data M = 1, 000 times, as we would to obtain percentile intervals. Next, we instead draw M samples from our copy model, compare the CIs implied by our copy model to those implied by the bootstrap, which we consider the gold standard. We plot this comparison in Figure 9 Evidently, the distributions we obtain from our copy model match the bootstrapped results quite well. as long as the number of trials is sufficient or the accuracy avoids floor- and
consistency
H Details on Model-vs-Human results
An idiosynerasy of the Model-vs- Modma-nv-Human benchmark that we have not expanded on so far is the fact that while the final scores (which are averages across the different corruptions) are stable, the uncertainty within each corruption can be quite large. In Figuue 11 we plot the CIs surrounding the EC values obtained for the phase-scrambling corruption as an example. In Figure 12 we plot pairwise rank correlations between all 17 corruptions, revealing that some, but not all corruptions are highly correlated, i.e. imply similar model rankings. In Ingigure 10 we plot the confidence intervals one would obtain under the alternative aggregation strategy of simply averaging across conditions,
18

---

0.600
Mean Limits
0.575
Error Consistency [Kappa]
0.550
0.525
0.500
0.475
0.450
0.425
0.400
10
10a
10a
Number of Bootstraps
Figure 8: Convergence EC estimates as a function number of bootstraps. Starting from single sequence of 1, 000 trials generated by our copy model (arbitrary target EC of 0.5, classifier accuracies 0.7 and 0.8 respectively) we chose number of bootstrap runs and bootstrapped 100 times for each, to obtain estimates of uncertainty. Shaded regions correspond 95% percentile intervals. Evidently, at the M 10, 000 bootstrap runs we use, the estimates have stabilized, thus justifying choice of M.
EC [Kappa] (Thals; 50)
0.4
-.2
EC from model
-0.4
EC from bootstrap
D
EC [Ksppa] [Taals: 1000)
0.6
0.0
-0.2
-0.4
0.5 Ground Truth (Accuracy: 0.95)
Ground Truth EC (Accuracy: 0.5)
Figure 9: Validating our copy model. We plot the distributions of EC estimates from the copy model to bootstrapped (gold standard) estimates of uncertainty. Top: 50 trials. Bottom: 1000 trials. Left: Accuracy 50%. Right: Accuracy 95%. Evidently, both the number of trials is low and accuracy approaches the ceiling of 100%, the model diverges from bootstrapped results, but error consistency is not well-behaved in this case anyway. Ideally, experiments should avoid this regime, which can be achieved by increasing the number of trials and calibrating their difficulty.
19

---

Algorithm 2 Bootstrapping EC confidence intervals without real trials
Require: Target marginals p
p(() over K classes, target error consistency EC, number of trials N, number of simulations M
Ensure: 95% confidence interval for eπror consisteney
Compute underlying parameters:
1-EK (1))(())
2: Peopy f EC
1- (()())
(y) (y) for all y {L, , K}
4: (()) (v) -Prooy PPspy-pppp (y))
for all y E { K}
1-Pop
5:
6: ++ 0
>> Initialize list of EC values
7: for = 1 to M do
8:
Generate responses for first classifier:
9: for j = to N do
10:
~ Categorical (P(1)
> Sample from marginal
11:
end for
12:
13:
Generate responses for second classifier:
14:
Newpy Peopy NJ Toopy f sample Noopy indices from , N without replacement
Number of responses to copy
15:
for to N do
17:
if; j E Zoppy then
(2) (1) ) )
Copy response
else
20:
end Categorical
> Sample independently
21:
22: end for
23:
24:
EC( (2)) LU
Compute error consistency
25:
end for
27:
[lowrr, upper] + Percentileasg, (C)
> Compute 95% interval
return [lower, upper]
without grouping by corruption first. Evidently, this would drastically affect the confidence intervals. To be clear, this is a deviation from the MvH protocol and thus "wrong", but the data itself the same, thus illustrating the point that there is large variance in the data, which is obfuscated by the hierarchical aggregation.
In Figure 2p, we explain how in many of the conditions within Model-vs-Human, the accuracy mismatch between models and human participants is so large that kmar takes on very small values. One might wonder how Model-vs-Human scores would change if one excluded conditions not as prescribed by the authors, but based on this mismatch. However, this would render the selection of conditions dependent the model. Hence, the benchmark would longer present a fair comparison of models, and final scores would no longer be comparable.
I Generating Synthetic Brain-Score Data
we estimate the size of confidence intervals surrounding models on the public Brain-Score benchmark. To estimate a model's EC humans within one condition c, we would technically have to create one set of binary trials that has the property of having the desired average EC to the human observers of Kyeal.e- To simplify, we drop this dependence and generate n independent sequences of trials, one for each human who saw images in this condition. Generating only one sequence of trials would drastically overestimate the variance, which reduced by taking the mean. For each of these sequences, need three inputs for the copy mopel: The
For the analysis conducted in Section 4.2
20

---

Consistency Humans [Kappa]
0.5
0.3
0.2
0.1
Models
Figure 10: EC to humans without hierarchical aggregation. If one were to simply define a model's EC humans its average EC over all conditions (without first grouping by corruption), would obtain the 95% confidence intervals depicted here, which clearly overlap for all models.
phase-scrambling conditions)
Consistency Humans [Kappa]
0.35
0.30
0.25
0.20
0.15
0.10
122925211111
20 3436 1024 16244341 323713 1728 Models
Figure 11:
ECs phase-serambled images. We plot CIs for the EC humans achieved by all models in the phase-scrambling corruption as an example, analogous to Figure 4
desired kappa (which is given by the Brain-Score data after removing the ceiling correction), the accuracy of the humans (which is given by the Model-vs-Hunee data) and the accuracy of the model in this condition. The latter is unknown, but bounded mathematically. We obtain the CI sizes in Figure by selecting the accuracy in the middle of the bounds, which seems reasonable. Selecting the lower bounds of the accuracy leads to the most optimistic estimate of CI size, which plot in Figure 14
21

---

1.00
0.26
0.39
contrast
0.28
1.30
.0.45
-0.05
-0.18
-0.23
9.11
-0.02
-0.04
-033
-0.30
-0.20
0.75
cue conflctt
0.14
0.45
1.00
0.36
0.33
0.10
0.49
0.32
0.08
0.48
0.19
0.28
0.27
0.62
0.59
0.72
0.37
0.56
eidelent
0.00
-0.05
0.33
0.24
1.00
0.30
0.54
0.001
0.03
0.41
0.40
0.28
0.32
0.39
0.41
0.28
0.16
eiddionil
0.31
0.08
0.10
0.05
0.30
1.00
0.24
0.13
-0.23
0.13
0.29
0.21
0.06
0.12
0.01
0.06
-0.16
0.25
Talse colour
0.39
0.34
0.32
0.31
0.01
0.13
0.17
1.00
0.10
0.299
0.04
0.04
0.11
0.31
0.36
0.35
0.34
Experiment
Nih--pas
-.0.08
<0.10
0.08
0.16
0.03
-0.23
0.02
-0.10
1.00
0.06
0.16
0.07
0.24
0.04
0.15
0.16
0.36
pass
phase-scramiling 0.14
0.11
0.19
0.16
0.40
0.29
0.29
.0.04
0.16
0.38
1.00
0.46
0.33
0.23
0.31
0.19
0.29
powerequalisatiion -00
-0.02
0.28
0.20
0.28
0.21
-0.04
0.07
0.35
0.48
0.33
0.36
0.29
0.30
0.50
sihouette -0.28 -0.31
0.62
0.54
0.39
8.12
0.44
-0.31
0.04
0.58
0.23
0.36
0.35
1.00
0.68
0.80
0.38
0.40
0.56
stylized
.0.43
0.72
10.35
1.00
uniform-noise -.28
-0.20
0.37
0.43
0.16
-0.16
0.28
-0.34
0.36
0.37
0.29
9.30
0.33
0.38
0.45
0.42
1.00
ralour
contrat
Cue annallle
edbioon
edconil-
sidobrill
high-pass
pass
phask-crambling -aaiiinnnnn
rntaton
sthouctt
styized
uniformnneie nole
11.00
false
Experiment
Figure 12: Spearman Correlation between Experiments. For all possible pairs formed by the 17 corruptions from Model-vs-Humail we compute the Spearman rank correlation between the rankings of models they imply. All rank correlations are statistically significant at 0.05 (without correcting for multiple comparisons).
22

---

colour
0.35
contrast
cue-conflict
0.30
edge
eidolon!
0.25
eidolon!l
eidolonili
D-value
false-colour
high-pass
0.15
low-pass
phase-scrambling
power-equalisation
0.10
rotation
silhouette
0.05
sketch
stylized
0.00
uniform-noise
0.1
EC
Figure 13: p-values for the EC between CLIP and human participants. For all experimental conditions (corruptions at certain strength) we scatter the EC against the p-value, finding that the majority of conditions is significant, while some values close to 0 are not, which is reasonable.
Index
Model Name
Index
Model Name
0
clip
26
vggl6 bn
1
BiTM resnetv2_101x1
27
mobilenet v2
2
BiTM resnetv2_1522 152x2
28
vggll bn
3
WWTMM_resett2_500
29
resnet101
resnet50_12 ____pp5_________________________
30
resnet152
5
resnet50_12 _________3__33__________________
31
mnasnet1 _
6
ResNeXt101 32x16dxxxggcc swsl
32
wide rede_ressetl01
7
152x4
33
resnext50_32xx 32x4d
8
BiTM_ 2
34
resnext101
9
resnet50_12 eps1
35
simclr reennt50x2
10
vit vit_ st__arge_paa prrge_patch16__cco 224
36
vgg13_br
11
vit ftase_patchll 224
37
simclr resnet50x4
12
vit hit_small_pa Saciallppachhh...1 224
38
simelr resnet50xl
13
munfformer_.11_IN2111
39
@ssnet50_1__ee
14
densenet201
40
MoCoV2
15
resnet50_
41
viidereenne50)
16
inception_vv v3
42
efficientnet_1__men
17
ransformer_Bll__NN
43
squeezenetl_1
18
resnet50
44
mnasnetO_5
19
densenet169
45
InfoMin
20
BiTM res1etv2_101x3
46
alexnet
21
resnet34
47
shufflenet_vv v2 v2_x0_5
22
resnet50
_
48
squeezenet1 _0
23
resnet18
49
MoCo
24
vgg19_bn
50
PIRL
25
densenet121 21
51
InsDis
Table 2: Model Indices for MyH. map model indices Figure 4 to their full names.
23

---

Error Consistency Humans [Kappa]
0.32
0.30
F I
0.28
0.26
0.24
0.22
0.20
Models
Figure 14: Most optimistic Confidence Intervals for Brain-Score ECs. Like in Figure plot the 95% confidence interval of the EC of the top 30 Brain-Score models, but assume the lowest mathematically possible model accuracy. The true CI size for these models probably looks more like Figure 5) but this is a lower bound on their size.
24

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: abstract and introduction, we claim to calculate confidence intervals for error consistency and use this to revisit earlier work, which then do.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our work, in particular the fact that we are limited to bootstrapping instead of relying on particular analytical solutions. The computational costs of our algorithms negligible. There are privacy and fairness concerns.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
25

---

Answer: [Yes]
Justification: The new theoretical contribution of our paper is the copy model in Section For didactic reasons, we first proof the correctness of the model for simpler case of equal accuracies, [Proposition I before moving on to the harder case of unequal accuracies in Proposition
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide code to reproduce all experiments in the supplemental material. The data sources for empirical experiments (Brain-Scone) and Model-vs-Hund vuldar-s-Humar))) are publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general, releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
human
26

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: See previous question. The submitted code is extensively annotated with instructions and contains an environment specification via requirements.ttt
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //.pps.cc/publi//ggug guw.es/ CoodeSummis ssi asssonoiicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification:
We do train DNNs in the context of this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The reporting of error bars the topic of our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA]
Justification; All experiments (bootstrapping) done for this work are at tiny scale and run on consumer laptops in seconds or a few minutes.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwwapps...cppppp//tttaaa
Answer: [Yes]
Justification: Our research is not in conflict with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdictio).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We feel this question is not applicable to our work, We do not release models that could be misused, or directly impact society.
28

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the works we investigate and whose data we use, such as Model-vs-
Human.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
29

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not release new assets. For the code we will release upon acceptance, we will follow standard procedures by releasing via Github etc.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do conduct experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not conduct experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
We do not make use of LLMs beyond standard programming aid for editing purposes.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) hutp:::/eearrppccc/ooffeeeeeeeee /c///onfeeeeeees//225////.. for what should or should not be described.
31


Reviews:
[{"review_id": "xKt7eWlDIV", "reviewer": "Reviewer_SWie", "summary": "This paper addresses the issue of quantifying uncertainty in Error Consistency (EC), which is a standard metric used to measure the behavioral alignment between ML models and human behaviors. The authors improve the standard EC in two ways:\n- They demonstrate how to obtain confidence intervals for EC using a bootstrapping technique, and derive significance tests for EC.\n- They propose a new generative model that simulates the experiment and estimates the resulting confidence intervals. This model can be used to provide practical guidance on the number of trials required for experiments.\n\nThe paper applies this methodology to existing experimental results. The authors emphasize that their method enables researchers to design experiments that can reliably detect behavioral differences and fosters more rigorous benchmarking of behavioral alignment.", "questions": "- What are the takeaways we can get from analyzing with the confidence intervals but cannot with point estimates?\n- It would be better if the authors can offer more discussion over the definition of EC and expand the results to other metrics besides Cohen's $\\kappa$.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 1, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly motivated: the measure of EC is inherently noisy, so quantifying uncertainty is necessary when analyzing the results.\n- The discussion with theoretical issues with EC offer a solid theoretical ground for the paper.\n- The solution (bootstrapping) offered by the paper is concrete and pragmatic for calculating confidence intervals for empirical EC values. The proposed generative model can be useful for practitioners to determine the sample size of experiment.\n- The re-analysis of existing experiments (Model-VS-Human and Brain-Score) provides empirical evidence for the issues of uncertainty and demonstrates the utility of their methodology.\n\nWeaknesses:\n- The assumptions of bootstrapping are not discussed, such as the sample need to be representative, which can be questionable for some scenarios of error consistency between human and ML classifiers. For example, when the sample of human is too small and cannot represent the population of interest.\n- The paper simply applies bootstrapping to EC, leaving the choice of parameters in bootstrapping that could impact the results dramatically. \n- The motivation for the definition of EC is not enough. Why is EC modeled with $p_{copy}$? Is it the only choice?", "was_revised": true, "final_justification": "Thank the authors for their responses during the rebuttal.\n\nI found that most of my concerns are the same with reviewer eJLm's. The most important contribution of this paper is the uncertainty quantification of EC, but I am still concerned that the method used by the authors is not enough. When you have access to the trail-level data, it is natural to quantify the uncertainty using bootstrap. The new method proposed by the authors applies to the case before the actual data is collected. However, it largely depends on the interpretation of EC and the current manuscript lacks validation on this.\n\nThe authors mentioned that they will add a result showing the match between simulated results and empirical results. This analysis can be important for validating the model. Based on the commitment to include this validation, I have raised my score from 3 to 4.", "rebuttal": "We would like to thank all reviewers for their valuable feedback. We very much appreciate their assessment of our work as containing a **“strong motivation”** (HmXf), being **“well-written and easy to follow”** (eJLm), offering **“novel insights”** (HmXf) and **“provid[ing] new tools”** (1TAb). It connects to previous work (SWie, 1TAb, eJLm) and adds **“a major theoretical result”** (1TAb) which improves our understanding of the popular Error Consistency metric. We have implemented most of the reviewers’ suggestions in the newest version of the manuscript, and hope our responses will clarify all remaining points. The reviewers’ feedback has greatly improved the submission. \nDue to the rebuttal rules this year, we cannot upload a new version of the manuscript, but can only promise to make changes in the camera-ready version, should the paper be accepted. \n\nDear Reviewer,\n\nThank you for taking the time to review our paper. We appreciate that you found our paper to be **clearly motivated**, and believe that our **generative model can be useful**. Based on your feedback, we have amended the manuscript to elaborate further on the details of bootstrapping. In summary, we follow standard procedures as suggested by e.g. [1], which are very robust to parameter choices, see below for a more detailed explanation. \n\n*Q: “The assumptions of bootstrapping are not discussed. [...] For example, when the sample of humans is too small and cannot represent the population of interest”*\n\nA: While you are right that a fundamental requirement for our approach to work is a well-conducted study, like the one by Geirhos et al. (2021), our method is concerned with something else: Once we have an empirical measurement for one pair of observers, how can we estimate the uncertainty in that measurement? The point of bootstrapping is orthogonal to the question whether the sample of human observers is representative, which is, of course, also an interesting question, but not one we address.\n\n*Q: “Why is EC modelled with $p_{copy}$? Is it the only choice?”*\n\nA: Thank you for this question. In principle, it is possible that other generative models of EC exist, but our model based on $p_{copy}$ has the significant advantage of being very interpretable, providing us with an intuition for what Error Consistency really means. This is itself a contribution of our work: Previously, the metric was not well understood, so there was a risk of people using it incorrectly. Our model also allows us to simulate trial data, which is very useful for planning experiments and for the analyses we conducted here (e.g. we can estimate confidence intervals for Brain-Score models without access to most of their data). Does this answer sufficiently address your concerns? The definition of EC, which you also asked about, is outside of our control: It is, as you write, an established metric already, which measures a quantity of interest that is not captured by the percentage of correct answers given by two classifiers.\n\n*Q: “What are the takeaways we can get from analyzing with the confidence intervals but cannot with point estimates?”*\n\nA: When ML models are evaluated regarding their behavioral alignment with humans, confidence intervals are crucial for proper evaluation of the measure. For example, say we have two models, each with EC = 0.5 (close to the alignment between humans). Critically, if the confidence interval of one was very wide, say [0.3, 0.7], and that of the other was very narrow, say [0.49, 0.51]. Our interpretations are substantially different in the two cases: For the first, we could assume that the high EC value came from random fluctuations (especially in our context, where multiple classifiers are evaluated and the best one is considered), and it is not ruled out that the ground truth is that this first model has a relatively low alignment. On the other hand, we could be reasonably certain that the second model is well aligned with human behavior; the results are certainly not due to measurement noise. Thus, analyzing EC with confidence intervals substantially changes interpretations of point estimates here: They tell us whether observed differences in the point estimates are meaningful or not.\n\n\n**Details of Bootstrapping**\n\nThank you for pointing out that our explanation of bootstrapping may have been too short. We have elaborated more on this point now, in summary: Given two sequences of length N from two observers with indices ${0,...,N-1}$, we first draw N indices randomly, with uniform probability and replacement. We then obtain bootstrapped trials by selecting the observers’ trials at the chosen index positions, yielding two new sequences of trials. From these new sequences, we can calculate new estimates of EC. Repeating this procedure multiple times yields distributions of EC values, thereby quantifying the uncertainty in the measurement.\nTo address your concern about parameters (which bootstrapping requires very few of), we have also added a paragraph justifying our parameter choices. Basically, there is only one variable parameter, which we typically denote as M, the number of bootstraps. While higher M is always better, we find that a value of 10,000 is sufficient, and have added a figure to the appendix which illustrates how bootstrapped EC values converge as a function of M. The other choices, such as sampling N items with replacement for a sequence of length N and drawing uniformly is how a non-parametric bootstrap is performed, see [1]. \n\n[1] Efron, B., & Tibshirani, R. J. (1994). An introduction to the bootstrap. Chapman and Hall/CRC."}, {"review_id": "wgxPFiYqk9", "reviewer": "Reviewer_1TAb", "summary": "This paper presents a study of error consistency (EC), a standard metric to measure the behavior alignment between ML models and human observers. \n\nA major theoretical result is a new computational model, which frames EC as the probability that one classifier copies the other, multiplied by a factor originating from the accuracy mismatch. This result serves two purposes. \n\nFirst, it provides an alternative interpretation of EC, offering a clearer picture that it can be decomposed into the two factors of the copying probability and mismatching accuracy. \n\nSecond and perhaps more importantly, it offers a means of computing confidence interval for EC, a missing component in previous studies that only provides a point estimation and hence may not be reliable. More importantly, while bootstrapping can be trivially applied for estimating confidence intervals, the technique from their Generative computational model can be useful for planning future experiments, as it helps to determine the number of trials needed to obtain statistically reliable results. \n\nThe authors applied their new methodology to re-analyze the results from major benchmarks, including Model-VS-Human and Brain-Score.", "questions": "I don't have any questions.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- The paper provides new insights into the EC metric with the decomposition into the copy probability and the accuracy mismatch. This can be important for understanding what is truly being measured by this metric. \n- The paper provides new tools for computing confidence intervals for EC, a missing piece in previous works.\n- The paper revisited previous experiments using their algorithm, and revealed new insights (e.g., a larger number of trials is essential to obtain statistically significant results).\n\nWeaknesses:\n\nI don't have any particular concerns. One suggestion is to put the algorithms in Sec 3.1 and Sec 3.2 in an algorithm box to improve on clarity.\n\nOverall, the paper has a nice flow that started with an in-depth analysis of the metric itself, revealing insights into what it is truly measuring; then went into the key objective of  calculating confidence intervals, which makes use of the aforementioned analysis. While I don't have experience in this area (hence cannot comment on related work), the paper does appear to offer a quite solid study of EC, that can benefit future studies in using it as a tool for measuring machine vs human consistency and beyond.", "was_revised": true, "final_justification": "By reading the other reviewers' comments, there are concerns about the significance of the contribution and generality of the theoretical result. I am unfortunately not knowledgeable enough in this area to evaluate those aspects. \n\nI rate the paper as acceptance mainly because the paper is well-written and I learned some new interesting insights, and that I believe that the general (non-expert) audience of the conference can benefit the same from this paper.", "rebuttal": "We would like to thank all reviewers for their valuable feedback. We very much appreciate their assessment of our work as containing a **“strong motivation”** (HmXf), being **“well-written and easy to follow”** (eJLm), offering **“novel insights”** (HmXf) and **“provid[ing] new tools”** (1TAb). It connects to previous work (SWie, 1TAb, eJLm) and adds **“a major theoretical result”** (1TAb) which improves our understanding of the popular Error Consistency metric. We have implemented most of the reviewers’ suggestions in the newest version of the manuscript, and hope our responses will clarify all remaining points. The reviewers’ feedback has greatly improved the submission. \nDue to the rebuttal rules this year, we cannot upload a new version of the manuscript, but can only promise to make changes in the camera-ready version, should the paper be accepted. \n\n\nDear Reviewer,\n\nThank you for your review. We were delighted to read that you believe our work **\"provides new insights\"** and that our new model is a **\"major theoretical result\"**. We welcome your suggestions to provide supporting explanations. For that, we have written explicit algorithms for the procedures we describe in sections 3.1 and 3.2, which we would include in the camera-ready version upon acceptance. This does indeed facilitate the understanding of the bootstrapping procedure, which other reviewers also asked about, so thank you for the valuable suggestion."}, {"review_id": "9KddxfXdXo", "reviewer": "Reviewer_eJLm", "summary": "Error consistency quantifies whether two classifiers make consistent decisions. Geirhos et al (2020) proposed to use Cohen’s Kappa (Cohen, 1960), a popular metric in behavioral science, to evaluate the error consistency of deep networks and humans. Previous studies focused on reporting the mean of the Cohen’s Kappa. The variability and the confidence interval associated with Cohen’s Kappa have been ignored. This paper proposed a procedure to compute the confidence interval of Cohen’s Kappa using bootstrapping, and applied it to evaluate two benchmark datasets. The paper also proposed a generative model of classifier consistency, which might allow one to perform simulations to decide on the sample sizes that would be needed in the experiments for achieving certain statistical power.", "questions": "The paper reported results on testing whether the EC between a network and humans was substantially different from that between two independent binomial observers in Fig. 7. While this test is useful, its interpretation is also limited. Another test (potentially more useful test) would be to test if one EC is larger (or smaller) than another EC. This would allow one to make claims about which neural network exhibits behavior that is more similar to humans. Would this be possible? (Note that Figure, right panel, seems to be imply that this test is already done between the different network models, but how that was done was not described in Section 3.3.)\n\nLine 135-144 mentioned that it would be important to ensure that human responses have uniform marginal distributions. Practically, what would be an effective procedure for achieve this?\n\nCan the authors please explain in more detail how the bootstrapping procedure in Section 3.1 was applied in oder to take into account that each human subject likely exhibit different behaviors and thus might have Cohen’s Kappa to networks?", "limitations": "One of the limitations of the work was acknowledged, i.e., a lack of analytical solutions.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "###Strengths: \n\nThe paper is well-written and is easy to follow.  \n\nThe proposed method for computing the confidence interval of Cohen’s Kappa is reasonable, and is a nice supplement to the work of Geirhos et al (2020). \n\nSome of the limitation of Cohen’s Kappa is highlighted. For example, the upper bound of Cohen’s Kappa strongly depends on the difference in the accuracy of the two classifiers. This raised some concerns on prior results.\n\n\n###Weaknesses: \n\nThe work is fairly incremental. While the ability to compute a confidence interval is useful, the proposed method based on bootstrapping is fairly standard. This by itself does not seem to be justified for a stand-alone contribution at NeurIPS.\n\n\nThe applications in Section 4 did not reveal substantial new insights. \n\nThe interpretation of the main results in Figure 1 is problematic. It was claimed that because the error bar was substantial, the gap between network v.s. human and human v.s. human should shrink. However, it could also be the opposite, that is, the gap was actually larger than what was implied in the difference in the mean of the Cohen’s Kappa. \n\n\nThe analytical results in Proposition 2 and Section 3.2 seem to be based on relatively restrictive assumptions. It is unclear whether a mixture of a copy of the response distribution of classifier 1 and another marginal distribution is sufficient to model the general response distribution of classifier 2. This needs to be clarified.", "was_revised": true, "final_justification": "I would like to thank the authors for their detailed response during the rebuttal. The rebuttal clarified a few points. The authors acknowledged the error in the interpretation of Fig 1. It was originally claimed that because the error bar was substantial, the gap between network v.s. human and human v.s. human should shrink. Now the authors agreed that that interpretation was incorrect. While the authors considered this correction would not cause a re-interpretation of the overall contribution, I think it decreased the significance of work. That is, if the original interpretation in Fig. 1 were to be correct, that would be a substantial result. Now with the correction, we only know some of model comparison results are inconclusive due to the large error bar. While this is useful to know, it is not a substantial new insight. It remains \n\nAfter reading the authors’s response on the assumption of theory, I still have concerns about it. I understand that if classifier 2’s behavior can be described as the mixture of the distribution of classifier 1 and its own marginal, the math will go through. However, the behavior of classifier 2 can be dependent on that of classifier 1 without a component that “copies” the response of classifier 1. The dependence between the two classifiers can take other form. I believe, to fully understand this question, the conditional distribution between the two classifiers need to be modeled and linked to EC. So for this point, I disagreed with what was claimed in the paper about the generality of the theoretical results. \n\nI noticed reviewer SWie shared some similar concerns.\n\nI do agree that the finding that the interpretation of EC depends on the accuracy of the two classifiers is a useful result, which casts doubts on the usefulness of EC in practice and motivate more carefully designed experiments. I also agreed that it’s useful to have confidence intervals. However, this contribution is rather incremental. in particular Geirhos et al 2020 also reported confidence interval. I only became aware of this from reading the review of Reviewer HmXf. The authors claimed the method in Geirhos et al 2020 is suboptimal.\n\nOverall, I felt that, while the paper contains some useful contributions, substantial concerns remain. I increased my overal ratings from 2 to 3, yet remain overall negative about the paper.", "rebuttal": "We would like to thank all reviewers for their valuable feedback. We very much appreciate their assessment of our work as containing a **“strong motivation”** (HmXf), being **“well-written and easy to follow”** (eJLm), offering **“novel insights”** (HmXf) and **“provid[ing] new tools”** (1TAb). It connects to previous work (SWie, 1TAb, eJLm) and adds **“a major theoretical result”** (1TAb) which improves our understanding of the popular Error Consistency metric. We have implemented most of the reviewers’ suggestions in the newest version of the manuscript, and hope our responses will clarify all remaining points. The reviewers’ feedback has greatly improved the submission. \nDue to the rebuttal rules this year, we cannot upload a new version of the manuscript, but can only promise to make changes in the camera-ready version, should the paper be accepted. \n\n\nDear Reviewer,\n\nThank you for providing valuable feedback on our work. We were delighted to read that you found our manuscript well-written and easy to follow and agree that being able to compute confidence intervals is useful. We believe that addressing your concerns has greatly improved the manuscript, especially figure 1 (see below).\n\n*Q: “The work is fairly incremental”*\n\nA: You are right to point out that bootstrapping to generate confidence intervals would not be novel. However, our contributions go beyond that: At the moment, there is influential work that uses EC to draw widely believed conclusions about behavioral alignment between ML models and humans. But the metric itself is still poorly understood. At this point, non-incremental, “disruptive” work (e.g. proposing yet another poorly understood metric) would probably not advance the field. Instead, we believe it to be crucial to improve interpretations of the existing measure. The novel contribution we make lies in presenting a well-interpretable, generative model. This allows understanding EC better (as a scaled copy-probability) and, as a practical application, to predict the width of confidence intervals in future experiments (as agreed upon by other reviewers). This will enable future work to interpret and use the metric correctly. Without our work, there is a real risk of people overinterpreting differences in rankings made with error consistency and drawing wrong conclusions.\n\n*Q: “The interpretation of the main results in Figure 1 is problematic. It was claimed that because the error bar was substantial, the gap between network v.s. human and human v.s. human should shrink. However, it could also be the opposite, that is, the gap was actually larger than what was implied in the difference in the mean of Cohen’s Kappa.”*\n\nA: Thank you very much for pointing this out, you are absolutely correct. The phrasing in Figure 1 was indeed wrong—be assured that we never interpret our results erroneously in the text; this was a graphical error in how we represent our results in Figure 1, not an error of substance. What we meant to say was of course that the uncertainty in the measurement is revealed, so the likelihood that the gap was found by chance increases. Thus, our main message does not change. Of course, we have now fixed the figure, to say “gap size uncertain”, and apologize for the mistake. Thanks again for raising this concern!\n\n*Q: “The analytical results in Proposition 2 and Section 3.2 seem to be based on relatively restrictive assumptions. [...] This needs to be clarified.”*\n\nA: Thank you for bringing this point to our attention, we were indeed not sufficiently clear. Actually, these assumptions are not restrictive: Since we place no constraints of any kind on the marginal distribution (e.g. we do not enforce it to have a certain shape), any overall distribution attainable for observer 2 can be represented as a weighted combination of the marginal of observer 1 and this latent marginal of observer 2; its whole purpose is to “cancel out” the deviation from the empirical marginal introduced by the marginal of observer 1. The misunderstanding might arise from the fact that when we talk about the distributions, we mean the (binary) distributions over correctness here, not the distributions over responses. We have now revised this point in the manuscript . Does this clarify our reasoning?\n\n*Q: “Another test [...] would be to test if one EC is larger (or smaller) than another EC. [...] Would this be possible?”*\n\nA: Thank you for this excellent question! We agree that testing the significance of a difference in EC-values is even more relevant than just rejecting a null-hypothesis of independence. Indeed, doing this is possible with our methodology. In the simple case with full data, we have responses to N images from humans and two models. We can then bootstrap jointly from these responses, compute ECs for both human-machine pairings, and compute a difference. Doing so for each bootstrap run, we obtain a distribution over those differences, which we can then evaluate as usual. We are currently looking into this and will add an example of such a calculation to our paper. Moreover, with our generative model, we can also evaluate differences even without access to the trials themselves, by first using the model to generate sequences of trials and then running the bootstrapping. Thank you very much for raising this point; discussing and answering it has made our submission stronger.\n\n*Q: “Practically, what would be an effective procedure for achieving [uniform marginal distributions of human responses]?”*\n\nA: Thank you for this question. What we had in mind was to control the difficulty of trials for humans by standard procedures, e.g. running pilot experiments to find stimuli sets on which human participants achieve 50% accuracy, or using adaptive procedures like staircase paradigms. Alternatively, there are various standard techniques in psychophysics that are used to modulate difficulty of fixed stimuli, such as decreased presentation time or using masking patterns. In any case, controlling the marginal distribution of one of the two classifiers is sufficient. We have elaborated on this point in section 2 of the manuscript. Does this answer your question?\n\n*Q: “Can the authors please explain in more detail how the bootstrapping procedure in Section 3.1 was applied?”*\n\nA: Thank you for raising this point. We have amended the manuscript to elaborate further on the details of bootstrapping. In Section 3.1 specifically, we apply the bootstrapping approach to the data from [1], which is a bit more complex than a standard experiment: Essentially, they collected multiple trial-sequence-pairs for every pair of observers. One might be tempted to concatenate all these sequences to then calculate one final kappa for each pair of observers. However, since the sequences were obtained from experiments with differing difficulties, doing so would overestimate the EC. So instead, one has to average the ECs themselves. When we bootstrap data, we of course have to follow the same procedure to obtain correct confidence intervals. Does this explanation sufficiently clarify our bootstrapping procedure?\n\nWe hope that our responses addressed your concerns satisfactorily, perhaps allowing you to reconsider the score you have given our submission. Of course, we would be very happy to engage in further discussion.\n\n[1] Geirhos, R., Narayanappa, K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F. A., & Brendel, W. (2021). Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34, 23885-23899."}, {"review_id": "3gKO7k6LES", "reviewer": "Reviewer_HmXf", "summary": "The paper investigates error consistency (EC) and highlights several improvements in how it could be used in the Literature. \nSpecifically, the authors address the lack of confidence intervals commonly reported alongside the EC, propose a decomposition of the EC into interpretable components, and apply their framework to re-analyze prior studies. \nThrough this lens, they uncover potential statistical limitations in existing evaluations of model performance.", "questions": "Major points:\nSee Weaknesses. \n\nClarifications:\n- Fig2.b: What would the figure look like if one included a lower boundary for -EC? Would it be the same or would the level curves be different?\n- L116: On the third line of the calculations, what is $\\tilde P_2$? Is it supposed to be $\\tilde P^{(2)}$? \n- Prop1: I believe there is a missing assumption: the independence between $\\hat y^{(1)}$ and $\\hat y^{(2)}$ when one is not copying the other. This is necessary to write the joint probability distribution as on L113, with a \"dependent\" and an \"independent\" parts. Also, I believe it would be _formally_ more correct to write $\\tilde P^{(\\cdot)}$ instead of $P^{(\\cdot)}$ in the equation, as the joint is defined in terms of the underlying distributions (as the authors already do in the proof of Prop2). \n- L170: What is a \"condition\" here? Does it correspond to the intensity of corruption?\n- L187, 189, possibly elsewhere: A binomial distribution has support on $\\mathbb N$, but $r$ is defined as a binary variable (L72). I think a Bernoulli distribution would fit the context better.\n-  Although one can infer them, Section 3.3 does not clearly state the null and alternative hypotheses.\n- L 220, 221: I do not understand this sentence and its implications. What are \"prior\" and \"posterior\" in this context? Why is the assumption of a uniform prior over the accuracies reasonable? Where does the Beta come from?\n\nSpelling and notation:\n- L70: Duplicated \"and\".\n- I suggest replacing $P^{(\\cdot)}$ with $P_\\cdot$, as it would make the math more readable and less cluttered.\n- I suggest replacing $y$ with $r$ in, e.g., Prop2, as it would make it clearer that, in the context of this paper, these results are applied to the \"correctness\" of the model, rather than the actual labels.\n- L214: The symbol $\\kappa$ is already used for Cohen's $\\kappa$.\n\nOpen discussion points:\n- Figure 3 shows how the confidence intervals are very wide even for relatively large samples. In the long run, I can imagine that, as the field advances, DNNs will become better at imitating humans, thus achieving closer EC and requiring prohibitively large samples. I would be interested in knowing the authors' take on whether EC is a good metric for this kind of experiments or whether the community should move on to something different.\n- Also related to Figure 3: using the decomposition of EC in Prop2, could one \"decompose\" the confidence intervals into the $p_{copy}$ and $f$ CIs? That could help understand where the high-variance of EC comes from.", "limitations": "I recommend moving them from Section 3.2 to a dedicated subsection of Section 5.\n.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## S1 - Strong motivation\nThe paper addresses a timely and important problem. \nThe motivation is clearly explained and compelling, especially thanks to  Figure 2.b.\n\n## S2 - Novel insights\nThe authors' novel interpretation of EC (Prop2) helps understanding the metric, its meaning, and its limitations. I found particularly impactful the remark that a DNN with higher-than-human accuracy might be penalized in the evaluation (L139--141). \n\n## W1 - Existing CIs in the related work\nGeirhos (2020), in Section 2.2, already seems to provide a confidence interval for EC. The paper lacks a discussion of existing related work in this direction.\n\n## W2 - Conclusions of Section 3.2\nSection 3.2 raises an important point about the limited reliability of the EC when samples are relatively small, as it is common in the field (L194--198). \nHowever, the analysis assumes a ground truth EC of 0.5, which, as shown in Appendix D, leads to the widest confidence intervals.\nThus, the authors' critique may be over-conservative, at least when applied to cases in which the EC lies further from 0.5. \nI recommend (1) adding a link to Appendix D in the main text (and to other sections of the Appendix too, when relevant); (2) either moderate the strength of the conclusions drawn, or (preferably) expand Figure 3 to include a range of ground-truth EC values, indicating where existing studies and results fall along the EC spectrum.\n\n## W3 - Informative value of the confidence intervals in Section 4.1\nIn Section 4.1, Figure 4 reports the average EC of individual models, averaged, among others, across humans (cf. Section 3.1). \nHowever, given that humans already disagree among each other (EC$\\approx$0.42), it is unclear to me whether the reported CIs are informative.\nTo clarify, I mean that the variability introduced by human disagreement may dominate the total variance and that the reported CIs may be very small compared to it, making them less useful.\nThis effect may become even more pronounced when other factors (such as corruption and condition) are also de-aggregated. \nThe paper should at least provide de-aggregated versions of Figure 4, showing error bars for the variation of EC for human and/or other factors.", "was_revised": true, "final_justification": "I am fully satisfied by the authors' rebuttal.", "rebuttal": "We would like to thank all reviewers for their valuable feedback. We very much appreciate their assessment of our work as containing a **“strong motivation”** (HmXf), being **“well-written and easy to follow”** (eJLm), offering **“novel insights”** (HmXf) and **“provid[ing] new tools”** (1TAb). It connects to previous work (SWie, 1TAb, eJLm) and adds **“a major theoretical result”** (1TAb) which improves our understanding of the popular Error Consistency metric. We have implemented most of the reviewers’ suggestions in the newest version of the manuscript, and hope our responses will clarify all remaining points. The reviewers’ feedback has greatly improved the submission. \nDue to the rebuttal rules this year, we cannot upload a new version of the manuscript, but can only promise to make changes in the camera-ready version, should the paper be accepted. \n\nDear Reviewer,\n\nThank you for reading our work with such attention to detail. We greatly appreciate the depth of your review and the specific suggestions you made, which have improved the manuscript considerably. \n\n*Q: “Geirhos (2020), in Section 2.2, already seems to provide a confidence interval for EC. The paper lacks a discussion of existing related work in this direction”*\n\nA: Thank you for this comment. You are right, Geirhos et al. indeed provided a basic variant of confidence intervals. But these confidence intervals were computed in a suboptimal way. They were based on the expected agreement $p_{exp}$ rather than the actual accuracies of the two classifiers, $p_1$ and $p_2$. This creates an ambiguity because, for example, $(p_1 = 0.5, p_2 = 0.5)$ as well as $(p_2 = 0.5, p_1 = 0.99)$ both produce the same expected agreement of $p_{exp}$ = 50% but different confidence interval widths. Moreover, these confidence intervals were necessarily centered around 0 because they were constructed only for independent observers. Our approach now enables calculating confidence intervals by making use of the observed accuracies, and for two dependent observers. We will add a section discussing this important difference in the paper. We appreciate your suggestion, which clearly improves our paper by making the difference to prior work more explicit!\n\n*Q: “The analysis assumes a ground truth EC of 0.5, which [...] leads to the widest confidence intervals. I recommend (1) adding a link to Appendix D in the main text [...]; (2) either moderate the strength of the conclusions drawn, or (preferably) expand Figure 3 to include a range of ground-truth EC values, indicating where existing studies and results fall along the EC spectrum”*\n\nA: Thank you for this actionable suggestion. We agree that this would improve the manuscript and have implemented (1) and (2). In fact, the EC values in most existing studies hover around 0.2 to 0.4, at which point the confidence intervals are very similar to the ones depicted, only the lower bound changes a bit. Unless you insist, we would stick with depicting the largest CIs, since we feel like otherwise we would be downplaying the importance of the result: Since we want to be a voice of caution, displaying the worst case scenario seems reasonable. We are more explicit about this now, though. Does this resolve your issue?\n\n*Q: “In Section 4.1, Figure 4 reports the average EC of individual models, averaged, among others, across humans”*\n\nA: You are absolutely correct, we are aware of this issue and delighted that you noticed it. In fact, an earlier draft of the manuscript included the figure you requested in the appendix, but we chose to exclude it out of fear that it would confuse less astute reviewers. Individual observers do indeed have somewhat different values, but our main message is not affected, and this is a consequence of  the specific experiment itself, not our methodology. We have now included such a figure. However, the CIs in figure 4 are informative in the sense that if one accepts the paradigm, these are conservative CIs, and they are still very large: Every other choice of aggregation would make confidence intervals even larger.\n\nWe have numbered your clarifications 1 to 7:\n\nC1: Yes, the figure would indeed look different, as the upper and lower bounds are not symmetric. In the context of the experiments by Geirhos et al, the lower bound is not of much interest because all values are positive, but we now include such a figure in the appendix.\n\nC2: Yes, thank you for catching this typo.\n\nC3: Yes, we now make the independence assumption more explicit. Additionally, we changed the notation in the proof to show underlying rather than observed distributions. Technically, both variants are correct here because, in this proof, both distributions are identical. We nevertheless changed it in line with your suggestion, which we appreciate because this change improves clarity.\n\nC4: Yes, exactly. By “condition” we mean one intensity level within a corruption, so for 3 corruptions with 2 intensity levels each, you’d have 6 conditions.\n\nC5: You are absolutely correct, thanks for the suggestion, we call it a Bernoulli distribution now.\n\nC6: Good point, we explicitly state the Null hypothesis now.\n \nC7: Thank you for the question, we may have glossed over some details here and assumed an unreasonable familiarity with Bayesian statistics. The problem here is the following: Our observers had a certain accuracy, the uncertainty of which depends of course on the number of trials (i.e. if you get 6/10 trials right, I’m less certain that your true latent accuracy is 60% than when you got 600/1000 right). When we model the independent observers, we need to account for our uncertainty about their accuracy, so instead of assuming some fixed accuracy, we draw it from a distribution, which is derived like this: We assume a binomial observer, i.e. an observer that draws each response to a sequence of binary questions from a Bernoulli distribution with an (unknown, ground-truth) parameter $p$, so the whole of its responses were drawn from a binomial distribution. Initially, we know nothing about the observer, so our prior, capturing our knowledge about $p$, is uniform over {0,1} (which is commonly considered the most conservative prior, so is adequate here). After having observed $\\alpha$ incorrect responses and $\\beta$ correct responses, the posterior distribution $P(p | data) = \\frac{P(data | p) P(p)}{P(data)}$ is given by $Beta(\\alpha, \\beta)$ (because the Beta distribution happens to be the conjugate prior for the binomial distribution, i.e. is the closed-form solution for the posterior, see https://en.wikipedia.org/wiki/Conjugate_prior, where this is the first example). In line 221, there is actually a typo, because we accidentally inverted $\\alpha$ and $\\beta$, which we have now fixed - thank you for directing our attention here!\n\nThank you for the spelling- and notation-suggestions, we have implemented most of them. As exceptions, we still use $\\kappa$ to denote EC, because it is indeed Cohen’s $\\kappa$ (specifically, over the correct / incorrect responses). For this reason, we also kept responses $y$ instead of correctness $r$ in proposition 2 — where we now improve clarity by making this explicit thanks to your suggestion. Lastly, we also kept the superscript notation $P^{(1)}$ instead of $P_1$ to be consistent with the notation for responses, where we have a superscript j for the classifier and a subscript i for the trial, $y^{(j)}_i$. This is not our invention but has been done in other papers as well (e.g., in Gwet, 2016). Having said that, if you insist, we will make this change for the final manuscript version. \n\nRegarding your discussion points (which are great points, so we partially address them now in the discussion section): \n1. Yes, to achieve a more precise resolution with EC, we would need way more trials, which will eventually be prohibitively expensive for human observers. A possible remedy would be to design experiments cleverly, by selecting stimuli that bring out the differences efficiently. Also, EC can not only be used for comparisons between humans and DNNs, but also to compare different DNNs with each other, in which case the slow convergence is less of a concern. As multimodal models become increasingly popular and we are no longer limited to pure classification outputs, more sophisticated methods of comparison will have to be developed, but for the near future, most (M)LLM evaluations employ multiple-choice paradigms, to which EC is applicable.\n\n2. This is an excellent question and suggestion. Indeed, given that EC can be separated into two factors, copy probability and accuracy-mismatch term, the uncertainty can be similarly decomposed. We are currently working on this based on standard error propagation methods, and preliminary results support our previous recommendations (e.g., that the accuracies of the classifiers should ideally be held close to 50% to maximize the information gain about the copy probability). Although this could be an interesting addition, we fear it goes beyond the scope of the current manuscript. We would defer a thorough analysis of this uncertainty decomposition to future work."}]

Author rebuttal:

Meta-review: This paper analyzes the behavior of error consistency (EC), a widely used metric for model–human behavioral alignment, and proposes methods to quantify its uncertainty for use in designing more reliable experimental comparisons. The contributions include: (i) bootstrapped confidence intervals and significance tests for EC, (ii) a generative model interpreting EC as a copy probability scaled by an accuracy-mismatch factor that enables experiment planning, and (iii) a python package realizing these methods. The authors re-analyze major benchmarks and show that many model-to-model comparisons are not statistically reliable despite differences being claimed in prior work.

Reviewers appreciated the strong motivation, clear writing, and the interpretability provided by the decomposition of EC, which stands to impact the use of this common metric. There were some concerns raised about novelty (bootstrapping being a standard technique), about assumptions that are made (e.g., about the two classifiers relation), and about a mistake in one figure. The rebuttal did a satisfactory job of clarifying and making the case for the contribution. The authors should follow through on all revisions they promised in the rebuttal, including validation of the model against bootstrap CIs, corrected misinterpretations, and promised expanded analyses and clearer discussion of assumptions.

Overall, this is a technically solid and practically minded critical methods paper addressing a timely gap in evaluation practice. All but one reviewer recommends acceptance. For the camera-ready, the authors should include the promised validation of their theoretical model against empirical results, clarify assumptions and relation to CIs in prior work (e.g., Geirhos et al. 2020), and add the expanded analyses (pairwise tests, broader EC ranges, disaggregated human variability) to strengthen clarity and impact.
Decision: Accept (poster)