Title: Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Error Consistency

Behavioral Alignment

Vision
Abstract: Benchmarking models is a key factor for the rapid progress in machine learning (ML) research. Thus, further progress depends on improving benchmarking metrics. A standard metric to measure the behavioral alignment between ML models and human observers is error consistency (EC). EC allows for more fine-grained comparisons of behavior than other metrics such as accuracy, and has been used in the influential Brain-Score benchmark to rank different DNNs by their behavioral consistency with humans.  Previously, EC values have been reported without confidence intervals. However, empirically measured EC values are typically noisy - thus, without confidence intervals, valid benchmarking conclusions are problematic. Here we improve on standard EC in two ways: First, we show how to obtain confidence intervals for EC using a bootstrapping technique, allowing us to derive significance tests for EC. Second, we propose a new computational model relating the EC between two classifiers to the implicit probability that one of them copies responses from the other. This view of EC allows us to give practical guidance to scientists regarding the number of trials required for sufficiently powerful, conclusive experiments.
Finally, we use our methodology to revisit popular NeuroAI-results. We find that while the general trend of behavioral differences between humans and machines holds up to scrutiny, many reported differences between deep vision models are statistically insignificant. Our methodology enables researchers to design adequately powered experiments that can reliably detect behavioral differences between models, providing a foundation for more rigorous benchmarking of behavioral alignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates error consistency (EC) and highlights several improvements in how it could be used in the Literature. \nSpecifically, the authors address the lack of confidence intervals commonly reported alongside the EC, propose a decomposition of the EC into interpretable components, and apply their framework to re-analyze prior studies. \nThrough this lens, they uncover potential statistical limitations in existing evaluations of model performance.\n\nSTRENGTHS AND WEAKNESSES:\n## S1 - Strong motivation\nThe paper addresses a timely and important problem. \nThe motivation is clearly explained and compelling, especially thanks to  Figure 2.b.\n\n## S2 - Novel insights\nThe authors' novel interpretation of EC (Prop2) helps understanding the metric, its meaning, and its limitations. I found particularly impactful the remark that a DNN with higher-than-human accuracy might be penalized in the evaluation (L139--141). \n\n## W1 - Existing CIs in the related work\nGeirhos (2020), in Section 2.2, already seems to provide a confidence interval for EC. The paper lacks a discussion of existing related work in this direction.\n\n## W2 - Conclusions of Section 3.2\nSection 3.2 raises an important point about the limited reliability of the EC when samples are relatively small, as it is common in the field (L194--198). \nHowever, the analysis assumes a ground truth EC of 0.5, which, as shown in Appendix D, leads to the widest confidence intervals.\nThus, the authors' critique may be over-conservative, at least when applied to cases in which the EC lies further from 0.5. \nI recommend (1) adding a link to Appendix D in the main text (and to other sections of the Appendix too, when relevant); (2) either moderate the strength of the conclusions drawn, or (preferably) expand Figure 3 to include a range of ground-truth EC values, indicating where existing studies and results fall along the EC spectrum.\n\n## W3 - Informative value of the confidence intervals in Section 4.1\nIn Section 4.1, Figure 4 reports the average EC of individual models, averaged, among others, across humans (cf. Section 3.1). \nHowever, given that humans already disagree among each other (EC$\\approx$0.42), it is unclear to me whether the reported CIs are informative.\nTo clarify, I mean that the variability introduced by human disagreement may dominate the total variance and that the reported CIs may be very small compared to it, making them less useful.\nThis effect may become even more pronounced when other factors (such as corruption and condition) are also de-aggregated. \nThe paper should at least provide de-aggregated versions of Figure 4, showing error bars for the variation of EC for human and/or other factors.\n\nQUESTIONS:\nMajor points:\nSee Weaknesses. \n\nClarifications:\n- Fig2.b: What would the figure look like if one included a lower boundary for -EC? Would it be the same or would the level curves be different?\n- L116: On the third line of the calculations, what is $\\tilde P_2$? Is it supposed to be $\\tilde P^{(2)}$? \n- Prop1: I believe there is a missing assumption: the independence between $\\hat y^{(1)}$ and $\\hat y^{(2)}$ when one is not copying the other. This is necessary to write the joint probability distribution as on L113, with a \"dependent\" and an \"independent\" parts. Also, I believe it would be _formally_ more correct to write $\\tilde P^{(\\cdot)}$ instead of $P^{(\\cdot)}$ in the equation, as the joint is defined in terms of the underlying distributions (as the authors already do in the proof of Prop2). \n- L170: What is a \"condition\" here? Does it correspond to the intensity of corruption?\n- L187, 189, possibly elsewhere: A binomial distribution has support on $\\mathbb N$, but $r$ is defined as a binary variable (L72). I think a Bernoulli distribution would fit the context better.\n-  Although one can infer them, Section 3.3 does not clearly state the null and alternative hypotheses.\n- L 220, 221: I do not understand this sentence and its implications. What are \"prior\" and \"posterior\" in this context? Why is the assumption of a uniform prior over the accuracies reasonable? Where does the Beta come from?\n\nSpelling and notation:\n- L70: Duplicated \"and\".\n- I suggest replacing $P^{(\\cdot)}$ with $P_\\cdot$, as it would make the math more readable and less cluttered.\n- I suggest replacing $y$ with $r$ in, e.g., Prop2, as it would make it clearer that, in the context of this paper, these results are applied to the \"correctness\" of the model, rather than the actual labels.\n- L214: The symbol $\\kappa$ is already used for Cohen's $\\kappa$.\n\nOpen discussion points:\n- Figure 3 shows how the confidence intervals are very wide even for relatively large samples. In the long run, I can imagine that, as the field advances, DNNs will become better at imitating humans, thus achieving closer EC and requiring prohibitively large samples. I would be interested in knowing the authors' take on whether EC is a good metric for this kind of experiments or whether the community should move on to something different.\n- Also related to Figure 3: using the decomposition of EC in Prop2, could one \"decompose\" the confidence intervals into the $p_{copy}$ and $f$ CIs? That could help understand where the high-variance of EC comes from.\n\nLIMITATIONS:\nI recommend moving them from Section 3.2 to a dedicated subsection of Section 5.\n.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nError consistency quantifies whether two classifiers make consistent decisions. Geirhos et al (2020) proposed to use Cohen’s Kappa (Cohen, 1960), a popular metric in behavioral science, to evaluate the error consistency of deep networks and humans. Previous studies focused on reporting the mean of the Cohen’s Kappa. The variability and the confidence interval associated with Cohen’s Kappa have been ignored. This paper proposed a procedure to compute the confidence interval of Cohen’s Kappa using bootstrapping, and applied it to evaluate two benchmark datasets. The paper also proposed a generative model of classifier consistency, which might allow one to perform simulations to decide on the sample sizes that would be needed in the experiments for achieving certain statistical power.\n\nSTRENGTHS AND WEAKNESSES:\n###Strengths: \n\nThe paper is well-written and is easy to follow.  \n\nThe proposed method for computing the confidence interval of Cohen’s Kappa is reasonable, and is a nice supplement to the work of Geirhos et al (2020). \n\nSome of the limitation of Cohen’s Kappa is highlighted. For example, the upper bound of Cohen’s Kappa strongly depends on the difference in the accuracy of the two classifiers. This raised some concerns on prior results.\n\n\n###Weaknesses: \n\nThe work is fairly incremental. While the ability to compute a confidence interval is useful, the proposed method based on bootstrapping is fairly standard. This by itself does not seem to be justified for a stand-alone contribution at NeurIPS.\n\n\nThe applications in Section 4 did not reveal substantial new insights. \n\nThe interpretation of the main results in Figure 1 is problematic. It was claimed that because the error bar was substantial, the gap between network v.s. human and human v.s. human should shrink. However, it could also be the opposite, that is, the gap was actually larger than what was implied in the difference in the mean of the Cohen’s Kappa. \n\n\nThe analytical results in Proposition 2 and Section 3.2 seem to be based on relatively restrictive assumptions. It is unclear whether a mixture of a copy of the response distribution of classifier 1 and another marginal distribution is sufficient to model the general response distribution of classifier 2. This needs to be clarified.\n\nQUESTIONS:\nThe paper reported results on testing whether the EC between a network and humans was substantially different from that between two independent binomial observers in Fig. 7. While this test is useful, its interpretation is also limited. Another test (potentially more useful test) would be to test if one EC is larger (or smaller) than another EC. This would allow one to make claims about which neural network exhibits behavior that is more similar to humans. Would this be possible? (Note that Figure, right panel, seems to be imply that this test is already done between the different network models, but how that was done was not described in Section 3.3.)\n\nLine 135-144 mentioned that it would be important to ensure that human responses have uniform marginal distributions. Practically, what would be an effective procedure for achieve this?\n\nCan the authors please explain in more detail how the bootstrapping procedure in Section 3.1 was applied in oder to take into account that each human subject likely exhibit different behaviors and thus might have Cohen’s Kappa to networks?\n\nLIMITATIONS:\nOne of the limitations of the work was acknowledged, i.e., a lack of analytical solutions.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a study of error consistency (EC), a standard metric to measure the behavior alignment between ML models and human observers. \n\nA major theoretical result is a new computational model, which frames EC as the probability that one classifier copies the other, multiplied by a factor originating from the accuracy mismatch. This result serves two purposes. \n\nFirst, it provides an alternative interpretation of EC, offering a clearer picture that it can be decomposed into the two factors of the copying probability and mismatching accuracy. \n\nSecond and perhaps more importantly, it offers a means of computing confidence interval for EC, a missing component in previous studies that only provides a point estimation and hence may not be reliable. More importantly, while bootstrapping can be trivially applied for estimating confidence intervals, the technique from their Generative computational model can be useful for planning future experiments, as it helps to determine the number of trials needed to obtain statistically reliable results. \n\nThe authors applied their new methodology to re-analyze the results from major benchmarks, including Model-VS-Human and Brain-Score.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper provides new insights into the EC metric with the decomposition into the copy probability and the accuracy mismatch. This can be important for understanding what is truly being measured by this metric. \n- The paper provides new tools for computing confidence intervals for EC, a missing piece in previous works.\n- The paper revisited previous experiments using their algorithm, and revealed new insights (e.g., a larger number of trials is essential to obtain statistically significant results).\n\nWeaknesses:\n\nI don't have any particular concerns. One suggestion is to put the algorithms in Sec 3.1 and Sec 3.2 in an algorithm box to improve on clarity.\n\nOverall, the paper has a nice flow that started with an in-depth analysis of the metric itself, revealing insights into what it is truly measuring; then went into the key objective of  calculating confidence intervals, which makes use of the aforementioned analysis. While I don't have experience in this area (hence cannot comment on related work), the paper does appear to offer a quite solid study of EC, that can benefit future studies in using it as a tool for measuring machine vs human consistency and beyond.\n\nQUESTIONS:\nI don't have any questions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue of quantifying uncertainty in Error Consistency (EC), which is a standard metric used to measure the behavioral alignment between ML models and human behaviors. The authors improve the standard EC in two ways:\n- They demonstrate how to obtain confidence intervals for EC using a bootstrapping technique, and derive significance tests for EC.\n- They propose a new generative model that simulates the experiment and estimates the resulting confidence intervals. This model can be used to provide practical guidance on the number of trials required for experiments.\n\nThe paper applies this methodology to existing experimental results. The authors emphasize that their method enables researchers to design experiments that can reliably detect behavioral differences and fosters more rigorous benchmarking of behavioral alignment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is clearly motivated: the measure of EC is inherently noisy, so quantifying uncertainty is necessary when analyzing the results.\n- The discussion with theoretical issues with EC offer a solid theoretical ground for the paper.\n- The solution (bootstrapping) offered by the paper is concrete and pragmatic for calculating confidence intervals for empirical EC values. The proposed generative model can be useful for practitioners to determine the sample size of experiment.\n- The re-analysis of existing experiments (Model-VS-Human and Brain-Score) provides empirical evidence for the issues of uncertainty and demonstrates the utility of their methodology.\n\nWeaknesses:\n- The assumptions of bootstrapping are not discussed, such as the sample need to be representative, which can be questionable for some scenarios of error consistency between human and ML classifiers. For example, when the sample of human is too small and cannot represent the population of interest.\n- The paper simply applies bootstrapping to EC, leaving the choice of parameters in bootstrapping that could impact the results dramatically. \n- The motivation for the definition of EC is not enough. Why is EC modeled with $p_{copy}$? Is it the only choice?\n\nQUESTIONS:\n- What are the takeaways we can get from analyzing with the confidence intervals but cannot with point estimates?\n- It would be better if the authors can offer more discussion over the definition of EC and expand the results to other metrics besides Cohen's $\\kappa$.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-07-11"}