Humanly Certifying Superhuman Classifiers
Authors: Qiongkai Xu
Christian Walder
Chenchen Xu
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Evaluation theory
Oracle accuracy
Superhuman classifier

Abstract: This paper addresses a key question in current machine learning research: if we believe that a model's predictions might be better than those given by human experts, how can we (humans) verify these beliefs? In some cases, this ``superhuman'' performance is readily demonstrated; for example by defeating top-tier human players in traditional two player games. On the other hand, it can be challenging to evaluate classification models that potentially surpass human performance. Indeed, human annotations are often treated as a ground truth, which implicitly assumes the superiority of the human over any models trained on human annotations. In reality, human annotators are subjective and can make mistakes. Evaluating the performance with respect to a genuine oracle is more objective and reliable, even when querying the oracle is more expensive or sometimes impossible. In this paper, we first raise the challenge of evaluating the performance of both humans and models with respect to an oracle which is $\textit{unobserved}$. We develop a theory for estimating the accuracy compared to the oracle, using only imperfect human annotations for reference. Our analysis provides an executable recipe for detecting and certifying superhuman performance in this setting, which we believe will assist in understanding the stage of current research on classification. We validate the convergence of the bounds and the assumptions of our theory on carefully designed toy experiments with known oracles. Moreover, we demonstrate the utility of our theory by meta-analyzing large-scale natural language processing tasks, for which an oracle does not exist, and show that under our mild assumptions a number of models from recent years have already achieved superhuman performance with high probability---suggesting that our new oracle based performance evaluation metrics are overdue as an alternative to the widely used accuracy metrics that are naively based on imperfect human annotations.

Published as a conference paper at ICLR 2023
HUMANLY CERTIFYING SUPERHUMAN CLASSIFIERS
Qiongkai Xu University of Melbourne Victoria, Australia qiongkai xu@unime lb edu, au
Christian Walder* Google Brain Montreal, Canada
cwaldez0googge..com com
Chenchen Xu
Amazon
Canberra, Australia
xuchench& ama zon. com
ABSTRACT
This paper addresses a key question in current machine learning research: if we believe that model's predictions might be better than those given by human ex- perts, how can we (humans) verify these beliefs? In some cases, this "superhu- man" performance is readily demonstrated; for example by defeating top-tier hu- man players in traditional two player games. On the other hand, can be challeng- ing to evaluate classification models that potentially surpass human performance. Indeed, human annotations are often treated as ground truth, which implicitly as- sumes the superiority of the human over any models trained human annotations. In reality, human annotators are subjective and can make mistakes. Evaluating the performance with respect to genuine oracle is when querying the oracle is more expensive or sometimes impossible. In this pa- per, first raise the challenge of evaluating the performance of both humans and models with respect to an oracle which is umobserved. We develop a theory for estimating the accuracy compared the oracle, using only imperfect human anno- tations for reference. Our analysis provides an executable recipe for detecting and certifying superhuman performance in this setting, which we believe will assist understanding the stage of current research classification. We validate the con- vergence of the bounds and the assumptions of our theory on carefully designed toy experiments with known oracles. Moreover, demonstrate the utility of our theory by aueta-analyzing large-scale natural language processing tasks, for which oracle does not exist, and show that under our mild assumptions number of models from recent years have already achieved superhuman performance with high probability subabblity--uuggestinn that new oracle based performance evaluation metrics are overdue as an alternative to the widely used accuracy metrics that are naively based on imperfect human annotations.
objective and reliable, even
INTRODUCTION
Artificial Intelligence (AI) agents have begun outperform humans on remarkably challenging tasks; AlphaGo defeated top ranked Go players (Silver et al. 2016 Singh et al. 2017). and Ope- nAI's Dota2 AI has defeated human world champions of the game (Berner et al. 2019). These AI tasks may be evaluated objectively, e.g., using the total score achieved in a game and the victory against another player. However, for supervised learning tasks such as image classification and sen- timent analysis, certifying a machine learning model as superhuman is subjectively tied to human judgments rather than comparing with an oracle. We focus on paving a way towards evaluating models with potentially superhuman performance in classification.
When evaluating the performance of a classification model, we generally rely on the accuracy of the predicted labels with regard to ground truth labels, which we call the oracle accuracy. However,
Work was done while the authors were with the Australian National University and Data61 CSIRO

---

Published as a conference paper at ICLR 2023
acle labels may arguably be unobservable. For tasks such as object detection and saliency detection, the predictions are subjective to many factors of the annotators, e.g., their background and physical experts may not be able to summarize an explicit rule for
mental state. For other tasks, the prediction, such as predicting molecule toxicity and stability. Without observing oracle labels researchers often resort two heuristics, i) human predictions aggregated human annotations
effectively treated as ground truth (Wang et al. 2018 Lin et al., 2014 Waag et al. 2019) to approxi- mate the oracle, and ii) the inter-annotator aggreement is taken the best possible machine learning model performance (for an extensive survey of works that make this claim without proof, see the works cited within (Boguslav Cohen| 2017 Richie al., 2022)). This heuristic approach suffers some key disadvantages. Firstly. the quality control of human annotation is challenging (Artstein 2017 Lampert et al., 2016). Secondly, current evaluation paradigms focus on evaluating the per- formance of models, but not the oracle accuracy of humans yet we cannot claim that machine learning model is superhuman without properly estimating the human performance as compared to the oracle. Thirdly, as machine learning models exceed human performance important tasks, becomes insufficient to merely report the agreement of the model to human annotations.
In this paper, that oracle labels are unobserved (see Fig- ure D. Within this setting is provided theory for estimating the oracle accuracy on classification tasks which formalises what empirical works have hinted towards (Richie et al. 2022), that machine learn- ing classification models may outperform the humans who provide them with train- ing supervision. Our aim is not to opti- mally combine machine learning systems, but rather to estimate the oracle accu- racy of a single machine learning system by comparing it with the results obtained from multiple human annotators. Our the- ory includes i) upper bounds for the av- eraged oracle accuracy of the annotators, ii) lower bounds for the oracle accuracy of the model, and iäi) finite sample analysis for both bounds and their margin which represents the model's outperformance. Based on our theory, we propose algorithm to detect competitive models and to report confidence scores, which formally bound the probability that given model outperforms the average human annotator. Empirically, observe that some exist- ing models for sentiment classification and natural language inference (NLI) have already achieved superhuman performance with high probability.
work on the setting
Oracle
(a) Unobserved
Pll =((=)..)
P(e
(b) Observed
Annotator
Annotator
P(L: = bg)
(i)
The relationship between a) the oracle accu- racy of the annotators, IP(4 f.), and b) the agree- ment between two annotators, P(C = Lÿ). la and l are labels given by annotator and j. l, the oracle label. In our setting, part a) is unobserved (gray) and part b) observed (black).
Figure 1:
2
EVALUATION THEORY
We now present our theory for human annotators and machine learning models with oracle labels.
2.1 PROBLEM STATEMENT
We are given K labels crowd sourced from K human annotators, {4 and some labels from model M. The probability of two annotators a; and a; possess matched annotations with the other is IP(e; = E5). Denote by fx the label of the "average" human annotator which we define the label obtained by selecting one the K human annotators uniformly at random. We seek to formally compare the oracle accuracy of the average human, IP(6x = l.). with that of the machine learning model. PflM = f.). where L, is the unobserved oracle label. Denote by Eg the label obtained by aggregating (say, by voting) the K human annotators' labels. We distinguish between the oracle accuracy P(eM = E.) and the agreement with human annotations P(eM = Eg). although these two concepts have been confounded in many previous applications and benchmarks.
2

---

Published as a conference paper at ICLR 2023
2.2 AN UPPER BOUND FOR THE AVERAGE ANNOTATOR PERFORMANCE
The oracle accuracy of the average annotator lx follows the definition of the previous section, and conveniently equals the average of the oracle accuracy of each annotator, i.e.
P(c:- (.)
K Σ = P(4=
By introducing an assumption as equation 2 we may bound the above quantity. Intuitively, anno- tators are likely to be positively correlated because i) they tend to have the same correct or wrong annotations on the same easy or difficult tasks respectively, ii) they may share similar backgrounds that affect their decisions, and etc. Note that this assumption also discussed in Section 3.2 (see RQ2) where we provide supporting evidence for it on a real-world problem with known oracle la- bels.
Theorem (Average Performance Upper Bound) Assume annotators are positively correlated,
P(4: E.le, f.) 2 P(4: f.).
Then, the upper bound of averaged annotator accuracy with respect to the oracle is
K
k
P((x = 2.) < -
P(4 eg).
(3)
K2
We observe that average annittor agreement will be over-estimated by including the self com- parison terms P44 = 42), which is always equal to one when = j. but that the total overestimation to 2/2 less or equal to 1/K (K out of K2 terms), and that the influence will reduce and converge to zero limit K - To provide more practical estimation, we introduce an empirically ap- proximated upper bound u((e), In contrast, U in equation is also noted as theoretical upper bound, t(0)
Definition 1 The empirically approximated upper bound.
K K
2( A
P(4 = =E))-
(4)
K(K 1)
Lemma 2 (Convergence of (((e)) Assume that Ej=1iti P(Le =e))
K-1 N.
where Ne is the
constant number of classes. The approximated upper bound u(e) satisfies
lim
(/ 1.
(5)
K
Therefore, with large K, U(e) converges to or (())
Empirical support for the convergence of ((()) ()) are demonstrated in Figure of Section 3.2
2.3 A LOWER BOUND FOR MODEL PERFORMANCE
For our next result, we introduce another assumption as equation 6 Given two predicted labels la and l, assume that la is reasonably predictive the assumption formally stated within the following theorem. Note that this assumption is rather mild in that even random guessing satisfies it, in this case the probability of choosing the correct label is equal to any other single wrong label. Once again, this key assumption is discussed and validated on human data with known oracle labels in Section (see RQ2).
those instances that gets wrong, as per
Theorem (Performance Lower Bound) Assume that for any single incorrect label fx + lu,
Pfls Ealla # (.) Pl,e- e 1 + P.).
Then, the lower bound for the oracle accuracy of l, is
2 P(C, = 4.) < P(G, t.).
In practice, accurate to gives tighter lower bound for l, and employ the aggregated human annotations for the former (letting f, = Eg) to calculate the lower bound of the machine learning model (letting la LM), demonstrated in Section
3

---

Published as a conference paper at ICLR 2023
Connection to traditional practice of accuracy caleulation. Generally, the ground truth of benchmark corpus constructed by aggregating multiple human annotations (Wang et al. 2018 2019). For example, the averaged sentiment score is used in SST (Socher et al. 2013) and majority of votes in SNLI (Bowman et al., 2015). Then, the aggregated annotations are treated as ground truth to calculate accuracy. Under this setting, the 'traditional" accuracy score evaluated the (aggregated) human ground truth can be viewed as a special case of our lower bound.
2.4 FINITE SAMPLE ANALYSIS
The results above assume that the agreement probabilities are known; we now connect them with the finite sample case: gin) denotes the label assigned to the n-th data point accordance to for 1,2,,..... .......... N. p(N) is the empirical probability given N observations, and limn P(N), We begin with standard concentration inequality (see e.g. (Boucheron et al. 2013 $ 2.6)).
Theorem (Hoeffding's Inqquality) Ler ; N be independent random variables with variance such that (xX,, 2 [a. for all 1 n S N. Let
N
X A
X
N
7=
then, for any > 0,
2Nt
P(X - E 2 +1) S exp
(a - 8)2
2N
P(X
< - < exp
(8)
(a 3)2
Combining this with Thereom
| we obtain the following.
Theorem (Sample Average Performance Upper Bound) With Theorem
assumptions and
(L: = eg) = 1 N
(x) = m
(9)
defining the empirical agreement ratio and letting
8m exp(-2211))
(10)
With probability least 1 Sw for any tu O.
Pllx -l) tu
((N) (()(=()))
(11)
Theorem (Sample Performance Lower Bound) With Theorem define
assumptions and equation
6, = exp(-2N1i)
(12)
With probability least 1 òr. for any t
TP(N) = (b) S 41 + P(es = t.).
(13)
2.5 DETECTING AND CERTIFYING SUPERHUMAN MODELS
We propose a procedure to discover potentially superhuman models based on our theorems.
1. Calculate the upper bound of the average oracle accuracy of human annotators, Uy, with
N data samples;
Here [-] is the Everson hracket.
4

---

Published as a conference paper at ICLR 2023
2. Calculate the lower bound of the model oracle accuracy CN using aggregated human notations as the reference with N data samples;
3. Check whether the finite sample margin between the bounds CN Ly is larger than zero
4. Give proper estimation of lu and l and calculate a confidence score of P(C - U 2 0).
Generally, larger margin indicates higher confidence out-performance. To formally check con- fidence for the aforementioned margin provide the following theorem and corresponding algo- rithms.
Theorem (Confidence of Out-Performance) Assume an annotator pool with agreement statistic UN of equation 34 and agreement statistic between model and aggregated annotations CN of Ux then for all 2 0, tu 2 0 and t 2 that satisfy
equation 39
LN 4 v + uh =
(14)
with probability at least 1 - 8 ó,, the oracle accuracy of the model exceeds that of the average annotator by T
P(P(MM= = (.) -P(e = -(,) 2 r) 21 51 6
(15)
where
5n = exp (-2Nt ,6 = exp (-2Nt)).
(16)
Confidence Score Estimation. The above theorem suggests the confidence score
= 51 6az
(17)
and we need only choose the free constants 41 Lu and т. Recall equation 14
(LN t4) Vtu +42
(18)
and remove one degree of freedom parameterise in fu
( 7) = LN T v + Uy--
(19)
We are interested in IP(C 14 2 0) so we choose O, and give two choices for f, and & Algorithm (Heuristic Margin Separation, HMS). We assign half of the margin to f..
CN UN
(20)
Then, with r 0 we calculate the corresponding
LN Un
t = LN
2 + +Uy
(21)
and compute the heuristic confidence score S.
Algorithm 2 (Optimal Margin Separation, OMS).
For locally (in tw) optimal score, perform gradient ascent (Lemaréchal 2012) on S(tu), where
S(tw) 8(t.) o(ti(tu, 0)),
(22)
with tu is initialized as (Cnv - Ux)/2 before optimization
We demonstrate that aggregating predictions by voting weighted averaging are effective in im- proving our bounds. We emphasize however that the aggregated predictions need not he perfect, as we do not assume that this aggregation yields an oracle.
larger deviation, say high positive value, is of more interest to our certification as it gives a higher confidence score to the outperformance.
We set the learning le-4, and iterated 100 times.
5

---

Published as a conference paper at ICLR 2023
3 EXPERIMENTS AND DISCUSSION
Previously, we introduced new theory for analyzing the oracle accuracy of set of classifiers using observed agreements between them. In this section, we demonstrate our theory on several classifi- cation tasks, to demonstrate the utility of the theory and reliability of the associated assumptions. Our code available https: tuttps://githbe com/ http://xwccogkkaauuueeeeeee
Eval.git
3.1 EXPERIMENTAL SETUP
We first consider two classification tasks with oracle labels generated by rules. Given the oracle predictions,
able to empirically validate the assumptions for our theorems and observe the convergence of the bounds. Then, we apply our theory on two real-world classification tasks and demonstrate that some existing state-of-the-art models have potentially achieved better performance than the (averaged) performance of the human annotators in reference to the (unobserved) oracle.
Classification tasks with oracle rules. To validate the correctness of our theory, we collect datasets with observable oracle labels. We construct visual cognitive tasks, Color Classification and Shape Classification, with explicit unambiguous rules to acquire oracle labels, as follows:
Color Classification: select the most frequently occurring color of the objects in an image.
Shape Classificattonn: select the most frequently occurring shape of the objects in an image.
we vary colors (Red, Blue and Yellow) and shapes (Triangle, Square, Pentagon, Hexagon and Circle) for the tasks, respectively.
For both tasks, object size ignored. As illustrated Figure 2
(a) Color
(b) Shape
Figure 2: Example a) Color Classification and b) Shape Classification. a) includes 40 objects of three colors, Red (14), Blue (15) and Yellow (11), with Blue the most frequent color and therefore the oracle label. b) includes 37 objects of five different shapes, Triangle (9), Square (10), Pentagon (7), Hexagon (6) and Circle (5), with Square the dominant shape and oracle label.
For each task, generated 100 images and recruited 10 annotators from the Amazon Mechanical Turkf to label them. Each randomly generated example includes 20 to 40 objects. We enforce that no objects overlap more than 70% with all others, and that there is only one class with the highest count, to ensure uniqueness of the oracle label. The oracle number of the colors and shapes are recorded to generate oracle labels of the examples. Note that our 2/2 the average agreement among annotators, and so is proportional to Cohen's 's Kappa coefficient which we report in Appendix D along with additional details about the guidelines and interface presented to the annotators.
Real-World Classification Tasks. We analyze the performance human annotators and machine learning models two real-world NLP tasks, namely sentiment classification and natural language inference (NLI). use Stanford Sentiment Treebank (SST) (Socher et al. 2013) for sentiment classification. The sentiment labels are mapped into two classes (SST-2 five classes (SST-5),
httpps: / /www. mturk. com
"Samples with overall neutral
excluded in Tai
2015

---

Published as a conference paper at ICLR 2023
0.00
1L5
P:
Pile
Number Amnotators
Number of Ammotators
(a) Color Classification
(b) Shape Classification
Figure 3: Comparison of sample lower bound LN for model oracle accuracy P(eM f.). Rela-
tively strong and weak models are indicated by M and MM(w), The aggregation of one annotator based on the labels provided by the single annotator, Another comparison of sample theoretical upper bound (1) and sample empirical upper bound ( of average oracle accuracy of annotators P(lx E.).
very negative( (0, 0.2),, negative ((0.2, 0.4)), (O.4 0.6), positive ((0.6, 0.8). and very posi- tive ((0.8, 1.0). We use the Stanford Natural Language Inference (SNLI) corpus (Bowman et al., 2015) for NLI. All samples are classified by five annotators into three categories, L.e. Contradic- tion (C), Entailment (E), and Neutral (N). More details of the datasets are reported in Appendix C In the latter part of this section, we focus on the estimated upper bounds on test sets, compare them with the performance of machine learning models generally evaluated on test sets.
intend to
Machine Learning Models. For both of the classification tasks with known oracles, we treat them as detection tasks and train YOLOv3 models (Redmon & Farhadi 2018) for them. The input image resolution is 608 x 608, and we use the proposed Darknet-53 as the backhone feature extractor. For comparison, we train two models, strong model and a weak model, on 512 and 128 randomly generated examples, respectively. All models are trained for maximum of 200 epochs until con- vergence. During inference, the model detects the objects and we count each type object to obtain the prediction. We compare several representative models and their variants for real- aolldorld classifi- cation tasks, such as Recurrent Neural Networks (Chen et al.] 2018 Zhou et al.] 2015). Tree-base based Neural Networks (Mou et al. 2016 Tai al. 2015), 5) and Pre-trained Transformers (Devlin al., 2019 Radford et al. 2018 Wang et al. 2020
Sun et al. 2020).
3.2 RESULTS AND DISCUSSION
We conduct several experiments to validate the convergence of the bounds and the validity of the assumptions. We then demonstrate the utility of our theory by detecting real-world superhuman models. We organize the discussion into several research questions (RQ).
RQI: Will the bounds converge given more annotators? We first analyze the lower bounds. We demonstrate lower bounds for strong (s) and weak (w) models in Figure 3 in black and blue lines respectively. Generally, i) lower bounds Ln are always under the corresponding oracle accu- racy; ti) the lower bounds grow and tend to get closer the bounded scores given more aggregated annotators. Then, we analyze the upper bounds. We illustrate theoretical upper bound 24 and empirically approximated upper bound (452), in comparison with average oracle accuracy of anno- tators TP = (.), in Figure We observe that i) both upper bounds give higher estimation than the average oracle accuracy of annotators; ii) the margin between u4 and (0 reduce, given more annotators incorporated; iii) (4(e) generally provides a tighter bound than 2 and we will use 4 as Uy calculate confidence score in later discussion.
Are the assumptions of our theorems valid? We verify the key assumptions for the up- per bound of Theorem [ and the lower bound of Theorem 3 by computing the relevant quanti- ties Table I The assumptions within these theorems are not concerned with agreement (or otherwise) particular training examples (which could be unrealistic). but rather statements
RQ2:

---

Published as a conference paper at ICLR 2023
Table 2: The sample theoretical upper bounds and sample empirically approximated upper bounds, U and ( of the average oracle accuracy of the human annotators, and the sample lower bounds Li of some representative models the SST and SNLI tasks. Those models with Lx higher than 2 or even 2 are highlighted with f or +.
SST S-CLASS
SST 2-CLASS
SNLI 3-CLASS
Classifier
Classifier
Scvre
Classifier
Score
24
Avg. Human
0.790 1
Avg. Human
0.960
Avg. Human
0.904
Avg. Human
0.660
Avg. Human
0.939
Avg. Human
0.879
CNN-LSTM
CNN-LSTM
BiLSTM
Zhou et 2015
0.492
Zhou el. 2015
0.878
et.
0.855
Constimunncy Tree-LSTM
Constimency Troo-LSTM
Treo-CNN 2016
(Tai etal. 2015
0.510
Tai al. 2015
0.880
0.821
BERT large
BERT large
LM MM-Prtrrined Transformer
Devlin al.
2019
0.555
Devlin 2019
0.949
(Radford al. 2018
0.899
RoBERTa-Self- Explainngg
StmetRRRT
SemBERT
Sun al. 2020
0.591
Wang et al. 2020
0.971
Zhang et al. 2020
0.919t
made in aggregate over all input data points. In words, Theorem T assumes that the proba- bility that annotator predicts the oracle label must increase when we assume that that any other annotator predicts the oracle label on average, over all classifier inputs and class labels. Table 1: Validating our assumptions for both upper bound Theorem and lower bound Theorem on Color and Shape.
(iteoremmaasuune that on average, over all classifier inputs and class labels, if the majority vote by the human is incorrect w.r.t. the oracle, then the machine learn- ing model is still more likely to predict the oracle label than any other specific label that disagrees with the oracle. The two as- sumptions clearly hold in our specially de- signed experiments with real human sub- jects, although we can only perform this analysis on the tasks with known oracle labels. However, the methodology behind Tis by design rather conservative, as we sum over all incorrect labels (see col- umn 2 of Table [[]). Despite this stricter setup, our assumption still holds on both experiments.
Task
P(G f.If, F.)
P(4 VV-=60 (.)
Color
0.850
0.836
Shape
0.586
0.542
(a) Theorem
assumes P(C. 6,1e) (.) > P(C iJ
Task 6
P(fs f.le, # (()
Ex, P(6, E2 (.)
Color M(w)
1.000
0,000
Color M
1.000
0.000
Shape ((())
0.579
0.421
Shape (
0.895
0.105
(b) Theorem 3 ussumes P(4 = f.16, (.)
Exfe, (.)
Disclaimer: while the assumptions appear reasonable, we recommend where possible to obtain some oracle labels to validate the assumptions when applying our theory.
RQ3: How to identify a 'powerful', or even superhuman, classification model? We first com- pare the LN with Uy in our toy experiments, in Figure 3 Overall, showing superhuman perfor- mance is more likely given more annotators. ((( outperforms both 24 and 24 given more than and 6 annotators for color classification and shape classification, respectively. When the model marginally outperforming the humans, see weak model for color classification, may not observe
a clear superhuman performance margin. (w) and (efe) are very close given more than annotators.
For real-world classification tasks, we
Table 3: Confidence score S for the certificated models that outperform human annotators in SST-2 and SNLI Model
calculate the average annotator upper bounds given multiple annotators' labels and ii) collect model lower bounds re- ported in previous literature. Results on SST and SNLI are reported in Table pre- trained language models provide signifi- cant performance improvement on those tasks. Our theory identifies some of these
Task
S(HMS)
S(OMS)
Devlin et al. 2019
SST-2
0
0
Wang et al. 2020
SST-2
0.4730
0.6208
Radford al. 2018
SNLI
0,8482
0.9267
Zhang et al. 2020
SNLI
0.9997
0.9999
8

---

Published as a conference paper at ICLR 2023
models that exceed average human performance with high probability, by comparing Uxx or the even more restrictive U//
RQ4: How confident are the certifications? We calculate our confidence score for the identified outperforming models via Uy, Lx, .N, and using HMS and OMS, as reported in Table Generally, the confidence scores for SNLI models are higher than those of SST-2 because the former has test set is more than five times larger, while more recent and advanced models achieve higher confidence scores as they have larger margin of LN - UN.
4 RELATED WORK
Classification accuracy is a widely used measure of model performance (Han et al. 2011), although there are other options such as precision, recall, F1-score (Chowdhury 2010; Sasaki et al. 2007),
Matthews correlation coefficient (Matthews 1975, Chicco & Jurman 2020), etc.. Accuracy mea- sures the disagreement between the model outputs and some reference labels. common practice is to collect human labels to treat as the reference. However, we argue that the ideal reference is rather the (unobserved) oracle, as human predictions are imperfect. We focus on measuring the oracle accuracy for both human annotators and machine learning models, and for comparing the two.
widely accepted approach is crowd source (Kittur al. 2008 Mason Suril 2012) a dataset for testing purposes. The researchers collect a large corpus with each examples labeled by multiple annotators. Then, the aggregated annotations are treated as ground truth labels (Socher et al. 2013
Bowman et al.] 2015). This largely reduces the variance of the prediction (Nowak & Rüger 2010 Kruger et al. 2014), however, such aggregated results are still not oracle, and their difference to oracle remains unclear. In our paper, prove that the accuracy as ground truth, could be considered as a special case of the lower bound of oracle accuracy for machine learning models. On the other hand, much work considers the reliability of collected data, by providing the agreement scores between annotators (Landis & Koch 1977). Statistical measures for the reliability of the inter-annotator agreement (Gwer) 2010), such as Cohen's 's Kappa (Pontius Jr & Millones (20I)) and Fleiss' Kappa (Fleiss) 1971). are normally based on the raw agreement ratio. However, the agreement between annotators does not obviously reflect the oracle accuracy; egg. identical predictions from two annotators does not mean they are both oracles. In our paper, we prove that observed agreement between all annotators could serve as an upper bound for the average oracle accuracy of those annotators. Overall, we propose theory for comparing the oracle accuracy of human annotators and machine learning models, by connecting the aforementioned bounds.
aggregated human prediction,
The discovery that models can predict better than humans dates back at least to the seminal work (Meehl 1954), which compared ad hoc predictions based on subjective information, to those based on simple linear models with (typically small) number of relevant numeric attributes. Subsequent work found that one may even train such a model to mimic the predictions made by the experts (rather than oracle), and yet still maintain superior out of sample performance (Goldberg) 1970). The comparison of human and a algorithmic decision making remains an active topic of psychology research (Kahneman al. 2021). Despite this, much work continues to assume without formal proof that the inter-annotator agreement gives an upper bound on the achievable machine learning model performance (Boguslav & Cohen 2017 Richie et al., (2022;; the mounting empirical evidence against which is now placed on a solid theoretical footing by the present work.
5 CONCLUSIONS
In this paper. we built theory towards estimating the oracle accuracy of classifiers. Our theory covers i) the upper bounds for the average performance of human annotators, üi) lower bounds for machine learning models, and iii) confidence scores which formally capture the degree of certainty to which we may assert that a model outperforms human annotators. Our theory provides formal guarantees even within the highly practically relevant realistic setting of finite data sample and no access to an oracle to serve as the ground truth. Our experiments on synthetic classification tasks validate the plausibility of the assumptions which our theorems are built. Finally, our meta analysis of existing progress succeeded in identifying some existing state-of the--ar models have already achieved superhuman performance compared to the average human annotator.

---

Published as a conference paper at ICLR 2023
BROADER IMPACT
Our approach can identify classification models that outperform typical humans in terms of clas- sification accuracy. Such conclusions influence the understanding of the current stage of research on classification, and therefore potentially impact the strategies and policies of human-computer collaboration and interaction. The questions we may help answer include the following: When should we prefer model's 's diagnosis over that of medical professional? In courts of law, should we leave sentencing to an algorithm rather than Judge? These questions and many more like them are too important to ignore. Given recent progress in machine learning we believe the work is overdue.
LIMITATIONS
Yet we caution that estimating a model's oracle accuracy in this way is not free. Our approach requires the results from multiple annotators and preferably also the number of annotators should be higher than the number of possible classes in the target classification task. Another potential challenge in applying our analysis is that some of our assumptions may not hold under some specific tasks or settings, e.g., collusion attack by group of annotators. We recommend those who apply our theory where possible to collect a small amount of 'oracle' annotations, to validate the assumptions in this paper. Our work focus on multi-class classification, which only admits a single answer for each task. A multi-label classification task can be transformed to multiple binary classification tasks before using our theorem.
REFERENCES
Ron Artstein. Inter-annotator agreement. In Handbook of linguistic annotation, pp. 297-313. Springer, 2017.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysaw Debiak, Christy Den- nison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning, arXiv preprint arXiv:
2019.
Mayla Boguslav and Kevin Cohen. Inter-annotator agreement and the upper limit on machine per- formance: Evidence from biomedical natural language processing. Studies in health technology
and informatics, 245:298 302, 01 2017.
S. Boucheron, G. Lugosi, and P. Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence. OUP Oxford, 2013. ISBN 9780191655500. URL com wtw.:a/./ooooks.ooooooooooo
google.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, 2015.
Qian Chen, Zhen-Hua Ling, and Xiaodan Zhu. Enhancing sentence embedding with generalized pooling. In Proceedings of the 27th International Conference on Computational Linguistics, pp- 1815-1826 2018
Davide Chicco and Giuseppe Jurman. The advantages of the matthews correlation coefficient (mcc) over fl score and accuracy in binary classification evaluation. BMC genomics, 21(1):1-13, 2020.
Gobinda Chowdhury. Introduction to modern information retrieval. Facet publishing. 2010.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 4171 41866, 2019.
Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378, 1971.
10

---

Published as a conference paper at ICLR 2023
L Goldberg. Man versus model of man: A rationale, plus some evidence, for method of improving on clinical inferences. pp. 422 2222-32,,4 1970.
Mitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S. Bernstein. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Sys- fems, CHI '21, New York, NY. USA, 2021. Association for Computing Machinery. ISBN 9781450380966. doi: 10.1145//4411764.34455522 URL https: http::://dc..org/////// 1145/
Kilem Gwet. Handbook of inter-rater reliability advanced analyties. 2010.
LLC, Gaithersburg, MD,
Jiawei Han, Jian Pei, and Micheline Kamber. Data mining: concepts and techniques. 2011.
Elsevier,
Kahneman, Sibony, and CR Sunstein. Noise: Flaw Human Judgment. William Collins, 2021.
Aniket Kittur, Ed H Chi, and Bongwon Suh. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 453-456, 2008.
Justin Kruger, Ulle Endriss, Raquel Fernandez, and Ciyang Qing. Axiomatic analysis of aggrega- tion methods for collective annotation. In Proceedings of the 2014 international conference on Autonomous agents and mulli-agent systems, pp. 1185-1192, 2014.
Thomas A Lampert, André Stumpf, and Pierre Gançarski. An empirical study into annotator agree- ment, ground truth estimation, and algorithm evaluation. IEEE Transactions on Image Processing, 255))22557 2016.
J Richard Landis and Gary Koch. The measurement of observer agreement for categorical data. biometrics, pp. 159-174, 1977.
Claude Lemaréchal. Cauchy and the gradient method. Doc Math Extra, 251 251(254):10, 2012.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: Improving fairness and accu- racy by learning to defer. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018 URL htt.:httss///mm
htttt:::://proceedinggooooo httpsstttttttttttt//to/ttooootot fi 1e/09d37c00ff7b 22.. mew.reiidiinnnnnnnni.r.r..... httpppps.ccppppee/21///////
Winter Mason and Siddharth Suri. Conducting behavioral research on amazon's mechanical turk. Behavior research methods, 44(1):1-23, 2012.
Brian W Matthews. Comparison of the predicted and observed secondary structure of 14 phage lysozyme. Biochimica Biophysica Acta (BBA)-Proteis Structure, 405(2):442-451, 1975.
Paul E. Meehl. Clinical versus statistical prediction: A theoretical analysis and a review of the evidence minneapolis: University of minnesota press. [reprinted with new preface. pp. 136-141, 1954.
Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. Natural language inference by tree-based convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 130-136,11 2016.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7076-7087.
PMLR, 13-18 Jul 2020.
11

---

Published as a conference paper at ICLR 2023
Stefanie Nowak and Stefan Rüger. How reliable annotations via crowdsourcing: study about inter-annotator agreement for multi-label image annotation. In Proceedings of the international conference on Multimedia information retrieval, pp. 557-566, 2010.
Robert Gilmore Pontius Jr and Marco Millones. Death kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment. International Journal of Remote Sensing, 32 (15):4407-442) 2011.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement.
arXiv preprint
arXiv:/8044.02267,2000 2018.
Russell Richie, Sachin Grover, and Fuchiang (Rich) Tsui. Inter-annotator agreement is not the ceiling of machine learning performance: Evidence from a comprehensive of simulations. In Proceedings of the 21st Workshop on Biomedical Language Processing, pp. 275-284, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.1863///20222bionlgg 26. http:ptc///cccaatooooo.ooooo wttp:///2/2.....pp1.11112222
Yutaka Sasaki al. The truth of the f-measure. 2007. URL: https:///ww. cs. odu. edu/ mukkalcs795sum000en htwp:////.S...eoooeeeeeeee////// -60 YS-260c107. pdf [accessed 2021-05-26), 2007.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwiesser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):444-489, 2016.
Satinder Singh, Andy Okun, and Andrew Jackson. Learning play go from scratch. Nature, 550 (7676)336-337,, 337, 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cessing, pp. 1631-1642, 2013..
Zijun Sun, Chun Fan, Qinghong Han. Xiaofei Sun, Yuxian Meng, Fei Wu, and Jiwei Li. Self- explaining structures improve nlp models. 2020.
Kai Sheng Tai, Richard Socher, and Christopher Manning. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 1556-15666 2015.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: a stickier benchmark for general-purpose language un- derstanding systems. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 3266-3280, 2019.
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Jiangnan Xia, Zuyi Bao, Liwei Peng, and Luo Si. Struct- bert: Incorporating language structures into pre-training for deep language understanding. In International Conference Learning Representations, 2020.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. Semantics-aware bert for language understanding, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 9628 9635, 2020.
Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Francis Lau. c-Istm neural network for text classification. arXiv preprint arXiv:1511.08630 2015.
12

---

Published as a conference paper at ICLR 2023
A PROOF FOR THEOREMS AND LEMMAS
Proof of Theorem I (Average Performance Upper Bound)
Proof For i # j and i,j E {1, 1.. K),, we have
P(C, = Eÿ) Ple, = =bile, = (.)P(e, = (.)+
P(L: = Eīle; + (,)P(e, # e.)
PP, = E,le, = (,) = (.)
=P(L. = L.K,, = (.)P((, = =.)
PPC, = (.) = C.).
(23)
While for i = J. we have P((e = {j) = Therefore.
P(C: C₁) 2 P(l. = (.) = C.).
(24)
Then, combining equation 23 and equation 24
K
K
P(lx = 6.)²
Pll:- = (.)) Σ
P(C, = (.)
(25)
K2
j=1
K
Σ
P(C: =(p)
(26)
K
P(ex (.) S
P(4= = EG).
(27)
K
=1
Proof of Theorem
3
(Performance Lower Bound)
Proof
P(la 4)) =P(l, Lale + +L) P(6 lale P() = (.)
<P(E, L. + (.) +4+ P(ss = Lakea = ()) = =l)
=P,l, ELlea # (() (4) + P(G, L. ()) la)
=P(Eo- = -..)
(28)
Proof of CemmmaaaCCovrggeeeeee of Empirically Approximated Upper Bound)
Proof By comparing the upper bound and empirical upper bound, we have
u
K
K
PlL, K,)
U(a
K
P(l, = En)
Li
1= $
Σx Pil = (s)
1+
i=1
K
Σ Ej=1 Pl(, = Ej)
i$1
K 1
K
(29)
K
K
Σ $$1
K
P(l, = (n)
For the first factor in equation 29
K K
lim t
1
(30)
For the second factor in equation 29 as both annotators address the same task, the annotator agreement should be better than guessing uniformly random, i.e. P(ce E,) 2 1/No, where N. is
13

---

Published as a conference paper at ICLR 2023
the number of categories in the classification task. Then, using looser constraint Σ Ej=1,44) P(Li
Eg) 2 K-1
we have
No
K
$ N
0 S
P(4 = Ej)
K-1
1
As lim o0 K-T
No = O,
lim
1
=1
(31)
K-+
Di=l
P(4
(
Combining equation 30 and equation 37
we have
u
lim
(32)
K K-+00 Ute)
Therefore, the empirically approximated upper bound converges to the theoretical upper bound when K grows larger.
Proof of Theorem 5 (Sample Average Performance Upper Bound)
Proof We apply Theorem 4 with
K
K
(m)
= (m)
X,
(33)
K²
i=1 j=1
obtaining X, E 10, ie a = 0, and B = 1. Let
K
K
Un -
P(N) = ES)
(34)
K2
i=1 j=1
Our choice equation 33 of X,, implies Ufz = X U2 R( and so by equation g
(Vrx - +2y su) dw.
P
(35)
Rewrite equation as
P(ex = (+)+4) SM
(36)
which implies
(v + +2 + P(1x L.) SP (V + su)
P
(37)
Combining equation
35 with equation 37 gives the result.
Proof of Theorem (Sample Performance Lower Bound) Proof We apply Theorem f with
X, ((()) p(1))
(38)
obtaining X,, E 1], i.e. O, and 3 1. Let
LN P p(N)) (ta = =la)-
(39)
Now equation 38 implies LN = X and L = P(l. = =66) = x
(CN ti 2 L) < dr.
(40)
Recall equation 7 Ple.- = 4)) < P(oo = t.), which implies
IP (CN - t 2 Pls = l.)) P ( - tq 2 L)
(41)
Combining equation 40 with equation 47 gives the result.
14

---

Published as a conference paper at ICLR 2023
Proof of Theorem (Confidence of Out-Performance)
Proof Recall Theorem 5 and Theorem 6
V lu + Ub S P((ex = (.)) s 8n
P
P(LN t, 2 P(em (+)) S 8t.
Then, have
P P(P(MM- (.) P((x T)
=P PleM (.) - P(lx ()) 2 LN -1 tu +2h
2P P((M (.) 2 LN 4 4np((x =4) S // +22y
21 P ((P(MM 6.) 5 LN -1)) -P [F(ex (.) 2 Vtu +23
21 - 61 8n.
(42)
B AN EXAMPLE FOR THE ASSUMPTIONS
Here, we provide running example to show that both assumptions for Theorem and 3 could reasonably hold with no conflict. common example demonstrated in Table this case, all annotators did decent job (generally more correct than incorrect in all conditions). For the more challenging condition (other annotators fail), the ratio of correct performance is slightly less, see the rows in Table 4a
b E.
Ex
P(e. = [.)
0.7
La L,
0.8
0.2
Ple, =
0.3
la = 8x
0.6
0.4
P(e. Ex)
(a) Pfos Eo fx)
Table 4: An example for the assumptions.
For assumption 1, all possible inequations hold:
* 0.74 = 0.56 + 0.18 P(4. = (.) - P(s. E, lea = fx) 0.80
0.26 = 0.14 + 0.12 = P(s = (x) s P(l, = Ex lea = +x) = 0.30
0.70 = 0.56 0.14 P(C, = s P(a. = l.16 (.) = 0.56/0.74 0.757
0.30 0.18 + 0.12 I.C,. (x) S P(c. lx 144 (x) 0.12/0.26 0.462
For assumption 2, all possible inequations hold:
. 0.6 14, (x) 2 P44 = Cx 1.. = (x) = 0.4
0.538 = 0.14/0..6 = P(e, = f. = fx) 2 P(la = Ex l6s = [x) = 0.462
Note that, if b should be decent MI. model or rational annotator who works better than random guessing, ie., 0.5 = P(l, = f.le. = (x) = PPC, le = 2x) = 0.5.
Binary classification is discussed for simplicity.
15

---

Published as a conference paper at ICLR 2023
C DETAILS FOR NLP DATASETS
Table Statistics SST and SNLI: the number of test samples, number of classes, and the number of annotators for each sample. Note that annotators are sampled from large and diverse pool.
Dataset
#Test
#Class
#Annot.
SST-2
Socher al.
2013
1,821
2
SST-5
Socher el al.
2013
2,210
5
SNLI
Bowman et
2015
10,000
3
D DETAILS FOR HUMAN ANNOTATION
We crowd source the annotations via the Amazon Mechanical Turk. The annotation interfaces with instructions for color classification and shape classification are illustrated in Figure Each example is annotated by A = 10 different annotators. For quality control, we i) offer our tasks only to experienced annotators with 100 more approved HITs; automatically reject answers from annotators who have selected an invalid option 'None of the above'.
We demonstrate the inter- anter-an agreement (Cohen's Kappa, Fleiss' Kappa and Krippendorff's Alpha) of collected annotations on Color and Shape, in Table Note that Cohen's 's Kappa compares only two annotators. different pairs of annotators. The results show that our collected human annotation datasets cover the cases for both strongly (Color) and weakly (Shape) correlated human annotations.
calculate the mean of Cohen's 's Kappa scores between all K(K 1)/2
Table 6: Inter-annotator agreements on classification tasks, Color and Shape.
Task
Cohen's 's Kappa (mean)
Fleiss' Kappa
Krippendorff's alpha
Color
0.6040
0.6036
0.5819
Shape
0.2386
0,2372
0.2330
16

---

Published as a conference paper at ICLR 2023
Instructions
Shortooute
Choose the most trequent color of the ciiles.
Instructions
X
Select an option
Choose the most trequent color the
Red
ciries.
Blue
circies ignore avample, are
Yellow
two
None the Above
Circles which overiap (ind partially
viaibae) ahould count same.
More Instructions
Submit
Color Classification
Instructions
Shortcute
Choose the most frqquent the azjects.
Instractions
X
Select an option
Triangie sides)
objects.
Please count the shapes and ignore their size. For ouample, there are three dircles and two triangles, select cirdee.
Square (4 sides)
Pentagon sides)
Cojects ovariao jand are partialy count same.
Hexagon (6 sides)
Circle (round)
None of the Above
Instructions
tt
Submit
(b) Shape Classification
Figure 4: Human annotation interface for the Color Classification and Shape Classification tasks.
17


Reviews:
[{"review_id": "eYLEyaaGU6T", "reviewer": "Reviewer_ofWq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses the idea that oracle labels for classification tasks may be unobservable, and as a result, researchers often resort to two heuristics: human predictions or aggregated human annotations are effectively treated as ground truth to approximate the oracle, and the inter-annotator agreement is taken as the best possible machine learning model performance. The authors argue that this approach has several disadvantages, including the challenges of quality control for human annotation and the fact that current evaluation paradigms focus on evaluating the performance of models, but not the oracle accuracy of humans. They propose a theory for estimating the oracle accuracy on classification tasks which formalizes that machine learning classification models may outperform the humans who provide them with training supervision. The theory includes upper bounds for the averaged oracle accuracy of the annotators, lower bounds for the oracle accuracy of the model, and finite sample analysis for both bounds and their margin which represents the model’s outperformance. Based on this, they propose a method to detect competitive models and to report confidence scores, which formally bound the probability that a given model outperforms the average human annotator. Empirically, it is observed that some existing models for sentiment classification and NLI have already achieved superhuman performance with high probability.", "summary_of_the_review": "The paper is well motivated, well-written and provides a detailed exploration of an important issue around classification tasks---to estimate bounds for the oracle performance and determine whether a model has outperformed human annotators. The paper can be confusing at times but it is something that I believe can be addressed in camera ready. I would also like to see more discussion around limitations of this work. The proposed algorithm could be useful in practice, particularly in scenarios where it is difficult to obtain accurate ground truth labels.", "strength_and_weaknesses": "- This paper is well motivated and addresses an important problem in machine learning. It would be widely of interest to the community.\n- The paper is mostly well-written and provides a detailed description of the proposed theory and algorithm. \n- The toy setup along with empirical results on real world datasets make the case for the validity of the theory. The results appear to be promising, with some existing models already achieving superhuman performance with high probability.\n- The theoretical setup can be confusing at times. I recommend the authors do a pen and paper read, go line by line and ask themselves, what does this mean?\n - As an example, it's not clear why certain notation has been redefined (l_{i} in first sentence of Section 2.1 and then l_{a} for the same thing in the next sentence).\n - There are also several typos in the paper (\"inter-annotator aggreement\", \"empirical works has\", etc.)\n- One question I would encourage the authors to think about: how does this work in cases where the labels may not be as objective as in sentiment or NLI. Take hate speech classification for example. What one person may think is acceptable speech, another might find offensive. Doesn't mean that either is wrong, all it means is that their labels rely on their lived experiences. How will your method work in those cases? And if it won't, why? What are the limitations of this approach?\n\nHappy to update my score post revision.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "arP-mpeS8x", "reviewer": "Reviewer_gbkc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a theory to address the problem of evaluating the performance of both humans and\nmachine learning models with respect to an oracle which is unobserved. The theory hinges on two assumptions: 1) human annotators are positively correlated and 2) the fact that a particular annotator wrongly labels an instance does not preclude the chance of the other annotator to be predictive and be correct.  These assumptions are shown to lead to an upper bound for the averaged performance across human annotators and a lower bound for the machine learning model performance.  A finite sample analysis is then provided to derive a practical estimator for these bounds and an algorithm for evaluating whether a machine learning model is super-human. The assumptions are validated on toy experiments with known oracles and the utility of the theory is illustrated by meta-analysis in sentiment classification and natural language inference. Based on the paper's definition of what constitutes superhuman performance, it is shown that current ML models may already by superhuman in these tasks. ", "summary_of_the_review": "Overall I like the paper and it is an important contribution.  There is enough in the paper to recommend acceptance. Revision of the paper to address clarity of the claims and discussion of limitations will strengthen the paper.", "strength_and_weaknesses": "Strengths:\nAddresses a very important problem of subjectivity and randomness among human annotators and how one can verify whether a machine learning model comes close to an ideal (unknown) oracle performance.  The theory, proofs and experiments adequately illustrate the potential utility of the methodology.\n\nWeaknesses:\nMy concern about the paper has to do with the definition of what constitutes superhuman and the comparison of the ML model against the bound on average human performance.  The source of variability in annotations is complex and involves many variables - the availability and cost of getting best annotations from human experts, the context sensitivity of tasks and of human expert performance variability depending on their experience, etc. Thus, I would expect that we ought to be deriving bounds for the performance achievable for the best picked human annotator for a given context and then contrasting the ML model performance against the best subset of humans for a given instance.  I am not absolutely clear about assumption (2) discussed in my summary above, as this in some sense seems contradicting to assumption (1) that the human annotators are correlated. If the human annotation results are indeed correlated, how is it that given that one annotation result that is incorrect another human annotator will have a higher likelihood of being correct to the oracle rather than being wrong. There is a mention in the paper that random guessing will do and this is a weak assumption. Can you elaborate on this?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MjXvl5FDsz", "reviewer": "Reviewer_xVdj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work investigates whether it is possible to assess when statistical models outperform humans in crowdsourcing tasks. In this context, the main issue is that ground truth is not available and human annotations are generally used as proxies. The authors find deterministic and probability bounds for the accuracy of what they call the \"average\" human annotator. They then propose a method to find whether the model outperforms humans and show its applicability in a series of experiments. ", "summary_of_the_review": "I have raised several concerns that I hope the authors will address in their rebuttal. ", "strength_and_weaknesses": "The work is interesting and I like the spirit of the paper. However, it was difficult for me to get through it mainly because of the notation used. I describe below some of my concerns. \n\nFirst, I don't fully understand why the authors focus on finding bounds for the accuracy of the ``average'' annotator ($l_K$) rather than of some other statistics of the distribution of crowdsourced labels, e.g., its mode. In practice, $P(l_K=l_*)$ is not of much interest. It would be way more interesting to study whether the wisdom of the crowds beats the model or vice-versa. One issue is that the proposed framework seems to assume that all workers have the same ability, which is certainly not the case. Most models used in crowdsourcing do not make such an assumption, which represents a limitation of this framework (and these annotators should not be considered at all!). If the authors have some specific use cases for the model that they propose, they should describe them (decision-making with experts could be one).\n\nThe notation is fairly confusing in my opinion. For example, 2.1 first introduces $l_i$ and then reintroduces $l_a$, which in 2.1 it is presented as the label assigned by the annotator while in 2.3 as the label predicted by the model. It should be mentioned that i and j are in ${1,\\dots, K}$. It would be useful to clarify that $P(l_i =l_j)$ can vary with i,j. The theoretical upper bound $U^{(t)}$ should be defined already in (3). Hoeffding's inequality is a standard inequality, thus there is no need to report its statement. It is unclear what the sample size $N$ refers in page 4: While it should indicate the number of data points (probably tasks? as explained in page 2), in equation (9) it seems that for each data point the annotations of two fixed annotators are considered for each data point (task?). Why? \n\nIn section 2.5, bullet point 3: What is the \"margin\"? It also seems that the model performance is assessed against some ``aggregated'' version of the human annotations; one should take into account the likelihood that the majority vote or some other sort of aggregate version may not correspond to the real ground truth. \n\nOther clarifications:\n* What does \"calibrate this overestimation\" mean?\n\n* For the case where the model outperforms humans only in certain regions of the feature space, how is this handled by the proposed framework? This seems to be an important consideration. \n\n* Assumption in (6) probably needs some discussion because it seems possible that if the tasks admits several plausible answers \n\n* Confidence bands in the experiments plots should be reported. \n\n* Details on IRB approval need to be reported, including how much workers were paid in terms of hourly wage. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CwNNI0AAYBi", "reviewer": "Reviewer_F2Uj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach to certify whether a given machine learning model achieves super-human performance when the dataset labels are (possibly erroneous) human annotations and not (unobserved) ground-truth labels. The proposed approached relies on proving the following results (given $K$ human annotators and infinite data):\n\n   1. (Theorem 1) The probability that a randomly selected human annotator labels all samples correctly is bounded above by the root mean squared inter-annotator agreement.\n\n   2. (Theorem 3) The probability that the machine learning model predicts the labels of all samples correctly is bounded below by the probability that the machine learning model predicts labels that match the aggregated human-annotated labels (aggregated via majority voting, for example).\n\nNote that the lower bound in Theorem 3 is basically the machine learning model accuracy with respect to the human annotations, as is traditionally reported. The paper also proves variants of the theorems above for finite data samples.\n\nThe theorems above rely on the following assumptions:\n\n   1. Human annotators are not independent; their annotations are \"positively correlated\".\n   2. Even if the aggregated human annotation is incorrect, the machine learning model is more likely to predict the correct label than the incorrect human annotation.\n\nThe proposed approach essentially compares the upper bound for human annotators (Theorem 1) with the lower bound for the machine learning model (Theorem 3). The paper also provides a way to construct confidence intervals on the difference between these two bounds. The paper concludes with an empirical evaluation of the proposed theory.", "summary_of_the_review": "This paper considers the important problem of certifying whether an ML model achieves superhuman performance, and provides a principled approach to doing so. The proposed approach is clear and transparent in its assumptions. However, some of the notation makes it difficult to follow the paper, and the restrictiveness of the assumptions in practice is insufficiently explored. While both these drawbacks are significant, I believe they are addressable.", "strength_and_weaknesses": "This paper has several strengths.\n\nIt considers an important problem: how can we theoretically guarantee that a given model exceeds human annotation performance? More importantly, it enables qualifying claims of superhuman performance by quantifying whether the observed/reported superhuman performance is statistically significant for a given number of annotators with a specific inter-annotator agreement.\n\nThe paper approaches this problem in a principled manner, by deriving upper bounds on human performance and lower bounds on model performance (without needing access to ground truth labels). The paper also derives finite sample variants of these bounds that are practically useful, and derives confidence intervals on the difference between these two bounds. All assumptions are clearly stated.\n\nThe paper concludes with an empirical evaluation of the proposed theory, which covers several important aspects of the theory such as the validity of the assumptions, whether the bounds are valid empirically, and how they vary as the number of annotators increases.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%