Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an approach to certify whether a given machine learning model achieves super-human performance when the dataset labels are (possibly erroneous) human annotations and not (unobserved) ground-truth labels. The proposed approached relies on proving the following results (given $K$ human annotators and infinite data):

   1. (Theorem 1) The probability that a randomly selected human annotator labels all samples correctly is bounded above by the root mean squared inter-annotator agreement.

   2. (Theorem 3) The probability that the machine learning model predicts the labels of all samples correctly is bounded below by the probability that the machine learning model predicts labels that match the aggregated human-annotated labels (aggregated via majority voting, for example).

Note that the lower bound in Theorem 3 is basically the machine learning model accuracy with respect to the human annotations, as is traditionally reported. The paper also proves variants of the theorems above for finite data samples.

The theorems above rely on the following assumptions:

   1. Human annotators are not independent; their annotations are "positively correlated".
   2. Even if the aggregated human annotation is incorrect, the machine learning model is more likely to predict the correct label than the incorrect human annotation.

The proposed approach essentially compares the upper bound for human annotators (Theorem 1) with the lower bound for the machine learning model (Theorem 3). The paper also provides a way to construct confidence intervals on the difference between these two bounds. The paper concludes with an empirical evaluation of the proposed theory.
strength_and_weaknesses: This paper has several strengths.

It considers an important problem: how can we theoretically guarantee that a given model exceeds human annotation performance? More importantly, it enables qualifying claims of superhuman performance by quantifying whether the observed/reported superhuman performance is statistically significant for a given number of annotators with a specific inter-annotator agreement.

The paper approaches this problem in a principled manner, by deriving upper bounds on human performance and lower bounds on model performance (without needing access to ground truth labels). The paper also derives finite sample variants of these bounds that are practically useful, and derives confidence intervals on the difference between these two bounds. All assumptions are clearly stated.

The paper concludes with an empirical evaluation of the proposed theory, which covers several important aspects of the theory such as the validity of the assumptions, whether the bounds are valid empirically, and how they vary as the number of annotators increases.
clarity,_quality,_novelty_and_reproducibility: I have a few concerns about the clarity and quality of this paper, enumerated below.

**1. Unclear random vs. deterministic quantities in the problem statement**

I think the distinction between what is random and what is deterministic in the problem statement could be clearer.

Since $\mathbb{P}(l_i = l_*)$ is defined as "the ratio of matched labels" in Section 2.1, I believe that the dataset of $N$ points is fixed and not a random sample. Hence, $l_*$ and each $l_i$ is deterministic, and $\mathbb{P}(l_i = l_*)$ is a deterministic ratio. $l_\mathcal{K}$, in contrast, is random. Hence, $\mathbb{P}(l_\mathcal{K} = l_*)$ is random. The usage of $\mathbb{P}$ for both random and deterministic quantities makes following the problem statement a bit difficult.

Later in Theorem 1, $\mathbb{P}(l_i = l_j)$ is treated like a probabilistic quantity, which suggests that $\mathbb{P}(l_i = l_j)$ is the probability that $i$ and $j$ agree on all $N$ labels for a randomly sampled dataset of $N$ points. However, this contradicts Section 2.1 which defines $\mathbb{P}(l_i = l_*)$ as "the ratio of matched labels".

Finally in Theorem 5, $\mathbb{P}^{(N)}(l_i = l_j)$ is defined as the empirical fraction of $N$ data points where $i$ and $j$ agree on the label, which suggests that $\mathbb{P}(l_i = l_*)$ is the "the ratio of matched labels" assuming $N \rightarrow \infty$.

It would help if the notation made the appropriate interpretation of each probability unambiguous.

**2. Unclear meaning of "$\mathbb{P}(l_i = l_j)$ is overestimated as 1"**

Based on Section 2.1, $l_i$ is given for $i=1,\dots,K$. Hence, $\mathbb{P}(l_i = l_j) = 1$ when $i=j$, and $\mathbb{P}(l_i = l_j)$ is a deterministic quantity (the fraction of labels over $N$ data points on which humans $i$ and $j$ agree).

What does it mean to "estimate" or "overestimate" $\mathbb{P}(l_i = l_j)$?

**3. Clarity and restrictiveness of the assumption in Lemma 2**

Lemma 2 assumes that $\mathbb{P}(l_i = l_j) \geq 1/N_c$. Should this be for every pair of humans $i$ and $j$?

I am also unclear on how to interpret this assumption. One interpretation is, assuming the dataset is a random sample, that $\mathbb{P}(l_i = l_j$ is the probability that $i$ and $j$ agree on all labels in the dataset. This will likely be pretty low, so the assumption is unlikely to hold in practice (given that $N_c$ is usually not very large). Another interpretation is that $\mathbb{P}(l_i = l_j)$ is the fraction of labels on with $i$ and $j$ agree on a fixed dataset, in which case this assumption is not restrictive (annotator agreement rates are typically upwards of 70% in practice).

**4. How are the bounds in Figure 3 calculated for the case of just one annotator?"

The formula for the lower bound does not make this clear.

**5. Possible conflict between the assumptions of Theorem 1 and 3**

Theorem 1 assumes that the human annotators are positively correlated. Theorem 3 assumes that even when the aggregated human annotation is incorrect, it is possible for the machine learning model to be predict the label correctly. Given that the machine learning model is trained to mimic human annotations, and that these annotations are correlated, it seems that these 2 assumptions are in conflict.

Considering an extreme case (eg. humans are completely incorrect and strongly correlated, completely incorrect and weakly correlated, etc.) may help illuminate these assumptions better, and help evaluate whether they are applicable to a specific setting in practice. I believe (but am not sure), taken together, the proposed theorems rely on the human annotations being reasonably accurate and reasonably correlated (but not too correlated).  

**6. Minor Grammatical/Spelling Errors:**

"Within this setting provide..."
"along some labels"
"for all of a pairs"
"affects their decisions, and etc."
"we introduce another assumption equation"
"in that as even"
summary_of_the_review: This paper considers the important problem of certifying whether an ML model achieves superhuman performance, and provides a principled approach to doing so. The proposed approach is clear and transparent in its assumptions. However, some of the notation makes it difficult to follow the paper, and the restrictiveness of the assumptions in practice is insufficiently explored. While both these drawbacks are significant, I believe they are addressable.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a theory to address the problem of evaluating the performance of both humans and
machine learning models with respect to an oracle which is unobserved. The theory hinges on two assumptions: 1) human annotators are positively correlated and 2) the fact that a particular annotator wrongly labels an instance does not preclude the chance of the other annotator to be predictive and be correct.  These assumptions are shown to lead to an upper bound for the averaged performance across human annotators and a lower bound for the machine learning model performance.  A finite sample analysis is then provided to derive a practical estimator for these bounds and an algorithm for evaluating whether a machine learning model is super-human. The assumptions are validated on toy experiments with known oracles and the utility of the theory is illustrated by meta-analysis in sentiment classification and natural language inference. Based on the paper's definition of what constitutes superhuman performance, it is shown that current ML models may already by superhuman in these tasks. 
strength_and_weaknesses: Strengths:
Addresses a very important problem of subjectivity and randomness among human annotators and how one can verify whether a machine learning model comes close to an ideal (unknown) oracle performance.  The theory, proofs and experiments adequately illustrate the potential utility of the methodology.

Weaknesses:
My concern about the paper has to do with the definition of what constitutes superhuman and the comparison of the ML model against the bound on average human performance.  The source of variability in annotations is complex and involves many variables - the availability and cost of getting best annotations from human experts, the context sensitivity of tasks and of human expert performance variability depending on their experience, etc. Thus, I would expect that we ought to be deriving bounds for the performance achievable for the best picked human annotator for a given context and then contrasting the ML model performance against the best subset of humans for a given instance.  I am not absolutely clear about assumption (2) discussed in my summary above, as this in some sense seems contradicting to assumption (1) that the human annotators are correlated. If the human annotation results are indeed correlated, how is it that given that one annotation result that is incorrect another human annotator will have a higher likelihood of being correct to the oracle rather than being wrong. There is a mention in the paper that random guessing will do and this is a weak assumption. Can you elaborate on this?
clarity,_quality,_novelty_and_reproducibility: The paper is a bit difficult to read and requires multiple passes. While the main message and methodology is clear, the clarity is hindered by the present structure of the paper.   I would have expected a more structured discussion of the nature of human annotation, central assumptions about the population of annotators and a definition of what it means to be superhuman and connect this to the specific derivation of the bounds.  The disclaimer about the application of the methodology and the need for careful validation could be articulated early on so that the reader is better positioned to clearly view what is claimed.

The paper is original and is of good quality. I like the theme and the overall approach.  Grounding the claims with clarity will help strengthen the paper.  There is no mention in the paper about providing the sources for reproducing the experiments.
summary_of_the_review: Overall I like the paper and it is an important contribution.  There is enough in the paper to recommend acceptance. Revision of the paper to address clarity of the claims and discussion of limitations will strengthen the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper discusses the idea that oracle labels for classification tasks may be unobservable, and as a result, researchers often resort to two heuristics: human predictions or aggregated human annotations are effectively treated as ground truth to approximate the oracle, and the inter-annotator agreement is taken as the best possible machine learning model performance. The authors argue that this approach has several disadvantages, including the challenges of quality control for human annotation and the fact that current evaluation paradigms focus on evaluating the performance of models, but not the oracle accuracy of humans. They propose a theory for estimating the oracle accuracy on classification tasks which formalizes that machine learning classification models may outperform the humans who provide them with training supervision. The theory includes upper bounds for the averaged oracle accuracy of the annotators, lower bounds for the oracle accuracy of the model, and finite sample analysis for both bounds and their margin which represents the model’s outperformance. Based on this, they propose a method to detect competitive models and to report confidence scores, which formally bound the probability that a given model outperforms the average human annotator. Empirically, it is observed that some existing models for sentiment classification and NLI have already achieved superhuman performance with high probability.
strength_and_weaknesses: - This paper is well motivated and addresses an important problem in machine learning. It would be widely of interest to the community.
- The paper is mostly well-written and provides a detailed description of the proposed theory and algorithm. 
- The toy setup along with empirical results on real world datasets make the case for the validity of the theory. The results appear to be promising, with some existing models already achieving superhuman performance with high probability.
- The theoretical setup can be confusing at times. I recommend the authors do a pen and paper read, go line by line and ask themselves, what does this mean?
 - As an example, it's not clear why certain notation has been redefined (l_{i} in first sentence of Section 2.1 and then l_{a} for the same thing in the next sentence).
 - There are also several typos in the paper ("inter-annotator aggreement", "empirical works has", etc.)
- One question I would encourage the authors to think about: how does this work in cases where the labels may not be as objective as in sentiment or NLI. Take hate speech classification for example. What one person may think is acceptable speech, another might find offensive. Doesn't mean that either is wrong, all it means is that their labels rely on their lived experiences. How will your method work in those cases? And if it won't, why? What are the limitations of this approach?

Happy to update my score post revision.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written. The experiments are clearly presented and easy to understand but the theory can be confusing with changing notations (and some not well defined). The paper offers a new perspective on super-human performance and its novelty lies in the same. Experiments should be reproducible without a ton of effort, though I haven't tried reproducing them (and the authors haven't submitted their code as a supplementary attachment).
summary_of_the_review: The paper is well motivated, well-written and provides a detailed exploration of an important issue around classification tasks---to estimate bounds for the oracle performance and determine whether a model has outperformed human annotators. The paper can be confusing at times but it is something that I believe can be addressed in camera ready. I would also like to see more discussion around limitations of this work. The proposed algorithm could be useful in practice, particularly in scenarios where it is difficult to obtain accurate ground truth labels.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work investigates whether it is possible to assess when statistical models outperform humans in crowdsourcing tasks. In this context, the main issue is that ground truth is not available and human annotations are generally used as proxies. The authors find deterministic and probability bounds for the accuracy of what they call the "average" human annotator. They then propose a method to find whether the model outperforms humans and show its applicability in a series of experiments. 
strength_and_weaknesses: The work is interesting and I like the spirit of the paper. However, it was difficult for me to get through it mainly because of the notation used. I describe below some of my concerns. 

First, I don't fully understand why the authors focus on finding bounds for the accuracy of the ``average'' annotator ($l_K$) rather than of some other statistics of the distribution of crowdsourced labels, e.g., its mode. In practice, $P(l_K=l_*)$ is not of much interest. It would be way more interesting to study whether the wisdom of the crowds beats the model or vice-versa. One issue is that the proposed framework seems to assume that all workers have the same ability, which is certainly not the case. Most models used in crowdsourcing do not make such an assumption, which represents a limitation of this framework (and these annotators should not be considered at all!). If the authors have some specific use cases for the model that they propose, they should describe them (decision-making with experts could be one).

The notation is fairly confusing in my opinion. For example, 2.1 first introduces $l_i$ and then reintroduces $l_a$, which in 2.1 it is presented as the label assigned by the annotator while in 2.3 as the label predicted by the model. It should be mentioned that i and j are in ${1,\dots, K}$. It would be useful to clarify that $P(l_i =l_j)$ can vary with i,j. The theoretical upper bound $U^{(t)}$ should be defined already in (3). Hoeffding's inequality is a standard inequality, thus there is no need to report its statement. It is unclear what the sample size $N$ refers in page 4: While it should indicate the number of data points (probably tasks? as explained in page 2), in equation (9) it seems that for each data point the annotations of two fixed annotators are considered for each data point (task?). Why? 

In section 2.5, bullet point 3: What is the "margin"? It also seems that the model performance is assessed against some ``aggregated'' version of the human annotations; one should take into account the likelihood that the majority vote or some other sort of aggregate version may not correspond to the real ground truth. 

Other clarifications:
* What does "calibrate this overestimation" mean?

* For the case where the model outperforms humans only in certain regions of the feature space, how is this handled by the proposed framework? This seems to be an important consideration. 

* Assumption in (6) probably needs some discussion because it seems possible that if the tasks admits several plausible answers 

* Confidence bands in the experiments plots should be reported. 

* Details on IRB approval need to be reported, including how much workers were paid in terms of hourly wage. 

clarity,_quality,_novelty_and_reproducibility: The paper brings an element of novelty, but the writing is not clear. Many clarifications are needed. In addition, it is not clear how the work builds upon prior literature that does not assume that all annotators are equally skilled. More explanations behind this emphasis on whether the model outperforms the human "average" annotator are needed. 
summary_of_the_review: I have raised several concerns that I hope the authors will address in their rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach to certify whether a given machine learning model achieves super-human performance when the dataset labels are (possibly erroneous) human annotations and not (unobserved) ground-truth labels. A majority of reviewers are in support of accepting the paper. They do provide several points for improvement that the authors should take into account when preparing the final version of the paper.
justification_for_why_not_higher_score: The paper still has some points for improvement especially regarding clarity/presentation, however, a majority of the reviewers agree that the paper addresses an important, understudied problem in a reasonable way.
justification_for_why_not_lower_score: The paper address an important, understudied problem in a reasonable way.
Decision: Accept: notable-top-25%