Title: DISCREPANCY RATIO: EVALUATING MODEL PERFOR-MANCE WHEN EVEN EXPERTS DISAGREE ON THE TRUTH

Abstract: In most machine learning tasks unambiguous ground truth labels can easily be acquired. However, this luxury is often not afforded to many high-stakes, realworld scenarios such as medical image interpretation, where even expert human annotators typically exhibit very high levels of disagreement with one another. While prior works have focused on overcoming noisy labels during training, the question of how to evaluate models when annotators disagree about ground truth has remained largely unexplored. To address this, we propose the discrepancy ratio: a novel, task-independent and principled framework for validating machine learning models in the presence of high label noise. Conceptually, our approach evaluates a model by comparing its predictions to those of human annotators, taking into account the degree to which annotators disagree with one another. While our approach is entirely general, we show that in the special case of binary classification, our proposed metric can be evaluated in terms of simple, closedform expressions that depend only on aggregate statistics of the labels and not on any individual label. Finally, we demonstrate how this framework can be used effectively to validate machine learning models using two real-world tasks from medical imaging. The discrepancy ratio metric reveals what conventional metrics do not: that our models not only vastly exceed the average human performance, but even exceed the performance of the best human experts in our datasets.

1. INTRODUCTION:

The canonical supervised machine learning paradigm assumes the presence of both inputs and their corresponding, unambiguous outputs. In practice, the vast majority of datasets exhibit some degree of label noise. Acknowledgement of this fact has resulted in the development of algorithms tailored to training models in the presence of noisy ground truth labels (Sukhbaatar et al., 2015; Frenay & Kaban, 2014; Zhu & Wu, 2004) . These approaches include cleansing noisy labels before training begins (Hodge & Austin, 2004) , making the training process robust to noise (Patrini et al., 2017; Vahdat, 2017; Rolnick et al., 2017) and actively modeling label noise (Sukhbaatar et al., 2015; Joseph et al., 1995) . These methods almost exclusively address only the problem of noisy labels during training and tend to assume that at least some noise-free ground truth is available at test time in order to properly evaluate different approaches. Indeed, without an unambiguous yard-stick, how can competing schemes possibly be evaluated in a standardized manner? There exist many domains, such as medical imaging (Rosenkrantz et al., 2013; Lazarus et al., 2006) and machine translation (Wang et al., 2018) , in which access to unambiguous ground truth, even at evaluation time, is not practically achievable. In these domains not only do annotators disagree with each other, but they often even disagree with themselves (Gulshan et al., 2016; Hansegard et al., 2009) . This, however, does not imply that the annotators are not trustworthy, but rather that these tasks are difficult even for domain experts. Datasets in these regimes are, in practice, quite different from those typically used in machine learning research (which are usually geared towards content that can be easily curated by lay people). As a consequence, studies utilizing public data often resort to simulating label noise. Indeed, a theoretical understanding of evaluation in the presence of ambiguous test data has remained largely unexplored.

In this work, we propose a new framework for evaluating the performance of machine learning models in the presence of high label noise. Our main contributions are as follows:

1. We propose a novel, task-independent metric, the discrepancy ratio, that allows us to a) quantify model performance, taking into account how much humans disagree with each other, and b) set a precise threshold at which the model performance can be said to exceed the average human performance.

2. We perform a detailed analysis of the discrepancy ratio for the special case of binary classification and demonstrate that it can be evaluated in terms of simple, closed-form expressions that depend only on aggregate statistics (mean and variance) of the annotations and not on any individual annotation.

3. We demonstrate the effectiveness of our approach on models that we trained on two important medical imaging tasks with real-world clinical data and show that it reveals that our models not only significantly exceed the average human performance, but even beat the best human experts in our datasets.

2. RELATED WORK:

There are several well-known strategies for dealing with ambiguous or noisy evaluation labels. The most popular approach is to simply use a single annotator (Jafari et al., 2019; Rajchl et al., 2017; Cobzas et al., 2007; Roth et al., 2018) . An approach taken by many popular, large-scale datasets, such as ImageNet (Deng et al., 2009) , Places (Zhou et al., 2014) , COCO (Lin et al., 2014) and Open Images V4 (Kuznetsova et al., 2018) , is to take the average or majority vote. These efforts typically employ crowdsourcing in some fashion, since the labeling of this data does not require expert-level knowledge. While majority vote reduces label noise, in domains that require specialized knowledge it is often not practical to get enough experts that the noise is removed to the point where the annotations can be assumed to be "correct" (in the sense that MNIST annotations are assumed to be correct). A related approach obtains a ground-truth dataset by enforcing a resolution criterion on labels where annotators disagree. This criterion can, for example, involve a live session where annotators deliberate and come to a consensus or an adjudication process where a third party (usually a more senior, expert annotator) is tasked with resolving disagreements (Krause et al., 2018; Hannun et al., 2019; Rajpurkar et al., 2018b) . This approach may be effective at removing erroneous labels (reducing variance) at the expense of annotator independence (increasing bias).

The techniques outlined above fall short in providing a unified method that can both a) be compared across different studies/tasks and b) answer the essential question of how the model performance relates to an average human annotator (i.e. how a prototypical human currently performs at a task).

In developing high-stakes, real-world machine learning systems, the goal of evaluating a model is often to determine whether performance is safely substitutable for individual human behavior rather than comparing to an idealized oracle.

Several studies have compared model performance with inter-annotator variability. Some have visualized model performance and inter-annotator variability (e.g. via ROC curves (Krause et al., 2018; Gulshan et al., 2016) or Bland-Altman plots (Hansegard et al., 2009; Altman & Bland, 1983 )) without providing an explicit metric. Several works have constructed a metric by comparing the model to the average annotator performance, though each has done so in a completely different way (Wei et al., 2019; Hannun et al., 2019; Rajpurkar et al., 2018a; b; 2017) . These metrics, however, generally depend on the number of annotators used in the study and/or require that all annotators label the entire dataset (see Section 4 and Appendix C for more details). Moreover, the metrics are task-specific and cannot be interpreted outside the context of a particular study.

3. THE DISCREPANCY RATIO METRIC:

In this section, we define the discrepancy ratio and show that it has numerous desirable properties that overcome the limitations of prior art detailed in Section 2.

3.1. DEFINITIONS:

Definition 1. For any two arbitrary sets of labels Y = {y 1 , ...y n } and Y = {y 1 , ...y n }, the mean pairwise deviation (MPD) is defined by:

EQUATION

where δ can be chosen to correspond to any measure of agreement appropriate to the task in question (e.g. squared error, absolute error, Jaccard Index (Jaccard, 1912 ), Cohen's kappa (Cohen, 1960) ).

Consider a dataset of N samples where we have a prediction m i , for each sample i, by the model we are trying to evaluate. Further assume that each sample is labeled by at least two annotators and that each annotator may label the same sample multiple times. Formally, let Y i j = {y i jk } be the set of annotations provided by annotator j on sample i where k indexes the annotation number. Finally, let A i indicate the number of annotators that have annotated sample i.

Definition 2. We define the annotator discrepancy, which quantifies the average annotator disagreement, to be the average of the MPD over all annotator pairs and all samples in the dataset:

EQUATION

Without access to unambiguous ground-truth labels, a natural way to define human performance is by comparing an annotator's labels to those provided by other humans on the same samples. Consequently, we use the annotator discrepancy as a measure of average human performance.

Definition 3. Similarly, the model discrepancy is defined to be

EQUATION

Definition 4. We define the discrepancy ratio to be the ratio of the model and annotator discrepancies:

EQUATION

Intuitively, this metric is the ratio of the average model-annotator disagreement and the annotatorannotator disagreement. For example, a discrepancy ratio of 0.8 would mean that the model is 20% closer to the annotators than they are to each other. When ∆ is above (below) 1, the model performance can be said to be worse (better) than the average human performance. Since the measure of agreement δ is arbitrary, the discrepancy ratio can be thought of as a "meta-metric" that reports how the model performs (relative to humans) with respect to that particular measure of agreement.

The dependence of the discrepancy ratio on the choice of δ is further explored in Section 4.2.2.

3.2. PROPERTIES:

The discrepancy ratio metric, as formulated above, has several desirable properties:

Dimensionless: Dimensionless quantity whose meaning is invariant across different task categories (regression, classification, segmentation etc.) and whose value is interpretable and highly intuitive.

Minimal dataset coverage requirements: The discrepancy ratio does not require that all annotators label all samples (see Fig. 2 and Appendix C), which is often not practical in real-world situations,

Sample Index Model Annotator 1 Annotator 2 ... Annotator A 1 0 0, 1 0, 0 ... 1, 0, 1 2 1 - 1, 1 ... 0 ... ... ... ... ... ... N 0 1 1, 1, 1 ... 1, 1, 1, 1, 1

Table 1 : An Example Dataset for a Binary Classification Task.

but only that each sample is labeled by at least two annotators (two annotators are needed since it is meaningless to measure the annotator disagreement with only a single annotator).

Insensitive to number of labels per-sample: The discrepancy ratio is expressed in terms of averages over all model/annotation pairs and therefore does not depend on how many annotators are used (see Fig. 2 ), nor how many times each annotator labels each image. This property makes it convenient to compare the discrepancy ratio across studies.

Since the discrepancy ratio is formulated as a relative error that the model makes over that of the annotators, it can be naturally incorporated as a test statistic in a non-inferiority framework (Ahn et al., 2013) , often used in medical research. The goal of non-inferiority testing is to show that a new method is not statistically worse than an existing standard, up to some chosen equivalence bound. For the discrepancy ratio, we can choose the equivalence bound to correspond to an acceptable fraction of the error that the model is allowed to make on top of the one that annotators already make (e.g. setting the equivalence bound to be 0.1 can be interpreted as saying that the model can make an error no more than 10% larger than the ones humans typically make). Additional properties of the discrepancy ratio, as well as comparisons to alternative formulations, are discussed in detail in Appendix C and E.

3.3. AN ILLUSTRATIVE EXAMPLE:

In this section, we compute expressions for the model and annotator discrepancies on a generic binary classification task using the squared error as an agreement measure. This calculation will serve to build intuition for these metrics and can be used as a blueprint for evaluating them on other tasks.

Consider a dataset (see Table 1 ) containing N samples, where each sample is given a binary score by a model and a set of A annotators (an annotator may label an image more than once, or not at all). We assume all samples are labeled by at least two annotators. In the analysis below we adopt the convention that y i jk denotes the kth label provided by the jth annotator on the ith sample, and removing an index denotes averaging over the values of that index. For example, y i j is the average of the labels provided by the jth annotator on the ith sample and y is the average annotation over all labels, annotators and samples. The order of the averages should be clear from the context.

As shown in Appendix B, given two sets of binary labels Y = {y 1 , y 2 , ..., y n } and Y = {y 1 , y 2 , ..., y n } the MPD is given by the expression y + y -2yy , where y = 1 n n k=1 y k . Using this result and Eq. 2, the annotator discrepancy for a single sample is then given by (Appendix B):

EQUATION

)

where σ 2 y i is the variance of the labels y i and A i is the number of annotators that have labeled sample i. For the important special case where each annotator provided no more than a single label per image, this expression further simplifies to

EQUATION

where we see that annotator discrepancy is simply proportional to the variance of the labels. Similarly we can write the single-sample model discrepancy as

EQUATION

Averaging this result over all samples, the model discrepancy becomes

EQUATION

where the covariance is taken over the samples i and m and y correspond to the model predictions and annotator labels averaged over the dataset. The expression in parentheses in Eq.8 is invariant under permutations of the model predictions m i and annotations y i . It can be interpreted as the best baseline performance that can be expected when the model and annotators are simply guessing.

The covariance term represents a correction to this baseline performance that takes into account the correlation of the predictions and labels (i.e. when the model and annotators provide correlated labels, the performance is improved). The expressions in Eqs. 5-7 only depend on aggregate statistics (i.e. mean and variance) of the labels and not on any particular annotation and can be used, for example, as metrics during model training.

4. EXPERIMENTS:

We illustrate the ideas discussed in Section 3 on three datasets. First, we use MNIST (LeCun et al., 2010) with simulated models and annotators to objectively compare different strategies for evaluating model performance in the presence of noisy test data. Next, we train and evaluate models on two real-world, medical image datasets, which exhibit high degrees of expert annotator disagreement, and evaluate them using our framework.

4.1. MNIST DIGIT CLASSIFICATION:

For our MNIST experiments, we use the standard train/test splits and original (10-class) labels. Rather than train models, we synthetically generate them by corrupting the ground truth labels. Each model is designated as model X where X designates the percentage of the ground truth which the model preserves. For example, model 1.0 leaves the ground truth as is whereas model 0.8 chooses 20% of the labels randomly from amongst the other 9 classes.

We explicitly differentiate between true labels and observed labels, which are noisy and may not match the true labels. Our goal is to understand how different evaluation strategies fare in the presence of label noise, e.g. when we do not have direct access to the true labels. To this end, we evaluate several strategies: Naive Evaluation, Majority Vote, Relative F1 and our Discrepancy Ratio metric. Here, the number of annotators refers to the total number used in the analysis (6 annotators are used for the hold-out set while the number used in the ground truth set is varied). Panel (d) shows that the relative F1 score (where we used 6 annotators in the ground-truth set and 3 hold-out annotators) also gets skewed when annotators are allowed to voluntarily skip images (or equivalently not all annotators get images of equivalent difficulty). For panels (c) and (d) we have fixed the swap probability at 0.3.

4.1.1. NAIVE EVALUATION:

In this regime, we obtain each observed label by either selecting the true label (probability 1 -p) or random sampling (probability p). As Fig. 1a illustrates, the degree to which model performance is underestimated increases monotonically and linearly with the label swap probability. The Naive Evaluation strategy is limited in its reliance on one label per sample. It has no way to evaluate the degree of label noise (a priori) nor a mechanism to compare model to annotator performance.

4.1.2. MAJORITY VOTE:

In this regime, we assume we have 9 different annotators, each of whom selects the true label with probability p or randomly samples a label with probability 1 -p. The final observed label is obtained by taking the mode of the annotators' labels. Fig. 1b shows the effect of this aggregation on the observed accuracy. Majority Vote has an advantage over Naive Evaluation in that the narrowing gaps between models as the label noise is increased can, to some extent, be recovered by averaging multiple labels. Additionally, one can use the entropy of the label distribution to get a sense of inter-annotator variability. However, although Naive Evaluation and Majority Vote are sufficient for the task of ranking models (deciding which one is better), these approach do not indicate whether the model performance is better or worse than the average human performance.

4.1.3. RELATIVE F1:

We additionally consider the method used in (Rajpurkar et al., 2018b; Hannun et al., 2019; Rajpurkar et al., 2018a) . Here, a model and a set of held-out annotators are compared (using the F1 score) to a gold-standard ground truth set obtained using the majority-vote of multiple annotators. The model F1 score is then compared to the average F1 score of all the annotators in the hold-out set. As can be seen in Fig. 1c , the difference in the model and average annotator F1 scores, which we refer to as Relative F1, is dependent on the number of annotators used in the ground truth set (see Appendix C for intuition about this effect).

The Relative F1 score metric has another major shortcoming. As we discuss in Appendix C, in many applications annotators must be allowed to voluntarily skip samples when they feel that there is not enough information to provide an accurate label. Consequently, as shown in Fig. 1d a single annotator can bias the relative F1 score by simply annotating only the easier samples (on which there is high agreement among annotators) and skipping the hard ones (on which the agreement is low).

Here we assigned to each image in the MNIST test set its own swap probability p, which ranges from lower values (i.e. images on which there is high agreement) to high values (i.e. images on which there is high disagreement). We find that as we increase the single annotator strictness (the fraction of the dataset they skip), the relative F1 score is correspondingly reduced. It should be noted that this same exact bias can arise when comparing the model and all annotators pairwise (as in the methods used in (Wei et al., 2019; Rajpurkar et al., 2017) ) rather than to the majority vote.

4.1.4. DISCREPANCY RATIO:

The discrepancy ratio introduces a number of improvements over the baselines. First, unlike Naive Evaluation and Majority Vote, the discrepancy ratio explicitly compares the model performance to the inter-annotator variability. As illustrated in Figs. 2a and 2b , when the swap probability is close to 0, the annotators are in almost perfect agreement and the discrepancy ratio is higher than 1 (for models other than model 1.0). When p increases past a certain point, however, the annotators become more discrepant relative to each other than relative to the models and the discrepancy ratio falls below 1. It is in this regime that the model performance can be said to be better than the average human performance. Once p approaches 1, the labels are completely random and the annotators are again as different from each other as they are from the model, yielding a discrepancy ratio of 1. Naturally, the discrepancy ratio for the ground truth curve (model 1.0) always lies below 1. We note that the large gap in performance between models in the low-noise regime represents an important and subtle feature of our approach: models that have similar performance as measured by the Naive Evaluation and Majority Vote methods can in fact be dramatically different when human performance is taken into account. As a specific example, consider models 1.0 and 0.9 at a label swap probability of 0.05. These models appear to only be different by ∼ 10% when evaluated using accuracy metrics (Figs. 1a and 1b ). However, as shown in Figs. 2a and 2b, model 0.9 is twice as far from annotators as they are from each other, while model 1.0 is twice as close, thus resulting in a discrepancy ratio performance gap of ∼ 400%.

Relative F1 does allow for evaluating the model by comparing it to annotator performance but the discrepancy ratio has several advantages. First, as illustrated in Fig. 2c , the discrepancy ratio is insensitive to the number of annotators making it a more robust metric to compare across different published studies. Second, the discrepancy ratio allows for a highly varying number of annotators per sample. This feature permits its use in scenarios where annotators have the ability to skip samples and makes it a more practically viable solution than Relative F1. The presence of strict annotators (who skip hard samples) has no effect on the discrepancy ratio (Fig. 2d ). The red and blue points occur at label swap probabilities of 0.1 * 10/9 and 0.2 * 10/9, respectively, which correspond precisely to when the accuracies of the simulated human annotators equal that of the models (0.9 and 0.8, respectively). The discrepancy ratios (Eq. 4) for the same three models are shown in (b). The blue and red points correspond to those shown in (a). The shaded region denotes the area where the model is superior to the average human performance. Panels (c) and (d) show that the discrepancy ratio is not sensitive to the number of annotators used, nor is subject to the strict annotator bias illustrated in Fig 1d . Here we have used the same parameters as in Figs 1c and 1d . 

4.2. ECHOCARDIOGRAPHIC IMAGE CLASSIFICATION AND REGRESSION:

An echocardiogram is a common medical test that uses high-frequency sound waves (ultrasound) to visualize the structure of the heart. In this work, we consider two important and ubiquitous tasks that are performed as part of virtually any echocardiogram (Lang et al., 2015) .

PLAX measurability classification::

The exam typically begins with acquiring a parasternal long axis (PLAX) view, which is used to get an overview of general cardiac function, as well as to make specific measurements (e.g. fractional shortening (Lang et al., 2015) , presence of pericardial effusion, etc). Before any measurement can be performed, however, a clinician must first perform a binary classification task and determine whether the image in question is of sufficient quality to be used in the calculation (a similar binary classification screening step must be performed for most medical imaging tasks).

LVEF regression::

The left ventricular ejection fraction (LVEF), which measures the fraction of blood the left ventricle expels with each heart cycle, is one of the key metrics used to quantify heart functionality. This quantity is often measured in the apical 4-chamber view of the heart by segmenting the left ventricular endocardial border at end-systole and end-diastole (see Appendix A).

Both tasks described above currently require the significant expertise of a trained sonographer. We have developed deep neural network models to automate these tasks. The datasets for these tasks are summarized in Tables 2 and 3 . All data was collected using the Butterfly iQ ultrasound probe. Details of the data collection and annotations for both tasks are described in Appendix A. shows that human experts exhibit a high amount of disagreement on which images meet the threshold for measurability (i. e. with perfect agreement all labels would be 0 or 1). This plot displays the normalized histogram of the average labels on the PLAX measurability classification dataset. The discrepancy ratio on the test set as the output threshold of the network output is varied is shown in (b). Panel (c) shows the labels (blue) and model predictions (red) on the LVEF regression task test set, sorted by the mean LVEF label. Each patient was labeled by an average of 5 annotators.

The black crosses signify the points on which the model predictions were artificially corrupted to create large errors.

4.2.1. EVALUATION USING DISCREPANCY RATIO:

In order to explore the properties of the discrepancy ratio in a real-world setting, we first consider the PLAX measurability classification dataset. The neural network used for this task is described in Appendix D. When evaluating the performance of our network using Naive Evaluation (where we evaluate on a single annotator, chosen at random), we find that we achieve an accuracy of 77.8%, while the majority vote method gives an accuracy of 88.0%. These numbers are quite poor by the standards of most machine learning applications. Indeed, we might otherwise conclude that the model performs at a sub-standard level.

However, when we evaluate using the discrepancy ratio and consider the inter-annotator variability, a very different story emerges. As can be seen in the distribution of the average label (y i from Eq. 5) in Fig. 3a , there is significant disagreement (∼ 25% on average) among the annotators as to whether a given image meets the threshold of measurability. While this level of disagreement may seem high, it is typical of that found in the medical applications literature (Wei et al., 2019; Hannun et al., 2019) .

Fig. 3b shows the discrepancy ratio as a function of the threshold of the network output. Contrary to the impression suggested by using the conventional metrics, we find that for a wide range of thresholds the model dramatically exceeds the average human performance (as defined by ∆ = 1).

Table 4 shows the discrepancy ratio, with 95% confidence intervals, for the model as compared to those achieved by the individual annotators on the test set. Here we find that the discrepancy ratio has another nice property: it can be used to evaluate the performance of individual annotators. We find that there is a significant spread in the annotator performance and that our model outperforms even the best annotator in the dataset.

4.2.2. DEPENDENCE ON δ:

As described in Section 2, the discrepancy ratio can be thought of as a "meta-metric" that evaluates the model performance for any desired measure of agreement δ. Here we use the LVEF regression dataset to explore this dependence on δ. The neural network we used is described in Appendix D. We consider three choices of δ: the absolute difference: δ = |y -y |, squared difference: δ = (y -y ) 2 and the hinge loss: δ = max(0, λ -|y -y |), where a penalty is incurred only past a minimum deviation threshold1 . As with the PLAX measurability classification task, we find that our model exceeds the performance of even the best annotator in our dataset on all three measures of agreement, obtaining discrepancy ratios of 0.89, 0.78 and 0.67 respectively. In order to demonstrate the dependence on δ, we then artificially corrupt the model so that it occasionally makes errors that are far outside the range of those made by humans (Fig. 3c ). How does the model fare with respect to human performance? It depends on the type of errors one cares most about. When we set δ to be the absolute difference, the discrepancy ratio is 0.95 and one may conclude that the model is performing well. If the squared difference is used for δ instead, the outliers receive higher cost and the discrepancy ratio becomes Here each annotator was individually taken out of the dataset and treated as if they were the model for the purpose of calculating their discrepancy ratio. The confidence intervals were obtained using the bootstrap method where the sampling was performed by patient. We used the validation set to choose the threshold corresponding to the lowest discrepancy ratio (as in Fig. 3b ).

1.02. When a hinge-loss is used for δ, where deviations below λ are considered acceptable, we observe that the performance is worse still (1.08).

5. LIMITATIONS:

While our approach provides for a simple and general way of quantifying model performance in the presence of label noise, it has several limitations. First, although the flexibility in choosing δ allows us to examine various aspects of the model performance (illustrated in the example above), our metric needs to be applied with caution since it is only as good as the underlying measure of agreement. Second, in situations when annotators are allowed to skip samples they feel unsure about, samples that get fewer labels may tend to be more ambiguous. Since the discrepancy ratio is undefined for samples with fewer than two labels, this can cause the discrepancy ratio to overweight easier examples. Consequently, one should ensure that all samples in the domain of interest are labeled by at least two annotators. Finally, the discrepancy ratio can yield misleading results when the labels are provided by poor annotators, though this is also true of any metric (e.g. the relative F1) that attempts to normalize model performance to account for annotator disagreement. To mitigate this, the annotators should be properly vetted as skilled for the task (see e.g. Hovy & Lavid (2010) or our vetting procedure in Appendix A) and allowed to skip when they are unsure of the label. In some situations annotators may adhere to several discrete "schools of thought" and hence their annotations may be more similar within each group than between groups. In these situations it may make sense to evaluate the discrepancy ratio independently within each group.

6. CONCLUSION:

We present a unified and principled approach to evaluating machine learning models when humans disagree on the truth. Our metric, the discrepancy ratio, is highly intuitive and can be adapted to virtually any task. We have shown that the discrepancy ratio has numerous desirable properties including insensitivity to annotator count and insensitivity to annotator/image overlap. At the same time, we have evaluated the shortcomings of alternative implementations and show that our method has numerous advantages that make it a convenient standard by which models can be evaluated in the presence of ambiguous ground truth labels. Finally, to demonstrate our method we have trained deep neural networks on two real-world tasks from medical imaging and have used our metric to show that we achieve performance better than even the best annotators in our datasets. Here A is the total number of annotators in the dataset and the sum over samples has to go over only those i for which the jth annotator has labeled it at least once (i.e. Y i j = ∅). Conceptually, this expression calculates a discrepancy score for each model/annotator pair, and then averages them together to arrive at an aggregated measure of agreement.

An analogous alternative definition can be constructed for the annotator discrepancy:

α = A j,j =1 j =j N i=1 Y i j ,Y i j =∅ ψ(Y i j , Y i j ) N A(A -1)

.

As in the case of the model discrepancy, this expression computes a discrepancy for each annotator pair and then averages all the pairs to yield an aggregate measure of annotator agreement. The methods presented in Wei et al. ( 2019); Rajpurkar et al. (2017) use essentially these definitions, though they do not compare them using a ratio.

When all annotators label all samples, these definitions are perfectly valid, and are in fact equivalent to µ and α. However, in most real-world scenarios, data come from disparate sources (e.g. different hospitals with different annotators) and it is not practical to ensure that the exact same set of annotators labeled all the samples. Worse still, in many domains it is necessary to allow human annotators to voluntarily skip samples when they feel there is not enough information to provide an accurate label.

In such a scenario, formulating the model and annotator discrepancies using the alternative definitions above allows an annotator to improve their own discrepancy score by simply being strict and only annotating "easy" samples. This will generally result in a bias being introduced into the discrepancy ratio. To illustrate this point, consider the following toy example:

Example C.1 A dataset contains one "easy" image (on which the MPD between any two annotators is δ e ) and one "hard" image (on which the MPD between any two annotators is δ h ). Similarly, a machine learning model gets an MPD of δ e on the easy image and δ h on the hard image. Three annotators (Alice, Bob and Charlie) are asked to label the data. Alice and Bob label both images, but Charlie wants to improve his discrepancy and only labels the easy image. Using the original definitions of α and µ, the discrepancy ratio is ∆ = µ α = (δ e + δ h )/2 (δ e + δ h )/2 = δ e + δ h δ e + δ h .

Note that the value of ∆ would have been exactly the same without Charlie. On the other hand, using the alternative definition the discrepancy ratio becomes biased and is given by

∆ = µ α =

(2(δ e + δ h )/2 + δ e )/3 ((δ e + δ h )/2 + 2δ e )/3 = 2δ e + δ h (5δ e + δ h )/2 .

In the special case where the annotators and model are in perfect agreement on the easy image (δ e = δ e = 0), Charlie was able to double the discrepancy ratio by simply skipping the difficult image:

∆ = 2δ h δ h = 2∆.

It should be noted that even if we weigh the average of annotators by the inverse of the number of annotations they provided, the bias does not go away, whereas the discrepancy ratio, as defined in the main text, is completely insensitive to this effect. The effect illustrated in the example above is not limited to the presence of an adversarial annotator, but can creep into the data whenever there exist any systematic differences in how the data got partitioned among annotators, which in practice may be challenging or even impossible to control.

:

We used λ = 15 as a clinically relevant measure of a significant Ejection Fraction error.

Figures and Tables:

Figure 1: Performance of baseline metrics on the MNIST test set. Curves show mean value and 2standard deviation bounds obtained from multiple simulations of the experiment. (a) Naive Evaluation exhibits linear decay of observed accuracy but cannot compare model to inter-annotator variability (b) Majority Vote is more invariant to label noise but also does not compare model to inter-annotator variability. Panel (c) shows that the relative F1 score depends sensitively on the number of annotators.Here, the number of annotators refers to the total number used in the analysis (6 annotators are used for the hold-out set while the number used in the ground truth set is varied). Panel (d) shows that the relative F1 score (where we used 6 annotators in the ground-truth set and 3 hold-out annotators) also gets skewed when annotators are allowed to voluntarily skip images (or equivalently not all annotators get images of equivalent difficulty). For panels (c) and (d) we have fixed the swap probability at 0.3.

Figure2: Comparison of the discrepancy ratio with baseline metrics on the MNIST test set. Curves show mean value and 2-standard deviation bounds obtained from multiple simulations of the experiment. Panel (a) shows the model discrepancy (for three models) and annotator discrepancy. The red and blue points occur at label swap probabilities of 0.1 * 10/9 and 0.2 * 10/9, respectively, which correspond precisely to when the accuracies of the simulated human annotators equal that of the models (0.9 and 0.8, respectively). The discrepancy ratios (Eq. 4) for the same three models are shown in (b). The blue and red points correspond to those shown in (a). The shaded region denotes the area where the model is superior to the average human performance. Panels (c) and (d) show that the discrepancy ratio is not sensitive to the number of annotators used, nor is subject to the strict annotator bias illustrated in Fig 1d.Here we have used the same parameters as in Figs1c and 1d.

Figure3: Panel (a) shows that human experts exhibit a high amount of disagreement on which images meet the threshold for measurability (i. e. with perfect agreement all labels would be 0 or 1). This plot displays the normalized histogram of the average labels on the PLAX measurability classification dataset. The discrepancy ratio on the test set as the output threshold of the network output is varied is shown in (b). Panel (c) shows the labels (blue) and model predictions (red) on the LVEF regression task test set, sorted by the mean LVEF label. Each patient was labeled by an average of 5 annotators. The black crosses signify the points on which the model predictions were artificially corrupted to create large errors.

Training and Evaluation Datasets for PLAX Measuraility Classification Task.

Table content: <table><tr><td>Dataset</td><td colspan="2">No. of Images No. of Patients</td></tr><tr><td>Train</td><td>60,411</td><td>1,915</td></tr><tr><td>Validation</td><td>5,996</td><td>200</td></tr><tr><td>Test</td><td>17,659</td><td>600</td></tr></table>

Training and Evaluation Datasets for LVEF Regression Task.

Table content: <table><tr><td>Dataset</td><td colspan="2">No. of Images No. of Patients</td></tr><tr><td>Train</td><td>890,541</td><td>2,964</td></tr><tr><td>Validation</td><td>407,151</td><td>1,180</td></tr><tr><td>Test</td><td>149,992</td><td>322</td></tr></table>

The discrepancy ratio ∆ for the model and annotators (who have labeled at least 100 patients) on the PLAX measurability test set.

Table content: <table/>

Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. Denoising neural machine translation training with trusted data and online data selection. Proceedings of the Third Conference on Machine Translation, 1:133-143, 2018.

Table content: <table><tr><td>Jason W Wei, Laura J Tafe, Yevgeniy A Linnik, Louis J Vaickus, Naofumi Tomita, and Saeed</td></tr><tr><td>Hassanpour. Pathologist-level classification of histologic patterns on resected lung adenocarcinoma</td></tr><tr><td>slides with deep neural networks. Scientific reports, 9(1):3358, 2019.</td></tr><tr><td>Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep</td></tr><tr><td>features for scene recognition using places database. In Advances in neural information processing</td></tr><tr><td>systems, pp. 487-495, 2014.</td></tr><tr><td>Xingquan Zhu and Xindong Wu. Class noise vs. attribute noise: A quantitative study of their impacts.</td></tr><tr><td>Artificial Intelligence Review, 22:177-210, 2004.</td></tr></table>

Peer review (iclr2020):
Summary: This paper proposes a novel metric *discrepancy ratio* for evaluating the performance of a model where the ground truth for each data point comes from many expert-yet-imperfect annotators. The authors suggested that this problem has remained largely unexplored. The proposed metric is intuitive and is easy-to-use. The authors suggested that this metric can be used for many applications. For example, to evaluate if a model is better than the annotators on average, to evaluate the annotator, to compare between models. ======================================================== Comments on clarity: The writing is good overall and I am able to understand the contribution and the key idea of this paper. Questions and comments on clarity: 1. While MPD is defined with an argument (Y, Y*), the definitions annotator discrepancy, model discrepancy and discrepancy ratio ignore the inputs of the function. It might be better to keep the arguments of the function for clarity. When people want to adopt the discrepancy ratio in their work and there are many ratios to discuss, then the proposed definition can be used conveniently. 2. In Eq. (5) sigma seems to be undefined. I believe it is a variance but it is kinder to readers to explicitly state it (I found the authors explicitly stated it in Appendix B). 3. In Eq. (8), is it intentional to have exists over m and y as written in the paper? 4. Does the discussion of the simplicity of the discrepancy ratio hold for other delta, i.e., not the squared loss? 5. In Section 4.1 (experiment with MNIST), did the authors binarize the data or simply discuss the multi-class problem? Because this follows section 3.3 which dedicates the whole page to discuss binary classification. It may be a bit sudden to move to multi-class classification and an additional sentence to clarify this might help. 6. I could not understand what is *single annotator strictness in Figures 1d and 2d. It is not explained if I did not miss it. 7. I think we should not resize the figure/table caption. The text size of captions are much smaller than the main text. 8. Is PLAX binary classification? (I understood as yes from Figure (3a) but I think it*s also nice to clarify it to avoid confusion (also for MNIST)). 9. What is the y-axis for figure 3a? 10. I think reporting the performance of the mean-per-class discrepancy ratio without defining it in the main text (it is defined in a natural language (English) in Appendix E) makes the main body of the paper not self-contained. I strongly suggest to either (1) remove mean-per-class discrepancy ratio entirely from the main body or (2) define it mathematically in the main body. Also, I don*t see reporting mean-per-class discrepancy ratio adds values to a paper for current experiments because the reported mean-per-class ratio is very similar to that of discrepancy ratio (Figures 3b (almost overlap) and Table 2 (same trend and the values are very close to the discrepancy ratio for all cases)). 11. Can we use F1 to evaluate whether a model is better than the average human performance? Minor points on clarity, which may just be a personal preference of a reviewer: 1. The authors may consider using _x0008_egin{definition} when defining three key values: the mean pairwise deviation (MPD), annotator discrepancy, and model discrepancy. So the readers can easily look up when getting lost in definitions. And it may help to make a definition more precise (e.g., given label Y, data points, MPD is defined as follows). 2. Eqs (1), (2), (3) may be easier to understand if we put a denominator out of the summation as much as we can. For example, for Eq. 2, N can be outside of the two sums, A^i(A^i-1) can be between the two sums. ======================================================== Significance: The problem that this paper addresses is highly important. I like the idea and I am convinced that discrepancy ratio is potentially useful for the community. And it also opens many possibilities to analyze the discrepancy ratio and to find potential applications for it. ======================================================== Other comments: On the limitation: I am satisfied that the authors clearly discuss the limitations of the proposed method and I agree with the discussion. On the experiments: 1. Why we use model 1.0, 0.95, 0.9 for Figures 1c, 1d, 2c, 2d but 1.0, 0.9, 0.8 for Figures 1a, 1b, 2a, 2b? Perhaps full results with many models should also be reported for completeness. 2. Isn*t it more natural to show a graph that has x-axis as label swap probability in the same figure for comparison? For example, Figure 1 only report performance with respect to each measure (Naive Evaluation, Majority Vote, Relative F1, Discrepancy ratio) with respect to label swap probability. We can fix the number of annotators to 9 for relative F1 and discrepancy ratio, to validate whether it also becomes increasingly difficult to differentiate model improvements or not for them. Then after suggesting that Naive evaluation is not good and other metrics look fine. Majority vote cannot evaluate that the model is better than human average performance, which we want to know. Then Figure 2 highlights the contribution of the discrepancy ratio and shows that it outperforms relative F1 in some perspectives. 3. For the MNIST benchmark dataset, it is possible to show the performance on the test ground truth too. And I think it is interesting to compare different models with the same label noise rate and confirm that discrepancy ratio is successful to evaluate the best model with respect to the ground truth accuracy. It would be very impressive to see if the best model with respect to the discrepancy ratio can successfully be the best also on the ground truth test set. On space usage: I feel there are many important things to be included in the main body but it is instead in the appendix. On the other hand, I think there are several parts that can be shortened in the main body. And it seems that the binary classification example (3.3) also consumed a lot of space. The derivation may be omitted and make it a Theorem and then let*s suggest the readers to read the appendix for the proof if they are interested. Then we can have more space to add important components in the main body of the paper. ======================================================== Decision: Although there are issues on the clarity of the paper, I believe these issues are not too difficult to fix in the final version. The idea is interesting and potentially impactful. For these reasons, I vote a weak accept for this paper. Update: I would like to thank the authors for the clarification. I have read the rebuttal. I read through the updated version of the paper. I believe that although there is still room for improvement, the contribution of this paper is sufficient as an important step towards solving this difficult yet highly relevant problem. This paper also stimulates several future directions. The authors clearly stated the limitations of the paper, which is highly useful for researchers to improve this work or find a better solution. The clarity of the paper is also improved. For these reasons, I increased my score from weak accept to accept.