Discrepancy Ratio: Evaluating Model Performance When Even Experts Disagree on the Truth
Authors: Igor Lovchinsky
Alon Daks
Israel Malkin
Pouya Samangouei
Ardavan Saeedi
Yang Liu
Swami Sankaranarayanan
Tomer Gafner
Ben Sternlieb
Patrick Maher
Nathan Silberman
Venue:  (2020)
Keywords: Evaluation Metrics
Medical Imaging

Abstract: In most machine learning tasks unambiguous ground truth labels can easily be acquired. However, this luxury is often not afforded to many high-stakes, real-world scenarios such as medical image interpretation, where even expert human annotators typically exhibit very high levels of disagreement with one another. While prior works have focused on overcoming noisy labels during training, the question of how to evaluate models when annotators disagree about ground truth has remained largely unexplored. To address this, we propose the discrepancy ratio: a novel, task-independent and principled framework for validating machine learning models in the presence of high label noise. Conceptually, our approach evaluates a model by comparing its predictions to those of human annotators, taking into account the degree to which annotators disagree with one another. While our approach is entirely general, we show that in the special case of binary classification, our proposed metric can be evaluated in terms of simple, closed-form expressions that depend only on aggregate statistics of the labels and not on any individual label. Finally, we demonstrate how this framework can be used effectively to validate machine learning models using two real-world tasks from medical imaging. The discrepancy ratio metric reveals what conventional metrics do not: that our models not only vastly exceed the average human performance, but even exceed the performance of the best human experts in our datasets.

Published as a conference paper at ICLR 2020
DISCREPANCY RATIO: EVALUATING MODEL PERFOR- MANCE WHEN EVEN EXPERTS DISAGREE ON THE TRUTH
Igor Lovehinsky ilovchinsky
Alon Daks adaks
Israel Malkin imalkin
Pouya Samangouei psamangouei
Ardavan Saeedi asaeedi
Yang Liu yliu
Swami Sankaranarayanan
Tomer Gafner
Benjamin Sternlieh bsternlieb
ssankaranarayanan
tgafner
Patrick Maher
Nathan Silberman
pmaher
nsilberman
Butterfly Network @butterflynetworkk.omm
ABSTRACT
In most machine learning tasks unambiguous ground truth labels can easily be acquired. However, this luxury is often not afforded to many high-stakes, real- world scenarios such as medical image interpretation, where even expert human annotators typically exhibit very high levels of disagreement with one another. While prior works have focused on overcoming noisy labels during training, the question of how to evaluate models when annotators disagree about ground truth has remained largely unexplored. To address this, we propose the discrepancy ratio: a novel, task-independent and principled framework for validating machine learning models in the presence of high label noise. Conceptually, our approach evaluates a model by comparing its predictions to those of human annotators, taking into account the degree to which annotators disagree with one another. While our approach is entirely general, we show that in the special case binary classification, proposed metric can be evaluated in terms of simple, closed- form expressions that depend only on aggregate statistics of the labels and not on any individual label. Finally, demonstrate how this framework effectively to validate machine learning models using two real-world tasks from medical imaging. The discrepancy ratio metric reveals what conventional metrics do not: that our models not only vastly exceed the average human performance, but even exceed the performance of the best human experts in our datasets.
be used
INTRODUCTION
The canonical supervised machine learning paradigm assumes the presence of both inputs and their
corresponding, unambiguous outputs. In practice, the vast majority of datasets exhibit some degree
of label noise. Acknowledgement of this fact has resulted in the development of algorithms tailored
training models the presence of noisy ground truth labels (Sukhbaatar et 2015 Frenay &
Kaban 2014 7hu & Wu. 2004). These approaches include cleansing noisy labels before training
begins Hodge & Austin, 2004). making the training process robust to noise (Patrini et al.) 2017
Vahdat 2017 Rolnick al. (017)) and actively modeling label noise (Sukhbaatar al. 2015 Joseph
ct al. 1995). These methods almost exclusively address only the problem of noisy labels during training and tend to assume that least some noise-free ground truth is available at test time in order to properly evaluate different approaches. Indeed, without an unambiguous yard-stick, how can competing schemes possibly be evaluated in standardized manner?
There exist many domains, such as medical imaging (Rosenkrantz et al. 2013: Lazarus et al. 2006) and machine translation (Wang al. 2018), in which access to unambiguous ground truth,

---

Published as a conference paper at ICLR 2020
at evaluation time, is not practically achievable. In these domains not only do annotators disagree with each other, but they often even disagree with themselves (Gulshan et al. 2016 Hansegard et al.) 2009). This, however, does not imply that the annotators are not trustworthy, but rather that these tasks are difficult even for domain experts. Datasets in these regimes are, in practice quite different from those typically used in machine learning research (which usually geared towards content that can be easily curated by lay people). As a consequence, studies utilizing public data often resort to simulating label noise. Indeed, theoretical understanding of evaluation in the presence of ambiguous test data has remained largely unexplored.
In this work, we propose a new framework for evaluating the performance of machine learning models in the presence of high label noise. Our main contributions are as follows:
1. We propose novel, indeenndeenddnt metrie, the discrepancy ratio, that allows us to a) quantify model performance, taking into account how much humans disagree with each other, and b) set a precise threshold at which the model performance can be said to exceed the average human performance.
2. We perform a detailed analysis of the discrepancy ratio for the special case of binary classification and demonstrate that it can be evaluated in terms of simple. closed-form expressions that depend only on aggregate statistics (mean and variance) of the annotations and not on any individual annotation.
3. We demonstrate the effectiveness of our approach on models that we trained on two important medical imaging tasks with real-world clinical data and show that it reveals that models not only significantly exceed the average human performance, but even beat the best human experts in datasets.
2 RELATED WORK
There are several well-known strategies for dealing with ambiguous or noisy evaluation labels. The most popular approach is to simply use a single annotator (Jafari et al., 2019 Rajehl et al., 2017] Cobzas et al.) 2007 Roth et al. 2018). An approach taken by many popular, large- scale datasets, such as ImageNet (Deng et al. 2009). Places (Zhou et al./ 2014). COCO (Lin et al.) 2014) and Open Images V4 (Kuznetsova al. 2018). is to take the average or majority vote. These efforts typically employ crowdsourcing in some fashion, since the labeling of this data does not require expert-level knowledge. While majority vote reduces label noise, in domains that require specialized knowledge it is often not practical to get enough experts that the noise is removed to the point where the annotations can be assumed to be "correct" (in the sense that MNIST annotations are assumed to be correct). A related approach obtains a ground- -tutth dataset by enforcing a resolution criterion on labels where annotators disagree. This criterion can, for example, involve a live session where annotators deliberate and come to consensus or an adjudication process where a third party (usually a more senior, expert annotator) is tasked with resolving disagreements (Krause et al.] [0188] [Hannun] let al. 2019; Rajpurkar et al. 2018b). This approach may be effective at removing erroneous labels (reducing variance) at the expense of annotator independence (increasing bias).
The techniques outlined above fall short in providing unified method that can both a) be compared across different studies/tasks and b) answer the essential question of how the model performance relates to average human annotator (i.e. how a prototypical human currently performs at task). In developing high-stakes, real-world machine learning systems, the goal of evaluating a model is often to determine whether performance is safely substitutable for individual human behavior rather than comparing to an idealized oracle.
Several studies have compared model performance with inter-annotator variability. Some have visualized model performance and inter-annotator variability (e.g. via ROC curves (Krause et al. 2018: Gulshan et al. 2016) Bland-Altman plots (Hansegard al. 2009 Altman & Bland 1983)
without providing an explicit metric. Several works have constructed a metric by comparing the model the average annotator performance, though each has done so in a completely different way (Wei et al., 2019 Hannun et al., 2019; Rajpurkar et al.) 201 8a by 2017). These metrics, however, generally depend on the number of annotators used in the study and/or require that all annotators label the entire dataset (see Section 4 and Appendix C for more details). Moreover, the metrics are task-specific and cannot be interpreted outside the context of a particular study.
2

---

Published as a conference paper at ICLR 2020
3 THE DISCREPANCY RATIO METRIC
In this section, define the discrepancy ratio and show that it has numerous desirable properties that overcome the limitations of prior art detailed in Section 2.
3.1 DEFINITIONS
Definition 1. For any two arbitrary sets of labels y = 1vn and 3 = 1....'..'' w.. the mean pairwise deviation (MPD) is defined by:
(V.)") ΣΣ VEVN'E
6(y, y²)
D
where 8 be chosen to correspond to any measure of agreement appropriate to the task in question (e.g. squared error, absolute error, Jaccard Index (Jaccard 1912). Cohen's 's kappa (Cohen) 1960)).
Consider dataset of N samples where we have a prediction m', for each sample by the model we are trying to evaluate. Further assume that each sample is labeled by at least two annotators and that each annotator may label the same sample multiple times. Formally, let ) [ufe] be the set of annotations provided by annotator on sample where indexes the annotation number. Finally, let A' indicate the number of annotators that have annotated sample
We define the annotator discrepancy, which quantifies the average annotator disagree- ment, be the average of the MPD over all annotator pairs and all samples in the dataset:
Definition 2.
o (()))) N 1=1
Σ ,0),)%)
A
10=11
J'
Without access unambiguous ground-truth labels, natural way define human performance by comparing an annotator's s labels to those provided by other humans on the same samples. Consequently, use the annotator discrepancy measure of average human performance.
Definition 3. Similarly, the model discrepancy is defined to be
A' (mm, )7).
u(m', (211))
N
Definition 4, We define the discrepancy ratio to be the ratio of the model and annotator discrepancies:
A
(4)
Intuitively, this metric is the ratio of the average prodel-annotare disagreement and the annotator- annotator disagreement. For example, discrepancy ratio of 0.8 would mean that the model is 20% closer to the annotators than they are each other. When is above (below) 1, the model performance be said to be worse (better) than the average human performance. Since the measure of agreement is arbitrary, the discrepancy ratio can be thought of as "meta-metric" that reports how the model performs (relative to humans) with respect to that particular measure of agreement. The dependence of the discrepancy ratio on the choice of is further explored in Section 4.2.2.
3.2 PROPERTIES
The discrepancy ratio metric, formulated above, has several desirable properties:
Dimensionless:
Dimensionless quantity whose meaning is invariant across different task categories (regression, classification, segmentation etc.) and whose value is interpretable and highly intuitive.
Minimal dataset coverage requirements: The discrepancy ratio does not require that all annotators label all samples (see Fig. and Appendix C), which often not practical in real-world situations,

---

Published as a conference paper at ICLR 2020
Sample Index
Model
Annotator
Annotator
Annotator
0
0,
0,0
1,0,
2
1
1,1,1
1,1,
Table An Example Dataset for Binary Classification Task.
but only that each sample is labeled by at least two annotators (two annotators are needed since it is meaningless measure the annotator disagreement with only single annotator).
Insensitive to number of labels per-sample: The discrepancy ratio is expressed in terms of averages over all model/annotation pairs and therefore does not depend how many annotators are used (see Fig. 2) nor how many times each annotator labels each image. This property makes it convenient to compare the discrepancy ratio across studies.
Since the discrepancy ratio is formulated as relative error that the model makes over that of the annotators, it can be naturally incorporated as test statistic a non-inferiority framework (Ahn) al. 2013), often used in medical research. The goal of non-inferiority testing is to show that new method is not statistically worse than an existing standard, up to some chosen equivalence bound. For the discrepancy ratio, we can choose the equivalence bound to correspond to acceptable fraction of the error that the model allowed to make on top of the one that annotators already make (e.g. setting the equivalence bound to be 0.1 be interpreted as saying that the model can make error more than 10% larger than the ones humans typically make). Additional properties of the discrepancy ratio, as well comparisons to alternative formulations, are discussed detail in Appendix C and
3.3 AN ILLUSTRATIVE EXAMPLE
In this section, compute expressions for the model and annotator discrepancies on a generic binary classification task using the squared error as an agreement measure. This calculation will serve to build intuition for these metrics and be used blueprint for evaluating them other tasks.
Consider a dataset (see T containing N samples, where each sample is given a binary score by model and set of annotators (an annotator may label an image more than once, not at all). We assume all samples are labeled by at least two annotators. In the analysis below we adopt the convention that VI2 denotes the ktth label provided by the jth annotator on the ith sample, and
removing an index denotes averaging over the values of that index. For example, y is the average of the labels provided by the jth annotator on the ith sample and y is the average annotation over all labels, annotators and samples. The order of the averages should be clear from the context.
As shown in Appendix B, given two sets of binary labels y {y1, 1/,,,. Yn and y (v1, V/2--. Vyrr) } the MPD is given by the expression y + 1/ 2yy', where y = - 1==1 Using this result and Eq. 2] the annotator discrepancy for a single sample is then given by (Appendix B):
20
a² = 2y' -n")
y'
A'
where o² is the variance of the labels y and A' is the number of annotators that have labeled sample For the important special case where each annotator provided more than single label per image, this expression further simplifies to
2 A 1
(6)
where we see that annotator discrepancy is simply proportional the variance of the labels. Similarly we can write the single-sample model discrepancy
A'
m + 2i 2m V3 = =m² + 3² 2m y/.
(7)
A
j=1
Averaging this result over all samples, the model discrepancy becomes
4

---

Published as a conference paper at ICLR 2020
1 = (m + y 2may) 2Cov[m,, ,/",,
(8)
where the covariance is taken over the samples i and m and y correspond to the model predictions and annotator labels averaged over the dataset. The expression in parentheses in Eq g is invariant under permutations of the model predictions m² and annotations y'. It can be interpreted as the best baseline performance that can be expected when the model and annotators are simply guessing. The covariance term represents correction to this baseline performance that takes into account the correlation of the predictions and labels (i.e. when the model and annotators provide correlated labels, the performance is improved). The expressions in Eqs. 5 only depend on aggregate statistics (i.e. mean and variance) of the labels and not on any particular annotation and can be used, for example, as metrics during model training.
4 EXPERIMENTS
We illustrate the ideas discussed in Section 3 on three datasets. First, we use MNIST (LeCun et al. 20T0) with simulated models and annotators objectively compare different strategies for evaluating model performance in the presence of noisy test data. Next, we train and evaluate models on two real-world, medical image datasets, which exhibit high degrees of expert annotator disagreement, and evaluate them using our framework.
4.1 MNIST DIGIT CLASSIFICATION
For MNIST experiments,
the standard train/test splits and original (10-class) labels. Rather than train models, we synthetically generate them by corrupting the ground truth labels. Fach model designated as model X where X designates the percentage of the ground truth which the model preserves. For example, model 1.0 leaves the ground truth as is whereas model 0.8 chooses 20% of the labels randomly from amongst the other classes.
We explicitly differentiate between true labels and observed labels, which are noisy and may not match the true labels. Our goal is to understand how different evaluation strategies fare in the presence of label noise, e.g. when we do not have direct access to the true labels. To this end, we evaluate several strategies: Naive Evaluation, Majority Vote, Relative F1 and our Diserepancy Ratio metric.
(a)
(b)
(c)
(d)
3.5
3.90
nlsarved accuracy
model model
observed accuracy
0.207
model 0.8
0.152
telalive F1
0.33
0.109
0.30
0.073
0-2
model
0.9
0.00
model 0.95
model 1.0
0.003
1.0
0:355
model
01225
0.9
10
066
label swap probability
label swap probability
number of annetators
single annotator strictness
Figure 1: Performance of baseline metrics on the MNIST test set. Curves show mean value and 2- standard deviation bounds obtained from multiple simulations of the experiment. (a) Naive Evaluation exhibits linear decay of observed accuracy but cannot compare model to inter-annotator variability (b) Majority Vote is more invariant to label noise but also does not compare model inter-annotator variability. Panel (c) shows that the relative Fl score depends sensitively on the number of annotators. Here, the number of annotators refers the total number used in the analysis annotators are used for the hold-out set while the number used in the ground truth set is varied). Panel (d) shows that the relative Fl score (where we used 6 annotators in the ground-true set and 3 hold-out annotators) also gets skewed when annotators are allowed to voluntarily skip images (or equivalently not all annotators get images of equivalent difficulty). For panels (c) and (d) we have fixed the swap probability at 0.3.
4.1.1 NAIVE EVALUATION
In this regime, we obtain each observed label by either selecting the true label (probability 1 p) or random sampling (probability p). As Fig. [ illustrates, the degree to which model performance is underestimated increases monotonically and linearly with the label swap probability. The Naive Evaluation strategy is limited in its reliance on one label per sample. It has no way to evaluate the degree of label noise (a priori) nor a mechanism to compare model to annotator performance.
5

---

Published as a conference paper at ICLR 2020
4.1.2 MAJORITY VOTE
In this regime, we assume we have 9 different annotators, each of whom selects the true label with probability p or randomly samples a label with probability 1 1-p. p. The final observed label is obtained by taking the mode of the annotators' labels. Fig. D shows the effect of this aggregation on the observed accuracy. Majority Vote has an advantage over Naive Evaluation in that the narrowing gaps between models as the label noise is increased can, to some extent, be recovered by averaging multiple labels. Additionally, one can use the entropy of the label distribution to get sense of inter-annotator variability. However, although Naive Evaluation and Majority Vote are sufficient for the task of ranking models (deciding which one is better), these approach do not indicate whether the model performance is better or worse than the average human performance.
4.1.3 RELATIVE F1
We additionally consider the method used in (Rajpurkar et al. 2018b; Hannun et al. 2019 Rajpurkar cct al. [2018a). Here, a model and a set of held-out annotators are compared (using the Fl score) to a gold-standard ground truth set obtained using the majority-vote of multiple annotators. The model Fl score is then compared to the average F1 score of all the annotators in the hold-out set. As can be seen in Fig. D.p. the difference in the model and average annotator F1 scores, which refer as Relative F1, is dependent on the number of annotators used in the ground truth set (see Appendix C for intuition about this effect).
The Relative F1 score metric has another major shortcoming. As we discuss in Appendix C, in many applications annotators must be allowed to voluntarily skip samples when they feel that there is not enough information to provide an accurate label. Consequently, as shown in Fig. [ a single annotator can bias the relative Fl score by simply annotating only the easier samples (on which there is high agreement among annotators) and skipping the hard ones (on which the agreement is low). Here we assigned to each image in the MNIST test set its own swap probability p. which ranges from lower values (i.e. images on which there is high agreement) to high values (i.e. images on which there is high disagreement). We find that as we increase the single annotator strictness (the fraction of the dataset they skip), the relative Fl score is correspondingly reduced. It should be noted that this same exact bias can arise when comparing the model and all annotators pairwise (as in the methods used in (Wei et al. 2019. Rajpurkar et al. 2017)) rather than to the majority vote.
4.1.4 DISCREPANCY RATIO
The discrepancy ratio introduces number of improvements over the baselines. First, unlike Naive Evaluation and Majority Vote, the discrepancy ratio explicitly compares the model performance to the inter-annotator variability. As illustrated in Figs. 2 and 2b. when the swap probability is close to O, the annotators are in almost perfect agreement and the discrepancy ratio is higher than 1 (for models other than model 1.0). When p increases past a certain point, however, the annotators become more discrepant relative to each other than relative to the models and the discrepancy ratio falls below 1. It in this regime that the model performance can be said to be better than the average human performance. Once p approaches 1, the labels completely random and the annotators as different from each other as they are from the model, yielding a discrepancy ratio of 1. Naturally, the discrepancy ratio for the ground truth curve (model 1.0) always lies below 1. We note that the large gap in performance between models in the low-noise regime represents an important and subtle feature of our approach: models that have similar performance as measured by the Naive Evaluation and Majority Vote methods can in fact be dramatically different when human performance is taken into account. As a specific example, consider models 1.0 and 0.9 a label swap probability of 0.05. These models appear to only be different by ~ 10% when evaluated using accuracy metrics (Figs. [ [p) However, as shown in Figs. 2 and pp. model 0.9 is twice as far from annotators as they are from each other, while model 1.0 is twice as close, thus resulting in a discrepancy ratio performance gap of 400%.
again
Relative F1 does allow for evaluating the model by comparing it to annotator performance but the discrepancy ratio has several advantages. First, as illustrated in Fig. 2 the discrepancy ratio is insensitive to the number of annotators making it more robust metric to compare across different published studies. Second, the discrepancy ratio allows for a highly varying number of annotators per sample. This feature permits its use in scenarios where annotators have the ability skip samples and makes it a more practically viable solution than Relative F1. The presence of strict annotators (who skip hard samples) has effect the discrepancy ratio (Fig. 21)

---

Published as a conference paper at ICLR 2020
(8)
(b)
(d)
0.05
discrepancy
0.5
0.70
0.70
1:23
discrepancy
9.66
0.000
1.00
3.00
6.4
mose 10
055
model b9
model 1.4
model 1.0
annotator
mosel 8.8
0.45
model 0.3
model 0.35
model 0.3
model 0.35
02 044 24
0.8
10
0.2 0.4 0.5
0.5 1.5
0.40
30
15
25
02 044 0. 0.
label swap prosability
label swap probability
number of annotators
single annotator strictness
Figure 2: Comparison of the discrepancy ratio with baseline metrics the MNIST test set. Curves show mean value and 2-standard deviation bounds obtained from multiple simulations of the experi- ment. Panel (a) shows the model discrepancy (for three models) and annotator discrepancy. The red and blue points occur at label swap probabilities of 0.1 10/9 and 0.2 10/9, respectively, which correspond precisely to when the accuracies of the simulated human annotators equal that of the models (0.9 and 0.8, respectively). The discrepancy ratios (Eq. 4 for the same three models are shown in (b). The blue and red points correspond those shown (a). The shaded region denotes the area where the model is superior to the average human performance. Panels (c) and (d) show that
the discrepancy ratio is not sensitive to the number of annotators used, nor is subject to the strict annotator bias illustrated in Fig Td. Here we have used the same parameters as in Figs and Td.
Dataset
No. of Images
No. of Patients
Train
60,411
1,915
Validation
5,996
200
Test
17,659
600
Table 2: Training and Evaluation Datasets for PLAX Mcasuraility Classification Task.
4.2 ECHOCARDIOGRAPHII IMAGE CLASSIFICATION AND REGRESSION
An cchocardiogram is a common medical test that uses high-frequency sound waves (ultrasound) to visualize the structure of the heart. this work, we consider two important and ubiquitous tasks that are performed as part of virtually any echocardiogram (Lang et al., 2015).
PLAX measurability classification: The exam typically begins with acquiring a parasternal long axis (PLAX) view, which is used to get overview of general cardiac function, as well as to make specific measurements (e.g. fractional shortening (Lang et al. (015p, presence of pericardial effusion, etc). Before any measurement can be performed, however, clinician must first perform binary classification task and determine whether the image in question is sufficient quality to be used in the calculation (a similar binary classification screening step must be performed for most medical imaging tasks).
LVEF regression: The left ventricular ejection fraction (LVEF), which measures the fraction of blood the left ventricle expels with each heart cycle, is one of the key metrics used to quantify heart functionality. This quantity is often measured in the apical 4-chamber view of the heart by segmenting the left ventricular endocardial border at end-systole and end-diastole (see Appendix A).
Both tasks described above currently require the significant expertise of trained sonographer. We have developed deep neural network models to automate these tasks. The datasets for these tasks summarized in Tables 2
and All data collected using the Butterfly iQ ultrasound probe. Details of the data collection and annotations for both tasks are described in Appendix A.
Dataset
No. of Images
No. of Patients
Train
890,541
2,964
Validation
407,151
1,180
Test
149,992
322
Table 3: Training and Evaluation Datasets for LVEF Regression Task.
7

---

Published as a conference paper at ICLR 2020
(a)
(b)
(c)
Armotattr Disggtmmmnt 20.4%
discrepancy ratio
dicrepancy ratio
1.8
70
2.0
2.6
60
1.5
1.4
1.2
1.0
annotators
0.5
model
0.0
0.2
0.4
0.6 0.8
1.0
0.0
0.2
0.4
0.5 0.8
1.0
20 40
60
D0
150
120
140
average label
output threshald
patient index
Figure 3: Panel (a) shows that human experts exhibit a high amount of disagreement on which images meet the threshold for measurability (i. with perfect agreement all labels would be 0 or 1). This plot displays the normalized histogram of the average labels on the PLAX measurability classification dataset. The discrepancy ratio on the test set the output threshold of the network output is varied is shown in (b). Panel (c) shows the labels (blue) and model predictions (red) on the LVEF regression task test set, sorted by the mean LVEF label. Each patient was labeled by an average of 5 annotators. The black crosses signify the points on which the model predictions were artificially corrupted to create large errors.
4.2.1 EVALUATION USING DISCREPANCY RATIO
In order to explore the properties of the discrepancy ratio in real-world setting, we first consider the PLAX measurability classification dataset. The neural network used for this task is described in Appendix D. When evaluating the performance of our network using Naive Evaluation (where evaluate on a single annotator, chosen at random), we find that we achieve an accuracy of 77.8%, while the majority vote method gives accuracy of 88.0%. These numbers are quite poor by the standards of most machine learning applications. Indeed, we might otherwise conclude that the model performs at sub-standard level.
However, when we evaluate using the discrepancy ratio and consider the inter-annotator variability, a very different story emerges. As can be seen in the distribution of the average label (y² from Eq. 5) in Fig. 3a, there is significant disagreement (~~ 25% on average) among the annotators as to whether given image meets the threshold of measurability. While this level of disagrecment may seem high, is typical of that found in the medical applications literature (Wei et al. 2019 Hannun et al. 2019)
Fig. 3p shows the discrepancy ratio as a function of the threshold of the network output. Contrary to the impression suggested by using the conventional metrics, we find that for a wide range of thresholds the model dramatically exceeds the average human performance (as defined by A = 1). T shows the discrepancy ratio, with 95% confidence intervals, for the model as compared to those achieved by the individual annotators on the test set. Here we find that the discrepancy ratio has another nice property: it can be used to evaluate the performance of individual annotators. We find that there a significant spread in the annotator performance and that our model outperforms even the best annotator in the dataset.
4.2.2 DEPENDENCE ON 5
As described Section 2, the discrepancy ratio can be thought of as a "meta-metric" that evaluates the model performance for any desired measure of agreement 8. Here to explore this dependence on 5. The neural network we used described in Appendix D. We consider three choices of 8: the absolute difference:
the LVEF regression dataset
ly y'l, squared difference: 8 = (v y/)² and the hinge loss: = max (0, X ly I')). where a penalty is incurred only past a minimum deviation threshold [ As with the PLAX measurability classification task, find that our model exceeds the performance of even the best annotator in our dataset on all three measures of agreement, obtaining discrepancy ratios of 0.89, 0.78 and 0.67 respectively. In order to demonstrate the dependence on we then artificially corrupt the model so that it occasionally makes enrors that are far outside the range of those made by humans (Fig. 3e). How does the model fare with respect to human performance? depends on the type of errors one cares most about. When we set 5 to be the absolute difference, the discrepancy ratio is 0.95 and one may conclude that the model is performing well. the squared difference is used for 8 instead, the outliers receive higher cost and the discrepancy ratio becomes
We used À as clinically relevant measure of significant Fjection Fraction emror.
8

---

Published as a conference paper at ICLR 2020
(95% CI)
A1
(1.17 to 1.213)
A2
0,904 (0.875 to 0.932)
A3
0.907 (0.858 to 0.964)
(0.896
A5
1.105 (1.052 to 1.161)
0.959 (0.914 1.010)
A7
0.952 (0.923 to 0.984)
A8
1.131 (1 066 to 1.215)
0,932 (0
AIU
1.030 (0.983 to 1.073)
(0.973 1.049)
A12
1.069 (1 029 to 1.1)
Model
0.805 (0.786 to 0.820)
Table 4: The discrepancy ratio for the model and annotators (who have labeled at least 100 patients) on the PLAX measurability test set. Here each annotator was individually taken out of the dataset and treated as if they were the model for the purpose of calculating their discrepancy ratio. The confidence intervals patient. We used the validation set to choose the threshold corresponding to the lowest discrepancy ratio (as in Fig. 3b).
obtained using the bootstrap method where the sampling was performed by
1.02. When a hinge-loss used for 8, where deviations below considered acceptable, we observe that the performance is worse still (1.08).
5 LIMITATIONS
While our approach provides for simple and general way of quantifying model performance in the presence of label noise, it has several limitations. First, although the flexibility in choosing allows us to examine various aspects of the model performance (illustrated the example above), our metric needs to be applied with caution since it is only good as the underlying measure of agreement. Second, in situations when annotators are allowed to skip samples they feel unsure about, samples that get fewer labels may tend be more ambiguous. Since the discrepancy ratio is undefined for samples with fewer than two labels, this can cause the discrepancy ratio to overweight casier examples. Consequently, one should ensure that all samples in the domain of interest labeled by at least two annotators. Finally, the discrepancy ratio can yield misleading results when the labels are provided by poor annotators, though this is also true of any metric (e.g. the relative F1) that attempts to normalize model performance to account for annotator disagreement. To mitigate this, the annotators should be properly vetted as skilled for the task (see e.g. Hovy & Lavid (2010) or our vetting procedure in Appendix A) and allowed to skip when they are unsure of the label. In some situations annotators may adhere to several discrete "schools of thought" and hence their annotations may be more similar within each group than between groups. In these situations it may make sense to evaluate the discrepancy ratio independently within each group.
6 CONCLUSION
We present unified and principled approach to evaluating machine learning models when humans disagree on the truth. Our metric, the discrepancy ratio, is highly intuitive and can be adapted to virtually any task. We have shown that the discrepancy ratio has numerous desirable properties including insensitivity to annotator count and insensitivity to annotator/image overlap. At the same time, we have evaluated the shortcomings of alternative implementations and show that our method has numerous advantages that make it a convenient standard by which models can be evaluated in the presence of ambiguous ground truth labels. Finally, to demonstrate our method have trained deep neural networks on two real-world tasks from medical imaging and have used our metric to show that we achieve performance better than even the best annotators in our datasets.
ACKNOWLEDGMENTS
The authors would like thank Gal Koplewitz for his input and several helpful discussions.

---

Published as a conference paper at ICLR 2020
REFERENCES
Soyeon Ahn, Scong Ho Park, and Kyoung Ho Lec. How to demonstrate similarity by using non- inferiority and equivalence statistical testing radiology research. Radiology, 267(():328-338) 2013.
Douglas G Altman and Martin Bland. Measurement in medicine: the analysis of method comparison studies. Journal of the Royal Statistical Society: Series D (The Statistician), 32(3):307-317, 1983.
Dana Cobzas, Neil Birkbeck, Mark Schmidt, Martin Jagersand, and Albert Murtha. 3d variational brain tumor segmentation using a high dimensional feature set. IEEE 11th International Conference Computer Vision, 2007.
Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1) )):37, 1960.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Benot Frenay and Ata Kaban. A comprehensive introduction to label noise. ESANN 2014 proceedings, European Symposium Artificial Neural Networks, Computational Intelligence and Machine Learning, pp. 23-25, 2014.
Varun Gulshan, Lily Peng, Mare Coram, Martin C. Stumpe, Derek Wu, Arumachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, Ramasamy Kim, Rajiv Raman, Philip C. Nelson, Jessica L.. Mega, and Dale R. Webster. Development and validation of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. The Journal of the American Medical Association, 316:2402-24111 2016.
Awni Y. Hannun, Pranav Rajpurkar, Masoumeh Haghpanahi. Geoffrey Tison, Codie Bourn, Mintu P. Turakhia, and Andrew Y. Ng. Cardiologist-leve arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nature Medicine, 25:65-69, 69, 2019.
Joger Hansegard, Stig Urheim, Ketil Lunde, Siri Malm, and Stein Inge Rabben. Semi-automated quantification of left ventricular volumes and ejection fraction by real-time three-dimensional echocardiography. Cardiovascular Ultrasound, 7, 2009.
Victoria Hodge and Jim Austin, A survey of outlier detection methodologies. Artif. Intell. Rev., 22: 85-126, 2004.
Eduard Hovy and Julia Lavid. Towards a science of corpus annotation: A new methodological challenge for corpus linguistics. International Journal of Translation, 22, 2010.
Paul Jaccard. The distribution of the flora in the alpine zone. New Phytologist, 11:37-50, 1912.
Mohammad H. Jafari, Hany Girgis, Nathan Van Woudenberg, Zhibin Liao, Robert Rohling, Ken Gin, Purang Abolmaesumi, and Terasa Tsang. Automatic biplane left ventricular ejection fraction estimation with mobile point-of-care ultrasound using multi-task learning and adversarial training. International Journal of Computer Assisted Radiology and Surgery, 2019.
Lawrence Joseph, Theresa W. Gyorkos, and Louis Coupal. Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard. Am. J. Epidemiol., 141: 263-272, 1995.
Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S. Corrado, Lily Peng, and Dale R. Webster. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Opthamology. 125:1264-1277 1272, 2018.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuser Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint arXiv: 181 009822 2018.
10

---

Published as a conference paper at ICLR 2020
Roberto M Lang, Luigi P Badano, Victor Mor-Avi, Jonathan Afilalo, Anderson Armstrong, Laura Ernande, Frank A Flachskampf, Elyse Foster, Steven A Goldstein, Tatiana Kuznetsova, et al. Recommendations for cardiac chamber quantification by echocardiography in adults: update from the american society of echocardiography and the european association of cardiovascular imaging. European Heart Journal-Cardee Cardiovascular Imaging, 16(3):233-271 271, 2015.
Elizabeth Lazarus, Martha B Mainiero, Barbara Schepps, Susan L Koelliker, and Linda S Livingston. Bi-rads lexicon for us and mammography: interobserver variability and positive predictive value. Radiology, 239(2):385-391 2006.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://lannn lecun. com/exdb/mnist, 2, 2010.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: a loss correction approach. CVPR, pp. 2017.
Martin Rajchl, Matthew C.H. Lee, Ozan Oktay, Konstantinos Kamnitsas, Jonathan Passerat-Palmbac Wenjia Bai, Mellisa Damodaram, Mary A. Rutherford, Joseph V. Hajnal, Bernhard Kainz, and Daniel Rueckert. Deepeut: Object segmentation from bounding box annotations using convolu- tional neural networks. IEEE Transactions on Medical Imaging, 36, 2017.
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Robyn I.. Ball, Curtis Langlotz, Katie Shpanskaya, Matthew P. Lungren, and Andrew Y. Ng. Chexnet: Radiologist-leve pneumonia detection on chest x-rays with deep learning. 2017.
arXiv preprint
Pranav Rajpurkar, Jeremy Irvin, Aarti Bagul, Daisy Ding, Tony Duan, Hershel Mehta, Brandon Yang, Kaylie Zhu, Dillon Laird, Robyn L. Ball, Curtis Langlotz, Katie Shpanskaya, Matthew P. Lungren, and Andrew Y. Ng. Mura: Large dataset for abnormality detection in musculoskeletal radiographs. arXiv preprint arXiv: 11w..11....1111 2018a.
Pranay Rajpurkar, Jeremy Irvin, Robyn L. Ball, Kaylie Zhu, Brandon Yang. Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Bhavik N. Patel, Kristen W. Yeom, Katie Shpanskaya, Francis G. Blankenberg, Jayne Seekins, Timothy J. Amrhein, David A. Mong, Safwan S. Halabi, Evan J. Zucker, Andrew Y. Ng, and Matthew P. Lungren. Deep learning for chest radiograph diagnosis: A retrospective comparison of the chexnext algorithm to practicing radiologists. PLOS Medicine, 15, 2018b.
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint arXiv: 10694v3. 2017.
Andrew B Rosenkrantz, Ruth P Lim, Mershad Haghighi. Molly B Somberg, James S Babb, and Samir S Taneja. Comparison of interreader reproducibility of the prostate imaging reporting and data system and likert scales for evaluation of multiparametric prostate mri. American journal of roentgenology, 201(4):W112-611 2013.
Holger R. Roth, Chen Shen, Hirohisa Oda, Masahiro Oda, Yuichiro Hayashi, Kazunari Misawa, and Kensaku Mori. Deep learning and its application to medical image segmentation. arXiv preprint arXiv:1803.08691 08691v1.... 2018
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. International Conference on Learning Representations, 2015.
Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. NIPS, 2017.
11

---

Published as a conference paper at ICLR 2020
Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. Denoising neural machine translation training with trusted data and online data selection. Proceedings of the Third Conference Machine Translation, 1:133-143, 2018.
Jason W Wei, Laura J Tafe, Yevgeniy A Linnik, Louis Vaickus, Naofumi Tomita, and Saeed Hassanpour. Pathologist- leveel classification of histologic patterns on resected lung adenocarcinoma slides with deep neural networks. Scientific reports, 9()):335,, 2019.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In Advances in neural information processing systems, pp. 487-495, 2014.
Xingquan Zhu and Xindong Wu. Class noise vs. attribute noise: A quantitative study of their impacts. Artificial Intelligence Review, 22:177-210, 2004.
12

---

Published as a conference paper at ICLR 2020
(a)
(b)
intraventricular septum
aortic valve
left ventricular posterior wall
mitral valve
Figure S1: Examples of images deemed measurable (a) and not measurable (b) by majority vote of annotators. Several important features characteristic of measurable image are noted in (a). These same features are obscured or entirely absent in (b).
A DATA COLLECTION, DATASETS AND ANNOTATIONS
In this section describe the data collection, dataset splits and annotation procedure for the tasks presented in Section 4.2 of the main text.
A.1 DATA COLLECTION
For model development and evaluation, images were collected in several hospitals in the United States patients presenting for standard echocardiogram exam. All data was collected using the Butterfly iQ probe. For the PLAX measurability classification dataset we have additionally included images taken on healthy adults outside the hospital setting (approximately 10% of the dataset).
A.2 PLAX MEASURABILITY CLASSIFICATION TASK
In order to collect labels, echocardiogram images were presented to 14 professional, US-licensed sonographers. The images in the received an average of labels per image. The sonographers were given the following task:
Mark as MEASURABLE the frames that satisfy ALL the following criteria:
The frame shows a parasternal long axis (PLAX) view of diagnostic quality.
The frame can be used for measurement of the left ventricular internal diameter (LVID), shown in the diagram. Note that the LVID can be measurable irrespective of where the heart is in its cycle.
Mark all other frames as NOT MEASURABLE.
The above prompt refers to a diagram showing a canonical image PLAX view. Examples of images deemed measurable and not measurable by the majority of annotators shown in Fig. ST
13

---

Published as a conference paper at ICLR 2020
(a)
(b)
ED segmentation mask
ES segmentation mask
Figure S2: example LVEF calculation in the apical 4-chamber view. Panel (a) shows the end-diastole frame with the overlaid segmentation mask while panel (b) shows the corresponding end-systole frame within the same heart cycle. The calculated LVEF from these segmentation masks is 66%
A.3 LVEF REGRESSION TASK
Here collected three separate types of labels:
1. Segmentation Masks: annotators asked to segment the endocardial border of the left ventricle.
2. Measurability: annotators were asked to classify frames into measurable/non-meet meassurabbe classes using a procedure virtually identical that used in the PLAX measurability classifi- cation task above.
3. LVEF labels: annotators were asked to provide LVEF labels by segmenting the endocardial border the end-systole and end-diastole frames.
Since the model predicts segmentations and measurability scores only as intermediate states (see Appendix D for more details), they are not relevant for the evaluation framework presented in this work and we do not consider these predictions further. Instead we focus on the LVEF regression predictions. For this task, short echocardiogram video clips containing several heart cycles were presented to 19 professional, US-licensed sonographers. The videos in the test set received an average of 5 labels per video. The sonographers given the following task:
14

---

Published as a conference paper at ICLR 2020
Apical Endocardial contours should be labeled according to ASE guidelines:
Identify heart cycle within the cine.
Navigate to the ED frame and trace the endocardial border of the left ventricle.
Navigate to the ES frame and trace the endocardial border of the left ventricle.
Make sure the ED frame is the left of the ES frame but part of the same heart cycle.
Observe the calculated EF just above the cine. If this EF score looks cor- rect, hit Submit. Otherwise, adjust the ED and ES frames accordingly.
The above prompt refers to "ASE guidelines", which are published consensus recommendations and can be found at Lanl al. 2015). An example of LVEF prediction shown Fig S2 where the model segments the end-systole and end-diastole frames and calculates the LVEF via the Modified Simpson's s Method (Lang et al. 2015).
A.4 ANNOTATOR INCLUSION CRITERIA
We relied two senior cardiac sonographers (each having than 15 years of experience) source annotators. For both tasks, each annotator was interviewed and given an onboarding assignment. Annotators that were deemed by our sonographers to comply with ASE standards were then cleared to label images. During the first several weeks of label collection, our sonographers conducted several detailed reviews of each annotator's images. Examples which our sonographers disagreed with the assessment of the annotator were discussed with the annotator in live sessions to ensure that the disagreements resulted in genuine differences of medical opinion, rather than annotation errors. In addition, the disagreement metrics for each annotator (as calculated Section 4.2. 1) were tracked over time and were used schedule additional review sessions with the annotators, as necessary.
B MPD AND ANNOTATOR DISCREPANCY USING SQUARED-ERROR AGREEMENT FUNCTION
As a preliminary result that is used in Section 3.3, we show that given any two sets of labels the MPD is given by the expression
y (y1, 12, ./., }n and y {v(
((v,3") - //- o
where y E Us and of ( Ek=1 v2) 222 the mean and variance of yk. respectively.
Proof.
n'
(yk Me.e )2
((),3)) =
nn!
72
7
Uk
n
Uk
2/1
yk!
nn'
k=1
k=
k=1
1
n'nioo + )²) + nn' non' + 122)
2nn' 2nn'yz/ w/
nn'
(y 2/ + 0 oz
15

---

Published as a conference paper at ICLR 2020
As a special case, when the labels can only take on the values 0 and 1, the variance is fully determined by the mean:
oy y y²
and the MPD simplifies to
(''')' = y y 2yyy
Using this result and the definition in Eq. the annotator discrepancy for a single sample is then given by:
- Vン. 2 A' 1)
a²
Σ
j-'=1 1サ8
A²
A
2(A 1)
A - 1)
jJ'=1
20 A'
= 2y y²) +
y
-
where og. is the variance of the labels y'. Here have used the fact that
1
1
u,un (A'y))²
(2)) j=1
i===1
For the special case where each annotator provided more than single label per image, the above relationship between the mean and variance can be used to further simplify the annotator discrepancy
2
A' A og
C ALTERNATIVE FORMULATIONS OF THE DISCREPANCY RATIO
In constructing the discrepancy ratio metric, we made several seemingly arbitrary design choices. In this section discuss the reasons for these decisions and why the alternative formulations problematic.
C.I THE ORDER OF THE AVERAGES OVER ANNOTATIONS AND SAMPLES
Consider an alternative definition for the model discrepancy, where we swap the order of the averages over samples and annotations:
1 N
(mmm, 27))
NA
j=1
Viz0
16

---

Published as a conference paper at ICLR 2020
Here A is the total number of annotators in the dataset and the only those for which the jth annotator has labeled it at least once (i.e. yj * Ø). Conceptually, this expression calculates discrepancy score for each model/annotator pair, and then averages them together to arrive at an aggregated measure of agreement.
samples has to go
An analogous alternative definition be constructed for the annotator discrepancy:
A
203 NA(A 1)
d'
j+'=1 jサ D Di
As in the case of the model discrepancy, this expression computes a discrepancy for each annotator pair and then averages all the pairs to yield an aggregate measure of annotator agreement. The methods presented in Wei et al. (2019;; Rajpurkar et al. (2017) use essentially these definitions, though they do not compare them using a ratio.
When all annotators label all samples, these definitions are perfectly valid, and are in fact equivalent to y and a. However, in most real-world scenarios, data come from disparate sources (e.g. different hospitals with different annotators) and it is not practical to ensure that the exact same set of annotators labeled all the samples. Worse still, in many domains it is necessary to allow human annotators to voluntarily skip samples when they feel there is not enough information to provide accurate label. In such a scenario, formulating the model and annotator discrepancies using the alternative definitions above allows annotator to improve their own discrepancy score by simply being strict and only annotating "easy" samples. This will generally result in a bias being introduced into the discrepancy ratio. To illustrate this point, consider the following toy example:
Example C.1 A dataset contains one "easy" image (on which the MPD between any two annotators is 5,) and one "hard" image (on which the MPD between any two annotators is 6τ). Similarly, a machine learning model gets an MPD of 6% on the easy image and 8/, on the hard image. Three annotators (Alice, Bob and Charlie) are asked to label the data. Alice and Bob label both images, but Charlie wants to improve his discrepancy and only labels the easy image. Using the original definitions of a and μ. the discrepancy ratio is
1r 66'6''+ +
8% +
a 6,)/2
8. 6
Note that the value of A would have been exactly the same without Charlie. On the other hand, using the alternative definition the discrepancy ratio becomes biased and is given by
1 (2(8) + 6,/22
28' + 8%
a' ((5. + +
(50. + 5n./22
In the special case where the annotators and model are in perfect agreement on the easy image (8. 8' 0), Charlic was able to double the discrepancy ratio by simply skipping the difficult image:
28
A'
2A.
should be noted that even we weigh the average of annotators by the inverse of the number of annotations they provided, the bias does not go away, whereas the discrepancy ratio, as defined in the main text, is completely insensitive to this effect. The effect illustrated in the example above is not limited to the presence of an adversarial annotator, but can creep into the data whenever there exist any systematic differences how the data got partitioned among annotators, which in practice may be challenging impossible control.
17

---

Published as a conference paper at ICLR 2020
C.2 COMPUTING PAIRWISE DEVIATIONS VS. COMPARISONS THE MEAN
In the definition of the annotator discrepancy, we compute the MPD over all pairs of annotators. A seemingly valid alternative would be to instead compare each annotator to the mean of all the other annotations. In this alternative definition, the annotator discrepancy is given by the expression
A
O0 w/)
a"
NA
j=
where
Σ J==1 8'
w
A -1
Here, w is scalar that represents the average of all annotations for the ith sample, leaving out the jth annotator. We can analogously construct an alternative definition for the model discrepancy:
N
(mmm,
n²
NA
Note that we cannot simply compare the model predictions to the average of all the annotations since then the number of annotators that go into the average for μ" would be one higher than that used in a", which would artificially lower the discrepancy ratio. The methods presented in Hannun et al. Rajpurkar et al. (201 Sa b) are similar to this averaging approach.
While μ² and a may seem like valid formulations for the model and annotator discrepancies, they lead to a major problem: The discrepancy ratio becomes dependent on how many annotators are used in the study.
To why this is, notice that w is an average over A' annotators and will generally scale 1/v A.. This implies that both n² and a" will get smaller as the number of annotators is increased. They will generally scale at different rates, however, depending on the distribution the model predictions and annotator labels, and therefore will make the ratio dependent on the number of annotators (for specific example, see Section 4 of the main text).
D ALGORITHM DEVELOPMENT AND TRAINING
The deep learning algorithm used for the PLAX measurability classification task 10-layer convolutional neural network, where each 2-layer block subsamples the inputs by a factor of 2. After the final layer we apply a global average pooling, followed by a sigmoid function that thresholded at particular level (see Section 4). We trained the network de novo using the Adam optimizer with learning rate of 0.00005 and mini batch size of 32.
The algorithm used for estimating the LVEF employs a U-Net style network tasked with segmenting the left ventricular endocardial border. The network consists of 6-layer encoder followed by 6-layer decoder where downsampling and upsampling by factor of 2 occur at each layer, respectively. We trained the network novo using the Adam optimizer with a learning rate 0.0001 and mini batch size of 64. The network contains an additional output head that predicts the measurability of the input image. The per-frame segmentations and measurability values are then used to find the end-systole and ind-diastole frames, which are in turn used calculate the LVEF via the Modified Simpson's Method (Lang et al. 2015). The per- fermframe measurability scores are aggregated to produce a quality score for each LVEE The model only outputs LVEF predictions for which the predicted quality is
18

---

Published as a conference paper at ICLR 2020
above predefined threshold.
E EXTENSIONS OF THE DISCREPANCY RATIO
When there exists an imbalance of labels the dataset, the discrepancy ratio metric can be stratified to give rebalanced measure of agreement.
For example, a classification task where the class labels are imbalanced, we can define the meen--er-cllss discrepancy ratio by separately computing A for each class and averaging the results together (in this case, the stratification into different classes can be determined by majority vote).
19


Reviews:
[{"review_id": "rJxRqQToFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors presented an interesting paper which tries to solve a practically important question. Biomedical and tech industries usually hire human labelers for machine learning tasks, whose labels are usually noisy. Therefore, it is important to have a metric that can distinguish the performance of models with noisy labels.\n\nIn this paper, the authors proposed to measure the model performance based on the ratio of the discrepancy between the model prediction and the labeler, with the discrepancy between the labelers. The authors showed that in binary classification settings, their proposed metric can be reduced to simple to analyze quantities. The authors demonstrated the performance of their proposal in synthetic data as well in two real-world medical image datasets.\n\nOverall, the paper is well motivated with a reasonable amount of novelty. The numerical experiments are well conducted, but I am not totally convinced by their results.\n\nDetailed comments:\nSignificance: The paper is trying to address a practically important issue in machine learning. \nNovelty: The mathematical formulation of the metric consists of simple sums and averages, which in itself is not novel. However, the authors' choice of using such formulations to assess model performance is novel.\nClarity: The authors could simplify their notations, and could periodically remind the readers about the notations. For example, I was stuck by the sigma^2 notation in (6) (undefined) and the m notation in (7) defined in the past page in small texts. It would help me a lot if the authors reminded me the definition of those notations when they appeared.\nNumerical Experiments: This is my biggest complain. I wasn't able to conclude that the authors proposal is better than majority voting based on Figures 1b and 2b. To me they look qualitatively the same. Is there any reason that the discrepancy ratio is superior to the majority vote? In addition, I didn't see whether the curves in Figures 1 and 2 are an average of numerous simulated samples. if not, the authors should use the average to mitigate randomness; and if the curves are averages, then the distribution of the metrics should be described, because I think ultimately, we don't care about whether the average black curve is above the average red curve, but the chance of black curve being above the red curve. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJxfhzwjtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed to evaluate model performance when the ground truth labels were not available and noisy labels provided by multiple uncertain experts were provided instead. The proposed evaluation metric, called discrepancy ratio, is defined as the ratio between the average model-annotator discrepancy and the average annotator-annotator discrepancy. It can be applied to compare 1) the relative performance of different models; and 2) the relative performance of average annotators and the model. Experimental results on the MNIST classification task showed the proposed metric can compare model performance under different noise levels and is robust w.r.t. the number of annotators and a single annotator's strictness. Experimental results on a real-world medical image classification was also presented. \n\nThis work is well motivated. In real-world applications, such as medical image classifications, obtaining ground-truth labels is difficult or even infeasible, only multiple noisy labels are available to evaluate the model performance. Therefore, the setup of this problem seems novel (besides Rajpurkar et al. 2018).\n\nIn the experimental results on MNIST classification, for Naive Evaluation and Majority Vote, as the author pointed out, the gap between three models (model 1.0, model 0.9, model 0.8) becomes narrower when the label swap probability increases, indicating it's becoming more difficult to differentiate model improvements. However, this is also true for the proposed discrepancy ratio, as is shown in Figure 2(b). Therefore, the discrepancy ratio didn't overcome those \"drawbacks\" associated with these two baselines, correct? In other words, in the high-noise regime, it would be natural to observe narrower gaps between those models. \n\nIn MNIST classification, the performances gap of those three models evaluated by Naive Evaluation and Majority Vote didn't decrease rapidly as the noisy level increases. This is consistent with the constant gap between their ground truth accuracies (gap=10% between 100%, 90%, 80%). However, for the proposed discrepancy ratio, the gap between these three models decreases rapidly when the noise level increases from 0 to 0.4. This is an undesirable behavior since it over-estimate the performance gap in the low-noise regime. Since most real-world datasets could belong to this low-noise regime, this rapid change of performance gaps seems problematic.\n\nIn section 4.1.4, the claim that \"It is in this regime that the model performance can be said to be better than the average human performance\" seems questionable. When the ratio is less than 1, it simply means the annotator-model agreement is larger than the annotator-annotator agreement. What's the definition of the \"average human performance\" in this context?\n\nIn the work, multiple assumptions were made regarding different annotators. First, multiple annotators are assumed to be conditionally independent given the ground-truth label. Second, each annotator should provide high-quality labels. Third, different experts should have similar expertise. However, in medical applications, any of these assumptions could be violated. For example,1) multiple experts could provide similar labels given their similar knowledge background. It's more likely to observe multiple groups of experts, where experts in the same group are similar. 2), experts could provide random labels when they are extremely uncertain 3) different experts could have varying levels of expertise. Could the author provide some ideas on how to relax some of these assumptions?\n\nComments after reading authors' reply\n-----------------------------------------------------\nI would like to thank the authors for their detailed reply to my questions. The key novelty of this work, i.e., evaluating model performance (in terms of comparing different models and model vs average annotator performance) in the presence of noisy labels, should be appreciated. I'll raise my score to 6: Weak Accept. In the future work, it would be interesting to apply the proposed metric (or develop new metrics) to study whether any of the assumptions (conditional independence, reasonably high accuracy of each individual expert, different expert accuracies) can be relaxed.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJg_rIUitr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a novel metric \"discrepancy ratio\" for evaluating the performance of a model where the ground truth for each data point comes from many expert-yet-imperfect annotators. The authors suggested that this problem has remained largely unexplored. The proposed metric is intuitive and is easy-to-use. The authors suggested that this metric can be used for many applications. For example, to evaluate if a model is better than the annotators on average, to evaluate the annotator, to compare between models. \n\n========================================================\nComments on clarity:\n\nThe writing is good overall and I am able to understand the contribution and the key idea of this paper.\n\nQuestions and comments on clarity:\n\n1. While MPD is defined with an argument (Y, Y'), the definitions annotator discrepancy, model discrepancy and discrepancy ratio ignore the inputs of the function. It might be better to keep the arguments of the function for clarity. When people want to adopt the discrepancy ratio in their work and there are many ratios to discuss, then the proposed definition can be used conveniently. \n\n2. In Eq. (5) \\sigma seems to be undefined. I believe it is a variance but it is kinder to readers to explicitly state it (I found the authors explicitly stated it in Appendix B).\n\n3. In Eq. (8), is it intentional to have $i$ exists over m and y as written in the paper?\n\n4. Does the discussion of the simplicity of the discrepancy ratio hold for other \\delta, i.e., not the squared loss?\n\n5. In Section 4.1 (experiment with MNIST), did the authors binarize the data or simply discuss the multi-class problem? Because this follows section 3.3 which dedicates the whole page to discuss binary classification. It may be a bit sudden to move to multi-class classification and an additional sentence to clarify this might help.\n\n6. I could not understand what is \"single annotator strictness in Figures 1d and 2d. It is not explained if I did not miss it.\n\n7. I think we should not resize the figure/table caption. The text size of captions are much smaller than the main text.\n\n8. Is PLAX binary classification? (I understood as yes from Figure (3a) but I think it's also nice to clarify it to avoid confusion (also for MNIST)).\n\n9. What is the y-axis for figure 3a?\n\n10. I think reporting the performance of the mean-per-class discrepancy ratio without defining it in the main text (it is defined in a natural language (English) in Appendix E) makes the main body of the paper not self-contained. I strongly suggest to either (1) remove mean-per-class discrepancy ratio entirely from the main body or (2) define it mathematically in the main body. Also, I don't see reporting mean-per-class discrepancy ratio adds values to a paper for current experiments because the reported mean-per-class ratio is very similar to that of discrepancy ratio (Figures 3b (almost overlap) and Table 2 (same trend and the values are very close to the discrepancy ratio for all cases)).\n\n11. Can we use F1 to evaluate whether a model is better than the average human performance?\n\nMinor points on clarity, which may just be a personal preference of a reviewer:\n\n1. The authors may consider using \\begin{definition} when defining three key values: the mean pairwise deviation (MPD), annotator discrepancy, and model discrepancy. So the readers can easily look up when getting lost in definitions. And it may help to make a definition more precise (e.g., given label Y, data points, MPD is defined as follows).\n\n2. Eqs (1), (2), (3) may be easier to understand if we put a denominator out of the summation as much as we can. For example, for Eq. 2, N can be outside of the two sums, A^i(A^i-1) can be between the two sums. \n\n========================================================\nSignificance: \nThe problem that this paper addresses is highly important. I like the idea and I am convinced that discrepancy ratio is potentially useful for the community. And it also opens many possibilities to analyze the discrepancy ratio and to find potential applications for it.  \n\n========================================================\nOther comments:\n\nOn the limitation:\nI am satisfied that the authors clearly discuss the limitations of the proposed method and I agree with the discussion.\n\nOn the experiments: \n1. Why we use model 1.0, 0.95, 0.9 for Figures 1c, 1d, 2c, 2d but 1.0, 0.9, 0.8 for Figures 1a, 1b, 2a, 2b? Perhaps full results with many models should also be reported for completeness.\n\n2. Isn't it more natural to show a graph that has x-axis as label swap probability in the same figure for comparison? For example, Figure 1 only report performance with respect to each measure (Naive Evaluation, Majority Vote, Relative F1, Discrepancy ratio) with respect to label swap probability. We can fix the number of annotators to 9 for relative F1 and discrepancy ratio, to validate whether it also becomes increasingly difficult to differentiate model improvements or not for them. Then after suggesting that Naive evaluation is not good and other metrics look fine. Majority vote cannot evaluate that the model is better than human average performance, which we want to know. Then Figure 2 highlights the contribution of the discrepancy ratio and shows that it outperforms relative F1 in some perspectives.\n\n3. For the MNIST benchmark dataset, it is possible to show the performance on the test ground truth too. And I think it is interesting to compare different models with the same label noise rate and confirm that discrepancy ratio is successful to evaluate the best model with respect to the ground truth accuracy. It would be very impressive to see if the best model with respect to the discrepancy ratio can successfully be the best also on the ground truth test set.\n\nOn space usage:\nI feel there are many important things to be included in the main body but it is instead in the appendix. On the other hand, I think there are several parts that can be shortened in the main body. And it seems that the binary classification example (3.3) also consumed a lot of space. The derivation may be omitted and make it a Theorem and then let's suggest the readers to read the appendix for the proof if they are interested. Then we can have more space to add important components in the main body of the paper.\n\n========================================================\nDecision:\nAlthough there are issues on the clarity of the paper, I believe these issues are not too difficult to fix in the final version. The idea is interesting and potentially impactful. For these reasons, I vote a weak accept for this paper.\n\n\nUpdate: I would like to thank the authors for the clarification. I have read the rebuttal.\nI read through the updated version of the paper. I believe that although there is still room for improvement, the contribution of this paper is sufficient as an important step towards solving this difficult yet highly relevant problem. This paper also stimulates several future directions. The authors clearly stated the limitations of the paper, which is highly useful for researchers to improve this work or find a better solution.  The clarity of the paper is also improved. For these reasons, I increased my score from weak accept to accept.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles an interesting problem: "How should we evaluate models when the test data contains noisy labels?". This is a particularly relevant question in the medical imaging domain where expert annotators often disagree with each other. The paper proposes a new metric "discrepancy ratio" which computes the ratio how often the model disagrees with humans to how often humans disagree with each other. The paper shows that under certain noise models for the human annotations the discrepancy ratio can exactly determine when a model is more accurate than humans, whereas commonly used baselines such as comparing with the majority vote do not have this property. Reviewers were satisfied with the author rebuttal, particularly with the clarification that the goal of the metric is to accurately determine when model performance exceeds that of human annotators, and not to better rank models. The metric should be quite useful, assuming users are cautious of the limitations described by the authors.
Decision: Accept (Poster)