Discrepancy Ratio: Evaluating Model Performance When Even Experts Disagree on the Truth

1 INTRODUCTION . The canonical supervised machine learning paradigm assumes the presence of both inputs and their corresponding , unambiguous outputs . In practice , the vast majority of datasets exhibit some degree of label noise . Acknowledgement of this fact has resulted in the development of algorithms tailored to training models in the presence of noisy ground truth labels ( Sukhbaatar et al. , 2015 ; Frenay & Kaban , 2014 ; Zhu & Wu , 2004 ) . These approaches include cleansing noisy labels before training begins ( Hodge & Austin , 2004 ) , making the training process robust to noise ( Patrini et al. , 2017 ; Vahdat , 2017 ; Rolnick et al. , 2017 ) and actively modeling label noise ( Sukhbaatar et al. , 2015 ; Joseph et al. , 1995 ) . These methods almost exclusively address only the problem of noisy labels during training and tend to assume that at least some noise-free ground truth is available at test time in order to properly evaluate different approaches . Indeed , without an unambiguous yard-stick , how can competing schemes possibly be evaluated in a standardized manner ? There exist many domains , such as medical imaging ( Rosenkrantz et al. , 2013 ; Lazarus et al. , 2006 ) and machine translation ( Wang et al. , 2018 ) , in which access to unambiguous ground truth , even at evaluation time , is not practically achievable . In these domains not only do annotators disagree with each other , but they often even disagree with themselves ( Gulshan et al. , 2016 ; Hansegard et al. , 2009 ) . This , however , does not imply that the annotators are not trustworthy , but rather that these tasks are difficult even for domain experts . Datasets in these regimes are , in practice , quite different from those typically used in machine learning research ( which are usually geared towards content that can be easily curated by lay people ) . As a consequence , studies utilizing public data often resort to simulating label noise . Indeed , a theoretical understanding of evaluation in the presence of ambiguous test data has remained largely unexplored . In this work , we propose a new framework for evaluating the performance of machine learning models in the presence of high label noise . Our main contributions are as follows : 1 . We propose a novel , task-independent metric , the discrepancy ratio , that allows us to a ) quantify model performance , taking into account how much humans disagree with each other , and b ) set a precise threshold at which the model performance can be said to exceed the average human performance . 2 . We perform a detailed analysis of the discrepancy ratio for the special case of binary classification and demonstrate that it can be evaluated in terms of simple , closed-form expressions that depend only on aggregate statistics ( mean and variance ) of the annotations and not on any individual annotation . 3 . We demonstrate the effectiveness of our approach on models that we trained on two important medical imaging tasks with real-world clinical data and show that it reveals that our models not only significantly exceed the average human performance , but even beat the best human experts in our datasets . 2 RELATED WORK . There are several well-known strategies for dealing with ambiguous or noisy evaluation labels . The most popular approach is to simply use a single annotator ( Jafari et al. , 2019 ; Rajchl et al. , 2017 ; Cobzas et al. , 2007 ; Roth et al. , 2018 ) . An approach taken by many popular , large-scale datasets , such as ImageNet ( Deng et al. , 2009 ) , Places ( Zhou et al. , 2014 ) , COCO ( Lin et al. , 2014 ) and Open Images V4 ( Kuznetsova et al. , 2018 ) , is to take the average or majority vote . These efforts typically employ crowdsourcing in some fashion , since the labeling of this data does not require expert-level knowledge . While majority vote reduces label noise , in domains that require specialized knowledge it is often not practical to get enough experts that the noise is removed to the point where the annotations can be assumed to be “ correct ” ( in the sense that MNIST annotations are assumed to be correct ) . A related approach obtains a ground-truth dataset by enforcing a resolution criterion on labels where annotators disagree . This criterion can , for example , involve a live session where annotators deliberate and come to a consensus or an adjudication process where a third party ( usually a more senior , expert annotator ) is tasked with resolving disagreements ( Krause et al. , 2018 ; Hannun et al. , 2019 ; Rajpurkar et al. , 2018b ) . This approach may be effective at removing erroneous labels ( reducing variance ) at the expense of annotator independence ( increasing bias ) . The techniques outlined above fall short in providing a unified method that can both a ) be compared across different studies/tasks and b ) answer the essential question of how the model performance relates to an average human annotator ( i.e . how a prototypical human currently performs at a task ) . In developing high-stakes , real-world machine learning systems , the goal of evaluating a model is often to determine whether performance is safely substitutable for individual human behavior rather than comparing to an idealized oracle . Several studies have compared model performance with inter-annotator variability . Some have visualized model performance and inter-annotator variability ( e.g . via ROC curves ( Krause et al. , 2018 ; Gulshan et al. , 2016 ) or Bland-Altman plots ( Hansegard et al. , 2009 ; Altman & Bland , 1983 ) ) without providing an explicit metric . Several works have constructed a metric by comparing the model to the average annotator performance , though each has done so in a completely different way ( Wei et al. , 2019 ; Hannun et al. , 2019 ; Rajpurkar et al. , 2018a ; b ; 2017 ) . These metrics , however , generally depend on the number of annotators used in the study and/or require that all annotators label the entire dataset ( see Section 4 and Appendix C for more details ) . Moreover , the metrics are task-specific and can not be interpreted outside the context of a particular study . 3 THE DISCREPANCY RATIO METRIC . In this section , we define the discrepancy ratio and show that it has numerous desirable properties that overcome the limitations of prior art detailed in Section 2 . 3.1 DEFINITIONS . Definition 1 . For any two arbitrary sets of labels Y = { y1 , ... yn } and Y ′ = { y′1 , ... y′n′ } , the mean pairwise deviation ( MPD ) is defined by : ψ ( Y , Y ′ ) = ∑ y∈Y ∑ y′∈Y ′ δ ( y , y′ ) |Y||Y ′| , ( 1 ) where δ can be chosen to correspond to any measure of agreement appropriate to the task in question ( e.g . squared error , absolute error , Jaccard Index ( Jaccard , 1912 ) , Cohen ’ s kappa ( Cohen , 1960 ) ) . Consider a dataset of N samples where we have a prediction mi , for each sample i , by the model we are trying to evaluate . Further assume that each sample is labeled by at least two annotators and that each annotator may label the same sample multiple times . Formally , let Yij = { yijk } be the set of annotations provided by annotator j on sample i where k indexes the annotation number . Finally , let Ai indicate the number of annotators that have annotated sample i . Definition 2 . We define the annotator discrepancy , which quantifies the average annotator disagreement , to be the average of the MPD over all annotator pairs and all samples in the dataset : α ( { Yij } ) = 1 N N∑ i=1 1 Ai ( Ai − 1 ) Ai∑ j , j′=1 j′ 6=j ψ ( Yij , Yij′ ) . ( 2 ) Without access to unambiguous ground-truth labels , a natural way to define human performance is by comparing an annotator ’ s labels to those provided by other humans on the same samples . Consequently , we use the annotator discrepancy as a measure of average human performance . Definition 3 . Similarly , the model discrepancy is defined to be µ ( mi , { Yij } ) = 1 N N∑ i=1 1 Ai Ai∑ j=1 ψ ( mi , Yij ) . ( 3 ) Definition 4 . We define the discrepancy ratio to be the ratio of the model and annotator discrepancies : ∆ = µ α . ( 4 ) Intuitively , this metric is the ratio of the average model-annotator disagreement and the annotatorannotator disagreement . For example , a discrepancy ratio of 0.8 would mean that the model is 20 % closer to the annotators than they are to each other . When ∆ is above ( below ) 1 , the model performance can be said to be worse ( better ) than the average human performance . Since the measure of agreement δ is arbitrary , the discrepancy ratio can be thought of as a “ meta-metric ” that reports how the model performs ( relative to humans ) with respect to that particular measure of agreement . The dependence of the discrepancy ratio on the choice of δ is further explored in Section 4.2.2 . 3.2 PROPERTIES . The discrepancy ratio metric , as formulated above , has several desirable properties : Dimensionless : Dimensionless quantity whose meaning is invariant across different task categories ( regression , classification , segmentation etc . ) and whose value is interpretable and highly intuitive . Minimal dataset coverage requirements : The discrepancy ratio does not require that all annotators label all samples ( see Fig . 2 and Appendix C ) , which is often not practical in real-world situations , 1 0 0 , 1 0 , 0 ... 1 , 0 , 1. but only that each sample is labeled by at least two annotators ( two annotators are needed since it is meaningless to measure the annotator disagreement with only a single annotator ) . Insensitive to number of labels per-sample : The discrepancy ratio is expressed in terms of averages over all model/annotation pairs and therefore does not depend on how many annotators are used ( see Fig . 2 ) , nor how many times each annotator labels each image . This property makes it convenient to compare the discrepancy ratio across studies . Since the discrepancy ratio is formulated as a relative error that the model makes over that of the annotators , it can be naturally incorporated as a test statistic in a non-inferiority framework ( Ahn et al. , 2013 ) , often used in medical research . The goal of non-inferiority testing is to show that a new method is not statistically worse than an existing standard , up to some chosen equivalence bound . For the discrepancy ratio , we can choose the equivalence bound to correspond to an acceptable fraction of the error that the model is allowed to make on top of the one that annotators already make ( e.g . setting the equivalence bound to be 0.1 can be interpreted as saying that the model can make an error no more than 10 % larger than the ones humans typically make ) . Additional properties of the discrepancy ratio , as well as comparisons to alternative formulations , are discussed in detail in Appendix C and E .

Summary: This paper proposed to evaluate model performance when the ground truth labels were not available and noisy labels provided by multiple uncertain experts were provided instead. The proposed evaluation metric, called discrepancy ratio, is defined as the ratio between the average model-annotator discrepancy and the average annotator-annotator discrepancy. It can be applied to compare 1) the relative performance of different models; and 2) the relative performance of average annotators and the model. Experimental results on the MNIST classification task showed the proposed metric can compare model performance under different noise levels and is robust w.r.t. the number of annotators and a single annotator's strictness. Experimental results on a real-world medical image classification was also presented.