Discrepancy Ratio: Evaluating Model Performance When Even Experts Disagree on the Truth

1 INTRODUCTION . The canonical supervised machine learning paradigm assumes the presence of both inputs and their corresponding , unambiguous outputs . In practice , the vast majority of datasets exhibit some degree of label noise . Acknowledgement of this fact has resulted in the development of algorithms tailored to training models in the presence of noisy ground truth labels ( Sukhbaatar et al. , 2015 ; Frenay & Kaban , 2014 ; Zhu & Wu , 2004 ) . These approaches include cleansing noisy labels before training begins ( Hodge & Austin , 2004 ) , making the training process robust to noise ( Patrini et al. , 2017 ; Vahdat , 2017 ; Rolnick et al. , 2017 ) and actively modeling label noise ( Sukhbaatar et al. , 2015 ; Joseph et al. , 1995 ) . These methods almost exclusively address only the problem of noisy labels during training and tend to assume that at least some noise-free ground truth is available at test time in order to properly evaluate different approaches . Indeed , without an unambiguous yard-stick , how can competing schemes possibly be evaluated in a standardized manner ? There exist many domains , such as medical imaging ( Rosenkrantz et al. , 2013 ; Lazarus et al. , 2006 ) and machine translation ( Wang et al. , 2018 ) , in which access to unambiguous ground truth , even at evaluation time , is not practically achievable . In these domains not only do annotators disagree with each other , but they often even disagree with themselves ( Gulshan et al. , 2016 ; Hansegard et al. , 2009 ) . This , however , does not imply that the annotators are not trustworthy , but rather that these tasks are difficult even for domain experts . Datasets in these regimes are , in practice , quite different from those typically used in machine learning research ( which are usually geared towards content that can be easily curated by lay people ) . As a consequence , studies utilizing public data often resort to simulating label noise . Indeed , a theoretical understanding of evaluation in the presence of ambiguous test data has remained largely unexplored . In this work , we propose a new framework for evaluating the performance of machine learning models in the presence of high label noise . Our main contributions are as follows : 1 . We propose a novel , task-independent metric , the discrepancy ratio , that allows us to a ) quantify model performance , taking into account how much humans disagree with each other , and b ) set a precise threshold at which the model performance can be said to exceed the average human performance . 2 . We perform a detailed analysis of the discrepancy ratio for the special case of binary classification and demonstrate that it can be evaluated in terms of simple , closed-form expressions that depend only on aggregate statistics ( mean and variance ) of the annotations and not on any individual annotation . 3 . We demonstrate the effectiveness of our approach on models that we trained on two important medical imaging tasks with real-world clinical data and show that it reveals that our models not only significantly exceed the average human performance , but even beat the best human experts in our datasets . 2 RELATED WORK . There are several well-known strategies for dealing with ambiguous or noisy evaluation labels . The most popular approach is to simply use a single annotator ( Jafari et al. , 2019 ; Rajchl et al. , 2017 ; Cobzas et al. , 2007 ; Roth et al. , 2018 ) . An approach taken by many popular , large-scale datasets , such as ImageNet ( Deng et al. , 2009 ) , Places ( Zhou et al. , 2014 ) , COCO ( Lin et al. , 2014 ) and Open Images V4 ( Kuznetsova et al. , 2018 ) , is to take the average or majority vote . These efforts typically employ crowdsourcing in some fashion , since the labeling of this data does not require expert-level knowledge . While majority vote reduces label noise , in domains that require specialized knowledge it is often not practical to get enough experts that the noise is removed to the point where the annotations can be assumed to be “ correct ” ( in the sense that MNIST annotations are assumed to be correct ) . A related approach obtains a ground-truth dataset by enforcing a resolution criterion on labels where annotators disagree . This criterion can , for example , involve a live session where annotators deliberate and come to a consensus or an adjudication process where a third party ( usually a more senior , expert annotator ) is tasked with resolving disagreements ( Krause et al. , 2018 ; Hannun et al. , 2019 ; Rajpurkar et al. , 2018b ) . This approach may be effective at removing erroneous labels ( reducing variance ) at the expense of annotator independence ( increasing bias ) . The techniques outlined above fall short in providing a unified method that can both a ) be compared across different studies/tasks and b ) answer the essential question of how the model performance relates to an average human annotator ( i.e . how a prototypical human currently performs at a task ) . In developing high-stakes , real-world machine learning systems , the goal of evaluating a model is often to determine whether performance is safely substitutable for individual human behavior rather than comparing to an idealized oracle . Several studies have compared model performance with inter-annotator variability . Some have visualized model performance and inter-annotator variability ( e.g . via ROC curves ( Krause et al. , 2018 ; Gulshan et al. , 2016 ) or Bland-Altman plots ( Hansegard et al. , 2009 ; Altman & Bland , 1983 ) ) without providing an explicit metric . Several works have constructed a metric by comparing the model to the average annotator performance , though each has done so in a completely different way ( Wei et al. , 2019 ; Hannun et al. , 2019 ; Rajpurkar et al. , 2018a ; b ; 2017 ) . These metrics , however , generally depend on the number of annotators used in the study and/or require that all annotators label the entire dataset ( see Section 4 and Appendix C for more details ) . Moreover , the metrics are task-specific and can not be interpreted outside the context of a particular study . 3 THE DISCREPANCY RATIO METRIC . In this section , we define the discrepancy ratio and show that it has numerous desirable properties that overcome the limitations of prior art detailed in Section 2 . 3.1 DEFINITIONS . Definition 1 . For any two arbitrary sets of labels Y = { y1 , ... yn } and Y ′ = { y′1 , ... y′n′ } , the mean pairwise deviation ( MPD ) is defined by : ψ ( Y , Y ′ ) = ∑ y∈Y ∑ y′∈Y ′ δ ( y , y′ ) |Y||Y ′| , ( 1 ) where δ can be chosen to correspond to any measure of agreement appropriate to the task in question ( e.g . squared error , absolute error , Jaccard Index ( Jaccard , 1912 ) , Cohen ’ s kappa ( Cohen , 1960 ) ) . Consider a dataset of N samples where we have a prediction mi , for each sample i , by the model we are trying to evaluate . Further assume that each sample is labeled by at least two annotators and that each annotator may label the same sample multiple times . Formally , let Yij = { yijk } be the set of annotations provided by annotator j on sample i where k indexes the annotation number . Finally , let Ai indicate the number of annotators that have annotated sample i . Definition 2 . We define the annotator discrepancy , which quantifies the average annotator disagreement , to be the average of the MPD over all annotator pairs and all samples in the dataset : α ( { Yij } ) = 1 N N∑ i=1 1 Ai ( Ai − 1 ) Ai∑ j , j′=1 j′ 6=j ψ ( Yij , Yij′ ) . ( 2 ) Without access to unambiguous ground-truth labels , a natural way to define human performance is by comparing an annotator ’ s labels to those provided by other humans on the same samples . Consequently , we use the annotator discrepancy as a measure of average human performance . Definition 3 . Similarly , the model discrepancy is defined to be µ ( mi , { Yij } ) = 1 N N∑ i=1 1 Ai Ai∑ j=1 ψ ( mi , Yij ) . ( 3 ) Definition 4 . We define the discrepancy ratio to be the ratio of the model and annotator discrepancies : ∆ = µ α . ( 4 ) Intuitively , this metric is the ratio of the average model-annotator disagreement and the annotatorannotator disagreement . For example , a discrepancy ratio of 0.8 would mean that the model is 20 % closer to the annotators than they are to each other . When ∆ is above ( below ) 1 , the model performance can be said to be worse ( better ) than the average human performance . Since the measure of agreement δ is arbitrary , the discrepancy ratio can be thought of as a “ meta-metric ” that reports how the model performs ( relative to humans ) with respect to that particular measure of agreement . The dependence of the discrepancy ratio on the choice of δ is further explored in Section 4.2.2 . 3.2 PROPERTIES . The discrepancy ratio metric , as formulated above , has several desirable properties : Dimensionless : Dimensionless quantity whose meaning is invariant across different task categories ( regression , classification , segmentation etc . ) and whose value is interpretable and highly intuitive . Minimal dataset coverage requirements : The discrepancy ratio does not require that all annotators label all samples ( see Fig . 2 and Appendix C ) , which is often not practical in real-world situations , 1 0 0 , 1 0 , 0 ... 1 , 0 , 1. but only that each sample is labeled by at least two annotators ( two annotators are needed since it is meaningless to measure the annotator disagreement with only a single annotator ) . Insensitive to number of labels per-sample : The discrepancy ratio is expressed in terms of averages over all model/annotation pairs and therefore does not depend on how many annotators are used ( see Fig . 2 ) , nor how many times each annotator labels each image . This property makes it convenient to compare the discrepancy ratio across studies . Since the discrepancy ratio is formulated as a relative error that the model makes over that of the annotators , it can be naturally incorporated as a test statistic in a non-inferiority framework ( Ahn et al. , 2013 ) , often used in medical research . The goal of non-inferiority testing is to show that a new method is not statistically worse than an existing standard , up to some chosen equivalence bound . For the discrepancy ratio , we can choose the equivalence bound to correspond to an acceptable fraction of the error that the model is allowed to make on top of the one that annotators already make ( e.g . setting the equivalence bound to be 0.1 can be interpreted as saying that the model can make an error no more than 10 % larger than the ones humans typically make ) . Additional properties of the discrepancy ratio , as well as comparisons to alternative formulations , are discussed in detail in Appendix C and E .

Summary: This paper proposes a novel metric "discrepancy ratio" for evaluating the performance of a model where the ground truth for each data point comes from many expert-yet-imperfect annotators. The authors suggested that this problem has remained largely unexplored. The proposed metric is intuitive and is easy-to-use. The authors suggested that this metric can be used for many applications. For example, to evaluate if a model is better than the annotators on average, to evaluate the annotator, to compare between models.