Goodness-of-fit tests and function estimators for receiver operating characteristic (ROC) curves: Inference from perpendicular distances

GOODNESS-OF-FIT TESTS AND FUNCTION ESTIMATORS FOR RECEIVER OPERATING CHARACTERISTIC (ROC) CURVES: INFERENCE FROM PERPENDICULAR DISTANCES R. Bradley Patterson, PhD George Mason University, 2012 Dissertation Director: Dr. John Miller Statistical inference with the receiver operating characteristic (ROC) curve, a tool to assess performance of classication methods, has traditionally been based on dierences between empirical and parametric curves in the vertical direction. New and unique in this dissertation is the use of differences in a perpendicular direction for goodness-of-t tests, function estimators, and condence regions for the ROC curve. Working along directions perpendicular to parametric binormal ROC curves, we designed a goodness-of-t test similar to existing statistics based on the empirical distribution function (EDF) for a single random variable. We initially demonstrate the poor performance of the direct application of the Cramér-von Mises family of goodness-of-t statistics, which operate in only the vertical direction, to ROC curves. rough large simulations, our new test exhibits uniformity of p-values under the null hypothesis, and we prove its consistency. This document is a research report submitted to the U.S. Department of Justice. This report has not been published by the Department. Opinions or points of view expressed are those of the author(s) and do not necessarily reflect the official position or policies of the U.S. Department of Justice. Additional simulations with alternatives also show the goodness-of-t test’s power in three general cases. First, the test can reject the binormal model estimated with a fully parametric approach and small samples. Second, the test can reject the binormal model estimated with either a semiparametric or a fully parametric approach and large samples. ird, the test can reject the binormal model with small samples if one of the underlying distributions follows a truncated, mixture, or heavy-tailed distribution. e original function estimators developed here minimize dierences in perpendicular directions between empirical and parametric ROC curves. One function estimator performs a total least squares t that follows simply from principal component analysis and runs quickly. is short calculation time permitted the implementation of a parametric studentized bootstrap condence region, a computationally intensive technique with smaller error than simpler bootstrap methods. Large simulations suggest the new estimator has the smallest integrated squared error among semi-parametric estimators for ROC curves that are not too steep. e simulations also demonstrate coverage probabilities of the new condence regions to be close to nominal levels. ree extensions of this work are possible now. First, dierent weight functions in the new goodness-of-t statistic might improve the test’s eciency and raise its power. Second, the totalleast-squares estimator could handle covariates through regression. ird, other parametric forms, such as the biexponential and bigamma, of the ROC curve share some of the same fundamental geometry of the binormal and could benet from adaptation of the new statistics here. This document is a research report submitted to the U.S. Department of Justice. This report has not been published by the Department. Opinions or points of view expressed are those of the author(s) and do not necessarily reflect the official position or policies of the U.S. Department of Justice. Chapter 1: Introduction is work focuses on measuring the goodness-of-t and estimating functions of parametric receiver operating characteristic (ROC) curves. e ROC curve assesses the performance of classication methods used to identify observations by type. For instance, the ROC curve may indicate the performance of (1) a medical diagnostic test which classies patients as healthy or sick, (2) a forensic DNA test which classies genetic samples as matching or not, (3) a soware lter which classies emails as spam or ham, or (4) an astrophysics technique which classies galaxies as elliptical or spiral. To produce an ROC curve, a sample of observations with known classes must be available. Still, a nite sample results in an ROCplot which appears stepped like a staircase, possiblymaking it dicult to use. Oen, the true ROC functionmay be a continuous curve which remains unknown. Statistical methods lead to smooth, parametric estimates of the true ROC function from the jagged empirical one. To measure the quality of smooth, parametric ROC function estimates, we have developed a unique statistical goodness-of-t test. Inspired by that development, we have also created two original estimators of parametric ROC functions. A summary of our research appears in this chapter aer general introductions to ROC curves, goodness-of-t tests, and parameter estimation methods. 1.1 Overview of ROC curves e ROC curve measures the performance of classication methods. ROC curves appeared in signal detection theory in the 1950s as Swets et al. (2000) and Krzanowski and Hand (2009) discuss. Today, a variety of elds use ROC curves. In medicine, ROC curves have widespread adoption, particularly for evaluating diagnostic tests (Zhou et al., 2002; Pepe, 2004; Lasko et al., 2005; Zou et al., 2007). 1 This document is a research report submitted to the U.S. Department of Justice. This report has not been published by the Department. Opinions or points of view expressed are those of the author(s) and do not necessarily reflect the official position or policies of the U.S. Department of Justice. e general introduction by Fawcett (2006) covers their utility in data mining andmachine learning. More of the development and history of the ROC curve comes in the subsequent chapter. Before establishing terminology to discuss the ROC curve in detail, we highlight the tool’s benets in assessing the performance of classication methods. e ROC curve depicts a classication method’s capability to distinguish classes and shows all attainable error rates across thresholds. Furthermore, the ROC curve’s standard axes of true positive and false positive rates facilitate comparisons of various classication methods. 1.1.1 Classication methods, scores, and error rates is brief background on ROC curves begins with a review of the outcomes for a classication method. Suppose that we have a classication method that evaluates observations to determine to which one of two classes they belong. For consistency with established terminology of ROC curves, label one class the negative and the other the positive. For a given observation, the classication method returns a number that we call a score. Over multiple observations, the classication method generates two distributions of scores, one for the negative class and one for the positive class. Imagine that high scores suggest the positive class and that low scores suggest the negative. We wish to measure the method’s performance in discriminating between observations from the two classes. Next, for the purposes of testing this performance, assume that we have observations whose true sources we know. We apply the classication method to this test data to obtain scores for all observations. en we choose a threshold t on the scores. If the score for a given observation is greater than t, we suppose the observation arose from the positive class. Otherwise, we favor the negative class as the source of the observation. If we support the wrong hypothesis, we commit an error. When we favor the negative class when in truth an observation comes from the positive class, we call the error a false negative. Similarly, when we favor the positive class when in truth an observation comes from the negative class, we call the error a false positive. Table 1.1 lists the possible outcomes for a specic threshold t on the scores. Withmultiple observations, we can compute the rates of these errors. Wewill emphasize the false positive rate and true positive rate, which dene the axes of the ROC plot. e observed false positive 2 This document is a research report submitted to the U.S. Department of Justice. This report has not been published by the Department. Opinions or points of view expressed are those of the author(s) and do not necessarily reflect the official position or policies of the U.S. Department of Justice. Table 1.1: Possible outcomes for a single threshold on scores. True class: positive negative Predicted class: positive true positives false positives negative false negatives true negatives rate refers to the fraction of times that the classicationmethod incorrectly assigns observations from the negative class to the positive class. at is, false positive rate = number of false positives number of true negatives + number of false positives . Similarly, the false negative rate refers to the fraction of times that the classication method incorrectly assigns observations from the positive class to the negative class. Sowe can express the observed false negative rate as: false negative rate = number of false negatives number of true positives + number of false negatives . e true positive rate, used on the ROC plot, is the complement of the false negative rate: true positive rate = 1 − false negative rate. e observed true positive rate refers to the fraction of times that the classication method correctly assigns observations from the positive class to the positive class. In the preceding, a single threshold on scores yielded one set of rates, but we could vary the threshold to nd all possible rates. Figure 1.1 demonstrates the concept of constructing the ROC curve from all possible thresholds. First we sort the scores from greatest to least and then consider all possible values of the threshold t. For each value of t, we nd the values of the false positive and 3 This document is a research report submitted to the U.S. Department of Justice. This report has not been publishe