Pap e r 210-31 

## Receiver Operating Characteristic (ROC) Curves Mithat Gönen, Memorial Sloan-Kettering Cancer Center

ABSTRACT Assessment of predictive accuracy is a critical as pect of evaluating and compar ing models, algorithms or technologies that produce the pr edictions. In the field of medical diagnosis , receiver operating characteristic (ROC) curves have become the standard tool fo r this purpose and its use is becoming increasingly common in other fields such as finance, atmospheric science and machine learning. T here are surprisingly few built-in options in SAS for ROC curves, but several procedures in SAS/ST AT can be tailo red w i th little effort to produce a w i de variety of ROC analy s es. T h is talk w ill focus on the use of SAS/ST AT procedures FREQ, LOGIST I C, MIX E D and NLMIX E D to perform ROC analy s es, including estimation of sensitivity and specificity
, esti mation of an ROC curve and computing the area under the ROC curve. In additi on, several macros w ill be introduced to facilitate graphica l presentation and complement ex isting statistical capabilitie s of SAS w i th regard to ROC curves. Real data from clinical applications w ill be used to demonstrate the methods.



### INTRODUCTION Receiver operating characteristic (ROC) curves are us eful for assessing the accuracy of predictions. Making predictions has become an essential part of every business enterpris e and scientific field of inquiry . A simple example that has irreversibly penetrated daily lif

e is the w eather forecast. Almost all new s sources, including daily new spapers, radio and television new s, provide detailed w eather fore casts. T here is even a dedicated television channel for
w eather forecasts in United States. Of course, the infl uence of a w eather forecast goes bey ond a city dw eller’s decision as to pack an umbrella or not. Inclement w eather has negative effects on many vital activities such as transportation, agriculture and construction. F o r this reason collecting data that helps fo recast w eather conditions and building statistical models to produce forecasts from these data have becom e major industries.

I w ill give examples from other areas w her e prediction play s a major role to motivate statisticians from diverse fields of application. Credit scoring is an ex cellent ex ample: w hen a potential debtor asks for credit, creditors assess the likelihood of default to decide w hether to loan the funds and at w hat interest ra te. An accurate assessment of chance of default for a given debtor play s a crucial role for credito rs to stay competitive. F o r this reason, the prediction models behind credit scoring sy stems remain proprietary but their predictive pow er needs to be continuously assessed for the creditors to remain competitive.

T he final example is concerned w i th the field of medical diagnostics. T he w o rd "prediction" rarely appears in this literature, but a diagnosis is a prediction of w hat might be w r ong w i th a patient producing the sy mptoms and the complaints. Most disease processes elicit a response that is manifested in the form of increased levels of a substance in the blood or urine. T here might be other reasons for such elevated levels , and blood or urine levels mis-diagnose a condition because of this. T he kind of analy s is one w ould perfo rm for w eather forecasts is similarly valid for these blood or urine "markers."
ROC curves provide a comprehensive and visually attractive w a y to summarize the accuracy of predictions. T hey are w i dely applicable, regardless of the sour ce of predictions. T he field of ROC curves is by and large ignored during statistics education and training. Most statisticians learn of ROC curves on the jog, as needed, and struggle through some of the unusual features. T o make matters w o rse fo r SAS users, very few direct methods are available for performing an ROC analy s is although many procedures can be tailored w i th little attempt to produce ROC curves. T here is also a macro available from the SAS Institute for this purpose. T he goal of this paper is to summarize the available features in SAS for ROC curves and ex pand on using other procedures for further analy s es.



### BASIC CONCEPTS: BINARY PREDICTOR One of the simplest scenarios for predi ction is the case of a binary predict or. It is important, not only pedagogically because it contains the most important building blocks of an ROC curve, but also prac tically because it is often encountered practice. I w ill use an example from w eather forecasting to illustra te the concepts and at the end of the section mention some situations from other prominent fields.

T he article by T hornes and Stephenson (2001) review s the conc epts of assessment of predi ctive accuracy from the perspective of w eather forecast products. T heir opening exam ple is very simple and accessible to all data analy s ts regardless of their training in meteorological sciences. T he example relates to frost forecasts produced for M62



### 1motorw ay betw een Leeds and Hull in United Kingdom during t he w i nter of 1995/1996. A fros t is defined as w hen the road temperature falls below 0 °C. F i rst, the forecast for each night is produced as a binary indicator F r ost or No F r ost. T hen actual surface temperature for the road is monitored throughout the night and the outcome is recorded as F r ost if the temperature dropped below 0 °C and as No F r ost if it did not drop below 0 °C. T he guidelines provided by the Highw ay s Agency mandate the reporting of results (both forecast and t he actual) in a consolidated manner (such as the follow i ng 2x2 contingency table) only for the day s for w h ich the act ual temperature w a s below 5 °C. T he example w a s concerned w i th the w i nt er of 1995-1996 w hen there w e re 77 ni ghts w hen the actual road surface temperature w a s below 5 °C. T he results are given T abl e 1. Such a tabular descrip tion is the standard w a y of reporting accuracy w hen both the predicti on and the outcome are binary

. It is vi sually appealing, simple to navigate and it contains all the necessary information.

T here w e re 29 nights w hen frost w a s forecast and a fros t w a s observed and there w e re 38 nights w hen no frost w a s forecast and no frost w e re observed. T hose tw o cells (the diagonal cells of the 2x2 tabl e, the shaded portion of T able 1) represent the tw o ty pes of correct forecast. A general terminology is true positives (T P) and true negatives (T N). T he roots of this terminology can be found in medical diagnos tic studies w hen a test is called positive if it show s disease and negative if it does not show disease. By analogy w e can consider frost to mean "positive" and no frost to mean "negative" in w h ich case w e w ill have 29 true positives and 38 true negatives.

T able 1: F o recast accuracy for road surface temperature example Observed Forecast F r o s t N o F r o s t T o t a l Fro s t 29 6 3 5 No Frost 4 3 8 4 2 T o t a l 3 3 4 4 7 7

W hat about the forecast errors? T here w e re 6 nights w hen a frost w a s forecast and none w e re observed. T here w e re 4 nights w hen no frost w a s forecast but a frost w a s observed. You can easily extend the terminology to call these tw o cells as false positives (F P) and false negatives (F N). T he follow i ng table is a generic repr esentation of T able 1 using the terminology introduced above.

T able 2: Reporting accuracy for binary predictions Observed Forecast P o s i t i v e N e g a t i v e T o t a l Positive T r ue Positive (T P) False Positive (FP) T P +FP Negative F a lse Negative (F N) T r ue Negative (T N) F N + T N T o ta l T P + F N FP+T N T P +FP+FN+T N

T here are a variety of w a y s that one c an summarize the forecast accuracy . One that comes to my mind first, perhaps, is the misclassification rate (MR) w h ich is the proportion of all misclassified nights, the sum of false negative and false positives, out of all nights:



## TN FP FN TP FP FN MR + + + +

=

One minus the misclassification rate is sometimes called "percent correct" or si mply "accuracy ." MR for the data in table 1 is 10/77= 13%.

W h ile misclassification rate is simp le to compute and understand, it is sometimes too crude for understanding the mechanism behind misclassification. It is also prone to be biased if one is not careful in how the information is assembled. Suppose, instead of follow i ng the Highw ay s Agency ’ s guidelines, the fo recast provider decides to include all nights in a calendar y ear. T here are 77 nights reported in T able 1 and by defin ition all of those nights w e re those
w hen the actual temperature dropped below 5 °C. T herefore the remaining 288 nights w e re all above 5 °C (no frost), bringing column marginal totals to 33 nights w i th frost (unchanged) and 332 nights w i th no frost. It is very possible that the MR for these 288 nights w here the temperature w a s above 5 °C w a s much less than the MR for the nights w here the temperature w a s below 5 °C. Suppose for the time being that the misclassification rate for these 288 nights w a s 5%, resulting in 15 misclassified nights (rounded up). T hen ther e w ould be a total of 25 misclassified nights out of 365 and the MR w ill now be 25/265= 7%. T able 3 show s several possibilities.



### 2It is clear that MR is sensitive to w h ich nights are included in the sample since the performance of the forecast is not homogeneous for all the nights in a y ear. It is also clear that as one includes more "easy

- to-forecast" nights in the sample, the MR w ill be smaller. One can safely make a ca se that for w a rm day s in spring and summer a forecast as to w hether the M62 w ill freeze the follo w i ng night is quite useless since most people can make the same prediction (quite accurately ) w i thout resorting to a scientifically obt ained forecast. T herein lies the logic of the Highw ay s Agency to restrict the reporting of accuracy only to thos e w here the actual temper ature w a s 5 °C or less.

T able 3: Putative MR for the 288 nights w hen the temper ature w a s above 5 °C and the corresponding MR for all the 365 nights MR for the 288 nights w hen the

temperature w a s above 5 °C 0 % 1 % 2 % 4 % 6 % 8 % 1 0 % 1 2 % 1 3 %

MR for all the 365 nights 2.7% 3.5% 4.3% 5

. 9 % 7

. 5 % 9

. 1 % 1 0

. 6 % 1 2

. 2 % 1 3

. 0 %

In unregulated areas such as credit scoring, w here scoring al gorithms remain mostly proprietary , there are no such rules, or even guidelines, as to how accuracy should be evaluated. In addition, consum ers of predictions are not alw a y s diligent or know ledgeable in interpreting the details of forecast accuracy . T herefore it is desirable to propose measures that are more robust.

T he most common w a y of reporting the accuracy of a binary prediction is using the true (or false) positives and true (or false) negatives separately

. T h is recognizes that, a false negative prediction ma y have different consequences than false positive ones. In our w eather forecast example, a false positive is probably less costly because its primary consequence may be more cautious and better-prepared driv ers. On the other hand a false negative may end up in under-preparation and may lead to accidents. T h is sugges ts reporting false positive and false negative rates separately

.



## TP FN FN FNR TN FP FP FPR + = + = ,

One can extend these def initions to include T P R= 1-F P R and T NR= 1-F NR. T he true positive rate (T RP) is sometimes called sensitivity and the true negative rate (T NR) is sometimes called specificity . W h ile these are generic terms that routinely appear in the statisti cs literature, each field has come up w i th it s ow n terminology . W eather forecasters, for example, use miss rate for F NR and false alarm rate for F P R.

T he F P R and F NR for the w eather forecast data are 4/ 33= 12% and 6/44= 14%. T he sensit ivity and specificity are 29/33= 88% and 38/44= 86%. In this instance F P R and F NR are bot h very close to each other. W hen this is the case, they w ill also be very close to MR. In fact one can s how that MR is a w e ighted average of FPR and FNR: MR =
w * F P R + (1-w )*F NR and the w e ight (w ) is the proportion of ni ghts w i th an observed frost. T h is is sometimes called the "prevalence" of frost.

Note that the denominators for F P R and F NR are the total observed positives and negatives. It is possible to define similar quantities using the forecast pos itives and negatives. In this case the ratios corresponding to sensitivity and specificity are called positiv e predictive value (PPV) and negat ive predictive value (NPV):



## FP TN TN NPV FN TP TP PPV + = + = ,

It is very important to understand the corre ct interpretation of sensitivity
, s pecificity
, PPV and NPV. I w ill start by predictive values first. T heir denominator s are the number of positiv e and negative forecasts. In the w eather forecast
ex ample PPV can be interpreted as the probability of an actual frost w hen a frost w a s forecast and NPV is the probability of observing no frost w hen no frost w a s foreca st. In contrast, the denomi nators for sensitivity and specificity are observed positives and negatives. T herefore s ensitivity is the pr obability that a night w i th a frost w ill be correct identified by the forecast out of all nights w i th a frost during the w i nter and similarly specificity is the probabilit y that a night w i thout a frost w ill be correct identified by the forecast out of a ll nights w i th no frost (and less than 5 °C) during the w i nter.
It is easy to imagine the utility of these probabilities as occasional consumers of a w eather forecast. If I w ill drive from Leeds to Hull on M62 only a few nights during a w i nter, all I ca re is w hether the forecast w ill be accurate on those few



### 3nights. On the other hand the municipality w ill be more interested in sens itivity and specificity w hen deciding w hether to pay for these forecasts since that speaks to t he entire "cohort" of nights in the upcoming w i nter.

It is relatively easy to compute all of these measur es in SAS®. T he follow i ng DA T A step prepares the data:

data m62; input forecast $ observed $ weight; datalines; Frost Frost 29 Frost No_Frost 6 No_Frost Frost 4 No_Frost No_Frost 38
; run;

and the follow i ng execution of PRC F R EQ provides the necessary calculations:

proc freq data=m62; table forecast*observed; weight weight;
run;

T he output is show n below and contains many useful pieces of information:

forecast observed Frequency‚ Percent ‚ Row Pct ‚ Col Pct ‚Frost ‚No_Frost‚ Total ƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆ Frost ‚ 29 ‚ 6 ‚ 35 ‚ 37.66 ‚ 7.79 ‚ 45.45 ‚ 82.86 ‚ 17.14 ‚ ‚ 87.88 ‚ 13.64 ‚ ƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆ No_Frost ‚ 4 ‚ 38 ‚ 42 ‚ 5.19 ‚ 49.35 ‚ 54.55 ‚ 9.52 ‚ 90.48 ‚ ‚ 12.12 ‚ 86.36 ‚ ƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆ Total 33 44 77 42.86 57.14 100.00

As a quick reminder to the PROC F R EQ output, the key to the four numbers in each ce ll is found in the upper left portion of the table. T he first number is the frequency
, i.e. T P , F P , F N and T N . T he second number uses the table sum (sum of column sums or sum of row sums) as the denominator. You can compute the misclassification rate using the numbers in the numbers in the second row

. T he third number is the row per centage, i.e. the proportion that use
row sums as the denominator so thes e w ill include the PPV and NPV. Finally
, the fourth number is the column frequency
, using column sums as the denominator. Sensitivity
, specificity
, FPR and FNR w ill be among the third row numbers.

For ex ample, the MR is the sum of the second numbers in the off-di agonals: 7.79+5.19=12. 98%. PPV and NPV are 82.86% and 90.48%. T he F P R and F NR are 13.64% and 12.12%.

It is easy to recognize that these are all binomial pr oportions if one considers t he corresponding denominators as fixed. T he technical term is conditioning on the denominators. T h is gives easy rise to the use of binomial functionality



### 4w i thin PROC F R EQ to compute interv al estimates. T he follow i ng exampl e uses BY processing and the BINOMIAL option in the T ABLE statement to produc e confidence intervals and hy pothesis test s for sensitivity and specificity

.

proc sort data=m62; by observed; run;
proc freq data=m62; by observed; table forecast / binomial; weight weight;
run;

T he first part of the output for observed= F r ost is given below . Here the denominator is observed positives so the report is focusing on sensitivity and F P R. T he listing tells us that there w e re 29 true positives (both forecast and observed w e re "F rost") and 4 false positives (frost w a s forecast but not observed).

observed=Frost

The FREQ Procedure

Cumulative Cumulative forecast Frequency Percent Frequency Percent

ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ Frost 29 87.88 29 87.88 No_Frost 4 12.12 33 100.00

Binomial Proportion for forecast = Frost

ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ Proportion 0.8788 ASE 0.0568 95% Lower Conf Limit 0.7674 95% Upper Conf Limit 0.9901

Exact Conf Limits 95% Lower Conf Limit 0.7180 95% Upper Conf Limit 0.9660

Test of H0: Proportion = 0.5

ASE under H0 0.0870 Z 4.3519 One-sided Pr > Z <.0001 Two-sided Pr > |Z| <.0001

T he output under "Binomial Proportion" c ontains the key information. Sensit ivity w a s 87.88% and the asy m ptotic standard error (ASE) for the estimate of standard error is 5. 68%. You should remind y ourself at this point that the standard error for the estimate of a binomial proportion p is given by



## n p p ASE ) 1 (

− =

5w here n is the denominator for the binom ial proportion. T he confidence lim its appearing under the ASE are based on asy m ptotic theory

. If n is large, then the 95% confidenc e interval can be calculated using


## ASE p ×

± 96

. 1

Exact confidence limits, in contras t, are based on the binomial distribution and they have better coverage in small samples and/or rare events. Since t hey are now calculated by default, ther e is no need to use the asy m ptotic confidence limits.

F i nally , the last part provides a test of w hether the sensitivity is equal to 50% or not. In essence this is a test of
w hether there is any information for predicting frosts sinc e 50% sensitivity can be obtai ned using a coin toss. T he zstatistic reported is computed by z = p/ASE(null) w here ASE(null) means the ASE under the null hy pothesis. T he difference from ASE above is in the use of observed p (estimate of s ensitivity ) versus the va lue under the null (0.5 in this case). T h is is a ty pical w a y of computing test statis tics, sometimes referred to as Wald test. In large samples z has a normal distribution under the nu ll hy pothesis so a p-value can be obtained by referring to a standard normal table. T h is results in a p value less than 0.0001, a highly significant result s uggesting that w eather forecasts in this example are much better than tossing a coin.

T he output below , w h ich is the second part of the PROC F R EQ output w i th BY statement, is for the F NR (not specificity ) . T h is is because the per centages computed by PROC F R EQ use the first level in alphanumeric order
w h ich happens to be "F rost" in this case. W hen observed= F r os t this matched w i th T P and the result w a s sensitivity but w hen observed= No_F rost it matches F NR.

observed=No_Frost

The FREQ Procedure

Cumulative Cumulative forecast Frequency Percent Frequency Percent

ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ Frost 6 13.64 6 13.64 No_Frost 38 86.36 44 100.00

Binomial Proportion for forecast = Frost

ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ Proportion 0.1364 ASE 0.0517 95% Lower Conf Limit 0.0350 95% Upper Conf Limit 0.2378

Exact Conf Limits 95% Lower Conf Limit 0.0517 95% Upper Conf Limit 0.2735

Test of H0: Proportion = 0.5

ASE under H0 0.0754 Z -4.8242 One-sided Pr < Z <.0001 Two-sided Pr > |Z| <.0001

T o compute the specificity one only needs to subtract the estimates fr om 1. T herefore t he specificity is 1= 0.1364= 86.36% and the confidence inte rvals are 1-0.2735= 72.65% to 1-0.0517= 94.83%. In terms of hy pothesis testing the only change is in the sign of z . Specifically , ASE and the p-values w ill be the same.

W h ile BY processing is simple and provides all answ e rs, albei t after some minor hand calculation, it is possible to



### 6completely automate the procedure. T he follow i ng code calls PROC F R EQ tw ice to accomplish this:
proc freq data=m62(where=(observed='Frost')); table forecast / binomial(p=0.8); weight weight;
run;

proc freq data=m62(where=(observed='No_Frost')); table forecast / binomial(p=0.75); weight weight;
run;

T he difference from BY processing is the ability to set t he null hy pothesis through the us e P= option in the T ABLE statement. Since there are tw o separate calls to PROC F R E Q, tw o different values of P can be used (0.8 and 0.75 in this example). T he output is not repeated as it is identical in form to those above, w i th the only difference being the null value of the sensit ivity and specificity

. 

### THE ROC CURVE W e are now ready to consider the problem that naturally leads to the use of the ROC curve. Suppose w e are try i ng to predict a binary outcome, as in the w eather forecast, but instead of a binary predictor w e have a continuous predictor. Most w eather forecasts are produced by statistical models w h ich generate a probability level for the outcome. T o simplify the use sometimes a pre-set threshold on this probab ility is used to dichotomize the forecast. Most w eather forecasts mention "Rain" in their summary but if one looks at details they also report a "chance of rain." How w ould w e assess the predictive accuracy of t hese probabilities (or, perhaps more appropr iately , the model that produced these predicted probabilities)?

Since w e know how to analy z e predictive accuracy for a binary predictor (using MR, F P R etc) w e may consider transforming the predicted probability into a dichotomy by using a threshold. T he results, how ever, w ould clearly depend on the choice of threshold. How about using severa l thresholds and reporting the results for each one? ROC curve is one w a y of doing this. ROC curves focus only on s ensitivity and (one minus) specif icity

. W e w ill soon see the reason for this.

I w a s unable to find a reasonable data set on w eather forecast to illustrate this point, so I w ill use an example from another field: medical diagnostics. One w a y to diagnose cancer is through the use of special scan called positron emission tomography (PET ). PET produces a measure call ed standardized uptake value (SUV) w h ich is an indicator of how likely the part of the body under consideration has cancer. SUV is a positive number. After SUV is measured the patients undergo a biopsy w hereby a small piece of tissue from the su spected area is removed and examined under the microscope to evaluate w hether it is cancer or not. T h is is called pathological verification and considered the gold standard.

T hese data are reported by W ong et al (2002). T here are 181 patients 67 of w hom are found to have cancer by the gold standard. Since w e are dealing w i th a continuous predictor it is no longer po ssible to report the data in a tabular form. Instead I use the follow i ng side- by -side histograms from PROC UNIVAR IAT E that can be obtained w i th the simultaneous use of CLASS and HIST OGRAM statements.



### 70 10 20 30 40 50

P
e r c e n t +

02468 1 0 1 2 1 4 1 6 1 8 2 0 0 10 20 30 40 50

P
e r c e n t -

su v G
o l d

S t
a n d
a r d

T he upper panel of the figure is the hi stogram of SUV for the gold standard pos itive patients (those w i th cancer) and the low e r panel is for gold standard negative patients (those w i thout c ancer), as denoted by the vertical axis. Patients
w i th cancer tend to have higher SUV values; there are only a few that have very low SUVs. A very high SUV (say , 10 or more) almost certainly implies cancer. T here is, how ever , some overlap betw een the dist ributions for SUVs in the middle range (roughly 4-10). So extreme values of SUV strongly imply cancer or lack of c ancer, but there is a gray area in the middle. T h is is in fact a very common picture for many continuous predictors. How accurate, then, is SUV for diagnosing cancer?

One crude w a y of approaching the problem is to compute the sensitivity and specificity fo r various thresholds. T he follow i ng graphical depiction is for using a threshold of 7:



### 80 10 20 30 40 50

P
e r c e n t +

02468 1 0 1 2 1 4 1 6 1 8 2 0 0 10 20 30 40 50

P
e r c e n t -

su v G
o l d

S t
a n d
a r d

SUV>7: Positive Prediction SUV<7: Negative Prediction

Dichotomization of SUV can be accomplis hed using the follow i ng data step command

suv7=(suv>7);

w h ich can produce undesirable behavior if the variable SUV contains missing values in w h ich case IF -T HEN statements should be employ ed. Using th is variable one can easily obtain the ki nd of 2x2 table considered in the previous section for the w eather fo recast example. F o r this data set it w ould look like the follow i ng:

T able 4: Accuracy of SUV> 7 in diagnosing cancer Gold Standard Diagnosis SUV>7 Diagnosis C a n c e r N o C a n c e r T o t a l Cancer 25 3 2 8 No Cancer 42 1 1 1 1 5 3 T o t a l 6 7 1 1 4 1 8 1

W e can now estimate the sensit ivity and specificity to be 25/67= 37% and 111/114= 97%. T h is analy s is can be repeated for various thresholds. One w a y to report the resu lts w ould be in tabular form, such as the follow i ng:

T able 4: Accuracy of SUV in diagnosing cancer for various thresholds T h r e s h o l d 1 3 5 7 Sensitivity 9 7 % 9 3 % 6 1 % 3 7 % Specificity 4 8 % 6 5 % 8 8 % 9 7 %



### 9T h is table makes clear the w i de range of sensitivity and specificit y that can be obtained by vary ing the threshold. It also makes clear the inverse relationship betw een the tw o measures: as one increases t he other decreases and vice versa. T h is can be visually seen from the histogram above. If one moves the das hed line to the left (to, say 5, instead of 7) more patients w ill be classified as positive: some of these w ill be gold st andard +
, increasing t he sensitivity
, but some of them w ill be gold standard –, decr easing specificity

. So the relationship w e observed in T able 5 is universal: It is not possible to vary the threshold so that both specificity and sensitivity increase.

A tabular form is limited in the number of thresholds it can accommodate. O ne can plot the s ensitivity versus specificity
, keeping in mind t hat each point on the plot corresponds to a s pecific threshold. For reasons that w ill become clear later the traditional ROC curv e plots sensitivity on the vertical axis and the spec ificity on the horizontal axis. F i gure 1 is the ROC curve corresponding to this data set.

S e

n s i t i v i t y

0. 0 0. 1 0. 2 0. 3 0. 4 0. 5 0. 6 0. 7 0. 8 0. 9 1. 0

1 - S p e c i f i c i t y 0

. 0 0

. 1 0

. 2 0

. 3 0

. 4 0

. 5 0

. 6 0

. 7 0

. 8 0

. 9 1

. 0

Each point on the ROC curve corresponds to a specific thre shold. In this graph the poi nts are connected, leading to this so-called empirical ROC curves. Each point corresponds to a threshold, although the va lue of thresholds is not evident from the graph. T he ROC cu rve above can be plotted using the OU T R OC option of PROC LOGIST I C:

proc logistic data=HeadNeck noprint; model gold_standard=suv / outroc=ROCData; run;
symbol1 v=dot i=join; proc gplot data=ROCData; plot _sensit_*_1mspec_; run;quit;
It is often useful to enhance ROC curve plots in a variety of w a y s . T he most common enhanc ement is the inclusion of a 45-degree line. T h is line repr esents the ROC curve for a useless test and a visual low e r bound for the ROC curve of interest to out-perform, by including reference lines and highlighting selected threshol ds. Another useful enhancement is the inclusion of a few selected thre sholds on the curve, along w i th referenc e lines that indicate the corresponding sensitivity and specificity

. F i nally it is customary to hav e the horizontal and vertical axes have the same length, resulting in a square plot, as opposed to the default re ctangular plot of SAS/GRAPH. T he follow i ng plots the ROC curve of SUV again, w i th the ment ioned enhancements. T he macro %ROCPLOT
, available from the SAS® Institute can be useful. T he follow i ng w a s generated by %PLOT R OC, available by e-mail from the author, w a s used to generate this plot.



### 100. 0 0. 1 0. 2 0. 3 0. 4 0. 5 0. 6 0. 7 0. 8 0. 9 1. 0

1 - S p

e

c i f i c i t y 0. 0 0

. 1 0. 2 0

. 3 0. 4 0

. 5 0. 6 0

. 7 0

. 8 0

. 9 1. 0

T here is a convenient mathemat ical representation of the RO C curve that y i elds further insight to many of its properties. Consider the (complement of the) distribution of the SUV in pat ients w i th cancer, to be denoted by SUV + : P(SUV + > t)= F (t) and the (complement of the) distribution of the SUV in patient s w i th no cancer, similarly denoted by SUV

– : P(SUV

– > t)= G (t) w here F ( .) and G(.) are unspecified functions . Since all patients in F ( .) are by definition gold standard positive, F ( t) describes the pr oportion of positive patients w hose SUV exceed y out of all positive patients. T h is is nothing but the sensitiv ity w hen t is used as the threshold. Similarly
, 1-G(t) w ould be the specificity
, hence G(t) represents one minus the specif icity

. T herefore F and G are the functions that w e need to represent the ROC curve. Now
, let x represent a point on the horizontal axis of the ROC curve (that is a fixed le vel of one minus the specificity )

. T he threshold corresponding to this point w ill be given by t= G -1 (x) and the sensitivity corresponding to t is F ( t). So the sensitivity corresponding to x is given by y = F ( G -1 (x)). T h is representation w ill be helpf ul in the sequel. It also explains
w h y the ROC curves focus on sensitiv ity and specificity instead of NPV and PPV. T he denominator of the latter tw o change w i th the threshold and do not lend t hemselves to notation and analy s es by the use of cumulative distribution functions of the predictor variable.



### AREA UNDER THE ROC CURVE W h ile the ROC curve contains most of the information about the accuracy of a continuous predictor, it is sometimes desirable to produce quantitative summary measures of the ROC curve. T he mo st commonly used such measure by far is the area under the ROC curve (AUC). In an empirical ROC curve this is usually estimated by the trapezoidal rule, that is by forming trapezoids usi ng the observed points as corners, com puting the areas of these trapezoids and then adding them up. T h is may be quite an effort for a curv e like the one in F i gure 1 w i th many possible thresholds. Fortunately
, AUC is connected to a c ouple of w e ll-know n statistical measur es that facilitates comparison and improves interpretation.

T he first of these measures is the pr obability of concordance. If, in a randomly selected pair of patients, the one w i th the higher SUV has cancer and the one w i th the low e r SUV has no cancer than this pair is said to be a concordant pair. A pair w here the patient w i th the higher SUV has no cancer but the one w i th the low e r SUV has cancer is said to be a discordant pair. Some pairs w ill have ties for the SU V. Finally some pairs w ill be non-informative, for example both patients may have cancer or both may have no cancer. It is not possible to classify these pairs as concordant or discordant. T he probability of concordance is defined as the number of concordant pairs plus one-half the number of



### 11tied pairs divided by the number of all informative pairs (i .e. excluding non-informative pai rs). In essence, each tied pair is counted as one-half discordant and one-half concordant. T h is is a f undamental concept in many rank procedures, sometimes referred to as "randomly breaking t he ties." An equivalent w a y to express concordance is P(SUV + >SUV

– ) w here SUV

– indicates the SUV of a gold standard negative patient and SUV + indicates the SUV of a gold standard positive patient. It tu rns out that AUC is equal to the probability of concordance.
AUC is also closely related to the W ilcoxon rank-sum test. T he rank-sum statistics is t he sum of the ranks in one of the groups (here group is defi ned by the gold standard). If y ou take the sum of the rank is t he gold standard positive group, then the number of concordant pairs is equal to the rank sum minus a constant that depends on the size of the tw o groups only

. All of these relationships highlight a f undamental property of the ROC curve that it really is a rank-based measure. In fact the empirical ROC curve is inva riant to monotone transformations of the predictor. So one w ould get the same ROC curve if log(SUV) or SUV 1/3 is used as the predictor.

PROC LOGIST I C reports the area under the curve in it s standard output under the heading "Association of Predicted Probabilities and Observed Responses." T h is part of the out put for the cancer exampl e looks like the follow i ng:
Association of Predicted Probabilities and Observed Responses

Percent Concordant 86.1 Somers' D 0.743 Percent Discordant 11.8 Gamma 0.758 Percent Tied 2.1 Tau-a 0.348 Pairs 7638 c 0.871

W e see that out of the 7638 (informati ve) pairs, 86.1% w e re concordant and 2.1% w e re tied. Proportion of concordant pairs plus one-half the proportion of c oncordant pairs is 87.1% and is separat ely reported as "c" for concordance. Also reported is Somers’ D statistic w h ich is also relat ed to concordance via D= 2*(c-0.5), Somers’ D is simply a rescaled version of concordance that ta kes values betw een -1 and 1, like a us ual correlation coefficient instead of 0 and 1.

AUC is estimated from the data, so ther e has to be a standard error associated w i th this estimation. Unfortunately it is not available in PROC LOGIST I C. For this w e turn to a macro provided by the SAS® institute. Macro %ROC, currently in version 1.5 is available at http ://support.sas.com/samples_app/00/sample00520_6_roc.sas.tx t. T he follow i ng is the call that c an be used for this example:

% roc (version,data=HeadNeck,var=SUV,response=gold,contrast=1,details=no,alpha=.05);

F i rst one needs to create a gold standard variable takes on va lues of 0 or 1 only
, t he macro does not have the capability of re-coding the observed bi nary classes. T he variable GOLD takes on the value 1 for gold standard positive patients and 0 for negativ e patients. VAR specifies the continuous predictor. CONT RAST w ill alw a y s be 1 for a single ROC curve, it is useful for comparing several cu rves. Alpha pertains to confidenc e limit estimation and details control the amount of out put printed. T he output from this call is given below . T he area under the curve is 87.13% (same as the concordance from PROC LOGIST C , as it s hould be). But this output has the standard error for this estimate (2.63%) and the associated asy m ptotic confidence intervals (81.98%- 92.28%). T h is is a more complete analy s is, as it is possible to judge the effects of sampling variability on the estimate of AUC.



### 12The ROC Macro

ROC Curve Areas and 95% Confidence Intervals ROC Area Std Error Confidence Limits

SUV 0.8713 0.0263 0.8198 0.9228



### THE BINORMAL ROC CURVE So far our focus w a s on the empirical curve, the one t hat is obtained by connecting t he observed (sensitivity
, 1- specificty ) pairs. T he empirical curve is heavily -used becaus e it makes no assumptions r egarding the distribution of the individual predictors. But there ar e occasions w here one w ould prefer a sm ooth curve. T he most common w a y of smoothing an ROC curve is using the binormal model.

T he binormal model assumes that SUVs of the patients w i th cancer follo w a normal distribution w i th mean μ 1 and variance σ 1 2 and SUVs of the patients w i th no cancer follow a normal distribution w i th mean μ 0 and variance σ 0 2

. T hen using the notation from t he empirical ROC curves, G(t)= Φ ((μ 0 -t)/ σ 0 ). It follow s that the threshold t can be w r itten as a function of x as follow s : t = μ 0 - σ 0 Φ -1 (x). Since a threshold t corresponds to the sensitivity F 9 T 0 w e can w r ite the functional form of the ROC curve as 

# ( ) ) ( ) ( ) ( 1

1 1

0 0 1

1 1 

## x b a x t t F − − 

## Φ + Φ = ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ Φ +

− Φ = ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛

− Φ = σ σ μ μ

σ μ

w here

1 0

1 0 1



## , σ σ

σ μ μ

= −
= b a

and are often referred to as "binormal parameters." Someti mes they are called intercept and the slope since plotting on normal probability paper w ould y i eld a straight line for the binormal curve w i th intercept a and slope b. T h is practice is not common any more, but the nomenclature continues.

T he area under the curve for the binormal m odel also has a closed-form expression: 

## ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛

+ Φ = 2 

## 1 b

a AUC

T o estimate a binormal model from data one needs to estima te the means and variances separately

. T h is can simply be accomplished using PROC MEANS w i th a CLASS statem ent. It can also be obtained using PROC NLMIX E D as follow s :
proc nlmixed data=hn(drop=m); parameters m1=0 m0=0 s1=1 s0=1; if gold=1 then m=m1;else if gold=0 then m=m0; if gold=1 then s=s1**2;else if gold=0 then s=s0**2; a=(m1-m0)/s1; b=s0/s1; model highs ~ normal(m,s); estimate 'a' a; estimate 'b' b; estimate 'AUC' probnorm(a/sqrt(1+b**2)));
run;

T he advantage of using NLMIXED is t hat, via the EST IMAT E statement, one c an directly obtain estimates, and perhaps more importantly , standard errors for a, b and A UC. T he follow i ng output from NLMIXED execution, under the title of "Additional Estima tes," is all that is needed:



### 13Additional Estimates

Standard Label Estimate Error DF t Value Pr > |t| Alpha Lower Upper

a 1.3096 0.1776 181 7.37 <.0001 0.05 0.9591 1.6600 b 0.6604 0.07189 181 9.19 <.0001 0.05 0.5185 0.8022 AUC 0.8628 0.02935 181 29.40 <.0001 0.05 0.8048 0.9207

Using this output the binorma l ROC curve is given by Φ (1.31 + 0.66 Φ -1 (x)) w here x takes on values from 0 to 1. T hat the ROC curve is defined for all point s betw een 0 and 1 is a feat ure of the model. Although not all thresholds are observed, the binormal model "interpolat es" and makes available a sensitivity for all possible values of one minus specificity

.
T he follow i ng is a graphical comparison of the empirical curv e (solid line) and the binormal curve (dotted line) using cancer example. Early on binormal ROC curve is above the empirical one, suggesti ng that sensitivity is overestimated
w hen compared w i th the observed values. Later, curves cross and binormal curve now underestimates sensitivity

. Most of the time the difference bet w een the tw o curves is less than 10%. T h is w ould be considered a reasonable fitting binormal ROC curve.

S

e

n

s i t i v i t y

0. 0 0. 1 0. 2 0. 3 0. 4 0. 5 0. 6 0. 7 0. 8 0. 9 1. 0

1 - S p

e

c i f i c i t y 0. 0 0

. 1 0. 2 0

. 3 0. 4 0

. 5 0. 6 0

. 7 0. 8 0

. 9 1

. 0



### 14T he area under the empirical ROC curve w a s 87%, slightly higher than 86% w h ich is the area under the binormal ROC curve. T he confidence lim its are also very similar: (82%, 92%) ve rsus (81%, 92%). By most accounts the binormal analy s is y i elds similar results to that of the empirical curve.



### COMPARING TWO ROC CURVES It is often of interest to compare predictions. How does an easy
- and inexpensive-to-obtain medical diagnosis differ from one that requires an invasive procedure or a radi ologic scan? One can construct ROC curves for each and compare them. Visual comparisons usually reveal several useful features, but statis tical significance betw een tw o ROC curves can also be obtained.
Continuing w i th the cancer example, all of the patients w ho had their SUV measured also have additional information available: size of the lesion under cons ideration (called tumor or T stage, an or dinal variable w i th four categories, 1 through 4), w hether ly mph nodes are in volved or not (called nodal or N st age, an ordinal variable w i th three categories, 0 through 2) and a binary vari able w h ich is the overall assessment of the phy sician w h ich is affected by SUV, T stage and N stage, but other , perhaps unquantifiable factors.

W hen multiple variables need to be combined to make a predict ion, a reasonable w a y is to build a logistic regression model. W e have simply used the main effects in the fo llow i ng code, more careful modeling might produce a better prediction model:

proc logistic data=hn; class tstage nstage overallint; model gold_standard=suv tstage nstage overallint; score out=predprobs outroc=combinedroc;
run;

T he output from the LOGIST I C procedure i ndicates that the area under the curve fo r this combined prediction rule is 90.2%, a slight increase from 87.1% w i th SUV alone. W i t hout access to standard errors it is not possible to judge
w hether this increase can be explained by sampling error or no t. In addition this is a paired- data situation, since every patient receives tw o predictions, one w i th SUV only
, and t he other w i th the combined ru le. T herefore the tw o ROC curves are not independent and any statisti cal procedure evaluating their signifi cance needs to assess the covariance betw een the tw o curves.

%ROC macro that w e used in above to obtain estimates of standard error can also be used to compare tw o ROC curves. First one needs to merge the PREDPROBS data se t created by the ex ecution of PROC LOGIST I C above
w i th the original data set that contains SUV. T h is data set is called MERGED her e and P_ is the variable that contains the predictions from the logistic model. CONT RAST macro variable expects input similar to the CONT RAST statement in PROC GLM, so 1 -1, as bel ow w ould refer to the difference of t he tw o variables stated in the VAR macro variable.

% roc (version, data = merged,var = p_ suv, response = gold, contrast = **1** - **1** );

T he output from this call is given below . W e first see t he area under the curve estimates for each of the ROC curves, along w i th their standard errors and (marginal) confidence interv als. T hen the contrast info rmation is printed. T he next piece of output is the one that compares the tw o ROC curves. T he tw o AUCs differ by about 2% w i th a standard error of 1.3%. T he confidence interval for this difference is -0.4 % to 4.6%, covering 0 and indica ting that the tw o predictions are not statistically distinguis hable. T he p-value for the corres ponding significance test is 0.1021.



### 15The ROC Macro

ROC Curve Areas and 95% Confidence Intervals ROC Area Std Error Confidence Limits

P__ 0.9023 0.0267 0.8499 0.9547 highs 0.8815 0.0293 0.8240 0.9389

Contrast Coefficients P__ highs

Row1 1 -1

Tests and 95% Confidence Intervals for Contrast Rows Estimate Std Error Confidence Limits Chi-square Pr > ChiSq

Row1 0.0208 0.0127 -0.0041 0.0458 2.6732 0.1021

Contrast Test Results Chi-Square DF Pr > ChiSq

2.6732 1 0.1021

Same comparisons can be performed w i thin the context of a binormal model. It is easier to w r ite the binormal model as a regression:



## )
, 0 (
~ )
, 0 (
~ ) exp( ) (

2 2 3 2 1 0



## σ ε τ ε β α α α α

N N U G U GP P G W E + + + + + =

In this model G= 0,1 represents the gol d standard and P= = 0 ,1 is the ty pe of predi ctor (SUV vs combination). U is a random effect modeling the covariance betw een SUV and combi nation results due to the fact that they w e re obtained on the same patients.

2 2 1 2 2 0 

## ) 2 exp( σ β σ σ σ

= =

T he ROC curves induced by this model for eac h ty pe are binormal w i th the same slope, b = σ 0 2 / σ 1 2 . T he intercept is different and given by a 0 = α 1 / σ 1 2 and a 1 = ( α 1 + α 3 ) / σ 1 2 . T herefore it is α 3 that represents t he difference betw een the tw o ROC curves.

T h is model is fit using PROC NLMIXED w i th the follo w i ng statements. T he data set required for PROC NLMIXED contains tw o row s per patient, one for P= 0 and one for P= 1. It needs to have a subject indicator, SUBID in this data set. T he binormal parameters and the A UCs are computed along w i th their standard errors using the EST IMAT E statements.



### 16proc nlmixed data=long; parameters s1=1 s0=1 sr=1 _g=0 _gt=0 _a=0; if gold=1 then s=s1**2;else if gold=0 then s=s0**2; mm=_a+_g*gold+_t*type+_gt*gold*type+u; a1=_g/s1;a2=(_g+_gt)/s1;b=s0/s1; auc1=probnorm(a1/sqrt(1+b**2));auc2=probnorm(a2/sqrt(1+b**2)); model result ~ normal(mm,s); random u ~ normal(0,sr) subject=subid; estimate 'a1' a1; estimate 'a2' a2; estimate 'b' b; estimate 'AUC1' auc1; estimate 'AUC2' auc2; estimate 'AUC1-AUC2' auc1-auc2;
run;

T he relevant output from NLMIXED is given below :

Additional Estimates

Standard Label Estimate Error DF t Value Pr > |t| Alpha Lower Upper

a1 1.8494 0.2396 142 7.72 <.0001 0.05 1.3758 2.3229 a2 1.3307 0.2188 142 6.08 <.0001 0.05 0.8981 1.7633 b 0.5677 0.06834 142 8.31 <.0001 0.05 0.4326 0.7028 AUC1 0.9461 0.02014 142 46.97 <.0001 0.05 0.9063 0.9859 AUC2 0.8764 0.03620 142 24.21 <.0001 0.05 0.8048 0.9480 AUC1-AUC2 0.06970 0.03163 142 2.20 0.0292 0.05 0.007174 0.1322

T he AUCs are estimated to be 94.6% and 87. 6%. Note that the estimate for the AUC of SUV is not identical to the one obtained from the binormal model that use only SUV. T he reason for that has to do w i th the variance parameters. In particular the slope now is common to both predictors and different from the slope in the previous model. T he difference betw een the tw o AUCs is given to be 7% and the p-value for this difference is 0.0292, suggesting a significant improvement over SUV alone by the combination score.

It is not uncommon that the binormal m odel and the empirical model results in different conclusions. T he difference betw een them is similar to the differ ence betw een rank tests and t-tests. If the assumptions underly ing the binormal model hold then the birnomal model has more pow er and this might explain the significant result. On the other hand if the model assumptions do not hold then the T y pe I error ma y be inflated w h ich w ould manifest itself in the form increased false positive results. It is r eally not possible to conclude w h ich one is the driving force here. Exploratory graphical analy s es, like the one performed above remain the bes t w a y of checking the assumptions of binormality

.
More than tw o predictors can be compared using dummy variabl es in the model. One can also adjust for covariates as needed. In summary binormal regression models are very pow erful and flexible but they may be sensitive to departures from model assumptions.

W e caution that despite its enormous popularity
, the binormal models make some strong assumptions. It is no longer rank-based, every single observation contributes to the estimation of the binormal slope and intercept.



### CONCLUSION Predictive models are used by virtually every field of science and almost all bus iness enterprises. ROC curves provide
a w a y of assessing the accuracy of these predictions. Predictions are increasingly being produced by statistical models implemented by computers and mo st of them generate pr edicted probabilities (from a logistic regression model, for example). ROC curves are most useful w hen predictions are continuous, as they w ould be in predicted probabilities. T he methods w e have consi dered in this article can be applied to any continuous prediction, regardless of w hether they are directly observ ed variables (such as SUV) or model- generated abstractions (such as a credit score or a predicted probability )

.



### 17ROC curves can be generalized to ordinal predictions as w e ll. T hese are usually subjective assessments of experts, or an empirical combination of various variables (suc h as cancer staging). T he empirical ROC curve can be generated using the principles outlined above and the binormal can be extended usi ng latent variables to obtain a smooth curve. PROC LOGIST I C and PROC NLMIX E D w ould be the engines of these analy s es and code is available from the author for these analy s es. Com paring tw o ordinal predictors or adjusti ng for covariates can all be done w i thin the regression framew ork that is eas ily implemented in PROC NLMIXED.

Another data structure that has received interest in the context of ROC curves is clustered data. T h is corresponds to multiple observations for each patien t, encountered in medical diagnostic contex ts. F o r example, a PET scan of the abdominal region may be evaluated for st omach, colon and liver separately y i elding clustered data. One can implement a hierarchical model in NL MIXED, although since NLMI XED is limited to one RANDO M statement it is not possible to fit a model to this data set.

It is important to remember that graphical methods, su ch as side-by -side histograms remain the best w a y of communicating y our results as w e ll as checking y our assumptions.



### REFERENCES T hornes JE and Stephenson DB (2001). How to judge the qua lity and the value of w eat her forecast products. Meteorological Applications , 8:307–314.

W ong RJ et al. (2002). Diagnostic and prognostic value of 18F -fluorodeoxy g lucose positron emission tomography for recurrent head and neck squamous cell carcinoma. Jornal of Clinical Oncology , 20:4199-4208.



### RECOMMENDED READING T here are tw o books that cover ROC curves extensively from a statistical pers pective: "T he Statistical Evaluation of Medical T e sts for Classification and Prediction" published by Oxford Press, w r itten by MS Pepe and "Statistical Methods in Diagnostic Medicine" published by Wiley
, w r itten by X H Zhou et al. In addition both SAS Press and Springer are in preparation to publish boo ks on ROC curves w i thin the next y ear.



### CONTACT INFORMATION Your code requests, comments and questions are valued and encouraged. C ontact the author at Mithat Gönen Memorial Sloan-Kettering Cancer Center 1275 York Avenue Box 44 New York, NY 10021 (646) 735-8111 gonenm@mskcc.org h t t p : / / www. m skcc.org/mskcc/html/3185.cfm
SAS and all other SAS Institute Inc. product or servic e names are registered tr ademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Other brand and product names are trademarks of their respective companies.



### 18