Re: “Problems With Risk Reclassification Methods for Evaluating Prediction Models”

In her recent Journal article, Pepe (1) takes a strong stance against reclassification, particularly the reclassification calibration test. Although I agree with several comments, I disagree with her dismissal of this test. Pepe's mathematical assumptions are based on the average predicted risk from the full model equaling the observed risk within reclassified categories. There are several reasons beyond sampling variation why this may not hold in practice. The model may not provide a good fit within the ranges of interest. This is particularly true when interest focuses on those at intermediate or high risk within generally low-risk population-based cohorts. The extent of reclassification may also be too small and random in the higher, more meaningful categories. Cook and Paynter (2) performed a series of simulations, similar to Pepe's, examining the performance of various measures of fit. When considering the full model with true predictors X and Y, they found that the reclassification calibration test will in general be rejected 5% of the time. However, a significant predictor does not always translate into an improvement in reclassification (3, 4). For example, even a statistically significant (P = 0.006) genetic risk score did not improve reclassification calibration (4). In the report by Cook and Ridker (5), although calibration was improved, lack of fit remained for 3 of 8 biomarkers. The focus is indeed usually on the base model. Cook and Paynter's simulations (2) show that, in the null case, the reclassification calibration test performs appropriately, with type I error near 0.05. When Y is associated with the outcome (odds ratio > 1), it is untrue that the test for the X-only model will “generally be rejected.” Simulations found that power was lower, as expected, for the reclassification tests than the test of association. The reclassification calibration test had power similar to that of the net reclassification improvement and the C statistic. Alternatively, power for the likelihood ratio test, the integrated discrimination improvement statistic, and the continuous net reclassification improvement were near each other because these are similar tests (6, 7). Although focus has been primarily on the addition of new variables to a model, reclassification tables can be much broader. They could be used to compare models with very different variables or characteristics, such as a neural network and a classification tree. These comparisons do not reduce to likelihood ratio tests, and it is not true that one model will be preferred over another. Pepe's mathematical assumptions will also not hold in a validation set, which Cook (8) and Ridker et al. (9) have used previously. In this setting, predicted probabilities do not reflect observed risk unless the model offers real improvement. In a model development set, any measure of fit, whether the area under the receiver operating characteristic curve, the net reclassification improvement, or the integrated discrimination improvement statistic, will improve when new variables are added to an existing model. Cook and Ridker (5) used 10-fold cross-validation to demonstrate this optimism. Model validation, whether through external data or resampling, is essential for all measures of model fit, including reclassification. If these methods test the same underlying hypothesis (1, 10), why not stop at the likelihood ratio test? The problem is that a positive association doesn't necessarily translate into clinical utility. From the late 1970s (11) through the present time (12), physicians have risk stratified patients to more appropriately assign treatments. Such stratification has been adopted into national guidelines, both for cardiovascular (13) and other (14, 15) diseases. The reclassification table is designed to explicitly address this need. Pepe's insistence on focusing on the margins misses the point that individuals change within the table. The goal is to see whether those who reclassify upward are actually at higher risk than those who move down. A better understanding of reclassification statistics is welcomed and should lead to better statistical practice. Ongoing work should lead to a better understanding of these and other measures of model fit.