Prediction Models: Revolutionary in Principle, But Do They Do More Good Than Harm?

It can sometimes seem as though we are drowning in prediction models. Every month brings a multitude of newly published risk calculators and nomograms to add to the multitude already in the literature—there are more than 100 prediction models on prostate cancer alone—and Web sites such as www.nomograms.org, www.nomogram.org, and www.cancernomograms.org continue to proliferate. As such, it is easy to become somewhat inured to prediction modeling and thus to forget that it constitutes an important shift in the way that medicine is practiced. Biopsy for prostate cancer is a good illustration of the potential role of prediction models. For many years, the typical approach has been to consider biopsy in men with a prostate-specific antigen (PSA) above 4 ng/mL; a positive clinical exam (the digital rectal examination); or perhaps a low free-to-total PSA ratio, such as less than 15%. There are two immediate problems with this approach. First, clinicians must somehow integrate multiple different numbers in their heads. For example, if PSA is just above the threshold, but free-to-total PSA ratio is normal, that might argue against biopsy; if PSA is well within the normal range and free-to-total ratio is abnormal, the appropriate decision might be less clear. If multiple genetic markers are introduced into the clinic at some point in the future, one might well imagine the clinician’s task becoming exponentially more complex. Secondly, the use of clinical cut points makes it difficult to choose thresholds rationally or amend them in the light of patient preference. If a clinician believes that a lower PSA threshold of 3 ng/mL should be used to indicate biopsy, it is unclear how to resolve a disagreement with a colleague who believes that a threshold of 4 ng/mL is preferable. Comparably, one might imagine that a patient averse to biopsy might be advised to watch and wait if his PSA was only slightly above the threshold (say, 4.2 ng/mL), but encouraged to accept a biopsy if his PSA was very high. Yet how high would be high enough for a clinician to insist on biopsy: 6 ng/mL? 7.5 ng/mL? 10 ng/mL? Prediction modeling takes multiple clinical and laboratory variables and converts them into a probability of disease. That probability can then be used in rational decision making. Instead of being asked whether 3 or 4 ng/mL is a better threshold, a clinician can be asked a clinical question: “Taking into account the benefits of finding prostate cancer early, the harms of biopsy such as infection, and the alternative—such as following PSA over time and deciding on biopsy at a later point—how many prostate biopsies would you do in order to find one cancer?.” The answer to this question can be easily converted into a risk threshold: clinicians who would do no more than five biopsies to find one prostate cancer have a risk threshold of 20%; those who would do no more than four have a risk threshold of 25%. Patients with risks higher than those thresholds from a prediction model would accordingly be advised to consider biopsy. Similarly, predicted risk can be used to individualize decision making. Given a patient averse to biopsy, the results of a prediction model can be used as part of shared decision making. Informing a patient that his risk is 60% versus 26% is far more conducive to decision making than reporting a PSA of 11 ng/mL versus one of 5 ng/mL. Despite their enormous promise, prediction models in cancer are currently beset by an important problem. In brief, we do not know whether prediction models do more good than harm. This stems, in large part, from the way in which models have been evaluated. One of the most fundamental principles is that models need to be tested using a different data set than that used to create the model. But with so many models being developed on a continuous basis, it is inevitable that only a minority have been subject to independent validation. The vast majority of models are created and validated on the same data set, with little further analysis. Instead of a limited number of models being proposed, and then evaluated on a wide range of different data sets, we currently have a large number of models that are not independently validated at all. Yet even when models have been evaluated on independent data sets, the statistical methods used have generally been rather unhelpful. Traditionally, statisticians have estimated two aspects of model performance, discrimination and calibration. Discrimination refers to the ability of a prediction model to distinguish between patients. A typical measure of discrimination is the concordance index, which is often described in the diagnostic setting as the area under the curve (AUC). The concordance index gives the probability that, for a randomly selected pair of patients, the model assigns a higher probability to the patient who had the event or who had shorter survival time. Hence the concordance index is measured on a scale from 0.5 (no better than chance) to 1 (perfect discrimination). Calibration refers to whether a prediction for an individual patient is close to his or her true risk. A model is well calibrated if, for every 100 patients given a risk of p%, close to p have the event. Calibration is typically assessed by plotting expected against observed probabilities; good models are close to the 45° line where observed and predicted probabilities are identical. JOURNAL OF CLINICAL ONCOLOGY E D I T O R I A L S VOLUME 29 NUMBER 22 AUGUST 1 2011