Validating predictive models of mortality: More than meets the eye*

I n this issue of Critical Care Medicine, the study by Dr. Render and colleagues (1) is a well thought out, comprehensive validation of a previously reported Veterans Administration (VA) mortality model (2, 3). For the patient population, this validation study examined two cohorts that collectively spanned 1999 through 2004; the model was originally developed on intensive care unit (ICU) admissions in 1996 and 1997. In addition, the authors expanded their original model to include substantially more diagnoses and tested this new model on one of the cohorts. Applying a previously constructed model to external and more contemporary data is the true test of any predictive model. Although it is clear that the VA model has retained excellent discrimination and yielded standardized mortality ratios near 1.00, this message is obscured somewhat by superfluous analyses. The comparison of the present model with that from the National Surgical Quality Improvement Project (4) is not a cogent one, as the two models predict outcomes at different time points. The sections on face and content validity merely affirm what has been established earlier. Of more interest is the model’s performance across diagnostic subgroups. There is substantial variation in standardized mortality ratios across diagnoses, with consistent overprediction of mortality for neurologic diagnoses (probably because the Glasgow Coma Scale score is not measured). However, among the 84 diagnostic groups in the updated model, 65 (77%) had a C statistic for discrimination of 0.80, which is commendable. Taken together with the excellent overall results, the VA model can be used to accurately benchmark ICU performance in VA hospitals. As with most good studies, the results presented here raise as many questions as they answer. How complex do critical care models of mortality need to be? In which populations can be the model be utilized? How often should predictive models be evaluated? There is a substantial range of complexity among contemporary critical care mortality models. The most basic is the Mortality Probability Model (MPM0-III, (5)), which includes yes/no responses to 17 questions and the patient’s age. All other major mortality models utilize patients’ physiology, usually via a weighted sum of physiologic scores (6 – 8). The number of diagnostic groups included in a model also contributes to its complexity. MPM0-III does not include any coefficients for diagnosis, the Simplified Acute Physiology Score (SAPS) III incorporates the contribution of 34 diagnoses to its score, the Intensive Care National Audit and Research Centre (ICNARC) model uses 101 diagnostic categories, and the Acute Physiology and Chronic Health Evaluation (APACHE) IV model contains 116 diagnostic groups. By comparison, the new VA model has 84 diagnostic categories. Except for MPM0-III, contemporary models have increased the number of diagnostic categories. The reason for this is straightforward: physiology and diagnosis consistently account for the majority of a model’s predictive power (6, 8). Although Dr. Render and colleagues (1) suggest that increasing the number of diagnostic subgroups had minimal effect overall, there is a major benefit in that units with an unusual case-mix will now be able to obtain more precise predictions. So, the general verdict seems to be that complex models are preferred over simplistic models, if (and it’s a big if) an ICU has the electronic capability to collect large amounts of data. Studies have consistently shown that models calibrate poorly when applied to external populations, although discrimination is not affected (9, 10). In the article by Dr. Render and colleagues (1), the new VA population can be considered external because of the different admission time and the expanded number of ICUs and hospitals. The discrimination of the VA model was 0.88, equivalent to the value obtained during model development. Calibration drifted, but not egregiously. Applying this model to populations other than VA ICUs may cause further erosion of calibration because case-mix at VA hospitals is substantially different from non-VA hospitals. Further, the VA electronic medical record does not capture vital signs, urine output, or the Glasgow Coma Scale score. On the positive side, the large pool of available patient data at VA hospitals, coupled with standardized electronic data capture and post–hospital discharge information on mortality, make this a rich source of data for outcomes research. Nevertheless to extend this model to other populations, the VA model should be evaluated in non-VA hospitals using sample sizes close to that in this study. The performance of predictive models erodes over time (11). Therefore they must be revalidated and, if necessary, remodeled at regular intervals. It is clear that 10 yrs is too long an interval (6–8), but what is not clear is what constitutes an acceptably short interval. Dr. Render and colleagues (1) revalidated their model using data collected up to 7 yrs after the original development data set. Their results indicate that their earlier model might still accurately benchmark performance in VA ICUs. Although the Hosmer-Lemeshow statistic was highly significant, it may not mean that the model is inaccurate. The current VA database was considerably larger than that used to initially develop the model (5). This alone could *See also p. 1031.