Population-based external validation of a competing-risks nomogram for patients with localized renal cell carcinoma.

TO THE EDITOR: Inarecentarticle,Kutikovetal describedanomogram predicting renal cell carcinoma (RCC) –specific, noncancer, and othercancermortality inpatientswithlocalizedsurgically treatedRCC.In their analysis, the authors relied on 17 registries of the Surveillance, Epidemiology, and End Results (SEER) database and included 30,801 patients treated surgically between 1988 and 2003. The results of their elegant analysis yielded a competing-risks nomogram in which patient age, race, sex, and tumor size predict the three outcomes of interest: RCC-specific, noncancer, and other-cancer mortality. The validity of the proposed model, as provided by Kutikov et al, was only supported with calibration data, which provide a graphical representation of the relationship between predicted and observed mortality rates. Calibration results only represent partial evidence of the model’s ability to predict the outcome of interest, and model accuracy (c-index) is needed to complement calibration data. The suboptimal nature of the validation offered by Kutikov et al is further compounded by its internal nature: the same population is used for model development and its validation, which may artificially inflate the model’s performance. Customarily, a formal validation should rely on an independent external cohort. Within an external validation, calibration results are not sufficient to provide adequate proof of validity. Discrimination (accuracy), which is quantified by the c-index, represents the most important validation benchmark and should invariably accompany calibration results. Accuracy values range from 0.5 to 1.0, where 0.5 represents random predictions and 1.0 represents perfect predictions. The importance of providing both metrics (discrimination and calibration) isdepictedbya largenumberofhighquality studieswhereprognostic models were externally validated. From a practical perspective, a model may be well calibrated, but may have low accuracy, in which case it is of no or limited value. Alternatively, a model may have high accuracy, but its calibration may be poor in a subset of patients. Therefore, it is important to provide both calibration and discrimination, if a model is to be used in clinical practice. BasedonlackofdiscriminationtestingoftheKutikovetalmodel,we decided to revisit the study end points. For this purpose, we relied on a more recent version of the SEER 17 database, where 47,192 patients were treated surgically between 1988 and 2006, for localized RCC. These patients were identified according to the criteria of Kutikov et al. This population was then randomly divided into development (n 23,639) and external validation (n 23,553) cohorts. We then replicated the previously reported competing-risks nomogram according to the specifications of Kutikov et al (Fig 1A). The goal was to test the calibration and discrimination properties of this nomogram within an external validation cohort. Our calibration results nearly perfectly replicated those reported by Kutikov et al, which confirmed the similarity between the two models. The calibration showed a virtually perfect agreement between predicted and observed mortality rates across the entire spectrum of predicted probabilities (Fig 1B). Moreover, in the external validation cohort, the discrimination (c-index) was 0.73 for the prediction of noncancer mortality, 0.70 for the prediction of RCC-specific mortality, and 0.71 for the prediction of other-cancer mortality. This implies that the model is reasonably accurate. This level of accuracy is highly likely to be superior to clinician ratings. For example, in breast cancer, clinicians were 54% accurate in predicting lymph node status versus 72% for nomogram-based predictions. Despite a lesser accuracy in the current model relative to another nomogram that can predict RCC-specific mortality after nephrectomy with a c-index from 86.7 to 89.2%, the current nomogram predicts three different end points, which otherwise cannot be predicted with any other tool. In consequence, the current nomogram should for now be considered as a prognostic standard. These results complement the crucial information that was missing from an otherwise elegant, pertinent, and extremely timely contribution authored by Kutikov et al. The combination of external validation results from our analysis with the original findings of Kutikov et al will provide clinicians with a prognostic tool that has proven calibration and discrimination abilities. These metrics are essential when a model’s predictions are considered for use in clinical practice.