Cautionary note regarding the use of CIs obtained from Kaplan-Meier survival curves.

Time to event or survival analyses are a staple of the medical literature and are thoroughly discussed in statistical texts and in methodologic series published by clinical journals. The motivation for this brief cautionary note stems from the original findings on the utility of computed tomography (CT) screening for the early detection of stage I lung cancer published by The International Early Lung Cancer Action Program (I-ELCAP) and the precipitating discussion in the literature. While the subsequent commentary raised many valid concerns regarding the findings, there is one concern, a statistical concern, that has not been expressed and could shed some additional insight into the interpretation of the data. Accordingly, this cautionary note will explore the validity of the reported CI for the estimated 10-year survival rate and offer a suggestion to mitigate the overinterpretation of the estimated survival rate and associated CI. Before a more formal examination of the considerations that may arise when analyzing survival data, it is necessary to examine the motivating example in more detail. The I-ELCAP study screened approximately 32,000 asymptomatic persons at risk for developing lung cancer between 1993 through 2005 using low-dose CT. A total of 484 diagnoses of lung cancer were made, and the median duration of follow-up was 40 months (range, 1 to 123 months). A total of 302 patients underwent surgical resection for the disease, and the 10-year survival rate in these patients was estimated to be 92% (95% CI, 88% to 95%). As this cautionary note will explain, this CI is likely not appropriate. The I-ELCAP study used the nonparametric Kaplan-Meier estimator to derive an empirical estimate of the survival function since the data were right censored. In particular, the median follow-up period was 40 months, and only approximately 10% of the subjects at risk at baseline were still included in the risk set after 6 years of follow-up. While not specifically mentioned in the methodology section of the I-ELCAP report, it is reasonable to assume the Greenwood estimator for the SE of the survival function was computed since it is the default formula in PROC LIFETEST (SAS Institute, Cary, NC), the procedure to calculate the Kaplan-Meier survival curve in SAS (SAS Institute). The Kaplan-Meier and Greenwood estimators share a common attribute adjustments to the estimates only occur where there is an event (a death in the case of the I-ELCAP study). More plainly, the estimated survival function and associated SE remain at the value associated with the most recent event until the next event occurs. A subsequent event causes a decrease in the estimated survival function thereby giving the Kaplan-Meier survival curve its familiar step-down appearance. When interpreting an estimated survival curve, one should ensure that the size of risk set is not too small. A small risk set size could result in large variation in the estimation and difficulty in interpretation. Moreover, if a treatment is expected to delay the event onset time, caution should be exercised to ensure that adequate follow-up time has been obtained. That is, one should cautiously extrapolate the estimated survival probability and associated CI far beyond the last observed event time, particularly when the risk set is small and majority of treatment subjects were censored. A very simple illustration of this caution can be drawn from the I-ELCAP study. In particular, only one of the 302 at-risk patients was followed for 10 years (123 months), but if this patient had an event, just 1 month later, the estimated survival probability at 124 months would be zero (according to the Kaplan-Meier product limit estimator, S (124) 0.92 (1-1)/1 0) and the associated CI would be from 0 to 0 since the SE would also be estimated to be zero. These estimates are in stark contrast to the reported 10-year survival estimates. Clearly, this sensitivity to single observation must be taken into account when the data are interpreted. In light of the potential for a misleading interpretation of estimated survival rate and associated CI, a general recommendation can be made. One should cautiously interpret estimates obtained from a Kaplan-Meier survival analysis when the risk set is small. Pocock et al recommend, in the context of the graphical display of the survival curve, ensuring that the risk set size is at least 10% to 20% of the participants that have not yet experienced the event. This suggestion would appear appropriate in this case as well. In addition, caution should be exercised in general with both the Kaplan-Meier and Greenwood estimators when the risk set is small (say, lower than 30) since they both rely on some asymptotic theory that requires large risk set size. When the risk set becomes small, the asymptotic theory used to derive the formulas may not be valid, so the obtained estimates may be questionable. In the context of the patients undergoing surgical resection in the I-ELCAP study, limited data is available to estimate the 10-year survival rate given the median length of follow-up for the entire cohort was only 40 months and only one of 302 patients was observed for more than 10 years. The risk set size dramatically decreases for each JOURNAL OF CLINICAL ONCOLOGY COMMENTS AND CONTROVERSIES VOLUME 27 NUMBER 2 JANUARY 1