Limits of observational data in determining outcomes from cancer therapy.

There has been a growing interest in using observational data to study cancer outcomes. This interest is driven in part by the availability of population-based data—in particular, data from the Surveillance, Epidemiology, and End Results (SEER) Tumor Registry that have been merged with Medicare charge data.1 These databases have the advantages of excellent external validity, and they allow for the study of populations that often are not included in clinical trials, such as the elderly, minorities, and patients with higher burdens of comorbidities. In addition, large administrative databases can provide information on patterns of care and treatment compliance2–4; can detect rare toxicities and assess treatment toxicities in representative, population-based cohorts5–9; and can permit the comparison of toxicities across different patient populations.5,8–10 However, more recently, administrative datasets are being used to compare the effects of different treatments on overall survival. This approach has been used across many tumor types, including breast, lung, colon, rectal, prostate, and ovarian cancers.11–19 Selection biases, particularly confounding by indication, are the primary threat to the validity of using observational data to estimate benefits of therapies.20,21 These biases can operate in several ways. For example, in a comparison between therapies where 1 therapy is considered potentially more efficacious (eg, adjuvant chemotherapy vs no chemotherapy), a bias may be expected whereby patients with poorer prognosis cancers would be more likely to receive that therapy. Alternatively, in a comparison involving potentially more toxic treatments versus less toxic treatments (eg, invasive surgery vs radiation treatment or chemotherapy vs no chemotherapy), a selection bias may be expected whereby patients with better underlying health—those considered more likely to tolerate the treatment—would be more likely to receive the more toxic therapy. Investigators clearly are aware of these potential biases and use statistical techniques to address them. Multivariate analyses, stratification, matching, restricting, and propensity analyses often are used adjusting for information available in the datasets, such as age; ethnicity; neighborhood socioeconomic level; and prior diagnoses, procedures, and hospitalizations.11–13,22,23 Nevertheless, unmeasured confounders are likely to persist. In this article, we explore the strong effects of selection biases in observational studies. We hypothesized that we would obtain results from observational analyses that were implausible when considered in the light of published data from clinical trials. We also hypothesized that the usual means of dealing with selection biases, such as controlling for patient and tumor characteristics using multivariate and propensity analyses, would not eliminate the improbable results. We present several examples, including reanalyses of previously published data, to illustrate the effects of common selection biases. For the first example, we selected a situation in which we believed that selection biases might produce implausible results compared with results from a randomized controlled trial. For the second and third examples, we reanalyzed previously published data. In all cases, we examined cancer mortality, noncancer mortality, and overall mortality. We reasoned that any real benefit of cancer therapy could be manifested only through differences in cancer-specific mortality but that selection biases might result in differences in noncancer mortality that would be as great or greater than the differences in cancer mortality calling into question the reliability of using mortality endpoints to assess treatment efficacy in nonrandomized data.