Rare adverse medical events in VA inpatient care: Reliability limits to using patient safety indicators as performance measures

Patient safety has become an issue of focal interest in the evaluation of health care quality (Kohn, Corrigan, and Donaldson 1999). Recently the federal Agency for Healthcare Research and Quality (AHRQ) has developed a comprehensive set of standardized patient safety measures, derived from hospital discharge data, which potentially might be used to compare performance within and between health care systems. In collaboration with the University of California-Stanford Evidence-based Practice Center, AHRQ has put forth 23 Patient Safety Indicators (PSIs) that use administratively obtained discharge data to calculate risk-adjusted rates of medical/surgical adverse events in hospitalized patients (AHRQ 2003a; Zhan and Miller 2003a). Hospitalizations involving adverse events identified by the PSIs have been associated with worse outcomes and higher costs in both the private sector (Zhan and Miller 2003b) and the Veterans Health Administration (VA; Rosen et al. 2005). AHRQ also has produced statistical software that calculates PSI rates and variances from administrative hospital discharge data, using comorbidities to risk-adjust them for cross-system comparisons. The intention is that the PSIs eventually may help health care managers improve quality and medical consumers choose higher quality services. Potentially, the PSIs may serve dual purposes, including (a) case-finding to explore the local root causes of adverse events for corrective intervention, and (b) establishing normative rates against which a system's overall performance can be compared. An implication of the second purpose, however, is that the reliabilities of the PSI rates must be demonstrated. Studies of PSI rates in the VA (Rosen et al. 2005, 2006a, 2006b), the nation's largest health care system, have generated interest in their potential application as performance measures. Because PSI rates are derived from administrative data, they have been promoted as easily obtained, inexpensive, and objective measures of medical quality. But if they are to serve as adequate performance measures they also must be both reliable and discriminating. As some health care systems consistently perform better than others, if PSI rates reflect underlying quality, they should be sufficiently sensitive to distinguish systems from one another, and their rank orders should show some stability over time. Rosen et al. (2006) showed that hospital-level PSI rates correlated after a 3-year lag, but the proportions of variance accounted for were low, ranging from 0.06 to 0.29; a shorter lag may have revealed better reliability. Fortunately for health care consumers, the adverse medical events that the PSIs identify are very uncommon occurrences. Unfortunately for health care analysts, the rareness of adverse events may limit the reliability of PSI rates as performance indicators. PSIs vary widely in terms of how uncommon the underlying events are. Some, such as Decubitus Ulcer, occur each year in nearly every hospital, but others, such as Death in Low Mortality DRGs, or Foreign Body Left during Surgical Procedure, are so rare that AHRQ's current algorithms do not even provide variance estimates for them from which risk-adjusted rates and confidence intervals can be derived. As PSI rates reflect rare adverse events, how well do they reflect changes in health care quality over time or differences among providers? To how large a health care system should they be applied to achieve reliable quality measures? To address these questions we calculated annual PSI rates from 9 years (1997–2005) of VA hospital discharge data. For each PSI, we assessed stability/change across time for the VA health care system overall, for each of the 22 regional Veterans Integrated Service Networks (VISNs) into which the VA system is organized (in 2002, two VISNs were merged into one, but we kept their data separate for these analyses), and for each VA Medical Center belonging to these VISNs. We anticipated that from year to year, PSIs would be more stable if they are calculated for larger health care systems, i.e., VISNs, than at the hospital level. We expected, as well, that those PSIs that are based on higher numbers of underlying adverse events would be more stable over time.