Improving the Statistical Approach to Health Care Provider Profiling

For reports on the performance of health care providers to be effective, profiling must be done using the best statistical methods. Commonly used profiling methods often contain some of the following deficiencies. They ignore important relevant information. They use statistical standards where medical standards would serve better. They use the probability that the observed outcome is extreme, assuming that a hospital's true performance is acceptable to identify providers with extreme rates; this is not the probability that the medical unit's true mortality rate exceeds a given standard. (The true mortality rate is the rate that would have occurred if the hospital had served a very large number of patients.) The 1993 report on coronary artery bypass graft surgery from the New York State Department of Health [1] listed mortality data (deaths before leaving the hospital) and profile statistics for 31 hospitals. These profiles aimed to identify hospitals that had excessively high or low mortality rates associated with coronary artery bypass graft surgery. In this article, we review 3 of the 31 hospitals to see how profile results improve with the use of more information. Two of the hospitals were chosen for their high observed mortality rates (1 had a mortality rate that was substantially higher than the 1992 statewide rate of 2.78%), and 1 was chosen for its low mortality rate. (It is important to keep in mind that the performance of the hospitals may have changed since 1992.) Hierarchical models use information from the available data obtained from all health care providers being examined. These models are so named because they apply to situations with two or more levels of random variation. In the mortality rate example to follow, the first level of the hierarchy specifies a distribution for the random number of observed deaths at a given hospital and the second level specifies possible distributions for the true mortality rates in several hospitals. To use terminology from analysis of variance, level 1 in the hierarchical model concerns the variation of rates within providers and level 2 concerns the variation between true rates of the hospitals. We urge the use of medical standards that specify the largest or smallest medically acceptable true mortality rate in the setting being profiled. How standards are set depends on their purpose; standards meant to encourage quality improvement, for example, may differ from standards meant to distribute pay incentives. Major improvements to profiles will result from the use of medically appropriate performance standards. Case-mix adjustments are made in almost all profile analyses to account for the differences in provider performances attributable solely to differences in the populations served. Hierarchical models accommodate these crucial adjustments. In addition, standard profiling procedures typically ignore units with small caseloads, such as those with fewer than 50 patients. This practice gives little information on the performance of low-volume providers and can provide unfair gaming opportunities (for example, a hospital with a very high mortality rate for 49 patients might refuse to admit further patients in the given category). Hierarchical models require no minimum sample size for a particular health care provider, provided that the ensemble of all providers analyzed has adequate data. Ensemble data are used to correct for regression-to-the-mean bias. In this paper, we review standard statistical profiling methods, showing how successively better results are obtained as more information is included. We recommend the use of hierarchical models to extract ensemble information and advocate the use of more directly relevant standards. The advantages of a hierarchical approach are as follows: 1) The probabilities of performance standards are calculated, 2) comparisons of units are based on medically relevant standards, 3) regression-to-the-mean bias is removed, and 4) providers with small sample sizes remain in the analysis. The benefits of hierarchical modeling apply not only to the mortality data considered here but also to profiles of patient satisfaction, referral rates, and other outcomes. Profiling Data We estimated the true mortality rates associated with coronary artery bypass graft surgery for 31 hospitals in New York State [1] by using data on the number of patients, the number of deaths, and the case-mix difficulty of the patients treated at each hospital. For this analysis, we focused on data for two hospitals (H1 and H2) that had high observed mortality rates. The simplest profile analysis would compare the number of deaths in each hospital. Because 22 deaths occurred at H2, it seems to be a much worse hospital than H1, at which 3 deaths occurred. However, the number of patients served should also be considered: that is, mortality rates, not raw counts, should be analyzed. The disparity in the number of deaths can be completely explained by the caseload of 484 coronary artery bypass graft procedures at H2 compared with 67 at H1. The mortality rates of 4.48% (3 of 67 patients) at H1 and 4.55% (22 of 484 patients) at H2 are almost indistinguishable. This improvement makes a fairer comparison by adjusting for caseload while retaining the basic concept of comparing the number of deaths. The improvements stem from obtaining and using appropriate additional information. An even better approach accounts for case-mix differences. Data from the New York state study [1] show that the patients who had coronary artery bypass graft surgery at H2 were less healthy than those at H1. On average, patients at H1 had 51.1% of the risk for death of all patients who had coronary artery bypass graft surgery (expected mortality rate, 1.42% for the case mix of patients at H1 compared with 2.78% statewide; 1.42 2.78 = 0.511). When we adjusted for this, the 3 deaths at H1 equivalently resulted from 34.2 (0.511 67) procedures done on patients with an average case mix. The risk-adjusted mortality rate is therefore 8.77% (3 34.2) at H1. The risk-adjusted mortality rate at H2 was 5.77%. (With many deaths and an exceptionally healthy patient case mix, it is possible for the relative risk case-mix adjustment method to produce adjusted rates exceeding 1.00.) Risk adjustments contribute vitally to reducing unfair profile evaluations. The need for risk adjustment has led to vigorous research on ways to account for case-mix differences [2-8]. For our purposes, we used the case-mix data and the risk adjustment methods as they were used in the report from New York State [1]. Models and Tests of Statistical Significance Developing good profiling procedures requires specifying probability distributions for the observed outcomes and choosing the standards of acceptable care that define the hypotheses to be tested. Standards that are based on input from medical professionals and from users of the profiles will be the most useful and the most meaningful. When statistical convention alone determines these choices, they are likely to produce inaccurate conclusions and lead to poor decisions. Probability Distributions Once the hospital rates have been adjusted for case mix, a probability distribution is needed to perform a statistical test. The observed count is governed by the hospital's true mortality rate; here, it is the rate that could be observed only if the hospital had treated a very large number of patients with the average case mix. The true mortality rates are unknown and must be estimated from the data. The Poisson distribution is appropriate for these data because the probability of death after coronary artery bypass graft surgery was small (2.78% state-wide). An alternative, the binomial distribution, is well approximated by the Poisson distribution in this case. If the expected number of deaths is very small or if some individual probabilities are large, then normal, Poisson, and binomial assumptions may not be valid and an alternate calculation may be necessary [9]. Commonly Used Decision Criteria Profile analyses often require tests of the null hypothesis that a provider's true mortality rate equals the average rate for all providers. The hypothesis is tested at a specified significance level. Following this convention, the New York State report [1] used the statewide mortality rate of 2.78% as the standard and set the significance level at 0.025. This hypothesis is not very useful: Taken literally, it means that if the true hospital mortality rates differ even by tiny amounts (which one would expect), many of the hospitals would have true rates that exceed the population mean. P Values When a distribution and a standard are specified, the P value can be calculated [10, 11]. A normal approximation to the Poisson distribution will work poorly for the profiles of New York State hospitals in which coronary artery bypass graft surgery was performed, because fewer than 10 deaths were expected at many of the hospitals. The P value for H1 is the probability of observing 3 or more deaths (because 3 deaths occurred at H1), assuming that H1 performed with a true mortality rate of 2.78% for patients with an average case mix. Had H1 performed at this average rate, 0.95 deaths would have been expected. A normal approximation produces a P value of 0.018. Small P values, such as this, identify high true mortality rates; H1 would therefore be identified by this approximate calculation. However, the exact P value based on the Poisson distribution is 0.072 and is too large to identify H1 as a poor performer. Profile procedures that use normal approximations result in incorrect profile estimates if the approximation is inaccurate. Errors such as this are unnecessary; many statistical computer packages make it easy to calculate exact P values for the Poisson and other common distributions. Hierarchical Bayesian Models for Profile Analyses A P value computed to test the hypothesis that a hospital