Comparing Hospital Mortality in Adult Patients with Pneumonia: A Case Study of Statistical Methods in a Managed Care Program

Comparing hospitals by their outcomes has become a popular method of drawing inferences about their relative quality. Outcome studies are now done regularly by employer- and insurer-sponsored managed care programs [1] and by federal [2] and state agencies [3-5]. Mortality, a well-defined outcome, has received the most study, and the public availability of raw mortality data has led to their frequent use as a measure of quality of care. Differences in mortality are known to be attributable to differences in patients' severity of illness [6, 7] or comorbid conditions [8]; thus, researchers have tried to develop methods to adjust raw data for these factors [9]. A well-described danger of such methods lies in our inability to identify all clinical predictors of mortality from computerized data sets [10-13]. In addition, a fundamental dispute remains about whether there is any correspondence between mortality and quality of care [14, 15]. The desire of health care payers for simple measures of hospital performance conflicts with the skepticism of health care providers about the validity and fairness of the assessment process [16, 17]. A less frequently discussed but equally vital issue in the comparison of patient outcomes is the appropriateness of the statistical methods used for analyzing data and interpreting results. Raw data are clearly insufficient for comparing hospitals or physicians; some statistical methods must be invoked. Ideally, using a set of techniques called standardization [18], the comparison of different samples of patients in varied clinical settings should be feasible. We examine one attempt to compare hospitals based on their observed mortality; this attempt was made in 1992 by consultants to a managed care program for a large corporation. The corporation sought to determine which of the hospitals serving the corporation's employees in central Pennsylvania delivered better quality of care as reflected in part by fewer in-hospital deaths in 1989 and 1990. Partly on the basis of the methods reported here, the corporation selected 10 of these hospitals to be eligible for its managed care network. Focusing on adult pneumonia, which was one of the diagnostic groups used by the consultants, we describe the consultants' methods and re-create their results for 1989 and 1990. Then, we reassess those results using more appropriate methods, and finally, we validate results using newly acquired 1991 data. (A glossary of statistical terms is given in Appendix 1.) Methods Choice of Diagnosis-Related Groups The study sample included data reported to the Pennsylvania Health Care Cost Containment Council on all inpatient hospitalizations for adult pneumonia (diagnosis-related groups 089-090) in 1989 and 1990, and, later, in 1991, for 22 hospitals in central Pennsylvania. Because we were using the consultants' methods, we excluded from analysis all patients older than 65 years of age, for whom Medicare would likely be the primary insurer. In addition, we excluded four patients with the acquired immunodeficiency syndrome and seven awaiting organ transplantation because only one hospital provided these specialized kinds of care. Examination of the 1991 data did not begin until after completion of the re-analysis for the years 1989 and 1990. The public-use data sets include coded discharge abstracts and the MedisGroups (Mediqual Systems, Westborough, Massachusetts) admission severity group, an indicator of severity of illness [19]. Admission severity group attempts to distill many key clinical findings at patient admission into a single, five-level severity score. In the years 1989-1991, admission severity group was not diagnosis specific; a patient received a given admission severity group score regardless of the cause of his or her illness. All hospitals in Pennsylvania must subscribe to the MedisGroups system, calculate admission severity groups retrospectively from medical records, and report these data and computerized discharge abstracts to the Cost Containment Council. The Managed Care Consultants' Analysis The consultants relied only on admission severity group to adjust for both severity of acute illness and comorbid conditions. Patient age, which is not an element in admission severity group, was not used. The consultants' statistical technique is best described as a stratified, standard, normalized analysis. First, they calculated death rates for each admission severity group within each institution. Next, they computed the unweighted mean and standard deviations for admission severity group death rates for all hospitals. They then computed a standard, normalized [20] difference between the hospital and the average death rate for each level of admission severity group by dividing the difference by the standard deviation of death rates. Finally, they combined these normalized differences across diagnosis-related group levels within each hospital by a weighted average. Weights depended on the fraction of the hospital's admissions in each admission severity group. The consultants did not calculate standard errors or P values for these weighted averages of normalized differences; they displayed their results along a horizontal line, indicating the location of the average normalized difference and the relative position of a given hospital along the line. Further details of our implementation of the consultants' methods are given in Appendix 2. Methods of Reanalysis Our reanalysis of the data on adult pneumonia from the same 22 hospitals sought to confirm or refute the consultants' results with conventional methods for data with binary outcomes, in this case, death or survival at discharge. We implemented model-based standardization [21] to compare hospital mortality adjusted for multiple patient risk factors. First, we verified the association between death and the patient's admission severity group by logistic regression as implemented in SAS, Proc Logistic (SAS Institute, Cary, North Carolina). Next, the hospital with the worst outcome in 1989-1990 (hospital 1) as calculated by the managed care consultants' method Table 1, was compared with the others by fitting two nested regression models: one with only admission severity group as a covariate, the other with admission severity group plus an indicator variable to detect any difference between hospital 1 and all other area facilities. The likelihood ratio statistic for the differences between the two models showed whether the hospital with the worst outcome, according to the consultants' analysis, was truly different from the other hospitals. Table 1. Mortality for Adults Patients with Pneumonia in 22 Central Pennsylvania Hospitals, 1989-1991* Also relevant to the outcome of many acute diseases is patient age, a factor readily available in the public-use data set but not used by the consultants. Based on a preliminary calculation showing that mortality increased with patient age, patients were grouped into three age ranges: 18 to 39 years, 40 to 54 years, and 55 to 64 years. For age to affect interhospital comparisons of pneumonia mortality, two conditions must be met: The distribution of ages must differ across hospitals and age must be associated with death after controlling for admission severity group. We tested the first condition by cross-classifying the three age categories and the 22 hospitals and calculating the Mantel-Haenszel chi-square statistic for differences in mean age across hospitals. We tested the second condition by using the likelihood ratio statistic for the difference in two logistic regression models: one with both admission severity group and age as factors and the other with only admission severity group as a factor. The last step in testing whether hospital 1 differed significantly from the others again involved the comparison of two nested models. For this comparison, the two models included both age and admission severity group as standardization factors, and one also included an indicator variable for hospital 1. The ratio of the odds of mortality and a P value were computed as previously described. After determining whether hospital 1 had an adjusted mortality higher than that of the other hospitals, we investigated whether another hospital among the 22 might have had significantly higher mortality than expected. Logistic regression also allows for the calculation of predicted probabilities of death for each patient with a given age and admission severity group level. By summing the number of deaths and this predicted probability of death over all patients with pneumonia within each hospital, we computed at the hospital level both an observed and a predicted (or expected) number of deaths. Hospital 4 showed the largest positive difference between observed and predicted numbers of deaths and thus became the test hospital for further analysis of differences in mortality. As before, we fit two nested regression models, one with admission severity group and age as covariates and the other with an additional single indicator variable for hospital 4, to test whether patient mortality was significantly higher there than at other facilities. Multiple Comparisons Contrasts among hospital death rates in the consultants' study, as in typical outcomes analyses, were determined by the data rather than planned in advance. Each hospital was compared in turn to determine whether it differed from all others. In this situation, a critical P value of 0.05 used repeatedly for each hospital comparison did not function to limit type I error, which was the probability of falsely classifying a hospital as a mortality outlier. Diehr and colleagues [22, 23] have commented on the issue of multiple comparisons, or multiple statistical tests, in a related context. The Bonferroni method [24], one of several available techniques [25], was used to adjust the critical P value to control type I error for multiple comparisons of hospitals. Computer Simulatio