Improving quality assessment through multilevel modeling: the case of nursing home compare.

Many health care quality assessment systems rely on data from patient-level outcomes in order to draw inference about provider performance. Providers having higher rates of adverse outcomes (e.g., infections, pressure sores, or mortality) are assumed to be delivering poorer quality care. Despite the hierarchical nature of these data (e.g., patients nested within health care facilities), there have been relatively few efforts to model health care quality explicitly from a multilevel perspective. Ignoring the multilevel nature of these data can lead to erroneous inferences about care quality. We apply multilevel modeling to a health care quality assessment system—Medicare's Nursing Home Compare—to show how it contrasts with conventional methods and leads to better estimates of care quality. The Nursing Home Compare website is designed to provide consumers with quality-related information on the nation's approximately 17,000 nursing facilities to help them make an informed choice about nursing home care. The nursing home quality measures (QMs) are a major component of this system in addition to results from the facility's licensure and certification report. The 12 chronic care QMs reported on this website are expressed as facility rates of the incidence or prevalence of potentially problematic care processes (e.g., restraints or catheters) or untoward outcomes (e.g., mobility decline or urinary tract infections). The Nursing Home Compare reporting system compares each facility's QM rate to the state and national averages. Facilities with higher than average QM rates are presumed to offer poorer quality care. The QMs could be of great benefit to consumers in helping to identify the best quality facilities, as well as to providers who could compare themselves against peers and identify areas for quality improvement. However, the current QM rates have fundamental problems (Mor, Angelelli et al. 2003; Mor, Berg et al. 2003; Zimmerman 2003; Arling et al. 2005). First, risk adjustment is used sparingly; by giving only minimal attention to differences in resident acuity or risk among facilities, the QMs may generate unfair or misleading facility comparisons. Facilities with residents at higher risk of poor outcomes (i.e., more medically unstable, functionally dependent, or cognitively impaired) are at a disadvantage when compared with facilities taking care of lower risk residents. Second, the QMs fail to adequately deal with estimation error. QM rates are presented in Nursing Home Compare with no information about their accuracy or precision. For example, no consideration is given to facility size even though QM estimates for small facilities are less precise and are more likely to take on extreme values than are estimates for large facilities. The General Accounting Office (2002) criticized the Nursing Home Compare QMs along these lines. This study proposes a better method for estimating facility QM rates. We develop and test multilevel models that produce risk-adjusted empirical Bayes (EB) estimated QM rates with credibility intervals (CIs). Our objectives are to (1) describe the multilevel modeling approach and why it is particularly well-suited to this application; (2) describe a set of multilevel models and produce new fully risk-adjusted EB QM estimates and CIs; (3) assess outcomes from these models—change in facility QM rate, rank, and likelihood of being flagged as having quality problems; (4) draw conclusions about effectiveness of the current versus new QM rates in minimizing estimation error and discriminating between facilities in the quality of their care; and (5) consider the practical implications of this approach.