Statistical benchmarks for health care provider performance assessment: a comparison of standard approaches to a hierarchical Bayesian histogram-based method.

The development and public reporting of performance standards are central to large-scale efforts to monitor and improve health care provider quality (McNamara 2006). Performance standards are also important for pay-for-performance programs (Rosenthal et al. 2005). These initiatives frequently involve characterizing provider performance using quality measures (QMs). For a given provider, a QM may summarize clinical processes of care, patient outcomes, or patient experiences with care. One example of such a data collection effort is the Healthcare Effectiveness Data and Information Set (HEDIS), which contains data on 76 QMs collected across five domains of care from over 90 percent of U.S. health care plans. The National Committee for Quality Assurance (NCQA) requires health plans to submit HEDIS measures for its accreditation process. For the HEDIS portion of the accreditation score, NCQA compares the plan's results to a national benchmark (the 90th percentile of national results) and to regional and national performance thresholds (the 75th, 50th, and 25th percentiles). Similarly, the Medicare Hospital Compare Web site reports the performance of hospitals meeting QMs covering process of care, patient experiences with care, and outcomes. The Medicare Hospital Value-Based Purchasing (VBP) program implemented in fiscal year (FY) 2013 provides additional payments to hospitals with relatively strong performance on a collection of QMs. An important component of determining the hospital bonus amounts is to compare each QM for a given hospital to a performance standard, which is defined as the 50th percentiles of provider performance on that QM. A hospital must exceed this threshold to receive any points for the QM in the bonus payment formula, with the maximum number of points given to those hospitals that exceed the performance benchmark, which is the mean performance of the top 10 percent of providers. Characterizing top performance using percentile cutoffs has also been advocated for smaller, local continuous quality improvement efforts. For example, the Achievable Benchmarks of Care (ABC) (Kiefe et al. 2001) approach defines the performance of the top 10 percent of providers as representing a “realistic standard of excellence” due to having been attained by some providers. Physicians who were randomized to receive feedback about their performance using the ABC method significantly improved their performance relative to others. Although the ABC method was initially developed for application to continuous quality improvement efforts, it has been employed elsewhere as a way to develop domain-wide performance benchmarks in areas such as mental health (Hermann et al. 2006) and primary care (Wessell et al. 2008). Despite the importance of statistical benchmarks to local and national quality improvement efforts, there has been very little exploration of the methodology used for their development. One exception is an analysis of data downloaded from the Hospital Compare Web site that found ABC and methods similar to the one used under the Medicare Hospital VBP program that identify the top 10 percent of providers tend to identify smaller, lower-volume hospitals as being in the top 10 percent, despite exploratory data analyses showing that higher-volume hospitals actually have better performance (O'Brien, DeLong, and Peterson 2008). One explanation provided for this finding is that direct (or raw) estimates of provider performance, such as the observed proportion of times a QM is met for a provider (numerator) out of all possible opportunities (denominator), have high variance, particularly for small providers. In response, O'Brien, DeLong, and Peterson (2008) estimated an alternative statistical benchmark of top 10 percent performance as the 90th percentile of posterior means of provider performance derived from a two-stage Bayesian hierarchical model. This model has gained popularity for profiling health care provider performance due to the greater stability of posterior means versus raw means, particularly for small providers, when the focus is on obtaining an estimate of true provider-specific performance (Christiansen and Morris 1997; Normand, Glickman, and Gatsonis 1997; Burgess et al. 2000; Landrum, Bronskill, and Normand 2000). This stability results from each provider's posterior mean being a shrinkage estimator, which is a weighted average of the provider's raw mean performance estimate and the overall (or “grand”) mean of the performance across all hospitals. Relatively greater weight is given to the overall mean in the hospital's posterior mean when its raw mean has large variance, while less weight is given to the overall mean when a hospital's raw performance estimate has relatively low variance. However, relative to the raw means approach, O'Brien, DeLong, and Peterson (2008) found that the relationship between hospital size and performance was reversed using posterior means, raising concerns that using posterior means over-compensates for small hospital size (higher variance) by weighting the grand mean so heavily for small providers that it becomes very unlikely for small providers to exceed the 90th percentile. Similar concerns have been raised when the goal is to estimate provider-specific performance for low-volume providers (Mukamel et al. 2010; Silber et al. 2010). Another explanation for the findings of O'Brien, DeLong, and Peterson (2008) that has implications regardless of hospital size is that estimating the 90th percentile of performance and identifying which providers exceed that performance are questions that pertain to the distribution of performance across all providers rather than summaries of hospital-specific performance (Paddock and Louis 2011). The posterior means are optimal under squared-error loss for estimating performance for individual providers but are under-dispersed relative to the distribution of the true, underlying provider parameters (Shen and Louis 1998). In addition, direct estimates of provider performance—that is, those calculated as simply the performance for each provider separately, rather than pooling data using hierarchical models—are over-dispersed for estimating the distribution of provider performance (Shen and Louis 1998). The histogram, or empirical distribution function, of provider-specific performance provides the optimal estimate of the distribution of provider-specific performance under integrated squared error loss. Figure​Figure11 illustrates this difference.1 The histogram of the posterior means is underdispersed relative to the true histogram of provider performance, which is denoted by the solid curve (Figure​(Figure1,1, left panel). The raw estimates (Figure​(Figure1,1, middle panel) are over-dispersed relative to the true histogram. The estimated histogram (Figure​(Figure1,1, right panel) correctly characterizes the true histogram of provider performance. Selecting the 90th percentile of provider performance by examining the right-most side of each histogram reveals that the 90th percentile of the posterior means would be too small, while that of the raw mean estimates would be too large. Figure 1 Illustration Using Simulated Data of the True Distribution of Provider-Specific Performance (solid line) versus Distribution (gray) Based on (a) Posterior Means, (b) Raw Estimates, and (c) Histogram Method Despite its appropriateness for estimating the distribution of performance for a population of providers, histogram-based statistical benchmarking has yet to be compared to widely used methods. In this article, their performance relative to other more widely used statistical benchmarking methods is examined to demonstrate whether and how their performance differs, both in terms of setting a performance benchmark and also in terms of which providers are identified as exceeding the performance benchmark. The stability of conclusions drawn about provider performance when different estimation procedures for provider-specific performance are compared with statistical benchmarks is also examined. These methods are illustrated using data on two processes-of-care measures reported on the Medicare Hospital Compare Web site and used in the Medicare Hospital VBP program.