Persistence of HMO performance measures.

Efforts to measure the performance of health plans on clinical dimensions have expanded rapidly over the past 10–15 years. Clinical performance measures can support plan quality improvement efforts, employer/government insurance contracting, and consumer plan enrollment decisions. The foundation of many of the plan measurement initiatives is the Health Care Effectiveness Data and Information Set (HEDIS), maintained by the National Committee for Quality Assurance (NCQA). Typically, reports based on HEDIS utilize data collected before enrollment or contracting decisions are made, and the decisions themselves generally do not take effect for some time after the decision date. For example, measures of health plan performance reported for a company's open health insurance enrollment period in (say) the fall of 2005 are based on data generated by the plan in the year 2004 or earlier, and decisions regarding plan choice made in the fall of 2005 do not take effect until 2006. Thus, there is typically at least a 2-year lag (e.g., 2004–2006) between plan performance and when any action based on that performance would take effect. For this reason understanding the stability of measures of performance over time is crucial to understanding how well these measures can be used to support decision making. Although internal quality improvement initiatives use individual HEDIS or CAHPS measures (Scanlon et al. 2001), decision making is often easier using aggregate measures. For example, from a consumers' viewpoint, having multiple measures of provider quality can be confusing and at times contradictory (Scanlon et al. 1998). For this reason it is important to develop aggregate measures of plan performance. Several studies have developed methods to estimate provider quality when multiple indicators exist (Landrum, Bronskill, and Normand 2000; Staiger and McClellan 2000; Roy and Lin 2002), with a subset of these also estimating the determinants of health plan quality (Scanlon et al. 2005). However, none of the aforementioned papers evaluates the extent to which provider or health plan performance is stable. If we consider only a single indicator of quality per provider, then a simple way to assess stability in health plan performance is to find the one-step transition probability. For example, we could calculate the fraction of plans that had performance scores above/below a certain threshold in (say) 2005 given that they were above/below the same threshold in (say) 2004. Because this estimate is derived without making any assumptions about the underlying distribution that generates the data, this is also called a nonparametric estimate of the transition probability. Despite its great simplicity, this approach is difficult to implement when multiple indicators of quality exist because there would be separate one-step transition probabilities for each individual measure. In this paper, we estimate a statistical model of the determinants of plan quality that allows for multiple indicators of performance in each period and for multiple periods. Health plan performance will reflect the degree of health plan management (e.g., disease or care management), the physician network, and characteristics of the patient population, including compliance with clinical advice. There are a variety of reasons why plan performance might not be persistent. This lack of persistence might stem from true variation in performance because outcomes are stochastic, underlying plan processes or market environments may change, and measurement error may exist. Finally, longitudinal changes in performance can result from plan quality improvement initiatives that vary over time and that may have lagged effects. We take a “parametric approach,” assuming performance is determined by a multivariate stochastic model. In particular, we estimate the parameters of a model that assumes that the multiple indicators in each period are linked via a latent variable that represents true performance. We further allow for measurement error in the individual performance indicators. True performance is assumed to be determined by a set of measured covariates that includes plan and market characteristics (e.g., profit status of the plan, HMO competition, etc.). A part of the variance in true performance is assumed to be driven by unmeasured heterogeneity. We use the estimated parameters to compute the degree to which plan performance is stable. The model is similar to those used in the literature to examine stability across poverty states among families and individuals (Lillard and Willis 1978; Gottschalk 1982). The use of a parametric model, as opposed to simply reporting the stability of a single measure over time, has several advantages. First, because there are multiple measures, estimates of the stability of each individual measure would need to be aggregated to provide a summary measure of stability. A model that allows this aggregation in a structured way is consistent with the statistics literature on aggregation of performance measures (Landrum, Bronskill, and Normand 2000). However, we note that aggregation of this sort only makes sense when the measures themselves are highly correlated both within and across periods. In cases where there is a low correlation among measures, we lose a lot of information that might be specific to a particular measure. Second, the use of a model allows us to address the small sample problem that would arise because we observe an unbalanced panel over a relatively short time frame. By parameterizing the model, we increase our ability to make inferences about the stability of performance over longer periods of time. A further advantage of the parametric model is that we can evaluate how the transition probabilities change as plan and market-level characteristics (i.e., profit status, market competition) change over time. The parameterized model allows us to adjust for changes in observed traits and to disaggregate changes in performance into transitory and persistent components. We apply the parametric model to HMO performance using six childhood immunization measures from NCQA's HEDIS data for the period 1998–2002. Childhood immunizations are crucial to maintaining public health and represent a cost-effective method for delivering preventative health services to children (National Institute of Health (NIH 2003). According to data from the National Immunization Survey 2006, only about 78 percent of the children receive the recommended schedule of immunizations by age two. In 1998, the National Vaccine Advisory Committee published a report that outlined strategies to sustain success in childhood immunizations (NVAC 1999). Their primary recommendation was for health plans to guarantee complete immunization of its eligible children using the schedule endorsed by the American Academy of Pediatrics. Luft and Romano (1993) provide the only other study assessing the stability of health performance measures. They examined the stability of hospital-level measures of mortality from cardiac surgery. That paper did not use a parametric model or attempt to aggregate multiple measures or relate stability to hospital traits. However, it did illustrate the many patterns of instability over time that might exist. No such work exists for health plan performance.