Profiling Provider Outcome Quality for Pay-for-Performance in the Presence of Missing Data: A Simulation Approach

The use of outcome measures to profile providers in pay-for-performance (P4P) programs is becoming increasingly common. Major P4P initiatives, including the Premier Hospital Quality Incentive Demonstration (Lindenauer et al. 2007), the California Pay for Performance Program (Integrated Healthcare Association 2009), and the United Kingdom's Quality and Outcomes Framework (Doran et al. 2008), have used outcome measures, in addition to process measures, to incentivize quality. Beginning in 2013, Medicare's Hospital Value-Based Purchasing program will also begin to incentivize performance on outcome measures for all acute care hospitals in the United States (Federal Register 2011). A potential problem associated with the use of outcome measures to profile providers in P4P programs, particularly in the outpatient setting, is that a substantial proportion of patients may be lost to follow-up or fail to have their outcomes measured and recorded at specified time points, resulting in missing data. There are two mechanisms through which missing data can undermine profiling. First, the loss of data decreases sample size, which in turn increases random sampling variation, ultimately decreasing measure reliability (Normand et al. 2007). Second, when measures are missing in a way that is correlated with patient outcomes, missing data can lead to systematically biased estimates of provider quality, biasing profiling. This problem, henceforth referred to as “systematically missing” data, undermines the validity of outcome measures. The extent to which missing data affects provider profiling in P4P programs may also depend on the specific approach used to profile providers—that is, to determine which providers receive financial incentives. One prominent approach is to use a relative, or tournament style threshold, in which providers who are ranked above a pre-determined percentile among the population of providers receive a P4P bonus payment (Institute of Medicine 2006; Lindenauer et al. 2007). Another common approach uses an absolute threshold, rewarding providers if they exceed a specified level of measured quality (e.g., providers with 50 percent or more patients achieving treatment goals) (Institute of Medicine 2006; Doran et al. 2008). There is reason to believe that absolute profiling methods may be more sensitive to missing data than relative profiling methods. While both profiling methods will be affected by the decrease in reliability associated with the reduction in sample size from missing data, systematically missing data may be more detrimental to absolute profiling methods. This is because relative profiling methods depend only on the relative ranking of providers, not the precise level of measured quality, while absolute profiling methods depend directly on the level of measured quality. Systemically missing data will always affect levels of quality, but they may not affect relative quality rankings if the effects of systematically missing data are similar across providers, thereby preserving provider rankings. While there is an extensive literature on the problems of missing data in longitudinal clinical studies (Laird 1988; Siddique et al. 2008; Gibbons, Hedeker, and DuToit 2010; Normand 2010), with an emphasis on nonrandom attrition from patient drop-out (Little 1995), we are aware of no research that has assessed the impact of missing data on the accuracy of provider profiling in pay-for-performance programs. We conducted a Monte Carlo simulation study to assess the impact of missing data on the accuracy of both absolute and relative provider profiling methods in the context of primary care depression treatment. Depression is strongly linked to poor adherence to treatment and impaired self-care. In the context of depression care, the effect of systematically missing data remains unknown: patients who remain depressed after the initial stage of treatment may think that treatment is ineffective or may face barriers to receiving treatment and therefore may be likely to drop out of treatment. At the same time, patients who respond to treatment may not see a need for the maintenance of treatment and may be more likely to drop out. Given this uncertainty, the potential impact of systemically missing data is of particular interest and relevance in depression care. Our Monte Carlo simulation study provides several advantages in addressing our research question. First, it allows us to know “true” provider quality rankings—because these are assigned in the simulation—and evaluate the accuracy of profiling methods against this known benchmark. By creating the data-generating process used in the simulation model, we can also partition profiling error into error resulting from random sampling variability and from missing data, which is impossible given an unknown data generating process. Second, our simulation study allows us to conduct sensitivity analysis to understand the conditions under which simulation results hold, and the key parameters to which results are sensitive. Third, the simulation allows us to conduct randomized experiments within the context of our model, allowing us to understand the causal relationship between key model parameters, such as the effect of systematically missing data on profiling error.