Creating the Evidence Base for Quality Improvement Collaboratives

Deficiencies in the quality and safety of health care remain a significant concern in the United States and abroad as evidence documenting gaps between actual and recommended clinical practices continues to accumulate (1, 2). The significance of the well-recognized health care quality chasm has been acknowledged by a broad array of stakeholders, who have responded with efforts to identify, understand, and correct specific shortcomings in health care delivery. The quality improvement collaborative (QIC) examined by Landon and colleagues in this issue (3) is arguably the health care delivery industry's most important response to quality and safety gaps; it represents substantial investments of time, effort, and funding. Largely developed and popularized by the Boston-based Institute for Healthcare Improvement (IHI) and best exemplified by IHI's Breakthrough Series collaborative program, the QIC method has been adopted on a large scale by the U.S. Health Resources and Services Administration (HRSA) (4) and the United Kingdom's National Health Service (NHS) (5). The Veterans Health Administration (6) and numerous smaller health care systems and individual hospitals and clinics worldwide have adopted the method on a smaller scale (7). The QIC method brings together a group of participating (collaborating) health care delivery organizations (typically between 20 and 40) and guides them in studying a specific health care quality problem, designing and implementing specific solutions, evaluating and refining these solutions, and disseminating findings to other organizations. Each participating organization is represented by a 3- or 4-person team; all teams meet together with a small faculty of experts in a series of 2 or 3 multiday collaborative learning session meetings, which take place over several months. During the meetings, the team members learn improvement techniques, exchange insights and advice, and generate enthusiasm and a shared sense of commitment to achieving common improvement goals and outcomes. Teams return to their organizations between learning sessions to apply their new knowledge and ideas in a PlanDoStudyAct framework (8). They conduct repeated cycles of quality problem diagnosis, development and implementation of small-scale improvement efforts, assessment of effects, and refinement and expansion of effective actions until desired outcomes are achieved. The QIC method is described in detail elsewhere (7, 9, 10). Although estimates of total investment and applications of the QIC method are not available, IHI reports that Since 1995, IHI has sponsored over 50 [Breakthrough Collaborative] projects on several dozen topics involving over 2,000 teams from 1,000 healthcare organizations (11). The IHI tally includes a large series of collaboratives conducted under HRSA sponsorship but excludes NHS-sponsored collaboratives and numerous other collaborative efforts conducted without IHI's direct involvement. The widespread acceptance and application of the QIC method are often questioned by observers citing the modest quantity and quality of published evidence that supports its effectiveness (12, 13). Although numerous reports of collaboratives have been published, they consist primarily of subjective or self-report assessments of QIC effects and qualitative summaries of key lessons learned by collaborative leaders and participants. Such first-hand reports offer important insights into the black box of improvement methods and processes (information generally absent from published evaluations of other quality improvement methods). However, they are incomplete without complementary, objective impact evaluations and are probably biased in favor of positive findings. This bias results from demand and supply factors. Demand-induced bias occurs because much of the published information is concentrated in management- and practitioner-oriented journals, such as The Joint Commission Journal on Quality and Safety and Quality Management in Health Care, whose mission and readership attract articles offering practical guidance and insights from only the successful quality improvement efforts. Supply-induced bias occurs when authors pursue quality improvement rather than research goals and document only successful collaboratives. Methodologic weaknesses in the individual studies exacerbate distortion in the published QIC evidence base caused by publication bias. Published assessments of collaboratives generally use uncontrolled pretestpost-test designs that cannot rule out plausible alternative explanations for observed improvements, such as secular trends. (A detailed table listing published assessments of collaboratives and describing key features of the evaluation design, methods, and measures in each published article is available from the author upon request.) Although QIC authors routinely acknowledge design limitations and suggest cautious interpretation of positive findings (14), such cautions are easily overlooked. Probable errors in measurement are also pervasive. They include outcome measures that rely on participants' unvalidated self-reports or collaborative leaders' subjective ratings of readily observed phenomena (such as team enthusiasm, commitment, and adherence to the collaborative process) rather than objective measures of clinical practice or outcome change. Participants' self-reported outcome measures are typically derived from nonrigorous, highly variable measurement efforts, which often focus on short-term effects measured during or immediately following the intensive collaborative period. This results in the capture of positive effects that may be temporary and driven by Hawthorne effects rather than fundamental, lasting change. Furthermore, subjective ratings provided by collaborative participants and leaders are subject to unintentional and unrecognized biases generated by common human decision and judgment heuristics. For example, expectation biases and the phenomenon of belief perseverance combine to produce systematic overweighting of evidence and observations that confirm a priori expectations and beliefs and underweighting or discounting of evidence that does not support the effectiveness of the QIC method (15). These phenomena reinforce faith and belief in the effectiveness of the QIC method, which contributes to further accumulation of published positive findings and an escalating cycle of belief and confirmatory evidence (12, 16). The apparent inconsistency between widespread belief in and use of the QIC method and the available supporting evidence heightens the importance of the study by Landon and colleagues. The authors reported on a large HRSA-sponsored collaborative that addresses the care of patients with HIV infection or AIDS. The HRSA has embraced the QIC method and has recently encouraged or mandated the participation of grantee clinics in a series of QIC projects (4). Landon and colleagues studied 44 HRSA-contracted clinics participating in the collaborative (some on a voluntary basis and some to meet contract requirements) and 25 comparison clinics. Although clinics were not randomly assigned, the authors selected comparison clinics through a careful matching process, which provided a basis for ruling out secular trends and other possible explanations of the positive effects often observed in uncontrolled pretestpost-test evaluations of collaboratives. Outcome measures were defined consistently across all sites, and comparable measures were obtained through chart review for representative patient samples from all sites. Although medical record reviewers were selected from the participating sites and the authors provide no information about chart reviewer training (or about reliability, validity checks, or other efforts to minimize measurement error), the level of rigor in the study by Landon and colleagues exceeds that of earlier evaluations of collaboratives. Any remaining bias seems more likely to produce false-positive rather than false-negative findings. Landon and colleagues' study demonstrates small and generally insignificant prepost improvements among both intervention and control sites in most of the 8 quality indicators measured. For the 2 indicators showing the greatest improvement (11% and 7.3%) among intervention clinics, comparable improvements were seen among control clinics as well. The small differences between the intervention and control clinics did not reach statistical significance and therefore did not preclude the attribution of observed improvement to secular trends. Several supplementary analyses compared subgroups of clinics (for example, newly funded vs. more established HRSA-funded clinics and clinics with low quality-of-care indicator scores at baseline vs. those with higher baseline scores) and subgroups of patients (for example, patients already receiving highly active antiretroviral therapy). The authors consider and rule out numerous potential explanations for their null findings, including the possibility that control clinics conducted their own improvement initiatives. They summarize many of the key design shortcomings in the existing evidence base and note that their own study would have reported (and attributed to the intervention) statistically significant improvements in 2 key measures in the absence of comparison group data. Landon and colleagues thoroughly discuss the limitations of their study but note that most seem unlikely to have produced their null results. Unfortunately, the data collected do not provide the information needed to understand and explain their findings. For example, the intervention sites may not have correctly diagnosed and understood the causes of targeted quality problems and may not have developed or fully implemented appropriate corrective actions. In addition, corrections may not have been sufficient to overcome the full spectrum of barriers to improvement. Thus, although the study by Landon and colleagues