Technical Review

Number 9 

Closing the Quality Gap: 

A Critical Analysis of 

Quality Improvement Strategies 

Volume 1—Series Overview and Methodology 

Prepared for: 

Agency for Healthcare Research and Quality 

U.S. Department of Health and Human Services 

540 Gaither Road 

Rockville, MD 20850 

www.ahrq.gov 

Contract No. 290-02-0017 

Prepared by: 

Stanford–UCSF Evidence-based Practice Center 

Kaveh G. Shojania, M.D. 

Kathryn M. McDonald, M.M. 

Robert M. Wachter, M.D. 

Douglas K. Owens, M.D., M.S. 

Series Editors 

Amy J. Markowitz, J.D. 

Managing Editor

AHRQ Publication No. 04-0051-1 

August 2004 
This document is in the public domain and may be used and reprinted without permission except 

those copyrighted materials noted, for which further reproduction is prohibited without the 

specific permission of copyright holders. 

Suggested Citation: 

Shojania KG, McDonald KM, Wachter RM, Owens DK. Closing The Quality Gap: A Critical 

Analysis of Quality Improvement Strategies, Volume 1—Series Overview and Methodology. 

Technical Review 9 (Contract No. 290-02-0017 to the Stanford University–UCSF Evidencebased Practices Center). AHRQ Publication No. 04-0051-1. Rockville, MD: Agency for 

Healthcare Research and Quality. August 2004.

ii
Preface 

The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-based 

Practice Centers (EPCs), sponsors the development of evidence reports and technology 

assessments to assist public- and private-sector organizations in their efforts to improve the 

quality of health care in the United States. This report, Closing the Quality Gap: A Critical 

Analysis of Quality Improvement Strategies, was requested and funded by the Agency for 

Healthcare Research and Quality. The reports and assessments provide organizations with 

comprehensive, science-based information on common, costly medical conditions and new 

health care technologies. The EPCs systematically review the relevant scientific literature on 

topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to 

developing their reports and assessments. 

To bring the broadest range of experts into the development of evidence reports and health 

technology assessments, AHRQ encourages the EPCs to form partnerships and enter into 

collaborations with other medical and research organizations. The EPCs work with these partner 

organizations to ensure that the evidence reports and technology assessments they produce will 

become building blocks for health care quality improvement projects throughout the Nation. The 

reports undergo peer review prior to their release. 

AHRQ expects that the EPC evidence reports and technology assessments will inform 

individual health plans, providers, and purchasers as well as the health care system as a whole by 

providing important information to help improve health care quality. 

We welcome written comments on this evidence report. They may be sent to: Director, 

Center for Outcomes and Evidence, Agency for Healthcare Research and Quality, 540 Gaither 

Road, Rockville, MD 20850. 

Carolyn M. Clancy, M.D. Jean R. Slutsky, P.A., M.S.P.H. 

Director Acting Director, Center for Outcomes 

Agency for Healthcare Research and Quality and Evidence 

 Agency for Healthcare Research and Quality 

iii
Acknowledgments

During the preparation of the series volumes, Dr. Owens, Dr. Goldstein, and Dr. Nayak were 

supported by the Department of Veterans Affairs. 

We are grateful to our many technical experts and content reviewers for their uniformly 

thoughtful comments. 

We give special thanks to our panel of technical advisors: Jeremy Grimshaw, MBChB, PhD, 

University of Ottawa, Cochrane Collaboration Effective Practice and Organisation of Care 

Group (EPOC); Andrew Oxman, MD, MSc, Department of Health Services Research, 

Norwegian Directorate for Health and Social Welfare, EPOC; Russell Glasgow, PhD, Kaiser 

Permanente, Colorado; Martin Eccles, MD, Centre for Health Services Research, University of 

Newcastle upon Tyne, EPOC; Harmon Jordan, ScD, New England Medical Center Evidencebased Practice Center; and Val Lawrence, MD, MSc, The University of Texas Health Science 

Center at San Antonio and South Texas Veterans Health Care System; whose attentive guidance 

from the beginning of the project was critical to the development of the methodology and 

structure for the series, and who were unstinting with their assistance under unusually tight time 

constraints. 

We are grateful to our panel of peer reviewers: Karen Adams, Institute of Medicine; Dale 

Bratzler, DO, MPH, Oklahoma Foundation for Medical Quality, Inc.; Lee Green, MD, MPH, 

University of Michigan; Norman Kaplan, MD, University of Texas, South Western Medical 

Center; Jerod Loeb, PhD, Joint Commission on Accreditation of Healthcare Organizations; Brian 

Mittman, PhD, Center for the Study of Healthcare Provider Behavior, VA Greater Los Angeles 

Health Care System; Peter Rudd, MD, Stanford University School of Medicine; William Rollow, 

MD, Centers for Medicare & Medicaid Services; Vincenza Snow, MD, American College of 

Physicians; William Caplan, MD, Kaiser Permanente, Patricia Barry, MD, MPH, Merck Institute 

of Aging and Health; Rodney Hornbake, MD, ProHealth Physicians MSO, Inc., Somerset Family 

Health Care; Marc Jaffe, MD, Kaiser Permanente Care Management Institute; Michelle Wong, 

MPH, MPP, Kaiser Permanente Care Management Institute; and Jim Zazzali, PhD, RAND 

Institute; who read early drafts of the chapters and provided timely and detailed comments. 

We are extremely indebted to our colleagues who worked together to develop a common 

approach across the clinical topics. In particular, we thank Mary Goldstein, MD, MS, Judith 

Walsh, MD, MPH, and Sumant Ranji, MD for sharing the wisdom they gained in researching 

their own topics with each others’ teams. 

Finally, we extend our thanks to Donna Neumark, PhD for her help in reviewing articles; Alan 

Bostrom, PhD for statistical analyses support; and Allan Korn, MD, FACP, Blue Cross and Blue 

Shield Association for providing our team with important background information and contacts. 

The views expressed are those of the authors, and not necessarily those of the Department of 

Veterans Affairs. The technical advisors and peer reviewers are not responsible for the accuracy 

of any of the content of this Volume.

iv
Structured Abstract 

Substantial evidence suggests that there is a wide gap between evidence-based best practices 

and those treatment practices actually used in day-to-day clinical medicine. To bring data to bear 

on this “quality gap” and the opportunities that exist to bridge it, the Agency for Healthcare 

Research and Quality (AHRQ) engaged the Stanford–UCSF Evidence-based Practice Center 

(EPC) to compile a critical analysis of the existing literature on quality improvement (QI) 

strategies for a selection of 20 disease and practice priorities identified in a 2003 Institute of 

Medicine report. 

In Volume 1 of Closing the Quality Gap, we provide an overview of our methods and the 

theoretical underpinnings of the field, which we will rely on to review and analyze the literature 

on the quality gap in a number of the IOM-identified priority areas that will appear in subsequent 

volumes. We describe the genesis of the quality implementation field, providing some historical 

perspective on the science of translating research into practice. We then set forth our 

methodology: our reviews generally are restricted to studies that are likely to have strong validity 

(randomized controlled trials, well controlled before–after studies, and interrupted time series 

studies). To ensure consistency across our reviews, we introduce a taxonomy for nine QI 

strategies (provider reminder systems; facilitated relay of clinical data to providers; audit and 

feedback; provider education; patient education; promotion of self-management; patient 

reminders; organizational change; and financial, regulatory, or legislative incentives). 

We hope the volumes in this series will be an essential source of accessible and critical 

analyses of the evidence regarding QI strategies that can help close the quality gap. 

v
This page intentionally left blank

vi
Contents

Structured Abstract ..........................................................................................................................v 

Summary..........................................................................................................................................1 

Chapter 1. An Introduction to the Report ........................................................................................5 

The Genesis of “Closing the Quality Gap” ..................................................................................5 

Origins of the Quality Movement.................................................................................................6 

Translating Research Into Practice: What Do We Know? ...........................................................7 

The Theoretic Underpinnings of Quality Improvement Efforts...................................................8 

What Conclusions Can Be Drawn From the Report’s Evidence? ..............................................10 

Organizational Framework of This Volume...............................................................................11 

Chapter 2. Evidence-based Review Methodology for the Closing the Quality Gap Series of 

Evidence Reports...........................................................................................................................13 

Definition and Scope ..................................................................................................................13 

Taxonomy of Quality Improvement Strategies..........................................................................14 

Types of QI Strategies................................................................................................................14 

Identification of Quality Improvement Strategies for Evaluation..............................................18 

 Search Strategy...........................................................................................................................19 

Inclusion and Exclusion Criteria ................................................................................................20 

Types of Evidence Assessed in the Review ...............................................................................21 

Evaluation of Quality Improvement Strategies..........................................................................23 

Quantitative Synthesis of Quality Improvement Strategies .......................................................24 

 Meta-regression Analyses...........................................................................................................25 

Chapter 3. Toward A Theoretic Basis for Quality Improvement Interventions............................27 

 Introduction ................................................................................................................................27 

Hierarchy of Theories and Models.............................................................................................28 

Conceptual Models and Grand Theories of Implementation Phenomena..................................29 

Classical Theories of Change .....................................................................................................30 

Limitations of Classical Theories...............................................................................................30 

Planned Models of Change.........................................................................................................30 

 Mid-range Theories....................................................................................................................32 

Social Psychological Theories....................................................................................................33 

 Organizational Theories .............................................................................................................34 

 Other Discipline-Based Theories................................................................................................36 

Micro or Situation-Specific Theory............................................................................................37 

 Conclusion..................................................................................................................................37 

Chapter 3 figures 

1. Ottawa Model of Research Use ..............................................................................................38 

2. Organizational Theories Examined for Evidence...................................................................38 

3. Examples of Theories that Could Begin to Populate the Ottawa Model of Research Use....39 

Notes..............................................................................................................................................40 

References......................................................................................................................................41 

vii
Appendixes 

A. National Priority Areas Summary with Key Associated Goals.............................................51 

B. Search Strategy Exemplar: Quality Improvement and Hypertension....................................55 

C. Article Review Triage Forms (Exemplar: Diabetes).............................................................61 

D. Full Article Review Abstraction Forms Stages 3 and 4 (Exemplar Diabetes) ......................65

viii
Summary

In early 2003, the Institute of Medicine (IOM) released its report, Priority Areas for National 

Action: Transforming Health Care Quality. The report listed 20 clinical topics for which “best 

practices” were strongly supported by clinical evidence. The rates at which these practices have 

been implemented in the United States has been disappointing low, at a cost of many thousands 

of lives each year. 

To bring data to bear on the quality improvement opportunities articulated in the IOM’s 2003 

report, the Agency for Healthcare Research and Quality (AHRQ) engaged the Stanford–UCSF 

Evidence-based Practice Center (EPC) to perform a critical analysis of the existing literature on 

quality improvement strategies for a selection of the 20 disease and practice priorities noted in 

the IOM Report. The focus of the commissioned investigations is translating research into 

practice–identifying those activities that increase the rate with which practices known to be 

effective are applied to patient care in real world settings. In other words, the EPC research 

effort aims to facilitate narrowing the “quality gap” that is in large part responsible for 

suboptimal health care practices and outcomes. In addition to furthering the IOM’s quality 

agenda, this analysis also has been prepared in support of the National Healthcare Quality Report 

(NHQR) (also see National Healthcare Disparities Report). In this, the first volume of Closing 

the Quality Gap, the authors introduce the series and its goal, while providing methodological 

and theoretical overviews for the quality improvement (QI) field of study. Subsequent volumes 

will address the relation of QI strategies to treatment practices for a number of the 20 priority 

areas identified in the IOM report. 

Target Audiences 

Closing the Quality Gap is intended to assist a wide range of users: 

• Policymakers can use the detailed evidence review to prioritize quality improvement

strategies and choose how best to close the quality gaps in their organizations. 

• Researchers can find detailed information about well-scrutinized areas of treatment, 

while learning of other areas in need of further exploration. 

• Clinicians and trainees can see a broad spectrum of approaches to improving the quality 

of care. Some of these approaches fall within the control of individual practitioners, while 

others will require major systemic changes at the local level or beyond. 

• Patients can learn quality improvement strategies that they can help to promote, while 

gaining a deeper understanding of the nature and extent of quality gaps, as well as the 

systemic changes necessary to close them. 

• Groups and individuals charged with funding research will be able to identify high-yield 

areas of concern that warrant future research support. 

Volume 1 consists of three chapters: 

 Chapter 1—reviews the genesis of the quality implementation field, providing some

historical perspective on the science of translating research into practice. 

1
 Chapter 2—sets forth the carefully designed methodology used to review the vast amount 

of existing quality literature on particular diseases. The methodology is the result of 

collaborative efforts of the editorial team, in consultation with several of the undisputed 

experts in the field. For this project, the following terms were defined: 

Quality of health care: The degree to which health services for individuals and 

populations increase the likelihood of desired health outcomes and are consistent with 

current professional knowledge. 

Quality gap: The difference between health care processes or outcomes observed in 

practice, and those potentially obtainable on the basis of current professional knowledge. 

The difference must be attributable in whole or in part to a deficiency that could be

addressed by the health care system. 

Quality improvement target: The outcome, process, or structure that the QI strategy aims

to influence, with the goal of reducing the quality gap. 

To ensure consistency in the review and evaluation of the literature, the editors developed a 

taxonomy of interventions that modifies several well-established classification systems, 

denominating the QI strategies as follows: 

1. Provider reminder systems 

2. Facilitated relay of clinical data to providers 

3. Audit and feedback 

4. Provider education 

5. Patient education 

6. Promotion of self-management 

7. Patient reminder systems 

8. Organizational change 

9. Financial, regulatory or legislative incentives 

 Chapter 3—provides the reader some context for the field of QI implementation with a 

summary of the theoretic underpinnings that may influence the development of QI interventions, 

and an overview of selective efforts that have been made to adopt and modify interventions from

outside of health care. The authors review a selection of the major theories that may influence 

the two dominant and parallel tracks of QI interventions: behavioral change and transfer or 

diffusion of knowledge. References to a number of pertinent theoretical models are cited. 

What Conclusions can be Drawn from the Report’s 

Evidence? 

The purpose of this report is to help readers assess whether the evidence suggests that a 

quality improvement strategy would work in their specific practice setting or with their specific 

patient population. Three important questions should be considered: 

1. Are the studies of the strategy valid? A study has validity (sometimes called “internal 

validity”) if its findings are likely to be true in the population in which the study was 

performed. The primary determinant of validity is the design and conduct of the study. 

2
2. For each quality improvement strategy that has been evaluated in multiple studies with

sufficient validity, does the weight of evidence indicate that the strategy is effective? 

3. Can the conclusions of a body of evidence be applied to a specific practice setting or 

population of interest?

Except where noted, the review is restricted to studies that are likely to have strong validity, 

(e.g., randomized controlled trials, well-designed and controlled before-after studies, and 

interrupted time series studies). The authors thought it important to find and analyze studies 

whose research methodologies were most likely to provide scientifically correct answers. When 

the same QI strategies have been evaluated in more than one study, assessing the weight of the 

evidence and whether it favors the strategy can be a complex matter. To help readers make this 

assessment, the findings of studies are summarized in tables showing the range of results for 

different strategies. 

Remarkably, despite the vast stakes—after all, the concern here is identifying which 

techniques have been shown to promote the adoption of evidence-based “best practices”—there 

has been remarkably little information about the most effective ways to translate research into 

practice. Even in the case of common disorders such as diabetes, hypertension, and cancer 

care—areas in which research has demonstrated some best practices that can save tens of 

thousands of lives—there has been only modest systematic study of the techniques and strategies 

that most successfully close the quality gap. Moreover, in the few areas that have benefited from

such studies, little consideration has been given to how practices may be “crosscutting” (i.e., how 

a practice that closes the quality gap in asthma, might be applicable to congestive heart failure). 

Closing the gaps will require new resources and focus from caregivers and institutions. 

Ultimately, pressure from patients (brought to bear through market choices, regulators, 

policymakers, or others) is crucial if we are to succeed. 

It is AHRQ’s hope that the Closing the Quality Gap series will become an essential source of

accessible and critical analyses of the evidence supporting techniques for implementing state-ofthe-art best practices, while stimulating ideas for ongoing quality improvement activity 

nationally, in individual health systems, and among individual caregivers. 

3
This page intentionally left blank 

4
Chapter 1. An Introduction to the Report 

Robert M. Wachter, M.D. 

University of California, San Francisco 

Douglas K. Owens, M.D., M.S. 

VA Palo Alto Health Care System

and Stanford University

The Genesis of Closing the Quality Gap

 Knowing is not enough; we must apply. 

 Willing is not enough; we must do.

 — Johann Wolfgang Von Goethe (1749-1832)

In early 2003, the Institute of Medicine (IOM) released its report, Priority Areas for National 

Action: Transforming Health Care Quality.

1

 The report listed 20 clinical topics for which best 

practice treatment guidelines are strongly supported by clinical evidence. Unfortunately, the 

report and a substantial quantity of other scientific literature show best practice implementation 

rates in the United States have been disappointing low, and at an annual cost of many thousands 

of lives. 

To bring data to bear on the quality improvement opportunities laid out in the IOM’s 2003 

report, the Agency for Healthcare Research and Quality (AHRQ) asked the Stanford–UCSF 

Evidence-based Practice Center (EPC) to perform a critical analysis of the existing literature on 

quality improvement strategies for a number of the 20 disease and treatment priorities noted in 

the IOM Report. Rather than concentrating on the specific clinical practices that appear to 

improve health outcomes, these analyses focus on the effort of translating research into practice–

identifying those activities that increase the rate at which effective practices are applied to 

patient care in real world settings. The overarching goal is one of narrowing the quality gap that 

is largely responsible for suboptimal health care practices and outcomes. This work also supports 

the recently released National Healthcare Quality Report (NHQR)2

 —and its companion 

document, the National Healthcare Disparities Report (NHDR).3 Based upon earlier 

recommendations of the IOM,4 Congress called upon AHRQ to deliver an annual report on the 

state of health care in the United States. The NHQR is intended to corroborate or refute 

widespread concerns related to health care quality, to document whether health care quality is 

stable, improving, or declining over time, and to provide national benchmarks with which 

individual states, health plans, and providers may compare their relative performance. 

This is the first volume in a series of reports intended to support these goals. A carefully 

designed methodology will be applied to the scientific literature for a number of medical 

conditions characterized by the IOM as high-level threats to health and longevity. It is AHRQ’s 

hope that the series will stimulate ideas for ongoing quality improvement activity nationally, as 

well as in individual health systems and among individual caregivers. 

5
Origins of the Quality Movement 

Although humans have long been intrigued and moved by the complex science of healing 

others, the science of measuring and improving the quality of delivered care is a relatively recent 

undertaking. Boston surgeon Ernest A. Codman (1869–1940) began his “end results system” a 

century ago, to track surgical outcomes and to improve surgical practice.5

 Codman’s work in this 

area ultimately led to the creation of the Joint Commission on Accreditation of Healthcare 

Organizations (JCAHO). 

Despite Codman’s pioneering work and several other individuals and organizations whose 

efforts extended through the middle of the 20th Century, the science of health care quality 

improvement truly took root only a generation ago. Several forces catalyzed this transformation. 

First, medicine transcended its status as an anecdotal, non-evidence-based enterprise to one in 

which good data led to the discovery of improved treatment practices. For example, in the mid1960s, 100 clinical trials were published each year. Thirty years later, that number had grown to 

more than 10,000.6,7

 

Second, as the public’s interest and investment in the “miracles” of modern medicine grew—

particularly in high technology specialties such as cardiac surgery and transplantation—so, too, 

did the public’s demand for greater provider accountability and positive patient outcomes. 

Although public awareness of patient safety and quality increased with the IOM’s seminal 

publication in 2000 of To Err is Human: Building a Safer Health Care System,8 and its broader, 

2001 indictment of health care quality (Crossing the Quality Chasm4

) the trend had already been 

established. In an increasingly consumerist society, people had become less inclined to simply 

trust that their caregivers would deliver the highest quality care. And the public’s skepticism

only grew with the cost-driven growth of managed care. 

Third, the expense of medical technology and the highly trained personnel needed to deliver 

that technology required the expansion of third-party payment systems, many of which were 

employer-based. These costs became a disproportionately large part of annual operating budgets 

and so employers, accustomed to making purchasing decisions based on value (quality and cost), 

found themselves without any information from the quality dimension of this equation. Their 

unwillingness to take it on faith that medicine’s “product” was of the highest quality only grew 

with the published evidence of huge regional variations in the numbers of common procedures 

(coronary bypass grafting, hysterectomies, trans-urethral prostate resections) that could not be 

explained by differences in patient populations nor justified by differences in outcomes.9,10 Other 

studies showed unacceptably high rates of “inappropriate” surgical procedures such as carotid 

endarterectomy,11 further fueling the skepticism regarding the quality of health care in America 

and increasing demands for accountability.12-17

 

These pressures were mounting during the same time that tools for measuring the quality of

evidence supporting clinical practice, such as clinical epidemiology, decision- and costeffectiveness-analysis, meta-analysis, and the like, were becoming more robust. Driven in part by 

sizable congressional allocations to the National Institutes of Health and by private investments 

on the part of pharmaceutical companies and others, the clinical research knowledge base grew 

as well. The use of computer-assisted health care management systems led to the creation of 

large databases that could be mined to provide information on the quality of care, as large and 

complex clinical trials became commonplace. Before long, specialties such as cardiology, for 

example, were transformed. Cardiologists witnessed a shift in the cultural context, their focus 

6
drawn away from the art of medicine and redirected toward the dozens of regularly published 

clinical trials and emerging evidence on best practice treatments and heart disease prevention. 

The probability of an American death by heart disease fell by 56 percent between 1950 and 

1996. Although many factors contributed to this decline, much of the success is attributable to 

advances in clinical care and medical science.18 

By the mid-1990s, the powerful influences of clinical treatment information, skepticism of 

the medical community’s ability to ensure high-quality health care, increased consumer and 

purchaser knowledge, and the science of quality measurement had come together. More and 

more studies revealed large gaps between the findings of scientific studies and their practical 

implementation, even in areas of medicine where the optimal clinical approach was assured. 

Several large and recent studies have confirmed sizable quality of care gaps in areas spanning 

preventive medicine, acute and chronic care, and care of elders.19,20-22 These and other studies 

emphasized the notion that research into quality health care does not, in itself, ensure the clinical

patient will receive the highest quality care. A new area of inquiry—how best to translate 

research into practice—was born.23-26

Translating Research into Practice: What do we Know? 

There are many reasons for the gaps that exist between the best, evidence-based 

understanding of high quality treatment practices, and the actual practices themselves. 

First, there may be a gap in the dissemination of knowledge. The large and growing number 

of clinical trials underway at any given time makes it impossible for any individual physician or 

system of care to stay fully abreast. There is an inevitable time lag between the publication of 

studies that demonstrate an effective practice and its implementation. There is sometimes a need 

for a consensus to emerge among specialists, or a diffusion from specialists to generalists. For 

example, a 1994 study by Ayanian showed that cardiologists were about 10 absolute percentage 

points more likely to prescribe therapies known to be effective, and less likely (by roughly the 

same margin) to prescribe ineffective therapy, than were their generalist peers.27 Similar lags 

have been demonstrated in the management of a variety of conditions,28 ranging from peptic 

ulcer disease29 to heart failure.30

Second, providers may be aware of a best practice, but fail to implement it because of 

skepticism surrounding the cost effectiveness of the practice (in terms of dollars or time needed 

to educate patients or adapt work processes). Or, they may have reservations regarding their 

treatment environment and the systems support (people or equipment) or changes in 

organizational culture needed to implement the practice. For example, new recommendations 

designed to provide improved glucose control in ICU patients31 would mandate an increase in 

ICU nurse staffing to facilitate more frequent blood glucose checks. The director of critical care 

services at the University of California, San Francisco estimates that such increased monitoring 

would consume an additional 2 hours per ICU nurse shift for a typical nurse caring for two 

patients (Michael Gropper, MD, personal communication, 2003). Not surprisingly, many ICUs 

have yet to adopt this practice, despite clear and compelling evidence of the clinical benefits.

Finally, while the treatment practice may have been proven effective in a special research 

setting, it may not be applicable to an individual provider’s setting. Clinical trials differ in many 

ways from real-life practice: staff members are attentive to the research protocols, personnel with 

specialized training may have been hired to provide additional support or patient education, 

7
patient selection may be related to the research protocol, and additional safety measures may be 

built into the trial. In addition, research studies are often carried out in specialized settings (e.g., 

a Veterans Affairs hospital or a large academic medical center) that may bear little resemblance 

to the smaller treatment setting of a physician considering the practice. This gap between 

efficacy (how well the practice works in the research environment) and effectiveness (how well 

it works in clinical practice–generalized to include a wide range of treatment settings, with 

providers who may not be committed to or expert in its application, and a broader array of 

patients) has been well appreciated in recent years.12-17 

As the quality gap has become more widely acknowledged, investigators have focused on its 

genesis and possible strategies for closing it. In one early analysis, Greco and Eisenberg32

described six possible interventions to improve uptake (adoption) of improved treatment 

practices: education, feedback, participation by physicians in efforts to bring about change, 

administrative rules, financial incentives, and financial penalties. 

In addition to those interventions that focus largely on the clinical behavior of individual 

providers (mostly physicians), more attention is being given to systematic changes in the practice 

environment, some of which (e.g., computerized rules and checklists, automatic stop orders) may 

bypass physicians entirely. A parallel movement is focusing on patients as the guardians of their 

own health care quality. One example cited frequently in the realm of patient safety involves 

patients asking their providers if they had washed their hands prior to the patient encounter.33

Whatever the method used to achieve the desired change, there is little doubt that the 

movement to base accountability and competition on metrics of quality has just begun. Business 

coalitions including the Pacific Business Group on Health and the Leapfrog Group are partnering 

with accreditation groups such as JCAHO to develop new quality-of-care standards. These 

standards will be made available to the public and can be used as the foundation for purchasing 

or payment decisions. The National Committee for Quality Assurance (NCQA) publishes its own 

“Report Card” for use by government agencies, employers, and consumers. Although the 

evidence regarding report card documents and their ability to characterize and improve overall 

health care quality is decidedly mixed,34-38 public reporting and the desire to avoid negative 

publicity has made certain hospitals and providers eager to receive good “grades.” As the case 

for improved quality in health care grows, so too does the realization that the best way to 

improve patient outcomes is a strict adherence to well-researched and respected quality 

improvement practices–to translate research into practice. 

 

The Theoretic Underpinnings of Quality Improvement Efforts 

Medicine has a long history of investigating what works in the clinical realm, and why. At 

the same time, we have a fairly limited understanding of the causal mechanisms of interventions 

to improve health care quality. Theories abound with regard to changing the behavior of patients, 

clinicians, and organizations for the better. These theories often are drawn from studies that try 

to isolate the effect of a single varied element or combinations of setting, interventions, and 

targets for change. The challenge for researchers rests in the accurate interpretation of this 

diverse literature regarding implementation. 

In an effort to provide the reader with some context relative to the field of quality 

improvement (QI) implementation, this report offers a brief summary of the theoretical 

underpinnings that influence the development of QI interventions, as well as identifying selective 

8
efforts that have been made to adopt and modify interventions from outside of health care. 

Readers interested in QI theory discussions of greater depth are encouraged to spend some time 

with Chapter 3, which reviews a selection of the major theories thought to influence the two 

dominant and parallel tracks of QI interventions: behavioral change, and the transfer or diffusion 

of knowledge. References to a number of pertinent theoretical models also are provided in this 

chapter.

An overarching theory for closing the quality gap may be neither feasible, nor desirable. 

Existing theories, including those from disciplines outside of health care, however, may be 

marshaled to design interventions for health care protocols in need of modification. Such theories 

have been applied in many ways—often borrowing techniques from industry such as those 

promoted by Juran and Deming39-43 —with varying degrees of success. The methods generally 

emphasize the importance of identifying a process with less-than-ideal outcomes, measuring the 

key performance attributes, using careful analysis to devise a new approach, integrating the 

redesigned approach with the process, and reassessing performance to determine if the change in 

process is successful. 

The mixed results produced by industry-oriented quality improvement programs (such as 

Total Quality Management [TQM] and Continuous Quality Improvement [CQI]) have taught 

managers and others the need to exercise caution before assuming that strategies drawn from

other industries automatically will work in health care settings, and demonstrated that additional 

attention that must be given to the forces that promote desired behavioral changes among frontline workers.

44-46 These forces are an outgrowth of human needs and desires: the altruism of 

most health care professionals, their desire for success and peer respect, their preference for 

avoiding embarrassment, and the goal of financial independence, to name but a few. These 

inspirations have prompted a more recent movement, in which the traditional quality 

improvement sensibilities of programs such as TQM or CQI are coupled with more modern 

approaches to behavior modification, such as performance auditing and feedback. An audit often 

will measure provider adherence to a specific process or treatment practice, and the providers

being studied will receive comparative data after the fact about their performance and how they 

stack up against their peers. In other types of audits, providers might receive financial rewards 

for their strict adherence to desired behaviors, or information regarding their performance and 

standing might be forwarded on to their patients (who can influence non-conforming providers to 

make the appropriate behavioral change, or choose to seek care elsewhere). 

Remarkably, considering the enormous stakes, there has been little information written about 

the most effective ways to translate research into practice. Even for common disorders like 

diabetes, hypertension, and cancer care—areas in which research has successfully demonstrated 

that some best practices can save tens of thousands of lives—there has been only modest 

systematic study of the techniques and strategies shown to close the quality gap. Moreover, in 

those few areas that have benefited from such studies, little consideration has been given to 

crosscutting practices (i.e., how a practice that closes the quality gap in asthma, might be 

applicable to congestive heart failure). 

9
What Conclusions can be Drawn from the 

Report’s Evidence?

This report is intended to help readers assess whether the available evidence suggests that a 

quality improvement strategy would work in their specific practice setting, or, within their 

specific patient population. Three important questions should be considered: 

 

1. Are the studies of the strategy valid? A study has validity (sometimes called internal 

validity) if its findings are likely to be true in the population on which the study was 

based. The primary determinant of validity is the design and conduct of the study. 

2. For the quality improvement strategies that have been evaluated in multiple studies with 

sufficient validity, does the evidence indicate that the strategy is effective? 

3. Are the conclusions of a body of evidence applicable to a practice setting or population of 

interest? 

Careful attention has been paid to the design of each included study (Chapter 2), as a means 

of assisting readers to better judge study validity. Except where noted, the review has been 

restricted to studies that are likely to have strong validity, i.e., randomized controlled trials, well 

controlled before–after studies, and interrupted-time-series studies. This has been done to 

acknowledge an important tension in the field of quality improvement. Given the challenges and 

constraints of studying change in complex organizations, some authorities consider some of the 

most relevant QI work to be that performed “in the trenches,” by front-line workers taking 

advantage of available resources to answer important, practical questions using simple designs 

(e.g., uncontrolled before-and-after studies). This point of view has relevance. However, in a 

report of this type, the authors placed a priority on finding and analyzing those studies with 

research methodologies most likely to give scientifically correct answers: randomized controlled 

studies, controlled before-and-after studies, and interrupted-time-series studies. They did so with 

the recognition that the relatively strict criteria may have led to the exclusion of some studies 

with potentially relevant findings. 

When specific QI strategies have been evaluated in the course of multiple studies, deciding if

the weight of the evidence favors the strategy can be a complex decision. To help readers make 

this assessment, the authors have used tables to indicate the range of results for different 

strategies. In those instances where studies were sufficiently similar in their design and sample 

size to justify combining the results, the authors used quantitative methods of analysis to 

synthesize their findings. When it was judged imprudent to combine studies quantitatively, the 

researchers made every attempt to highlight important findings and, when possible, they noted 

whether the findings are consistent across studies. The methods used in the course of these 

analyses are described in greater detail in Chapter 2. 

Perhaps the most difficult challenge facing the authors of this report and its readers concerns 

the applicability of a study’s results to a particular treatment setting or a patient population other 

than that used in the study itself. Studies vary in terms of the disease process considered, the 

population sample, the type of quality improvement intervention scrutinized, the behavior 

addressed by the intervention, and the time frame of the study. Each of these factors affects the 

applicability (sometimes called “generalizability”) of the study. For example, if a study showed 

that audit and feedback improved prescribing practices for hypertension in a managed care 

treatment setting, would these findings hold true in a fee-for-service practice? Would they hold 

10
true for diabetes care? If audit and feedback was effective in a general medicine clinic, would 

the same improvement strategy prove equally effective for a specialty clinic? 

Caution is warranted with respect to any study’s results and their applicability across settings 

or diseases, as the specific conditions of any user’s practice are certain to differ from those of the 

study population. The factors with the greatest effect on the applicability of study findings are 

not yet known, but the final evidence report of the series will describe the EPC’s findings and 

experience in the hope that the reader will be able to evaluate any common findings across 

different disease processes. 

The Organizational Framework of this Series 

Volume 1 contains this introduction to the series, the evidence-based methodology that 

unifies and underlies each of the treatment condition reports in the series (Chapter 2), and the 

theories thought to influence QI and implementation (Chapter 3). 

 

Volumes 2 and 3 will review the evidence regarding the effectiveness of QI implementation 

practices in the treatment of diabetes and hypertension, respectively. These volumes, and those to 

follow, will feature the same detailed organizational framework: 

Introduction – The authors identify the general background and clinical context for the 

disease or condition, they illustrate the primary quality gap(s) for the topic, and provide a 

means of benchmarking outcomes for these problems. The best treatment practices also are 

provided, as are the strategies for quality improvement. 

Methodology – The scope of material reviewed for the topic is delimited, noting studies that 

have been excluded, and specifying the primary outcomes of interest. Some information 

pertaining to the methodologic process and analysis appears in the Methods section of 

Volume 1 as well as in Volume 2 (Diabetes) and Volume 3 (Hypertension). This redundancy 

was planned for the reader’s convenience, since each of the volumes dealing with priority 

conditions may be read as a stand-alone analysis. 

Findings Overview – A summary of the reviewed literature is provided, along with two 

separate analyses: one delineated by outcome and one by quality improvement strategy. An 

Appendix for each volume provides tables of included studies and results. 

Discussion – An analysis will be included for each of the studied priority conditions, with a 

list of the strategies best supported by the available evidence, as well as obvious gaps and 

suggestions for future research. 

Subsequent volumes in the series, to be produced over the next two years, will consider the 

evidence behind QI practices for a select number of conditions from the IOM’s 2003 quality 

report. Evidence for the impact of individual QI practices in specific diseases or care settings will 

be considered in condition- or setting-specific volumes. Global analysis of the QI practices 

across diseases or settings will likely be addressed in the final volume in the series. The last 

volume also may be used to describe broad themes that emerge from the project. Finally, 

attempts will be made to quantify and prioritize the benefits of the various QI strategies, to the 

extent that the published evidence permits. 

11
This page intentionally left blank 

12
Chapter 2. Evidence-based Review Methodology for 

the Closing the Quality Gap Series 

Kaveh G. Shojania, M.D. 

University of California, San Francisco 

Kathryn M. McDonald, M.M. 

Stanford University 

Douglas K. Owens, M.D., M.S. 

VA Palo Alto Health Care System

and Stanford University

Definition and Scope 

The Stanford University–UCSF Evidence-based Practice Center (EPC) performed a 

comprehensive review of the evidence relating to a broad range of quality improvement (QI) 

strategies and their utility in a variety of clinical areas. The topic areas were chosen from a group 

of 20 priority conditions identified by the IOM1

 (see Appendix A). For this project, the authors 

defined the following terms: 

Quality of health care: The degree to which health services for individuals and patient 

populations increase the likelihood of desirable health outcomes and are consistent with current 

professional knowledge.47 

 Quality gap: The difference between health care processes or outcomes observed in practice, 

and those thought to be achievable with the most current and effective professional knowledge. 

The difference must be attributable in whole or in part to a deficiency that could be addressed by 

the health care system. An example of a process-level quality gap for hypertension involves the 

62 percent of clinical visits during which physicians failed to introduce evidence-based, 

guideline-concordant drug therapy to patients with a systolic blood pressure of 140mm/HG or 

higher.48 

An example of an outcome-level quality gap for myocardial infarctions involves a disparity 

in survival rates. Despite numerous new therapies that have substantially decreased mortality 

over the past 25 years, survival gains have occurred mainly in males and in younger patients, 

with less gain in women and the elderly.49 The resolution of such outcome gaps generally entails 

detailed analyses of relevant treatment and care processes, in an effort to explain their genesis 

and identify targets for action. 

Quality improvement (QI) strategy: Any intervention aimed at reducing the quality gap for 

a group of patients representative of those encountered in routine practice. For the purposes of 

their literature search, the authors considered a study to include a QI strategy evaluation if any of 

the following applied:

• The intervention targeted implementation of a particular process of care (or set of 

processes) believed to benefit patients with the priority condition(s); i.e., interventions

13
designed to improve provider adherence to a clinical best practice guideline, or those 

intended to increase the proportion of patients who received recommended care. 

• The intervention targeted implementation of a structural or organizational feature 

believed to benefit patients with the priority condition; i.e., interventions that changed the 

care provider, added supplemental personnel, or made clinical information systems part 

of the treatment protocol. 

• The intervention attempted to improve outcomes for a broad and relatively unselected 

group of patients with the priority condition; i.e., interventions designed to improve the 

delivery of care for all patients with diabetes or hypertension at a specific clinic. 

• The intervention targeted a subset of patients that typically is excluded from clinical

research; i.e., frail elders, minorities, the economically disadvantaged, or those with 

multiple comorbid conditions. 

• The intervention involved any of the specific QI strategies falling within a taxonomy of 

approaches to QI that the authors developed, based on evaluations of various quality 

improvement interventions50-59 and authoritative definitions

60 (see below for taxonomy). 

Quality improvement target: The outcome, process, or structure that the QI strategy is 

intended to influence, with the goal of reducing the quality gap. A target typically would be a 

measure of disease control, including direct health outcomes (morbidity or mortality), or 

intermediate outcomes proven to influence direct outcomes (such as blood pressure or 

hemoglobin A1C control). Targets also may involve adherence to accepted processes of care, 

either by clinicians (i.e., guideline recommendations and performance measures) or by patients 

(i.e., adherence to prescribed medications, recommended self-management). 

 

Taxonomy of Quality Improvement Strategies 

To ensure consistency in their review and evaluation of the literature, the authors developed a 

taxonomy that modifies several well-established classification systems.50-54 A recent and 

systematic review of disease management studies combined QI strategies and targets, classifying 

interventions as: provider education, provider feedback, provider reminders, patient education, 

patient reminders, and patient financial incentives.54 The Cochrane EPOC data collection 

instrument uses four broad classifications (professional interventions, organizational 

interventions, financial interventions, and regulatory interventions), each of which has detailed 

subcategories. An alternative taxonomy, described in a recent systematic review of interventions 

to promote immunization and cancer screening,52 specifies three dimensions for characterizing 

QI strategies – the type of QI strategy (e.g., education, audit and feedback, organizational 

changes, financial incentives), mediators of the intervention (e.g., use of local opinion leaders, 

involvement of top management, identification of barriers to change), and target audience (e.g., 

patients, providers, health care delivery systems). The authors of this series modified the various 

taxonomies to better facilitate their review of the evidence. 

Types of QI Strategies 

Nine types of QI strategies are outlined below, along with key substrategies. These categories 

are broad, and, in some cases, combine multiple interventions. The authors explored this 

heterogeneity in their analyses to assess the possibility of making inferences and judgments 

about the success of the strategy as a whole, or whether further subdivision would be needed. 

14
Where relevant, the analyses also take into consideration the fact that many interventions are 

multifaceted and employ more than one type of QI strategy. 

1. Provider reminder systems—the investigators defined a reminder system as any 

patient- or clinical encounter-specific information, provided verbally, in writing, or by 

computer, to prompt a clinician to recall information, or intended to prompt consideration 

of a specific process of care (i.e., “This patient last underwent screening mammography 3 

years ago”). The reminder also may include information prompting the clinician to follow 

evidence-based care recommendations (e.g., to make medication adjustments, or to order 

appropriate screening tests). The phrase “clinical encounter-specific” in the definition 

serves to distinguish reminder systems from audit and feedback, where clinicians 

typically receive performance summaries relative to a process or outcome of care 

spanning multiple encounters (e.g., all Type 2 diabetic patients seen by the clinician 

during the past 6 months). 

2. Facilitated relay of clinical data to providers—used to describe the transfer of clinical 

information collected directly from patients and relayed to the provider, in instances 

where the data are not generally collected during a patient visit, or using some format 

other than the existing local medical record system (i.e., the telephone transmission of a 

patient’s blood pressure measurements, from a specialist’s office). The EPOC group uses 

the term “patient mediated” to describe such interventions, but the authors regard the 

above label as more descriptive. Some overlap with provider reminder systems was 

expected, but the strategies were kept separate at the abstraction stage. This decision 

allowed for the possibility that the data could be subsequently analyzed with and without 

collapsing the two strategies. 

3. Audit and feedback—the researchers defined audit and feedback as any summary of 

clinical performance for health care providers or institutions, performed for a specific

period of time and reported either publicly or confidentially to the clinician or institution 

(e.g., the percentage of a provider's patients who achieved or did not achieve some

clinical target, such as blood pressure or HbA1c control over a certain period). 

Benchmarking is a term referring to the provision of performance data from institutions

or providers regarded as leaders in the field. These data serve as performance targets for 

other providers and institutions. The authors included benchmarking as a type of audit 

and feedback, so long as local data were provided for comparison with the benchmark 

data. 

4. Provider education—used to describe a variety of interventions including educational 

workshops, meetings (e.g., traditional Continuing Medical Education [CME]), lectures

(in person or computer-based), educational outreach visits (by a trained representative 

who meets with providers in their practice settings to disseminate information with the 

intent of changing the providers’ practice). The same term also is used to describe the 

distribution of educational materials (electronically published or printed clinical practice 

guidelines and audio-visual materials). The investigators further captured information 

about the intensity (i.e., duration and number of educational sessions) and format (i.e., 

lectures delivered live, via teleconference, or pre-recorded) in a free-text mode, for each 

of these substrategies. Early plans to capture these and other predictors in a structured

form were abandoned after the authors and their technical advisors agreed the judgments 

15
were too subjective. This was due in large part to a relative lack of detail surrounding the 

interventions in the vast majority of studies. 

5. Patient education—this strategy is centered on in-person patient education, either 

individually or as part of a group or community, and through the introduction of print or 

audio-visual educational materials. Patient education may be the sole component of a 

particular quality improvement strategy, or it can be one part of a multifaceted QI 

strategy. It should be noted that the authors evaluated only those strategies in which 

patient education was regarded as one component of a multifaceted strategy. A future 

volume in this series may address the topic of patient education as a singular intervention, 

along with its relative effects on a variety of chronic diseases. 

6. Promotion of self-management—this strategy includes the distribution of materials (i.e., 

devices for blood pressure or glucose self-monitoring) or access to a resource that 

enhances the patients' ability to manage their condition, the communication of useful 

clinical data to the patient (e.g., most recent HbA1c or lipid panel levels), or followup 

phone calls from the provider to the patient, with recommended adjustments to care. The 

authors expected some overlap with regard to patient education (strategy 5) and patient 

reminders (strategy 7). They elected to keep the strategies separate at the abstraction

stage, to allow for the possibility that the data could be analyzed after the fact, with and 

without collapsing the two strategies. 

7. Patient reminders—used to define any effort directed by providers toward patients that 

encourages them to keep appointments or adhere to other aspects of the self-management 

of their condition. 

8. Organizational change—this strategy included any intervention having features 

consistent with at least one of the following descriptions, each of which represents a 

substrategy of organizational change that was abstracted for incorporation in the analysis: 

a) Disease management or case management – the coordination of assessment, 

treatment, and referrals by a person or multidisciplinary team in collaboration with, or

supplementary to, the primary care provider. 

b) Team or personnel changes – adding new members to a treatment team (e.g., the 

addition of a diabetes nurse, a clinical pharmacist, or a nutritionist to a clinical 

practice), creating multidisciplinary teams within a practice, or revising the roles of 

existing team members (e.g., a clinic nurse is given a more active role in patient 

management), or the simple addition of more nurses, pharmacists, or physicians to a 

clinical setting. 

c) Communications, case discussions, and the exchange of treatment information 

between distant health professionals (i.e., telemedicine). 

d) Total Quality Management (TQM) or Continuous Quality Improvement (CQI) 

techniques for measuring quality problems, designing interventions and their 

implementation, along with process re-measurements. 

e) Changes in medical records systems—adopting improved office technology (e.g., 

computer-based records, patient tracking systems). 

16
Although the definition used for this strategy is consistent with prior reviews,52 the 

authors recognized the potential heterogeneity of included interventions and accordingly 

planned to analyze this strategy with respect to the aforementioned substrategies. 

9. Financial, regulatory or legislative incentives—this strategy encompassed any 

intervention having features consistent with at least one of the following descriptions: 

a) Positive or negative financial incentives directed at providers (e.g., regarding 

adherence to some process of care or achievement of target patient outcome). 

b) Positive or negative financial incentives directed at patients. 

c) System-wide changes in reimbursement (e.g., capitation, prospective payment, shift 

from fee-for-service to salary). 

d) Changes to provider licensure requirements. 

e) Changes to institutional accreditation requirements. 

The authors further abstracted information about the use of clinical information systems, 

including their role in identifying eligible study participants for QI interventions, for generating 

clinical reminders, for enabling decision support, and their ability to cultivate data for audit and 

feedback. 

Table 1 presents the major types of QI strategies in the first column, with examples of 

corresponding substrategies in the second column. The table illustrates the manner in which 

some QI strategies and substrategies target a single audience, while others attempt to influence 

multiple audiences, such as patients and health care delivery systems. Many QI strategies 

evaluated in the literature combine substrategies and audience targets, a situation that makes for 

challenging analyses of effectiveness. Such combinations often limit the ability of researchers to 

interpret the active component(s) of a particular intervention. 

Table 1. Taxonomy of QI strategies with examples of substrategies 

QI strategy Examples 

Provider reminder systems • Reminders in charts for providers 

• Computer-based reminders for 

providers 

• Computer-based decision support 

Facilitated relay of clinical data to providers • Transmission of clinical data from 

outpatient specialty clinic to 

primary care provider by means 

other than medical record, e.g., 

phone call or fax 

Audit and feedback • Feedback of performance to 

individual providers 

• Quality indicators and reports

• National/State quality report cards 

• Publicly released performance 

data 

• Benchmarking – provision of 

outcomes data from top performers 

for comparison with provider’s own

data 

17
QI strategy Examples 

Provider education • Workshops and conferences 

• Educational outreach visits (e.g., 

academic detailing) 

• Distributed educational materials 

Patient education • Classes 

• Parent and family education

• Patient pamphlets 

• Intensive education strategies

promoting self-management of

chronic conditions 

Promotion of self-management • Materials and devices promoting 

self-management 

Patient reminder systems • Postcards or calls to patients

Organizational change • Case Management, Disease 

Management 

• TQM, CQI techniques 

• Multidisciplinary teams 

• Change from paper to computerbased records 

• Increased staffing 

• Skill mix changes 

Financial incentives, regulation, and policy Provider-Directed: 

• Financial incentives based on 

achievement of performance goals 

• Alternative reimbursement systems 

(e.g., fee-for-service, capitated 

payments) 

• Licensure requirements 

Patient-Directed: 

• Co-payments for certain visit types 

• Health insurance premiums, user 

fees 

Health System-Directed: 

• Initiatives by accreditation bodies 

(e.g., residency work hour limits) 

• Changes in reimbursement 

schemes (e.g., capitation, 

prospective payment, salaried

providers) 

18
Identification of Quality Improvement 

Strategies for Evaluation

The medical conditions selected for evaluation were taken from the IOM National Priorities 

report,1

 and were based on the priorities of stakeholders, the quality of evidence in relation to the 

usefulness of QI strategies, the expertise of the EPC, and available resources. As described in the 

Introduction to this volume, the selected topics will be analyzed in a series of volumes to be 

published over the course of the next two years. The final volume may be used to examine 

crosscutting analyses of selected QI strategies for many of the disease topics presented in the 

series. 

Search Strategy 

The authors initially reviewed QI strategies for hypertension and diabetes to help formulate 

their methodologic approach. They searched the MEDLINE® database from 1980-present, the 

Cochrane databases, and the Cochrane registry for the selected topics. The general search 

strategy was consistent across these two topics. Appendix B illustrates the search strings for 

hypertension. They searched terms relevant to care coordination and disease management, 

quality improvement (including Total Quality Management and Continuous Quality 

Improvement), continuing medical education, educational outreach, audit and feedback, financial 

incentives, information technologies, telemedicine, and the specific condition under 

consideration (e.g., hypertension). Additional searches were undertaken for systematic reviews 

and manual searches also were done, when appropriate, for relevant references. The 

bibliographies of all articles that met final inclusion criteria were scanned by hand for the 

project, as were the bibliographies for all relevant systematic reviews and meta-analyses. In 

cases where no systematic review was found to exist for a given topic, the authors searched the 

bibliographies of traditional (narrative) review articles, editorials, and news items that appeared 

to describe QI studies involving outpatient diabetic care. 

These searches were supplemented with reviews of citations from the Cochrane EPOC 

registry of quality improvement strategies. Each of the Collaborative Review Groups within the 

Cochrane Collaboration works to prepare and maintain systematic reviews of the prevention, 

treatment, and rehabilitation of a particular health problem or groups of problems, known as the 

'scope' of the group. EPOC’s mandate is the systematic review of educational, behavioral, 

financial, organizational, and regulatory interventions designed to improve health professional 

practice and the organization of health care services, using the most statistically reliable 

methods, and across all clinical areas. 

 

The EPOC registry has been developed using a highly sensitive search strategy to identify

studies within EPOC’s scope. The registry is updated quarterly and is derived from a search of 

more than 200,000 citations in the MEDLINE, EMBASE®, and CINAHL® databases, last 

updated prior to this report on June 14, 2003, August 6, 2002, and May 28, 2003, respectively. 

As of this writing, the registry contains approximately 2,500 studies, with another 3,000 studies 

pending full text assessment. The registry includes the full bibliographic reference (including 

MEDLINE index terms) and details the type of study, interventions considered, and targeted 

behavior. With the assistance of EPOC, the authors developed searches within the registry using 

the applicable clinical area MESH terms. 

 

19
This approach differs from EPOC in one significant respect: it is EPOC policy to exclude 

interventions that do not involve provider or organizational change (e.g., patient education, selfmanagement, and behavior change). In part because of this difference in scope, the authors 

conducted independent MEDLINE and hand searches for the first two priority conditions: 

diabetes and hypertension. 

Inclusion and Exclusion Criteria 

To begin, teams consisting of one or two senior reviewers (including an editor), trained two 

or more junior reviewers (junior faculty, fellows, and research assistants) to perform literature 

searches, conduct content reviews, and abstract data. The searches undertaken by these 

individuals revealed several thousand abstract titles for each priority condition. Stage 1 centered 

on the triage process for the article titles and/or abstracts (see Appendix C—triage forms), to 

determine if an article described an actual QI strategy. At this stage of the review process, 

randomized controlled trials, quasi-randomized trials, controlled before–after studies, interrupted 

time series, and before–after comparisons all were considered evaluations. A senior reviewer 

confirmed exclusion decisions using a random sample of 500 citations from the articles excluded 

at Stage 1–roughly a 20% sample. If the exclusion sample revealed any articles that should have 

been passed on for a full-text review, all the excluded citations were re-reviewed. The 

investigators included studies that examined the use of single or multiple QI strategies, with one 

exception: studies that used only patient education interventions were excluded because these 

studies likely will become the focus of a subsequent review. Studies were identified as relevant 

to quality improvement for this project if any one of the following applied: 

1. The intervention was designed to increase the proportion of patients receiving 

recommended processes of care (e.g., those demonstrated to improve outcomes for 

patients with the condition of interest), including aspects of diagnosis and screening, 

therapeutic interventions, and patient education or counseling. 

2. The intervention implemented organizational or structural features likely to benefit 

patients with the condition of interest. 

3. The intervention attempted to improve outcomes for a broad and relatively unselected 

group of patients with the condition(s) of interest [e.g., “all patients with diabetes (or 

asthma, hypertension, etc.] who receive care at a clinic”). 

4. The intervention targeted a subset of patients that is typically excluded from clinical 

research (i.e., frail elderly, minorities, homeless). 

5. The intervention involved any of the specific QI strategies or sub-strategies noted in 

Table 1: provider reminder systems, facilitated relay, audit and feedback, provider 

educational interventions, patient educational interventions, promotion of selfmanagement, patient reminders, organizational change, and financial, regulatory, or 

legislative incentives and interventions. 

The authors set out to assess QI strategy effectiveness. The inclusion/exclusion criteria did 

not consider whether there was an established evidence-based guideline for the priority condition 

being studied. Nor did they review the evidence for the underlying quality improvement target. 

For example, the reviewers did not attempt to correlate the evidence for tight blood pressure 

20
control with improved diabetes outcomes. Rather, they examined the evidence for QI 

interventions that have a positive effect on blood pressure control. 

In Stage 2, a senior reviewer reconfirmed the description of a QI strategy in each included 

report and identified the study design (see Appendix C—triage forms). Determinations were 

made with regard to the study designs suitable for Stage 3 abstraction, based on the availability 

of the highest quality studies for that priority condition. Any study that was not excluded in 

Stage 2, on the basis of the title or abstract, was advanced to Stage 3 and a full text review. 

In fact, the articles that remained part of the study at Stage 3 were scrutinized independently 

by two reviewers. Each reviewer abstracted information from the complete article about the QI 

strategy employed, the study design, and the outcomes evaluated (see Appendix D—Stage 3 

abstraction forms). The forms used for the abstractions were tailored to each of the priority 

conditions, while still containing some common elements. Given the data available in the 

published literature, an emphasis was placed on information relevant to the effectiveness of the 

strategy and the aspects of study design most pertinent to the applicability of the study. The goals 

of health care quality (safety, effectiveness, patient-centeredness, timeliness, efficiency, and 

equity), outlined in the IOM’s Quality Chasm report4

, also served to guide the reviewers. 

Unfortunately, most of these dimensions generally are not reported in studies assessing the 

efficacy of quality improvement strategies. Comparative data has been included, where available. 

The purpose of Stage 3 was to ensure the exclusion of articles that were deemed to be 

something other than evaluations of QI strategies, and to allow an assessment of the amount and 

types of evidence available for a given priority condition. This information guided decisions 

regarding the breadth of the analysis to be undertaken, and how best to create discrete 

substrategies for synthesis. Any conflicts that arose in Stage 3 were resolved by consensus 

opinion between a junior reviewer and a senior reviewer. 

Once the study designs and QI strategies were identified for a specific priority condition, 

articles meeting these final criteria underwent Stage 4 review. A junior reviewer conducted a 

detailed abstraction of relevant data (e.g., patient population, QI strategy, outcomes) from all 

included articles within the defined scope (see Appendix D—Stage 4 abstraction forms). A 

senior reviewer further confirmed the accuracy of the data abstraction. 

Types of Evidence Assessed in the Review 

The highest quality evidence available was used to assess the value of the QI strategies. Each 

of the study designs for the different QI strategies was assessed with respect to the conditions 

under consideration (i.e., hypertension). The reviewers also assessed important features of study 

conduct and analysis including concealment of allocation, patient blinding, provider blinding, 

and the unit of analysis relation to the unit of randomization. The hierarchy of study designs in 

Table 2 was used to guide the selection of study types for detailed data abstraction. Randomized 

controlled trials were considered the most persuasive source of evidence and so were deemed 

Level 1. If there were few or no randomized trials for a given strategy, the researchers evaluated 

Level 2 studies. Additionally, Level 2 studies also were reviewed to determine if findings about 

QI strategies were consistent across different study designs. Upon completing their initial 

literature review for each priority condition, the authors determined if sufficient studies existed 

meeting either Level 1 or Level 2 criteria. If they did, no detailed review of the study designs 

was performed at Level 3 (see Table 2). This is because biases commonly appear in Level 3 

21
studies that make interpretation difficult, despite any insights they might provide with regard to 

applicability (e.g., external validity). Level 4 evidence was excluded; as such, no uncontrolled 

studies were considered. The studies also were catagorized by types of outcomes measured. 

Studies that did not report any of the outcome types specified in Table 3 also were excluded. 

Table 2. Hierarchy of study designs

Level 1. Randomized controlled trials

 

Level 2. Controlled Before–After (CBA)—contemporaneous observation periods for control 

and intervention groups before and after an intervention 

 Interrupted time series (ITS)—well-defined time period for intervention 

implementation and at least three time points both before and after intervention 

 Quasi-randomized trials—contained at least two cohorts of patients assembled 

prospectively based on an allocation procedure that was non-random, but arbitrary, 

in the sense of bearing no apparent connection to patient or provider factors that 

might affect intervention outcome (e.g. alternation, date of birth, even/odd character 

of provider or patient identification) 

Level 3. Observational studies with controls—includes before-after and time series not meeting 

strict definitions of CBA and ITS (above), case–control studies, cohort studies with

controls. 

Level 4. Observational studies without controls (e.g., cohort studies without controls and case

series) 

Table 3. Outcomes relevant to inclusion criteria* 

 

Measures of Disease 

Identification 

Measures of 

Disease Control 

Measures of 

Provider Adherence 

Measures of Patient 

Adherence 

Included 

studies 

reported one 

of more of the 

outcomes 

presented here

Proportion of eligible 

patients receiving 

appropriate 

screening (i.e., blood 

pressure

measurements,

cancer screening) 

Clinical outcomes 

Intermediate 

outcomes with

established 

connections to 

clinical outcomes 

(e.g., HbA1c, blood 

pressure, lipid 

levels) 

Performance of

specific processes 

of care with 

established 

connections to 

patient outcomes 

Adherence to wellrecognized practice

guidelines (e.g., 

from professional 

societies) 

Biochemical assays

(e.g., blood or urine 

drug levels, urine 

cotinine for smoking 

cessation) 

Pharmacy data 

(e.g., refilled 

prescriptions) 

Home or office pill 

counts 

Patient interviews

*Measures of provider knowledge, patient understanding, self-efficacy, or other intermediate outcomes were included

only when they accompanied outcomes listed in the above table. For instance, a study reporting a measure of patient 

adherence with care as well as changes in patient understanding, self-efficacy, or empowerment would be included. 

Similarly, articles reporting only measures of satisfaction with care or resource use were not included unless they also 

reported measures of disease identification, disease control, or provider or patient adherence. 

22
Evaluation of Quality Improvement Strategies 

Most of the reported information addressed QI strategy effectiveness. There was a paucity of 

available data on the safety, equity, and applicability of the various approaches. 

A number of factors may influence the success of a QI strategy. Table 4 summarizes many of 

these factors and organizes them into three categories. Relatively little information on the 

features of the QI target was obtainable, due to time restrictions. This is a potential limitation of 

the analysis. The authors have noted in the table those factors for which information was 

obtained, as well as those factors included in the synthesis. 

Table 4. Features that may affect success of QI interventions

Features of the Study Features of the QI Intervention Features of the QI Target 

Study setting 

√ Study period

√ Country (√√ US vs. non-US) 

__ Financial/organizational structure of 

health care system 

√ Type of clinical setting (e.g., general vs. 

specialty clinic, community-based, work 

site intervention) 

√ patient population (e.g., specific 

disease being studied (diabetes vs. 

hypertension vs. asthma, etc.), early vs. 

advanced illness, significant comorbid

conditions, underserved, poor adherence) 

√ magnitude of local quality gap 

Type of QI strategy 

√√ Broad category of QI strategy (e.g., 

patient education, provide education, 

audit & feedback, etc.) 

√√ Number of QI strategies employed in 

the intervention 

* Intensity of QI strategy (e.g., number of 

educational sessions, frequency of audit 

& feedback cycles, extent of case 

management) 

__ Involvement of top management and 

other forms of institutional support 

* Format in which QI strategy delivered 

(e.g., face-to-face, dissemination of 

printed materials) 

√√ Use of an information system 

Content 

__ Attitude of clinicians toward target 

(driven in turn by guideline complexity, 

evidence base, concordance with existing 

practice) 

__ Complexity of action required by

provider (e.g., making a referral, ordering 

a test, adjustment of medication regimen, 

performing specific aspects of history or 

physical) 

__ Baseline level of adherence with

target 

__ Difficulty in achieving target (e.g., 

achieving a specific goal such as blood 

pressure below a certain value vs. 

process performance irrespective of 

outcome) 

 

Study methodology 

√√ Trial design (e.g., RCT, quasi-RCT, 

CBA, ITS)†

√√ Concealment of allocation

√√ Blinding (patients, providers) 

√√ Agreement in unit of randomization 

and unit of analysis 

†

 RCT = randomized controlled trial; CBA = controlled before–after study; ITS = interrupted time series. 

__ Indicates no information collected 

* Indicates information captured in text answers by reviewers rather than structured format 

√ Indicates data collected relevant to this feature 

√√ Indicates data collected and included in summary analysis, when feasible for a given topic 

23
Quantitative Synthesis of Quality Improvement Strategies 

Quantitative evaluations of the QI effect were performed for the various strategies, when 

possible. These evaluations were done only in situations when: 1) a sufficient number of studies 

with similar outcomes were available, and 2) the studies were sufficiently homogeneous in their 

design and population to provide a valid quantitative sample.

 

Calculation of summary effect for studies. In addition to the descriptive and qualitative 

investigations, two additional forms of analysis were planned for inclusion in the review. The 

first involved calculation of the median effect for outcomes within a given category (i.e., all 

provider adherence outcomes reported by a given study) so that studies with the same features 

could be compared using a common metric. Following the method employed in a recent 

systematic review of strategies for guideline implementation,61 researchers identified for each 

study the adherence outcome that indicated the median improvement attributable to the 

intervention. For example, if a study reported one outcome involving adherence to a guideline 

for checking HbA1c, another relating to managing cardiovascular risk factors, and another for 

delivery of patient education, a calculation of the net improvement attributable to the 

intervention for each outcome then would be done. The net improvements then were ranked for 

all of the outcomes and the median net improvement was used as a summary measure for the 

study. 

The net improvement in adherence was calculated as (Post-intervention adherence – Preintervention adherence)Study group – (Post-intervention adherence–Pre-intervention adherence)Control 

group. Outcomes were not combined for measures of disease control, so for example, the authors 

simply reported the net reduction in HbA1c, systolic blood pressure (SBP) or diastolic blood 

pressure (DBP) attributable to the intervention.

For instance, the net reduction in SBP attributable to the intervention was calculated as: 

(Post-intervention SBP – Pre-intervention SBP)Study group – (Post-intervention SBP – Preintervention SBP)Control group.

To characterize the impact of a particular type of QI strategy (i.e., provider education) or 

study feature (i.e., trial design), a calculation was made of the median effect achieved in studies 

with the feature of interest. For instance, all trials with interventions that included some aspect of 

provider education and also reported a change in mean SBP for the study groups were identified. 

Next, the median net reduction in SBP for these trials was computed and compared to the median 

effect for all trials, as well as the median effect for trials with interventions having no component 

of provider education. The median improvement in adherence across different QI types was 

compared similarly. 

The use of median effects, rather than average effects, prevented the skewing of summary 

measures based on outliers with particularly large or small effect sizes. This was regarded as 

particularly important because, if publication bias were present, small studies with relatively 

large effect sizes would more likely be published than small studies with more modest effect 

size. Thought was given to a weighted median, with weights based on sample size, to avoid 

giving equal weight to all studies regardless of size. Weighted medians are not as straightforward 

as weighted means, especially when attempting to preserve the original significance of the effect 

size (e.g., the observed reduction in HbA1c or SBP in the units used for those outcomes). So 

24
rather than attempting a weighting function, the authors chose instead to examine the median 

effect sizes using different strata of study sample size (e.g., comparing the median effect among 

studies with sample sizes in the lowest quartile vs. those in the highest quartile, or lower half vs. 

upper half). 

Adjustments for unit of analysis errors. The “clustering effect,” in which the unit of 

analysis and unit of allocation differ (i.e., providers or clinics randomized, but patient level 

outcomes analyzed) was anticipated in a significant number of studies. The significance of 

clustering is that patients within a cluster are not independent (e.g., patients at one clinic 

resemble one another in more ways than they resemble patients at other sites, or those cared for 

by other providers in the trial). Unit-of-analysis errors do not affect point estimates for effect 

sizes, but they can have a spurious narrowing effect on the associated confidence interval, 

causing potentially false-positive trial results.62-69 To prevent the same false precision in this 

analysis, an effective sample size* was calculated for each study for the meta-regressions 

described below. Moreover, the degree to which investigators acknowledged or accounted for 

cluster effects did not affect the analysis, apart from the fact that investigators who did consider 

cluster effects in the design or analysis of their trial were more likely to report data such as the 

number of providers randomized, rather than reporting only the total numbers of patients in each 

group. The same investigators also might provide more technical details, such as values for the 

intra-cluster coefficient (ICC). 70-72

Meta-regression Analyses 

For the more involved quantitative analyses—meta-regression analysis of included studies—

the investigators used a more conventional measure of effect size, defined as the difference 

between the mean values for the intervention and control arms, divided by the pooled estimate of 

groups within the standard deviation.†

 The researchers constructed these formal effect sizes, as 

well as the above median effect measures, such that a positive result always reflected 

improvement (e.g., a positive reduction in average HbA1c or a positive improvement in 

adherence). 

The regression models aimed to evaluate the relative effectiveness of different intervention 

components and the impact of study features such as trial design and study period. Specifically, 

the investigators constructed regression models using the pre-intervention effect size (ESPre) as a 

predictor variable. Initially, each methodological feature or QI strategy was modeled with ESPre 

to evaluate its effect on the post-intervention effect size (ESPost); subsequently, the researchers 

developed multivariate models using multiple components as an individual feature’s covariates, 

in order to independently assess the effect of an individual feature after adjustment for other 

components. Linear regression was carried out as Y = β0 + β1X1 + β2X2, with X1= ESpre and the

dependent variable, Y, corresponding to the outcome of interest—a measure of disease control 

such as HbA1c, or the summary measure of adherence outcome described above. The approach 

retained ESpre as a predictorin all analyses because baseline differences between the study and 

 

* Effective N = (km) / (1 + (m-1)r) where k is the number of clusters and m is the number of observations per cluster and r is the 

intra-cluster coefficient. When r = 0, then N = km. When r = 1, then N = k

† Effect size = (X

_

 I - X

_

 C) / Sp where X

_

 I is the mean for the intervention group, X

_

 C is the mean for the control group and Sp is the 

pooled-within-groups standard deviation, which is calculated from: Sp

2

 = ( (NI – 1) SI

2

 + (NC – 1) SC

2

) / (NI + NC – 2). NI and NC

are the intervention and control sample sizes and SI and SC are the intervention and control standard deviations. 

25
control groups were expected to act as important covariates, even when these differences did not 

meet conventional thresholds for statistical significance. 

26
Chapter 3. Toward a Theoretic Basis for Quality 

Improvement Interventions 

Kathryn M. McDonald, M.M. 

Stanford University 

Ian D. Graham, Ph.D. 

University of Ottawa 

and Ottawa Health Research Institute 

Jeremy Grimshaw M.B.Ch.B. Ph.D. FRCGP 

Ottawa Health Research Institute 

and University of Ottawa

Introduction 

While medicine has a long history of investigating what works and why, we have a fairly 

limited understanding of the causal mechanisms of interventions to improve health care quality. 

Implementation research has been defined as the scientific study of methods to promote the 

uptake of research findings for the purpose of improving quality of care.73 It includes the study 

of factors that influence the behavior of health care professionals and organizations, and the 

interventions that enable them to use research findings more effectively. Closing the quality gap 

relies on implementation research to create effective quality improvement (QI) interventions. 

Implementation research is referred to by many names, including action research, research 

utilization, practice guidelines implementation research, diffusion of innovation, translating 

research into practice, quality improvement research, knowledge translation, knowledge transfer, 

knowledge mobilization, and knowledge exchange. Research has followed two primary and 

related tracks: transfer or diffusion of knowledge and behavioral change. 

Researchers and analysts from diverse disciplines have studied variables that influence the 

diffusion of innovations. In addition, numerous theories of behavioral change have been 

developed from a variety of perspectives-- psychology, sociology, economics, marketing, 

education, organizational behavior, and others. In QI, theories abound regarding ways to change 

the behavior of patients, clinicians, and organizations. However, these “parts” are not integrated 

into one overarching “whole.” Most theories are descriptive, and do not predict the components 

that are required for effective design and implementation of quality improvement strategies. 

What are the important variables in the sequence of steps that will result in improved quality? 

How do these variables interact? Which variables are likely to matter, and under what 

circumstances? The application of theory to quality improvement strategies may be analogous to 

the use of biologic plausibility in designing new clinical treatments; a process by which 

researchers determine the most likely target for a treatment, and interpret study results 

accordingly. Developing a theory that helps to explain and predict a particular set of behaviors 

would allow QI implementers to zero in on an appropriate target for change, and craft the 

intervention to effect that change. 

Development of theories with greater predictive value also would help researchers to design 

interventions to overcome barriers to behavioral change. Barriers to change may stymie even the 

most laudable (and seemingly obvious) effort to correct a health care quality problem. Ongoing 

27
attempts to enhance clinicians’ compliance with standard hand hygiene practice is a case in 

point.74 Barriers to change exist at each stage in the causal pathway of noncompliance, from the 

psychological, to the cognitive, to the physical layout, across diverse clinical settings. 

For hand hygiene, or any intervention to be effective it must be tailored to the target behavior 

or process, as well as to the environment itself. Cognitive theory suggests that clinicians may 

simply be unaware of the consequences of hygiene noncompliance, while adult-learning theorists 

would argue that clinicians must experience the negative consequences of their action before 

their behavior will change.75 Behaviorists might suggest an intervention designed to provide 

feedback and reinforcement, while social theory adherents and organizational theorists would 

turn their attention to the culture of the organization that fails to support compliance, or to 

disseminate an appropriate “message,” to adopt the vernacular of marketing theorists.76 All of 

these are credible suppositions. Without an understanding of the theoretical construct(s) that 

explain mechanisms of change, efforts are likely to be scattershot, challenging even the most 

enthusiastic QI implementer to transfer innovations into practice efficiently and consistently. 

As the hand hygiene example illustrates, numerous descriptive theories—and some

predictive theories—are available. Unfortunately, they have been put to little use in the field. A 

literature review of surveys of physician behavior identified nearly 300 barriers to physician 

adherence with clinical practice guidelines, sorted into seven general categories, and further 

classified according to behavioral and other theories.77 However, a recent review of guideline 

implementation efforts reveals that of the 235 included studies, merely 10% of the authors 

provided an underlying theoretical rationale for the particular intervention they had selected.78 

For readers with an interest in QI’s formative and ongoing theoretical discussion, this chapter 

introduces a general hierarchy that organizes various levels of theory from the macro to the 

micro. The authors then introduce selective examples of the theories that some researchers in the 

field of QI have considered as they design interventions to modify interactions among individual 

patients, health care providers, and the organizations they function within. The reader should 

note that this selective overview renders only a sampling of the numerous theoretical schools of 

thought from which health care quality improvement strategies are or could be designed. A 

comprehensive review of the entire body of theoretic work in the field of QI exceeds the scope of 

this review. In addition, the omission of any particular theoretic basis (and the inclusion of 

others) does not imply that one lacks importance or should not be accorded due consideration. 

Rather, the selection presented reflects the combined expertise of the EPC’s technical advisors, 

and represents a variety of some of the more commonly utilized theoretic underpinnings that 

inform or show promise for QI efforts in the field of health care. Sources for more in-depth 

coverage of the theoretical models discussed, and others, are noted. 

Hierarchy of Theories and Models

A theory describes and explains what is observed and why it happens. Various taxonomies, 

including the one used for the QI reviews of diabetes (Closing the Quality Gap, Vol. 2) and 

hypertension (Closing the Quality Gap, Vol. 3), answer the question of “what” strategies could 

be used to try to change behavior or transfer knowledge. Theories expand on the “what” question 

by addressing “how” and “why” these QI strategies or their components might or might not be 

effective, and under what conditions (“when” and “where”). Thus, theories provide a framework 

for interpreting a study’s findings in its own setting, and potentially, in other settings. A model 

can be conceptual, as described below, or action-oriented with process steps to provide an 

28
approach to QI. Following is a selection of models and theories that have been used to design 

some quality improvement strategies, or which may be useful in future work. Given the 

rudimentary state of the art and science of implementation research it is helpful to review what is 

sometimes referred to as the structural hierarchy of knowledge, or the levels of theory from the 

most abstract to the most specific.77

 

 The terms conceptual models, conceptual frameworks, and conceptual systems often are used 

synonymously and represent global ideas about a phenomenon. They are used to clarify, 

describe, and organize ideas,78 and could be viewed as the top level of the hierarchy. It should be 

noted, however, that conceptual models/frameworks and theories vary in their levels of 

abstraction and a continuum exists within the structural hierarchy of knowledge. Boundaries 

between conceptual models and levels of theory sometimes overlap, making it difficult to clearly 

differentiate among them within the hierarchical structure. 

 A theory is an organized, heuristic, coherent, and systematic set of statements related to 

significant questions that are communicated in a meaningful whole.79 It describes observations, 

summarizes current evidence, proposes explanations, and yields testable hypotheses—all within 

specific assumptions and constraints. Theories can be described in terms of their scope, with 

wider scopes reflecting generally higher levels of abstraction in the knowledge hierarchy. 

 A grand or macro theory is a very broad theory that provides a general framework for the 

nature and goals of a discipline. Grand theories are non-specific and are made up of relatively 

abstract concepts that lack operational definitions and relatively abstract propositions that are not 

amenable to direct empirical testing.80,81 They tend to be developed through thoughtful and 

insightful appraisal of existing ideas, or creative leaps beyond existing knowledge. While widely 

applicable, grand theoretical statements lack the detail that is necessary to fully understand the 

relationships between constructs and variables in specific situations. A mid-range theory is more 

limited in scope, addresses specific phenomena, and is intended to guide empirical inquiry as 

well as action or practice. It encompasses a limited number of concepts and a limited aspect of 

the real world. Mid-range theories are made up of relatively concrete concepts that are 

operationally defined and can be empirically tested. They are less abstract than grand theory and 

more abstract than empirical generalizations or micro theory.82 A micro, practice, or situationspecific theory (sometimes referred to as prescriptive theory) has the narrowest range of interest, 

and focuses on specific phenomena that reflect clinical practice, and that are limited to specific 

populations or to a particular field of practice. 

At all levels, the purpose of theory is to organize and communicate information 

parsimoniously and clearly, in order to describe, discover, and predict phenomena seen in the 

world. Whether a theory is “good” or not depends on both its ability to withstand efforts to 

disprove it (i.e., “falsifiability”), and its usefulness.83 In the subsequent descriptions of selected 

theories, organized roughly by their level in the hierarchy, no attempt has been made to evaluate 

the theories, apart from the commentary on their potential applicability to QI strategies. 

Conceptual Models and Grand Theories 

of Implementation Phenomena

Conceptual models and grand theories of implementation phenomena are essentially models 

or theories of change. Change theories/models may be classified as either classical or planned.78 

29
 

Classical Theories of Change 

Classical theories/models of change (sometimes referred to as descriptive or normative 

theories) are passive; they explain or describe the naturalistic process of change or diffusion of 

innovation. Perhaps the most prominent example of a classical theory of change is Everett 

Rogers’ diffusion of innovation theory.84-85 Some of the better known observations deriving from 

Rogers’ work are the innovation-decision process, which describes how potential adopters’ 

perceptions of the attributes or characteristics of an innovation influence diffusion of the 

innovation, and the relationship between adopter types and diffusion. The innovation-decision 

process consists of five stages that potential adopters pass through as they decide to adopt an 

innovation. These stages are: knowledge (becoming aware of the innovation), persuasion 

(developing positive attitudes toward the innovation), decision (making a cognitive decision to 

adopt the innovation—i.e., developing an intention to adopt), implementation (using the 

innovation), and confirmation (continuing to use the innovation, adapting the innovation, or 

abandoning it). 

Rogers posited that innovations are more quickly adopted when they are compatible with 

current values, beliefs, and ways of doing things; are seen to be more advantageous than the 

current practice (relative advantage); are easy to do or use (low complexity); are observed by 

others to be in use (observability); and can be easily tested before being formally adopted 

(trialability). Another important contribution of Rogers’ work has been the observation that 

potential adopters may fall into one of a number of adopter types, which relate to diffusion: 

innovators (the fastest adopter group, venturesome, cosmopolitan, socially disconnected), early 

adopters (opinion leaders, respected, locally well connected, self-conscious experimenters), early 

majority (deliberate, local observers, have watched early adopters), late majority (skeptical, more 

conservative, wait for majority’s adoption of the innovation before adopting it themselves), and 

laggards (traditionalists, socially isolated, slowest to change). Readers should also consider 

Jonathan Lomas’s Coordinated Implementation Model,86-87 another implementation theory that is 

more descriptive in nature and focuses explicitly on the medical context. For example, the Lomas 

model proposes that better knowledge transfer occurs when passive continuing medical 

education is replaced with active dissemination that takes into consideration a broad range of 

interacting factors that may promote or hinder adoption (e.g., economic, personal, administrative, 

and community-based incentives).88

Limitations of Classical Theories of Change 

While classical theories/models of change may help to identify potential determinants of 

change, they provide little information on the best way to either accelerate or hinder natural 

diffusion (e.g., they provide no direction for operationalizing opinion leaders). For this reason, 

QI implementers and researchers tend to be more interested in planned change theories/models, 

which are specifically intended as guides, or to cause change.78 

Planned Models of Change

A planned change model/theory is a set of logically interrelated concepts that explain, in a 

systematic way, the means by which planned change occurs. These models predict how various 

forces in an environment will react in specified change situations, and help QI implementers 

control variables that increase or decrease the likelihood that change will occur.89-90 Planned 

30
change, in this context, refers to deliberate (not haphazard) efforts to engineer change in groups 

that vary in size and setting. Those who use planned change theories/models may work with 

individuals, but their objective is to alter ways of doing things in social systems. Examples of 

planned change models/theories are Green’s Precede-Proceed model,91-92 the social marketing 

model, Berwick’s rules for dissemination,93 and the Ottawa Model of Research Use,94 to name

several. 

 Precede–proceed. Precede–Proceed specifies the steps that precede an intervention and 

suggests ways to proceed with its implementation, including subsequent evaluation.91 In the 

precede stages, the implementer first specifies the problem and then identifies the factors that 

contribute to it. These factors are categorized theoretically as predisposing, enabling, or 

reinforcing, and then rated in terms of importance and amenability to change. Predisposing 

factors include attitudes, beliefs, and perceptions, all of which provide the impetus for change. 

Enabling factors include the resources, facilities, and skills that must be present for change to 

actually occur. Reinforcing factors include rewards or incentives, such as positive feedback, that

encourage change.92 The key proceed stages are implementation and evaluation of the 

intervention. The evaluation stage examines the degree to which the protocol was implemented, 

and the effect the intervention had on behavior change, and on predisposing, enabling, and 

reinforcing factors. 

 Social Marketing. Social marketing is a planning model that consists of several stages: 95

planning and strategy, when research is conducted with the target group and resources available 

for the intervention are assessed; selecting the relevant channels and materials for intervention,

when specifications of the program structure and relevant outcomes are made, and the target 

group is ‘segmented’ to create homogeneous subgroups for tailoring messages and distribution 

methods; developing and piloting materials with the target audience to determine their 

relevance, comprehensibility, and likely impact; and finally, implementation, evaluation, and 

feedback after which the intervention may be refined. Social marketing has largely focused on 

bringing about health behavior change at the community level, but has also been used as the 

basis for some other quality improvement strategies. For example the principles for academic 

detailing proposed by Soumerai and Avorn96 are based upon social marketing approaches. In 

academic detailing, implementers conduct interviews to investigate baseline knowledge and 

motivations for current practice; focus programs on specific categories of physicians as well as 

opinion leaders; define clear educational and behavioral objectives; establish credibility through 

a respected organizational identity, reference authoritative and unbiased sources of information, 

and present both sides of controversial issues; stimulate active physician participation in 

educational interactions; use concise graphic educational materials; highlight and repeat the 

essential messages; and provide positive reinforcement for improved practices in followup visits. 

Donald Berwick’s Rules for Dissemination. While not proposed as a model per se, 

Berwick93 has proposed a series of seven ‘rules’ for translating research into practice, largely 

derived from the theoretical work of Rogers and Van de Ven.85-97 The seven rules require an 

implementer to: 1) find sound innovations, 2) find and support innovators, 3) invest in early 

adopters, 4) make early adopter activity observable, 5) trust and enable reinvention, 6) create 

slack for change, and 7) lead by example. The IOM National Priority Report (which provides the 

topic areas for this Series) reflects the first rule, in that the IOM committee identified clinical 

topics for which sound, evidence-based innovations exist, which could be implemented more 

broadly. This Series could be viewed as fulfilling the second rule, by finding and reviewing the 

research that innovators have reported. 

31
Ottawa Model of Health Care Research Use. The Ottawa Model of Research Use94-98

offers a comprehensive, interdisciplinary framework of elements that affect the process of health 

care knowledge transfer, and is derived from published literature. Although not explicitly linked 

to Donabedian's germinal work describing health care quality production in terms of structure, 

process, and outcomes,99 the model captures these characteristics, along with important social 

factors. The elements considered central to the research use process are: the evidence-base of the 

innovation, potential adopters, practice environment, strategies for transferring the evidence into 

practice, the use of the evidence, and outcomes of the process (e.g., related to patient health, 

practitioner issues, and economic implications). A particular advantage of the model is that it 

may be applied at any level in the delivery of care (e.g., individual professional or team behavior 

change, hospital behavioral change, health care system change), and that the patient is 

incorporated as a key component of each element. The model is dynamic, meaning that each 

element is assumed to influence and be influenced by the others (e.g., depicted by double arrows 

that create multiple loops in Figure 1). Finally, in planning research transfer activities, the model 

relies on a process of assessment, monitoring, and evaluation (AME) of each element before, 

during, and after the decision is made to promote an innovation (broadly defined as research 

evidence that is new to the potential adopter). In brief, the model directs QI implementers to 

conduct a barriers assessment of the evidence-based recommendations, the potential adopters, 

and practice environment to determine factors that might be expected to hinder or support the 

uptake of the recommendations. Next, this information is used to select and tailor interventions 

to overcome the anticipated barriers or enhance the supports. The introduction of the 

interventions is then monitored, to ensure that they are being delivered as expected, are 

addressing the identified barriers, and that unexpected barriers have not emerged since the time

of introduction that must now be addressed. Monitoring the adoption of the recommendations 

during the implementation phase can help determine whether the dose of intervention has been 

sufficient to bring about the desired change in practice, or whether more of the same or new 

interventions may be required. Finally, the impact of the implementation process is evaluated 

and the iterative process begins again. Figure 1 summarizes the key features of the model, with 

the QI strategies' elements bolded to emphasize the focus of this report. 

Limitations of planned change models. Planned change models provide broad frameworks 

for planning implementation activities, but are less helpful when considering which specific 

interventions to use. This may be due to the fact that these theories are as yet insufficient to 

relate specific intervention components to a predicted effect on knowledge transfer or behavior. 

In addition, potentially relevant theories have not yet been validated for health care professional 

or organizational change. 

Mid-range Theories 

Mid-range theories are potentially more predictive of behavior and behavioral change at 

different levels. They are commonly discipline-specific, however, and their applicability to 

individual health behaviors and/or health care professional behavior has not been well tested in 

many instances. 

Ferlie and Shortell100 have suggested several levels at which interventions to improve the 

quality of health care might operate: the individual health professional and health care groups or 

teams; organizations providing health care; and the larger health care system or environment in 

which individual organizations are embedded. Different theories will be relevant to interventions 

32
at different levels, for example, psychological theories will be more relevant to interventions 

directed at individuals and teams, while theories of organizational change will be more relevant 

to interventions directed at hospitals and other organizations. A full scientific rationale for 

interventions to improve quality requires exploration of theories relevant to interventions 

directed at each of these levels. 

Social Psychological Theories 

Individual health behaviors and health care professional behaviors are forms of human 

behavior that are influenced at the level of the health care system, the health care organization, 

the individual provider, and by the characteristics of clinical best practices. Theories of human 

behavior that have been successfully adopted in similar settings may provide a basis for 

developing a theoretical basis for understanding health care professional and organizational 

behavior. 

Social influence theories. Social influence theory recognizes the importance of shared 

beliefs and assumptions, group norms, and organizational culture as determinants of individual 

and professional behavior. Social influence interventions attempt to promote behavioral change 

by influencing group perspectives. Mittman and colleagues

101 have proposed a classification 

scheme of potential social influence interventions for practice guideline dissemination and 

implementation that depends on the size of the target audience, and includes educational 

outreach and opinion leaders. 

Motivational theories (including social cognition models). These theories propose that 

motivation determines behavior, and therefore the best predictors of behavior are factors that 

predict or determine motivation (or intention). Examples include Albert Bandura’s social 

cognitive theory,

102 and the theory of planned behavior103. Bandura’s social cognitive theory102

proposes that behavior is determined by incentives and three kinds of expectancies (situationoutcome expectancies, outcome expectancies, and self-efficacy expectancies). Self-efficacy 

expectancies are beliefs about one’s ability to perform the behavior (e.g., I can stop smoking) 

and have been found to be the most important construct in empirical studies104as well as a 

consistent predictor of behavioral change.105 The theory of planned behavior (TPB) proposes that 

the strength of an individual’s intention (or motivation) to engage in a behavior, and the degree 

of control they feel they have over that behavior (perceived behavioral control), are the proximal 

determinants of engaging in it.103 The theory of planned behavior also proposes that intention 

strength is determined by three variables: attitudes towards the behavior (a product of beliefs 

about its consequences and evaluations of those consequences), subjective norms (a product of 

perceptions of the views of other individuals or groups about the behavior, and the strength of 

the individual’s desire to gain approval of these groups), and perceived behavioral control (a 

function of beliefs about factors likely to facilitate or inhibit the behavior— these might include 

organizational constraints and patient preferences). The TPB has been shown to predict a range 

of individual health related behaviors with some success. Recent meta-analyses have suggested 

that the TPB can account for approximately 30% of the variance in health behavior.106 The 

application of TPB to our understanding of providers’ adherence to evidence-based advice about 

their practice has been limited. However, early studies suggest that it is a useful, systematic tool 

to identify barriers to, and facilitators of, change and is helpful in selecting appropriate forms of 

intervention.107-108

33
 Action theories. These theories may include motivational elements, but postulate that other 

factors are necessary to predict behavior. Examples include operant conditioning and 

implementation intentions. Operant conditioning proposes that behaviors that have positive 

consequences for the individual (such as remuneration) are likely to be repeated, whereas those 

that have unpleasant consequences will become less frequent.109 The principle that positive 

consequences promote repetition of behavior is well established and has been widely and 

successfully used to understand behavior and behavioral change. As rewarded behaviors are 

repeated, and may become “habitual,” the frequency of past behavior can be a powerful predictor 

of future behavior. Gollwitzer110 distinguishes between goal intentions (conceptually similar to 

intentions within the theory of planned behavior) and implementation intentions, which are 

explicit plans about when and where a goal intention will be achieved. Gollwitzer argues that by 

creating an implementation intention, people effectively transfer control of the behavior to the 

environment—establishing cues to action. This is a relatively new concept in health behavior 

research, however, experimental studies suggest that people who have formulated plans like 

these are more likely to translate their intentions into action than those who have not.111 For 

example, women are told to specify where and when they will perform breast self-examination, 

and subsequently are found to have followed through more frequently than those with no plan 

regarding their intention to implement.111 While studies thus far have utilized this theory for 

patient behavior change, it may be just as applicable to provider behavior change (e.g., a 

provider education QI strategy might incorporate a step that asks the clinician to formulate a plan 

for implementing a change in practice).112 

 Stage theories. These theories propose that at a given point in time, individuals are at 

different stages in a linear progression toward behavioral change (for example, an awareness 

stage occurs before an action stage), and that predictors of behavior must account for the 

different stages of progression. Examples include the transtheoretical model of change113 and the 

precaution–adoption process model.114 From a stage theory perspective, interventions to 

facilitate change will be most effective if they are tailored to the stage the individual has reached 

within this process. Stage theories have been widely used to develop interventions in behaviors 

such as smoking cessation.114 Despite their face validity and (relatively) widespread use, a recent 

systematic review found that there was little evidence that interventions based on stage theories 

were more effective than non-stage theory interventions.115 

Organizational Theories 

To understand organizational change theories, it is helpful to define organizational behavior, 

and to present some background on organizational theory on a macro level. Organizational 

behavior is the study of individual and group attitudes and actions within an organizational 

setting, and describes how the resultant behavior affects the goals of the organization. Macro 

theories regarding how organizations function and behave fall into three major typologies: 1) 

rational system theories, which focus on the internal structures and processes of an organization, 

2) natural-system theories, which also focus on internal workings, but emphasize the 

organization as a social system, noting the importance of unplanned processes and events, human 

relations, and integration of individual and organizational goals, and 3) open-system theories,

which emphasize the ways in which an organization’s environment relates to its structure and 

behavior.116 Shortell posits that the health care system can be best understood in light of all three 

of these theories, because of its complex inner, social, and external structures.117-118

34
 Organizational change theories, particularly the rational system models, may be most directly 

applicable to managing and achieving change within the health care organization and its 

environment. Alternatively, institutional models (described below) likely also apply to prominent 

QI strategies. The following section describes these mid-range models generally in terms of 

definition, important variables, perspective on organizational change, and strategies for change 

within organizations.118-119

 

 Rational models. These models emphasize four stages in the process of organizational 

change: awareness of a problem or quality gap, identification of action to solve the problem or 

narrow a gap, implementation of the action, and finally institutionalization, where all relevant 

parties accept the change. At each stage, the models adopt three perspectives of the change 

process, any of which may need to be addressed, depending upon circumstances. First, the 

behavioral perspective assesses the manner in which issues connect to attitudes, culture, values, 

and norms within the organization. The structural perspective focuses on the design of activities 

and roles of individuals and work groups, considerations of power and influence, and variables 

such as complexity, centralization and formalization. Finally, the contingency perspective

considers the interactions between behavior and structure, as well as the particular timing and 

circumstances of the target change. Kalunzy and colleagues120 diagram factors influencing a 

change process in three dimensions, with a spiral through time showing the stages from 

recognition through institutionalization, with each twist in the spiral affected differentially by 

personnel, processes, and structure of the organization, as well as the attributes of the change. 

For example, at the implementation stage of an innovation that is costly or has widespread 

consequences for the organization, each of the factors—personnel, process, and structure—are 

likely to be influential. At the earlier problem recognition phase, only personnel may be 

involved, regardless of the nature of the change. Throughout all phases, an organization is likely 

to be influenced by internal and external pressure for change, as well as potential examples from 

other organizations about intervention possibilities. 

 

 Institutional models. In contrast to rational models, which assume that management has the 

freedom to implement change according to specific economic and strategic interests of the 

organization, institutional models specify that legitimacy seeking behaviors of organizations 

drive the implementation of organizational change. Legitimacy is acquired relative to the norms

of the time and industry for each organization. These models may be particularly pertinent to QI 

because they emphasize the role of social factors including pressures to conform from outside of 

an organization (e.g., regulatory bodies, professional organizations). One illustration of how 

institutional theory explains important factors in the adoption and implementation of a QI 

strategy involves Total Quality Management (TQM), an organizational intervention 

encompassing a range of philosophies and activities. Institutional theorists hypothesize that as 

TQM has become more accepted, the reason for its adoption, and therefore the style of

implementation is motivated by a need for legitimacy, as opposed to a strategic or economic 

imperative. The menu of TQM philosophies includes commitment to identifying, meeting, and 

then exceeding the needs of both internal and external stakeholders; continuous improvement, 

including raising standards; structured, problem-solving processes using statistical and other 

tools; and employee empowerment to improve quality, including training across all levels, 

functions, and areas of an organization. Activities associated with TQM include assessments of 

community needs; benchmarking performance; training in principles and methods of TQM; 

brainstorming; use of cause and effect diagrams (also known as Ishakawa or “Fishbone” 

diagrams), control charts, and other analytic tools. Health care organizations have either adopted 

their own versions of TQM, or followed a standard approach named for its progenitors (e.g., 

35
Juran, with its emphasis on quality audits, Crosby, with a focus on training, or Deming, with an 

orientation toward statistical and process tools). Westphal and colleagues121 examined Total 

Quality Management programs introduced in standard or customized ways over an eight year 

time period to determine if implementation of this QI strategy differed depending on timing. 

They observed that earlier adopters customized the form of TQM in ways that improved 

organizational performance, as expected in a rational system theory, while later adopters more 

often used standardized TQM programs, apparently motivated by external factors such as 

accreditation ratings, and fitting with precepts of institutional theory. Thus, both the form and 

consequences of a particular QI strategy are influenced by the timing of its implementation, and 

the organization’s place, relative to the health care industry as a whole. 

Iles and Sutherland122 conducted a comprehensive review of several decades of 

organizational change theories for the British National Health Services. They provide summaries 

of a large number of models, and cluster them around a small number of key questions for 

implementation of quality improvement strategies. The organizing questions or topics for each 

cluster of theories, and the specific theories and models examined in the review are shown in 

Figure 2. 

According to this review, most of the evidentiary base for organizational theories in the 

health care sector is comprised of case studies, with only one cohort study on TQM examining 

the link between the strategy and the effect on health care quality.123 Several case studies also 

examine the link between the particular theory and health outcomes, but most focus on other 

outcomes (e.g., job security, cost-competitiveness, position, etc.). Generally, the organizational 

literature related to promoting change, or "change management" is descriptive and infrequently 

applied to the health care setting. It is certainly conceivable that the type of evidence useful in 

the management area may differ from that applied to clinical practice. The complexity of health 

service delivery and organization may require alternative methods for assessing what qualifies as 

evidence. Ultimately, however, more evaluative research in this area may be useful to those 

implementing quality improvement strategies by highlighting potential levers for change. 

Other Discipline-Based Theories 

A number of other disciplines have contributed theories to patient, provider, and 

organizational behavior change. For example, adult learning theory from educational research, 

emphasizes the role of intrinsic personal motivation,124 and creates interventions based on 

consensus development and problem-based learning.125 Provider education strategies that 

incorporate these approaches may be more likely to work than those that do not address intrinsic 

motivation. In contrast, marketing approaches have been used to promote health to the general 

public,126 as well as to target physician or organizational change. The advent of direct-toconsumer marketing is providing a natural experiment to determine the significance of marketing 

theories. 

Economic theories provide insight about the policies and funding mechanisms that may 

contribute to efficiency (e.g., avoiding waste in achieving quality) and equity (e.g., providing the 

same quality regardless of individual patient characteristics). Value or quality-based purchasing 

activity in the U.S. marketplace is an example of a QI strategy derived from economic theory. 

This approach is intended to have either a direct influence on the decisions of consumers or 

health care organizations by using payment incentives, or an indirect influence by reducing 

asymmetric information between different stakeholders in a market (e.g., health plans and 

36
consumers).127 Theories drawn from economics therefore play a role in considerations about 

designing policies to increase incentives for quality. 

Micro or Situation-specific Theory

As a relatively new field of inquiry, micro theories of knowledge translation and behavior 

change have yet to be articulated in relation to health care. These theories, should they eventually 

be developed, would predict the uptake of a specific behavior by a particular health professional 

group or subgroup when exposed to a given intervention (e.g., audit and feedback may be 

effective for changing radiologic ordering behavior of family physicians, but may not influence 

nurses’ use of compression bandaging guidelines). 

Conclusion 

Transferring knowledge, diffusing innovation, changing behavior, or getting the system to do 

the “right thing,” are different ways of stating the ultimate objective: to close quality gaps. 

Concurrent consideration of theory and actual implementation practice may yield mutually 

useful results. Figure 3 depicts a conceptual framework of the interactions among various midrange theories, and incorporates some of the factors that may predictably bring about change. 

The applicability of these theories to behavior change and knowledge transfer at the individual 

health, health care professional, and organizational level remains uncertain. There are 

considerable challenges to the development of interventions based upon these theories. To 

improve our understanding of the causal mechanisms, effect modifiers, and applicability of a 

variety of QI strategies, QI implementers and researchers may benefit from working together to 

rigorously develop and test predictive theories of professional and organizational theory at the 

mid-range and practice levels. Insight about why a QI strategy fails or succeeds may be more 

easily accomplished when theory and implementation are tested side by side. 

37
Chapter 3 Figures: 

ASSESS MONITOR EVALUATE

Evidence-based

Recommendations

•recommendation 

attributes 

•development process 

Strategies Adoption

HOW CAN CHANGE HAPPEN? 

•Organizational development (OD) 

•Organizational learning

•Action research

•Project management 

ADDRESSING COMPLEXITY, INTERDEPENDENCE AND FRAGMENTATION 

•Weisbord’s Six-Box Organizational Model 

•7S Model [Strategy, Structures, System, Staff, Style, Shared Values, Skills] 

•Content, Context and Process Model 

•Soft Systems Methodology (SSM) 

•Process modeling 

•Theory of Constraints (TOC) 

ANALYZING NEED FOR CHANGE 

•SWOT [Strengths, Weaknesses, Opportunities and Threats] analysis 

WHO AND WHAT CAN CHANGE? 

•Force field analysis 

•Organization-level change: 

 Total Quality Management (TQM) 

 Business Process Reengineering 

•Group-level change: 

 Parallel learning structures

 Self-managed teams 

 Figure 2. Organizational theories examined for evidence (adapted from Iles and Sutherland) 

Potential Adopters

•knowledge 

•attitudes 

•skills and habit

•diffusion (passive) 

•dissemination (active) 

•implementation (active 

and systematic) 

•overcome barriers 

•provide supports 

•intention/

 decision 

•use 

Outcomes

•patient 

•practitioner 

•system 

(economic) 

Practice Environment

•patients 

•structural 

•social 

Figure 1. Ottawa Model of Research Use (Logan and Graham, 2003)

•economic 

•other 

38
ASSESS MONITOR EVALUATE

Evidence-based

Recommendations

•Diffusion of innovations 

Potentia Strategies l Adopters •Marketing theories Adoption Outcomes •Theory of planned behavior •Advertising theories •Social cognitive theory •Change agents theories •Learning theories •Communications theory •Stages of change •Persuasion theories 

•Education theories 

Practice Environment

•Organizational change 

theory 

•Power and politics theory 

Figure 3. Examples of theories that could begin to populate the Ottawa Model of Research Use

•Attitudes and behavior of 

groups theory 

•Economic theory

39
Chapter 3 Notes 

1. We would like to acknowledge the many collaborators who have contributed to our 

understanding in this area, in particular Martin Eccles, Professor of Clinical Effectiveness, 

University of Newcastle upon Tyne, UK; Marie Johnston, Professor of Psychology, University 

of Aberdeen, Scotland; Jo Logan, Associate Professor, School of Nursing, Faculty of Health 

Sciences, University of Ottawa, Canada; Dr Anne Walker, formerly Senior Research Fellow, 

Health Services Research Unit, University of Aberdeen, Scotland. 

40
Appendixes 

49
This page intentionally left blank 

50
Appendix A 

National Priority Areas Summary 

with Key Associated Goals 

51
This page intentionally left blank 

52
Appendix A: National Priority Areas Summary 

with Key Associated Goals 

Care coordination (cross-cutting area)—About 60 million Americans live with multiple chronic conditions, such as 

hypertension and diabetes. Clinicians and institutions should actively collaborate and communicate to ensure an 

appropriate exchange of information and coordination of care. 

Children with special health care needs—Children with a chronic physical, developmental, behavioral, or emotional 

condition, or an increased risk of developing one, require more than the typical level of pediatric care. This vulnerable

population requires caregivers to work closely with families to develop and coordinate care plans. 

Diabetes—Diabetes is the fifth-leading cause of death in America, predisposing people to serious, long-term medical 

complications, including heart disease, hypertension, and blindness. There are several well-known and effective 

models for improving the delivery of care, with the goal of preventing progression through early and proper 

management. 

End of life with advanced organ system failure—Heart, lung, and liver failures account for about one-fifth of all 

fatalities in America. Care should minimize symptoms and reduce the rate of exacerbations of organ malfunction. 

Improving care requires continuity of care over time and across settings, close monitoring, and rapid responses. 

Evidence-based cancer screening—Cancer is the second-leading cause of death in the United States. Screening can 

significantly reduce death rates for several forms of cancer, especially colorectal and cervical cancer. Goals should 

be to increase the number of people who receive screenings and to provide timely followup. 

Frailty associated with old age—With more Americans living longer, more people will experience the multiple mental 

and physical health challenges associated with advanced age. Health care efforts should focus on preventing falls 

and pressure ulcers, maximizing function, and developing advanced care plans. 

Hypertension—Although this disease affects one in four adults in the United States, nearly a third of people with high 

blood pressure are undiagnosed. Untreated hypertension can lead to life-threatening complications, including stroke, 

heart attack, and kidney failure. Interventions should emphasize early detection and management. 

Immunization—Timely vaccination could prevent the deaths of about 300 children and between 50,000 and 70,000

adults annually. Influenza and pneumonia account for most of the adult deaths. Vaccination efforts should target 

nursing-home residents, who are susceptible to contagious illnesses because of advanced age and close living 

quarters. Also, new strategies should be developed to reach out to black and Hispanic adults, as well as low-income, 

inner-city children – populations that tend to have lower-than-average immunization rates. 

Ischemic heart disease—Ischemic, or coronary, heart disease, is the leading cause of death among adults in the 

United States. Efforts should focus on preventing heart disease and reducing recurrence of heart attacks through 

promotion of healthy lifestyle changes and use of cholesterol-lowering drugs, surgery, and timely administration of 

medications after a heart attack. In addition, efforts should ensure that those with heart disease are functioning at 

their greatest capacity. 

Major depression—Treatment rates for depression are significantly lower than those for many other chronic 

conditions; fewer than half of individuals with depression are correctly diagnosed. National rates of screening and 

treatment should be improved.

Medication management—Efforts should focus on preventing medication errors, particularly through greater use of 

computer technology. In addition, educational interventions that warn physicians and patients about problems 

associated with overuse of antibiotics have been successful. 

Nosocomial infections—Hospital-acquired infections kill nearly 90,000 patients in the United States each year, and 

cost an additional $5 billion to treat. Wider implementation of the nosocomial infection guidelines from the Centers for 

Disease Control and Prevention would save more than 40,000 lives annually, reduce infection rates by up to 50 

percent, and save nearly $2.75 billion each year. 

Obesity (emerging area)—Each year more than 300,000 deaths can be attributed to obesity. The condition eventually 

could become the nation's single most preventable cause of premature death and disability. Changes in social norms 

and national policies to promote physical activity and healthy diets are essential. Effective national strategies for 

obesity prevention, treatment, and control will require a combination of public health and clinical interventions. 

53
Pain control in advanced cancer—Twenty percent of Americans die from cancer, often after months of painful, 

progressive illness. Effective pain control programs have been developed. Efforts should emphasize cooperation in 

protocols across care settings, advance planning for changes in settings as well as heightened pain, and public 

education regarding the merits of opioid medications in this area. 

Pregnancy and childbirth—The quality of prenatal care and care related to labor and delivery should be enhanced to 

boost the long-term health of women and their children. Some key goals should be to increase the number of women 

who start prenatal care in the first trimester and to screen more pregnant women for sexually transmitted diseases. 

Self-management/health literacy (cross-cutting area)—Public and private entities should systematically provide 

educational programs and interventions that boost patients' skills and confidence in managing and assessing their 

health problems. Higher levels of health literacy allow people to understand and act on health care information. 

Severe and persistent mental illness—The quality of mental health care in the public sector, including state hospitals, 

community mental-health centers, and various federal and state programs should be improved. The federal 

government should play a larger role to assure higher standards of care across states. 

Stroke—Stroke is the third-leading cause of death in the United States. Efforts should focus on seamlessly integrating 

care across health care settings and clinical disciplines. Beginning rehabilitation as soon as possible after a stroke 

also helps patients regain their abilities. 

Tobacco-dependence treatment in adults—Tobacco use and dependence are the nation's most preventable causes 

of disease and death. Many successful efforts to improve health care in this area have used multilayered

interventions that include systems to remind caregivers to discuss tobacco use with patients, as well as providereducation programs centered on best practices. 

Adapted from National Academy of Medicine Press Release, January 7, 2003 “Officials Should Target 20 Key Areas 

to Transform Health Care System” http://www4.nas.edu/news.nsf/isbn/0309085438?OpenDocument; 

accessed April 11, 2003. 

54
Appendix B: Search Strategy Exemplar:

Quality Improvement and Hypertension 

55
This page intentionally left blank 

56
Appendix B: Search Strategy Exemplar:

Quality Improvement and Hypertension 

I. MEDLINE search 

Search Search String Citations‡

#1 Care

coord, 

Disease 

Mx, etc. 

Disease Management [mh] OR Patient Care Planning [mh] OR Patient-Centered Care

[mh] OR Primary Health Care [mh] OR Progressive Patient Care [mh] OR Critical 

Pathways [mh] OR Delivery of Health Care, Integrated [mh] OR Health Services 

Accessibility [mh] OR Managed Care Programs [mh] OR Product Line Management

[mh] OR Patient Care Team [mh] OR Patient-Centered Care [mh] OR Behavior Control

[mh] OR Counseling [mh] OR Health Promotion [mh] OR Patient Compliance [mh] OR

After-Hours Care [mh] OR ((coordination [ti] OR coordinated [ti] OR Multifactorial [ti] 

OR Multi-factorial [ti] OR Multicomponent [ti] OR Multi-component [ti] OR 

multidisciplinary [ti] OR multi-disciplinary [ti] OR interdisciplinary [ti] OR interdisciplinary [ti] OR integrated [ti] OR community-based [ti] OR organized [ti]) AND 

(care [ti] OR approach [ti] OR intervention [ti] OR strategy [ti] OR strategies [ti] OR 

management [ti] OR managing [ti] OR center* [ti] OR clinic*[ti])) OR Organization and

Administration [mh]

683,000 

#2 TQM, CQI Total Quality Management [mh] OR Quality control [mh] OR TQM [ti] OR CQI [ti] OR 

(quality [ti] AND (continuous [ti] OR total [ti]) AND (management [ti] OR

improvement [ti])) 

28,087

#3 CME,

educ

outreach 

Education, Continuing [mh] OR (Education [ti] AND Continuing [ti] AND (medical [ti] 

OR professional* [ti] OR nursing [ti] OR physician* [ti] OR nurse* [ti])) OR (outreach

[ti] AND (visit*[ti] OR educational [ti]) OR (academic [ti] AND detailing [ti])) 

35,276

#4 Diffusion of Innovation [mh] OR (Diffusion [ti] AND (Innovation [ti] OR technology

[ti])) 

4,889 

#5 Audit,

feedback

financial

incentive 

Medical audit [mh] OR ((Audit [ti] OR feedback [ti] OR compliance [ti] OR adherence 

[ti] OR training [ti]) AND (improvement* [ti] OR improving [ti] OR improves [ti] OR 

improve [ti] OR guideline* [ti] OR practice* [ti] OR medical [ti] OR provider* [ti] OR 

physician* [ti] OR nurse* [ti] OR clinician* [ti] OR practice guidelines [mh] OR 

academic [ti] OR visit* [ti])) OR Reminder Systems [mh] OR Reminder* [ti] OR 

((financial [ti] OR economic [ti] OR physician* [ti] OR patient*) AND incentive* [ti]) 

OR Reimbursement Mechanisms [mh] 

36,852

 

‡

 Numbers of citations reflect search results from July 8, 2003 

57
#6 IT,

telemed

Medical Informatics [mh] OR computer [ti] OR (decision [ti] AND support [ti]) OR 

Telemedicine[mh] OR Telemedicine [ti] OR telecommunication* [ti] OR Internet [mh]

OR web [ti] OR modem [ti] OR telephone* [ti] OR telephone [mh]

306,703 

#7 #1 OR #2 OR #3 OR #4 OR #5 OR #6 988,356 

#8 #7 AND (Hypertension [mh] OR Hypertension [ti] OR (blood [ti] AND pressure [ti])) 7,574 

#9 #8 AND (systematic review search string§ OR original research string**) 3,698

#10 #9 Limit to English 3,144 

#11 #10 Limit to Pub since 1980 2,942 

#12 Main

result 

#11 BUTNOT (editorial [pt] OR comment [pt] OR letter [pt]) 2,842

#13 Additional

yield of

journal 

search 

(#8 AND Journal Search String ††) BUTNOT (#9 OR editorial [pt] OR comment [pt] OR

letter [pt]) [Limited to English, 1980] 

220 

 

§

 ((meta-analysis [pt] OR meta-analysis [tw] OR metaanalysis [tw]) OR ((review [pt] OR guideline [pt] OR consensus [ti] OR 

guideline* [ti] OR literature [ti] OR overview [ti] OR review [ti] OR Decision Support Techniques [mh]) AND ((Cochrane [tw]

OR Medline [tw] OR CINAHL [tw] OR (National [tw] AND Library [tw])) OR (handsearch* [tw] OR search* [tw] OR 

searching [tw]) AND (hand [tw] OR manual [tw] OR electronic [tw] OR bibliographi* [tw] OR database* OR (Cochrane [tw] 

OR Medline [tw] OR CINAHL [tw] OR (National [tw] AND Library [tw]))))) OR ((synthesis [ti] OR overview [ti] OR review

[ti] OR survey [ti]) AND (systematic [ti] OR critical [ti] OR methodologic [ti] OR quantitative [ti] OR qualitative [ti] OR 

literature [ti] OR evidence [ti] OR evidence-based [ti]))) BUTNOT (case report [mh] OR case* [ti] OR report [ti] OR editorial

[pt] OR comment [pt] OR letter [pt]) Æ 38,865 MEDLINE records 

** Randomised [ti] OR Randomized [ti] OR Controlled [ti] OR intervention [ti] OR evaluation [ti] OR impact [ti] OR

effectiveness [ti] OR Evaluation [ti] OR Studies [ti] OR study [ti] Comparative [ti] OR Feasibility [ti] OR Program [ti] OR 

Design [ti] OR Clinical Trial [pt] OR Randomized Controlled Trial [pt] OR Epidemiologic Studies [mh] OR Evaluation Studies 

[mh] OR Comparative Study [mh] OR Feasibility Studies [mh] OR Intervention Studies [mh] OR Program Evaluation [mh] OR 

Epidemiologic Research Design [mh] —> 2,551,486 MEDLINE records 

†† N Engl J Med [ta] OR JAMA [ta] OR Ann Intern Med [ta] OR Am J Med [ta] OR Arch Intern Med [ta] OR J Gen Intern Med

[ta] OR BMJ [ta] OR Lancet [ta] OR CMAJ [ta] OR Clin Invest Med [ta] OR Arch Fam Med [ta] OR J Fam Pract [ta] OR Fam

Pract [ta] OR Ann Med [ta] OR Br J Gen Pract [ta] OR J Intern Med [ta] OR Med J Aust [ta] OR South Med J [ta] OR West J 

Med [ta] OR Aust N Z J Med [ta] OR Med Care [ta] OR Health Serv Res [ta] OR Inquiry [ta] OR Milbank Q [ta] OR Health Aff 

(Millwood) [ta] OR Health Care Financ Rev [ta] OR Med Care Res Rev [ta] OR eff clin pract [ta] OR eval health prof [ta] OR Jt

Comm J Qual Improv [ta] OR Qual Saf Health Care [ta] OR Int J Qual Health Care [mh] OR Qual Health Care [ta] OR Qual 

Health Res [ta] OR Rep Med Guidel Outcomes Res [ta] OR Am J Manag Care [ta] OR Am J Med Qual [ta] OR J Contin Educ 

Health Prof [ta] OR Prev Med [ta] OR Am J Prev Med [ta] OR Patient Educ Couns [ta] OR Ann Behav Med [ta] OR J Hum

Hypertens [ta] OR Hypertension [ta] OR Am J Hypertens [ta] OR Clin Exp Hypertens [ta] OR J Clin Hypertens [ta] OR J

Hypertens [ta] OR Am J Cardiol [ta] OR Am Heart J [ta] OR circulation [ta] OR J AM Coll Cardiol [ta] OR Can J Cardiol [ta]

OR Heart Lung [ta]

58
#14 Additional

yield of

author 

search 

(#8 AND author search‡‡) BUTNOT (#13 OR editorial [pt] OR comment [pt] OR letter 

[pt]) [Limited to English, 1980] 

29 

#14 3,070

references

total (322

of these 

were same

as in the 

Diabetes 

search) 

 

‡‡ (Berwick D [au] OR berlowitz d [au] OR davis d [au] OR kiefe c [au] OR wagner e [au] OR glasgow r [au] OR boddenheimer t 

[au] OR Hulscher M [au] OR grol r [au] OR grimshaw j [au] OR haynes b [au] OR haynes rb [au] OR sackett d [au] OR goldberg 

h [au] OR Hirsch I [au] OR nash d [au] OR roper w [au] OR weingarten s [au]) --> 6,401 

59
This page intentionally left blank 

60
Appendix C: Article Review Triage Forms 

(Exemplar: Diabetes) 

61
This page intentionally left blank 

62
Appendix C: Article Review Triage Forms 

(Exemplar: Diabetes) 

Note: Items in Italics are topic-specific 

Diabetes Triage Forms 

Stage 1: 

1. Does the article report or evaluate the results of an intervention (whether performed by the investigators or not)? 

o Yes 

o No <exclusionary answer> 

o Can't Tell 

 

2. Does the article involve quality improvement or a QI strategy?

o Yes - involves quality improvement or a QI strategy

o Yes - systematic review of evaluations of a QI strategy 

o No <exclusionary answer>

o Can't Tell 

Stage 2: 

1. Should this article proceed to article abstraction stage for this topic? 

o Yes - evaluates a QI strategy involving diabetes 

o No - focused on diabetes in pregnancy, Type I DM only, screening for/preventing diabetes, hospital care

only <exclusionary answer>

o No - but involves other EPC topic(s) <exclusionary answer>

o No - not an evaluation or not QI <exclusionary answer>

o Can't tell - need article 

o No - but useful background article <exclusionary answer>

 

2. What type of study design was used? 

o RCT or quasi-RCT

o Prospective clinical trial, CBA* or ITS ** 

o Cohort study; before-after or time series not meeting CBA* or ITS** definitions <exclusionary answer for 

this topic>

o Observational (e.g., cross-section, case-control) <exclusionary answer>

o Can't tell (need article) 

o Systematic review or meta-analysis 

o Economic or decision analysis, modeling <exclusionary answer>

o Non-research (commentary, review, news) <exclusionary answer>

o Qualitative research (e.g., focus groups) <exclusionary answer>

o Guideline or consensus statement <exclusionary answer>

 

63
This page intentionally left blank

64
Appendix D: Full Article Review Abstraction Forms 

Stages 3 and 4 (Exemplar: Diabetes) 

65
This page intentionally left blank 

66
Appendix D: Full Article Review Abstraction Forms 

Stages 3 and 4 (Exemplar: Diabetes) 

Note: Items in Italics are topic-specific 

Stage 3: 

1. Does this article merit abstraction at Level 3? 

o Yes 

o No – not QI or not an evaluation of a QI strategy <exclusionary answer>

o No – study design below Level 2 <exclusionary answer>

o No - excluded topic (focused only on pregnancy, Type I DM, or screening) <exclusionary answer>

o No – no eligible outcomes* <exclusionary answer>

 

*Eligible outcomes include measure of disease control, provider adherence, or patient adherence. Excluded are: 

measures of provider or patient understanding, satisfaction, self-efficacy; costs and resource use. Also excluded are 

articles reporting no outcomes specifically related to diabetes (e.g. smoking only). 

2. Does this article present data overlapping with another article? 

o Exclude this article as a duplicate publication (identify included citation being duplicated) <exclusionary

answer>

o Include this article, but obtain listed citation to help with abstraction (e.g., separate methods paper; identify 

required citation ) 

o No or N/A 

3. What category of study question is addressed by the article? 

o Can screening for or awareness of diabetes be improved? 

o Can provider treatment of diabetes be improved? (e.g., increased adherence to recommended care)? 

o Can patient glycemic control or diabetic complications be improved? 

o Can patient adherence, education or self management be improved? 

o Not sure or Other (describe) 

o N/A 

 

4. Describe the QI strategy used and its salient features. 

 

5. Did the QI strategy involve a provider reminder system* or facilitated relay of clinical data ** back to providers? 

o Chart based reminder system* for providers 

o Computer based reminder* or decision support for providers 

o Facilitated relay of clinical data to providers** 

o Not sure 

o No or N/A 

* Patient or provider encounter specific information, provided verbally, on paper or on a computer screen, which is 

intended to prompt provider to recall information (e.g., the last time the patient had a HbA1c checked and its value, the 

last time the patient underwent screening colonoscopy and the result). 

** Clinical information collected directly from patients and given to the provider using some format other than the 

conventional chart system. 

 

6. Did the QI strategy involve provider audit and feedback*? 

o Confidential feedback to individual provider 

o Non-confidential feedback to provider and colleagues in same clinic or institution 

o Public reporting of performance data 

o Benchmarking**

o Not sure or other 

o No or N/A 

 

*Any summary of clinical performance of health care over a specified period of time. E.g., the percentage of a 

provider's patients who have achieved or have not achieved some clinical target (e.g., BP or HbA1c in certain range), 

have or have not been offered some diagnostic test. 

**Benchmarking refers to the provision of performance data from institutions or providers regarded as "leaders in the 

field." These data provide targets for other providers and institutions to emulate. 

67
7. Did the QI strategy involve provider education? 

o Educational workshops, meetings (e.g., traditional CME), lectures (live or computer based) 

o Educational outreach visits (Use of a trained person who met with providers in their practice settings to give 

information with the intent of changing the provider's practice) 

o Distribution of educational materials (Distribution of published or printed recommendations for clinical care, 

including clinical practice guidelines, audio-visual materials and electronic publications) 

o Not sure or other 

o No or N/A 

8. Did the QI strategy involve patient education or promote self-management? 

o In-person patient education individually or as a part of a group or community 

o Distribution of printed or audio-visual educational materials 

o Patient reminders (e.g., to keep appointments or comply with other aspect of care) 

o Provision of clinical data back to the patient (e.g., your most recent HbA1c or lipid panel was such and such) 

o Distribution of materials or access to a resource that enhances patients' ability to manage their condition 

o Not sure or other 

o No or N/A 

9. Did the QI strategy involve organizational change? 

o Case management, disease management --coordination of assessment, treatment and arrangement for 

referrals by a person or multidisciplinary team in collaboration with or supplementary to the primary care

provider 

o Adding new members to team (e.g., adding a diabetes nurse, clinical pharmacist, or nutritionist to clinic) or 

creating multidisciplinary teams (creation of a new team of health professionals of different disciplines or 

additions of new members to the team who work together to care for patients) 

o Communication and case discussion between distant health professionals (e.g., telemedicine) 

o TQM/CQI - cycles of measurement of quality problems, design of interventions, implementation and remeasurement 

o Changes in medical records systems -- e.g. changing from paper to computerized records, patient tracking 

systems 

o Revision of professional roles ('professional substitution', 'boundary encroachment') - the shifting of roles

among health professionals (e.g., nurse midwives providing obstetrical care) 

o Increased staffing without changes in roles (e.g., adding more nurses) 

o Skill mix changes (changes in types or qualifications of personnel - e.g., changing from LVN to RN or RN to 

NP, but also changing from GP to specialist) 

o Not sure or other 

o No or N/A 

 

10. Did the QI strategy involve financial, regulatory or legislative incentives or actions? 

o Positive or negative financial incentives directed at providers 

o Positive or negative financial incentives directed at patients 

o System-wide changes in reimbursement (e.g., capitation, prospective payment, shift from fee for service to 

salary) 

o Changes to provider licensure requirements 

o Changes to institutional accreditation requirements 

o Not sure or other 

o No or N/A 

 

11. Did a clinical information system play a role in design or implementation of intervention (regardless of QI strategy

type)? 

o Identification and/or group allocation of eligible patients or providers 

o Reminders generated by existing clinical information system 

o Decision support at point of care (e.g., for provider order entry) 

o Facilitated communication between providers (e.g., generated emails between members of care team) 

o Audit data gathered from clinical information system to design QI strategy (e.g., audit and feedback, TQM, 

provider education, financial incentives) 

o Not sure or Other 

o No or N/A 

12. Who or what was targeted by the intervention? 

o Patients 

o Providers (i.e., individual clinicians) 

o Ambulatory clinics or practices 

68
o Inpatient units or hospitals 

o Public health systems, healthcare delivery systems, policy makers 

o Not sure or Other 

o N/A 

13. Among the target group, what was the number of participants? (i.e., study size) 

 

14. What type of study design was used? 

o RCT or quasi-RCT 

o Prospective clinical trial, CBA* or ITS** 

o Cohort study, retrospective before-after, or time series not meeting ITS definition** <exclusionary answer

for this topic>

o Not sure or other 

o N/A 

 

*Controlled Before After (CBA) requires contemporaneous observation periods for control and intervention groups 

AND judgment that control represents a comparable group or setting 

** Interrupted time series (ITS) requires statement of well-defined time period for intervention implementation AND at 

least three time points both before and after 

 15. What were the outcome types? 

o Measure of disease control (clinical outcomes, HbA1c, glucose control, lipids) 

o Provider adherence (adherence to a guideline or recommended practice) 

o Patient adherence 

o Patient or provider understanding, self-efficacy, empowerment <exclusionary answer for this topic> 

o Not sure or other 

o N/A 

 

16. What specific measures of disease control were used? 

o Serum glucose values (mean or percent of patients in certain range) HbA1c (mean or percent of patients in 

certain range) 

o Cardiovascular risk factor modification (hyperlipidemia, hypertension, smoking cessation) 

o Microvascular complications (retinopathy, neuropathy, microalbuminuria, foot ulcers) 

o Macrovascular complications (MI, stroke, renal failure, amputation) 

o Not sure or other 

o None or N/A 

17. For studies reporting measures of clinician adherence, what specific measures were used? 

o Adherence to guideline targets for assessment of glycemic control (e.g., measuring HbA1c at certain 

intervals) 

o Adherence to recommended screening practices for ophthalmologic complications (e.g., performance of or 

referral for dilated retinal exam) 

o Adherence to recommended screening practices for renal complications (e.g., checking urine microalbumin) 

 

o Adherence to recommended screening practices for neuropathy or foot complications (e.g., performance of 

or referral for foot examination) 

o Adherence to treatment choices for achieving glycemic control (e.g., medication choices) 

o Adherence to guideline targets for managing blood pressure or cardiovascular disease 

o Adherence to recommendations for patient education or counseling re: diet, exercise, smoking, or other 

lifestyle factors 

o Not sure or other 

o N/A 

 

18. For studies reporting measures of patient adherence, how was adherence assessed? 

o Laboratory confirmation (e.g., detection of drug or metabolite in blood or urine; including biochemical assays

for smoking cessation) 

o Pharmacy data (e.g., filled or refilled prescriptions) 

o Specially designed dispensers that record medication use 

o Home medication counts 

o Office medication counts (e.g., patients bring in bottles with unused pills) 

o Patient self report (via interview or survey) 

o Not sure or other 

o N/A 

69
Stage 4: 

1. Does abstraction of this study require information from methods or results reported in other citations. 

o Yes (specify) 

o No 

2. Does the article report data for more than one comparison (i.e., should it be abstracted as more than one study)? 

o Yes (specify which comparison is being abstracted here and which others will be abstracted elsewhere) 

o No 

A) Study Setting and Participants 

3. In what country did the study take place? 

o US only 

o Non-US (specify) 

4. Were the dates of the study period reported? 

o Yes – give dates exactly as indicated in paper 

o No – indicate duration of study in month or years if reported. 

5. In what setting did the study intervention take place?

o Primary care clinic 

o Specialist clinic (e.g. diabetes or endocrinology practice) 

o Community

o Multiple or Other (describe) 

o Not stated or not clear 

6. Were INCLUDED patients selected on the basis of any of the following? 

o Poor compliance with medications or clinic attendance (describe) 

o Poor glycemic control (describe) 

o Presence of specific comorbid conditions or illnesses (specify/describe – e.g., HTN, hyperlipidemia, 

coronary artery disease, obesity, tobacco use)

o Presence of specific diabetic complications (specify/describe – e.g., renal failure, albuminuria, neuropathy, 

retinopathy) 

o Other (explain)

o None of above

o Not applicable (no patient involvement in study – e.g., study of provider-based intervention and provider 

outcomes only). 

7. What type of care was provided to the control population? 

o No intervention or usual care

o Some form of low intensity intervention (describe) 

o No true control – just two or more different types of intervention (discuss with other reviewers; study may 

need to be excluded) 

B) Study Design 

8. What was the study design?

o Randomized trial – state method of randomization if described and any descriptive phrases (e.g., “randomly

assigned”) 

o Quasi randomly trial – state basis for treatment allocation (e.g., alternating patients, calendar date, even or 

odd identification numbers) 

o Controlled before-after study 

9. Did the study have a cross over design? (Patients randomized to a sequence of interventions such as treatment A 

followed by treatment B in one group B in one group and treatment B followed by treatment A in the other group). 

o Yes (describe) 

o No 

o Not sure – clarify with other review

10. What was the unit of randomization or treatment allocation? 

o Patient 

o Episode of care

o Clinic day 

o Provider 

o Practice 

o Firm (describe)

70
o Institution 

o Community

o Other 

11. For the unit of treatment allocation (above), state sample size in each group (If sample size differs for outcomes, 

detail differences in “Not stated or not clear” text box): 

o Control group 

o Intervention group 

o Not stated or not clear (explain) 

12. If unit of analysis differed from unit of treatment allocation (e.g., providers randomized, but patient outcomes 

analyzed, state sample size in each group: (Use text box for “Not applicable” if sample size for any outcomes 

reported is different-give details) 

o Control group 

o Intervention group 

o Not stated or not clear 

13. If unit of analysis differed from unit of treatment allocation, did authors acknowledge this issue and/or make 

appropriate adjustments? 

o Yes (describe)

o No 

o Not applicable (unit of analysis did not differ from unit of treatment allocation) 

14. Was the adequate concealment of treatment allocation?

o Yes – (unit of allocation was institution, team or professional and any random process explicitly described,

e.g., use of random number tables, OR unit of allocation was patient or episode of care and some form of 

centralized randomization scheme or sealed, opaque, serially numbered envelopes used) 

o Not clear (only partially meets above criteria) or not stated – specify which 

o No – inadequate concealment (enrollment of patient in alternation or through use of even/odd identifying

numbers OR unit of allocation was patient or episode of care and reported use of any allocation process that 

is entirely transparent before assignment (e.g., open list of random numbers) OR allocation was altered by

investigators, professionals or patients) 

15. Were patients blind to intervention/treatments allocation? 

o Yes 

o No 

o Not sure (explain) 

o Not applicable (patients not actively involved in study – e.g., provider-focused intervention with patient level 

data obtained retrospectively from charts) 

16. Were providers blind to intervention/treatment allocation?

o Yes 

o No 

o Not sure (explain) 

o Not applicable – (explain) 

17. Do any methodologic aspects of the study design not captured above seriously undermine appropriateness of 

inclusion? 

o Yes (explain) 

o No (use text box to document any non-fatal, but still noteworthy methodological features) 

C) Quality Improvement Attributes of Intervention 

18. Did the study involve PATIENT Education? 

o Yes (describe what was taught, where it occurred, duration and frequency of sessions) 

o No 

19. Did the intervention include access to a resource or provision of a device that promoted Patient SelfManagement? (excluding patient reminder systems) 

o Yes (describe)

o No 

20. Did the intervention involve a PATIENT REMINDER system? 

o Yes (specify target of reminder – appointments, compliance with meds or recommendations for self-care)

o No 

71
21. Did the intervention involve PROVIDER education? 

o Yes (describe nature of education, who administered the education, how often did it occur, etc) 

o No 

22. Did the intervention involve a PROVIDER REMINDER system? 

o Yes (describe content of reminders and how delivered) 

o No 

23. Did the intervention involve a Facilitated Relay of clinical information to providers? 

o Yes (describe type of information – e.g., recent glucose or HbA1c, and method of relaying information) 

o No 

24. Did the intervention involve provider AUDIT and FEEDBACK? 

o Yes (describe what was fed back, how often, etc) 

o No 

25. Did the intervention involve ORGANIZATIONAL Change (e.g., disease or case management, creation of 

multidisciplinary teams or expansion of professional roles, TQM/CQI, telemedicine, change in medical record 

system)? 

o Yes 

o No 

26. If the intervention involved Disease Management or Case Management, which of the following apply? 

o Intervention specifically described as involving “case management” or “disease management” 

o Someone other than physician actively participated in ongoing patient management using guidelines or 

systematic approach to care (protocols/algorithms to guide practitioner and patient decisions in specific 

clinical circumstances (specify type of person playing role of case manager) 

o Person or system actively tracked, scheduled and coordinated patients’ appointments 

o Other basis for describing intervention as disease/case management (describe) 

o Not applicable – no component of disease/case management 

27. Did intervention involve changes to make up of healthcare team or roles of providers? 

o Yes – Creation of multidisciplinary team, addition of new team member, expansion of roles, automatic 

referral for periodic visit with specific provider type (e.g., podiatrist or ophthalmologist) 

o Revision/expansion of roles or “shared care” (e.g., nurse or pharmacist operated actively managed 

medications without consulting physician) 

o Other (describe) 

o No changes to team/personnel

28. Did the intervention involve changes to medical records systems? 

o Change from paper to computerized records

o Implementation of computerized provider order entry (CPOE) 

o New patient tracking system 

o Other (describe) 

o Not applicable – No change to medical record system 

29. Did intervention involve any type of organizational change not captured by above questions? 

o Yes (describe)

o No 

 

30. Did a clinical information system play a role in design or implementation of intervention?

o Identification and/or group allocation of eligible patients or providers 

o Reminders generated by existing clinical information system

o Decision support at point of care (e.g., for provider order entry) 

o Facilitated communication between providers (e.g., generated emails between members of care team) 

o Audit data gathered from clinical information system to design QI strategy (e.g., audit and feedback, TQM, 

provider education, financial incentives) 

o Other 

o No role for a clinical information system 

D) Results 

31. For unit of treatment allocation (e.g., clinics, providers, patients), were results reported for at least 80% of 

participants? 

o Yes (state %) 

o No (state %) 

o Not stated 

72
32. If unit of analysis differed from unit of treatment allocation (e.g., providers randomized, but patient level outcomes 

analyzed), were results reported for at least 80% of participants? 

o Yes (state %) 

o No (state %) 

o Not stated or not clear 

o Not applicable (unit of analysis same as unit of treatment allocation) 

Measures of Disease Control 

33. Did the study report outcomes involving measures of disease control?

o Yes 

o No 

34. Did one measure of disease control involve HbA1c reported as mean and standard deviation in intervention and 

control groups? 

o Yes 

o No 

35. For the outcome of disease control involving mean HbA1c, provide the following information for patients in 

CONTROL group; indicate not reported by typing "NR”

o Mean HbA1c before intervention 

o Standard deviation for HbA1c before intervention 

o Mean HbA1c after intervention 

o Mean difference between pre- and post-intervention HbA1c values 

o Standard deviation for difference between pre- and post-intervention HbA1c values 

o Not applicable (no measure of HbA1c) 

36. For the outcome of disease control involving mean HbA1c, provide the following information for INTERVENTION 

group; indicate not reported by typing "NR”

o Mean HbA1c value before intervention 

o Standard deviation for HbA1c before intervention 

o Mean HbA1c value after intervention 

o Standard deviation for HbA1c after intervention 

o Mean difference between pre- and post-intervention HbA1c values 

o Standard deviation for difference between pre- and post-intervention HbA1c values 

o Not applicable (no measure of HbA1c) 

37. Did study report any measures of disease control involving HbA1c outcomes not captured above (e.g. median

HbA1c or % of patients with HbA1c in certain range)? 

o Yes (describe)

o No 

38. For articles reporting changes in SYSTOLIC BLOOD PRESSURE using mean and standard deviation, provide 

the following information for patients in CONTROL group (indicate not reported by typing NR) 

o pre-intervention SBP (state mean and standard deviation) 

o post-intervention SBP (state mean and standard deviation)

o difference between pre- and post-intervention values (state mean and SD)

o Not applicable - no disease control outcomes involving SBP as mean and SD

39. For articles reporting changes in SYSTOLIC BLOOD PRESSURE using mean and standard deviation, provide 

the following information for patients in INTERVENTION group (indicate not reported by typing NR) 

o pre-intervention SBP (state mean and standard deviation) 

o post-intervention SBP (state mean and standard deviation)

o difference between pre- and post-intervention values (state mean and SD)

o Not applicable - no disease control outcomes involving SBP as mean and SD

40. For articles reporting changes in DIASTOLIC BLOOD PRESSURE using mean and standard deviation, provide 

the following information for patients in the CONTROL group (indicate not reported by typing NR) 

o pre-intervention DBP (state mean and standard deviation) 

o post-intervention DBP (state mean and standard deviation)

o difference between pre- and post-intervention values (state mean and SD)

o Not applicable - no disease control outcomes involving DBP as mean and SD

73
41. For articles reporting changes in DIASTOLIC BLOOD PRESSURE using mean and standard deviation, provide 

the following information for patients in INTERVENTION group (indicate not reported by typing NR) 

o pre-intervention DBP (state mean and standard deviation) 

o post-intervention DBP (state mean and standard deviation)

o difference between pre- and post-intervention values (state mean and SD)

o Not applicable - no disease control outcomes involving DBP as mean and SD

42. Did study report any measures of disease control involving blood pressure outcomes not captured above (e.g. 

median SBP/DBP or % patients with BP in certain range)? 

o Yes (describe)

o No 

43. Indicate results for measures of disease control no captured above: 

o Serum blood glucose 

o Other CV risk factor (e.g. total cholesterol, HDL-C, LDL-C, triglyceride, lipid, smoking, weight) 

o Microalbuminuria or renal failure 

o Other microvascular complications (e.g. foot lesions, retinopathy, neuropathy) 

o Clinical outcomes (e.g. mortality, MI, stroke, amputation) 

o Other (explain)

o Not applicable - no other outcomes of disease control 

Measures of clinician adherence 

44. Did the study report outcomes related to clinician adherence?

o Yes 

o No – none reported or none in usable form (explain) 

ADHERENCE TO GUIDELINES FOR ASSESSING GLYCEMIC CONTROL USING HBA1C 

45. Did one of the outcomes of clinician adherence involve proportion of patient with HbA1c measure at least once 

during a certain time period?

o Yes 

o No – none reported or none in usable form (explain) 

46. For the adherence outcome involving measurement of HbA1c, indicate all that were reported or calculable for 

control group (All results should reflect % patients in designated group with HbA1c checked according to stated 

definition); indicate not reported by typing NR

o pre-intervention adherence (% patients) 

o post-intervention adherence (% patients) 

o difference between pre- and post-intervention values (% patients) 

o Not applicable - no adherence outcome involving measurement of HbA1c in this format 

47. For the adherence outcome involving measurement of HbA1c, indicate all that were reported or calculable for 

intervention group: 

o pre-intervention adherence (% patients) 

o post-intervention adherence (% patients) 

o difference between pre- and post-intervention values (% patients) 

o Not applicable - no adherence outcome involving measurement of HbA1c in this format 

48. Did study report any outcomes of clinician adherence involving checking HbA1c that are not captured above? 

o Yes (describe; give results) 

o No 

ADHERENCE TO OTHER GUIDELINES INVOLVING PERFORMANCE OF LABORATORY TESTS 

49. Did the article report outcomes for change in clinician adherence to a guideline for obtaining any lab 

measurements other than HbA1c? 

o Yes - specify definition (if more than one, report below for outcome with median effect attributable to 

intervention) 

o No - none reported or none in usable form

50. For the adherence outcome involving measurement of other lab values, indicate all that were reported or 

calculable for control group (All results should reflect % patients in designated group with other lab values

checked according to stated definition); indicate not reported by typing NR

o pre-intervention adherence (% patients) 

74

o post-intervention adherence (% patients) 
o difference between pre- and post-intervention values (% patients) 

o Not applicable - no adherence outcome involving measurement of other lab values in this format 

51. For the adherence outcome involving measurement of other lab values, indicate all that were reported or 

calculable for intervention group: 

o pre-intervention adherence (% patients) 

o post-intervention adherence (% patients) 

o difference between pre- and post-intervention values (% patients) 

o Not applicable - no adherence outcome involving measurement of other lab values in this format 

52. Were there any adherence outcomes for obtaining lab measurements not captured above? (If you had to choose 

outcome with median effect, use textbox for “Yes” answer to list the other adherence outcomes.) 

o Yes (list) 

o No 

ADHERENCE TO GUIDELINES FOR ASSESSMENT OR MANAGEMENT OF HYPERTENSION AND/OR 

CORONARY ARTERY DISEASE 

53. Did the article report outcomes for change in clinician adherence to a guideline for assessment or management of

HTN and/or CAD? 

o Yes - specify definition (if more than one, report below for outcome with median effect attributable to 

intervention) 

o No - none reported or none in usable form

54. For the adherence outcome involving assessment or management of HTN and/or CAD, indicate all that were 

reported or calculable for control group (All results should reflect % patients in designated group with stated 

guideline performed); indicate not reported by typing NR 

o pre-intervention adherence (% patients) 

o post-intervention adherence (% patients) 

o difference between pre- and post-intervention values (% patients) 

o Not applicable - no adherence outcome involving assessment or management of HTN and /or CAD in this

format 

55. For the adherence outcome involving assessment or management of HTN and/or CAD, indicate all that were 

reported or calculable for intervention group:

o pre-intervention adherence (% patients) 

o post-intervention adherence (% patients) 

o difference between pre- and post-intervention values (% patients) 

o Not applicable - no adherence outcome involving assessment or management of HTN and /or CAD in this

format 

56. Were there any adherence outcomes for assesment or management of HTN and/OR CAD not captured above?

(If you had to choose outcome with median effect, use textbox for “Yes” answer to list the other adherence 

outcomes.) 

o Yes (list) 

o No 

ADHERENCE TO GUIDELINES FOR ASSESSMENT OF DIABETIC COMPLICATIONS INVOLVING THE EYE OR 

FOOT 

57. Did the article report outcomes for change in clinician adherence to a guideline for referral for or performance of 

foot exam? 

o Yes - specify definition (if more than one, report below for outcome with median effect attributable to 

intervention) 

o No - none reported or none in usable form

58. For the adherence outcome involving referral for or performance of foot exam, indicate all that were reported or 

calculable for control group (All results should reflect % patients in designated group with feet checked according 

to stated definition); indicate not reported by typing NR 

o pre-intervention adherence (% patients) 

o post-intervention adherence (% patients) 

o difference between pre- and post-intervention values (% patients) 

o Not applicable - no adherence outcome involving referral for or performance of foot exam in this format 

75
59. For the adherence outcome involving referral for or performance of foot exam, indicate all that were reported or 

calculable for intervention group: 

o pre-intervention adherence (% patients) 

o post-intervention adherence (% patients) 

o difference between pre- and post-intervention values (% patients) 

o Not applicable - no adherence outcome involving referral for or performance of foot exam in this format 

60. Were there any adherence outcomes for referral for or performance of foot exam not captured above? (If you had 

to choose outcome with median effect, use textbox for “Yes” answer to list the other adherence outcomes.) 

o Yes (describe)

o No 

61. Did the article report outcomes for change in clinician adherence to a guideline for referral for or performance of 

eye exam? 

o Yes - specify definition (if more than one, report below for outcome with median effect attributable to 

intervention) 

o No - none reported or none in usable form

62. For the adherence outcome involving referral for or performance of eye exam, indicate all that were reported or 

calculable for control group (All results should reflect % patients in designated group with eyes checked according 

to stated definition); indicate not reported by typing NR 

o pre-intervention adherence (% patients) 

o post-intervention adherence (% patients) 

o difference between pre- and post-intervention values (% patients) 

o Not applicable - no adherence outcome involving referral for or performance of eye exam in this format 

63. For adherence outcome involving referral for or performance of eye exam, indicate all that were reported or 

calculable for intervention group: 

o pre-intervention adherence (% patients) 

o post-intervention adherence (% patients) 

o difference between pre- and post-intervention values (% patients) 

o Not applicable - no adherence outcome involving referral for or performance of eye exam in this format 

64. Were there any adherence outcomes for referral for or performance of eye exam not captured above? (If you had 

to choose outcome with median effect, use textbox for “Yes” answer to list the other adherence outcomes.) 

o Yes (describe)

o No 

ADHERENCE TO GUIDELINES FOR PATIENT COUNSELING OR DELIVERY OF PATIENT EDUCATION 

65. Did the article report outcomes for change in clinician adherence to a guideline for patient counseling or delivering 

of patient education?

o Yes - specify definition (if more than one, report below for outcome with median effect attributable to 

intervention) 

o No - none reported or none in usable form

66. For the adherence outcome involving patient education or counseling, indicate all that were reported or calculable 

for control group (All results should reflect % patients in designated group counseled or educated according to 

stated definition); indicate not reported by typing NR 

o pre-intervention adherence (% patients) 

o post-intervention adherence (% patients) 

o difference between pre- and post-intervention values (% patients) 

o Not applicable - no adherence outcome involving patient education or counseling in this format 

67. For the adherence outcome involving patient education or counseling, indicate all that were reported or calculable 

for intervention group: 

o pre-intervention adherence (% patients) 

o post-intervention adherence (% patients) 

o difference between pre- and post-intervention values (% patients) 

o Not applicable - no adherence outcome involving patient education or counseling in this format 

68. Were there any adherence outcomes for patient education or counseling not captured above? (If you had to 

choose outcome with median effect, use textbox for “Yes” answer to list the other adherence outcomes.) 

o Yes 

o No 

76
69. Did the article report outcomes for change in clinician adherence to any OTHER guideline?

o Yes (describe and give results) 

o No 

Patient compliance outcomes 

70. Describe results for any outcomes involving patient compliance 

o Compliance with self-care measures (e.g. self-monitoring of blood glucose), complying with diet or exercise, 

keeping appointments 

o Compliance with medications 

o Other (describe) 

o No patient compliance outcomes 

o Not sure (explain) 

71. Use textbox to state any important study features or results not captured above. 

72. Has a senior reviewer checked this Level 4 abstraction? 

o Yes - completely (indicate which senior reviewer) 

o Partially (indicate where re-review was left off, i.e. question #) 

o No (indicate any important questions/comments for senior reviewer) 

o Not applicable (first reviewer is a senior reviewer) 

77
References 

1. Priority Areas for National Action:

Transforming Health Care Quality. 

Washington, DC: National Academy 

Press; 2003. 

2. National Healthcare Quality Report: 

Update. Rockville, MD: Agency for 

Healthcare Research and Quality; 2002 

June. Report No.: 02-P028. 

3. National Healthcare Disparities Report. 

Background. 2002 November; 

Available from: 

http://www.ahrq.gov/qual/nhdr02/nhdr

prelim.htm. Accessed 2003. 

4. Institute of Medicine. Crossing the 

Quality Chasm: A New Health System

for the 21st Century. Washington, DC: 

National Academy Press; 2001. 

5. Mallon B, Mallon W. Ernest Amory 

Codman: The End Result of a Life in 

Medicine. Philadelphia: WB Saunders; 

2000. 

6. Dickersin K, Rennie D. Registering 

clinical trials. JAMA 2003;290(4):516-

23. 

7. Chassin MR, Galvin RW. The urgent 

need to improve health care quality. 

Institute of Medicine National 

Roundtable on Health Care Quality. 

JAMA 1998;280(11):1000-5. 

8. Kohn L, Corrigan J, Donaldson M, 

Committee on Quality of Health Care

in America IoM, editors. To Err Is 

Human: Building a Safer Health 

System. Washington, D.C.: National 

Academy Press; 1999. 

9. Wennberg J, Gittelsohn A. Variations

in medical care among small areas. Sci 

Am 1982;246(4):120-34. 

10. Wennberg J, Gittelsohn A. Small area 

variations in health care delivery. 

Science 1973;182(117):1102-8. 

11. Merrick NJ, Brook RH, Fink A, 

Solomon DH. Use of carotid 

endarterectomy in five California 

Veterans Administration medical 

centers. JAMA 1986;256(18):2531-5. 

12. Brook RH, Lohr KN. Efficacy, 

effectiveness, variations, and quality.

Boundary-crossing research. Med Care 

1985;23(5):710-22. 

13. Roper WL, Winkenwerder W,

Hackbarth GM, Krakauer H. 

Effectiveness in health care. An 

initiative to evaluate and improve 

medical practice. N Engl J Med 

1988;319(18):1197-202. 

14. Krumholz HM, Radford MJ, Wang Y, 

Chen J, Heiat A, Marciniak TA. 

National use and effectiveness of betablockers for the treatment of elderly 

patients after acute myocardial 

infarction: National Cooperative 

Cardiovascular Project. JAMA 

1998;280(7):623-9. 

15. Wennberg DE, Lucas FL, Birkmeyer 

JD, Bredenberg CE, Fisher ES. 

Variation in carotid endarterectomy 

mortality in the Medicare population: 

trial hospitals, volume, and patient 

characteristics. JAMA 

1998;279(16):1278-81. 

16. Katzan IL, Furlan AJ, Lloyd LE, Frank 

JI, Harper DL, Hinchey JA, et al. Use 

of tissue-type plasminogen activator 

for acute ischemic stroke: the 

Cleveland area experience. JAMA 

2000;283(9):1151-8. 

17. Flum DR, Morris A, Koepsell T, 

Dellinger EP. Has misdiagnosis of 

appendicitis decreased over time? A 

population-based analysis. JAMA 

2001;286(14):1748-53. 

18. Decline in deaths from heart disease 

and stroke─United States, 1900-1999. 

41
MMWR Morb Mortal Wkly Rep 

1999;48(30):649-56. 

19. Schuster MA, McGlynn EA, Brook 

RH. How good is the quality of health

care in the United States? Milbank Q 

1998;76(4):517-63, 509. 

20. Jencks SF, Cuerdon T, Burwen DR, 

Fleming B, Houck PM, Kussmaul AE, 

et al. Quality of medical care delivered 

to Medicare beneficiaries: A profile at 

state and national levels. JAMA 

2000;284(13):1670-6. 

21. Wenger N, Solomon D, Roth C, 

MacLean C, Saliba D, Kamberg C, et 

al. The quality of medical care 

provided to vulnerable communitydwelling older patients. Ann Intern 

Med 2003;139(9):740-747. 

22. McGlynn EA, Asch SM, Adams J, 

Keesey J, Hicks J, DeCristofaro A, et 

al. The quality of health care delivered 

to adults in the United States. N Engl J 

Med 2003;348(26):2635-45. 

23. Leape LL. Translating medical science 

into medical practice. Do we need a 

national medical standards board?

JAMA 1995;273(19):1534-7. 

24. Davis DA, Taylor-Vaisey A. 

Translating guidelines into practice. A 

systematic review of theoretic 

concepts, practical experience and 

research evidence in the adoption of 

clinical practice guidelines. CMAJ 

1997;157(4):408-16. 

25. Bero LA, Grilli R, Grimshaw JM, 

Harvey E, Oxman AD, Thomson MA. 

Closing the gap between research and 

practice: an overview of systematic 

reviews of interventions to promote the 

implementation of research findings. 

The Cochrane Effective Practice and

Organization of Care Review Group. 

BMJ 1998;317(7156):465-8. 

26. Grol R, Grimshaw J. Evidence-based 

implementation of evidence-based 

medicine. Jt Comm J Qual Improv 

1999;25(10):503-13. 

27. Ayanian JZ, Hauptman PJ, Guadagnoli 

E, Antman EM, Pashos CL, McNeil 

BJ. Knowledge and practices of 

generalist and specialist physicians 

regarding drug therapy for acute 

myocardial infarction. N Engl J Med 

1994;331(17):1136-42. 

28. Donohoe MT. Comparing generalist 

and specialty care: discrepancies, 

deficiencies, and excesses. Arch Intern 

Med 1998;158(15):1596-608. 

29. Breuer T, Goodman KJ, Malaty HM, 

Sudhop T, Graham DY. How do 

clinicians practicing in the U.S. 

manage Helicobacter pylori- related 

gastrointestinal diseases? A 

comparison of primary care and 

specialist physicians. Am J 

Gastroenterol 1998;93(4):553-61. 

30. Chin MH, Friedmann PD, Cassel CK, 

Lang RM. Differences in generalist and 

specialist physicians' knowledge and 

use of angiotensin-converting enzyme 

inhibitors for congestive heart failure. J 

Gen Intern Med 1997;12(9):523-30. 

31. van den Berghe G, Wouters P, 

Weekers F, Verwaest C, Bruyninckx F, 

Schetz M, et al. Intensive insulin 

therapy in the critically ill patients. N 

Engl J Med 2001;345(19):1359-67. 

32. Greco PJ, Eisenberg JM. Changing 

physicians' practices. N Engl J Med 

1993;329(17):1271-3. 

33. 20 Tips to Help Prevent Medical 

Errors. Patient Fact Sheet [online]. 

AHRQ Publication No. 00-P038. 2000 

February [cited 2003 November 30]; 

Available from: 

http://www.ahrq.gov/consumer/20tips.

htm.

34. Mehrotra A, Bodenheimer T, Dudley 

RA. Employers' efforts to measure and 

improve hospital quality: determinants 

42
of success. Health Aff (Millwood) 

2003;22(2):60-71. 

35. Krumholz HM, Rathore SS, Chen J, 

Wang Y, Radford MJ. Evaluation of a 

consumer-oriented internet health care 

report card: the risk of quality ratings 

based on mortality data. JAMA 

2002;287(10):1277-87. 

36. Gandhi TK, Francis EC, Puopolo AL, 

Burstin HR, Haas JS, Brennan TA. 

Inconsistent report cards: assessing the 

comparability of various measures of

the quality of ambulatory care. Med 

Care 2002;40(2):155-65. 

37. Poses RM, McClish DK, Smith WR, 

Huber EC, Clemo FL, Schmitt BP, et 

al. Results of report cards for patients 

with congestive heart failure depend on 

the method used to adjust for severity. 

Ann Intern Med 2000;133(1):10-20. 

38. Hofer TP, Hayward RA, Greenfield S,

Wagner EH, Kaplan SH, Manning 

WG. The unreliability of individual 

physician "report cards" for assessing 

the costs and quality of care of a 

chronic disease. JAMA 

1999;281(22):2098-105. 

39. Deming W. Out of the Crisis: Quality, 

Productivity and Competitive Position. 

New York: Cambridge University 

Press; 1982. 

40. Deming W. Sample Design in Business 

Research (Wiley Classics Library).

Hoboken, NJ: John Wiley & Sons Inc; 

1990. 

41. Juran J. Quality Control Handbook. 

New York: McGraw-Hill; 1951. 

42. Managerial Breakthrough: The Classic 

Book on Improving Management 

Performance/30th Anniversary Edition. 

revised ed. New York: McGraw-Hill; 

1995. 

43. Quality Planning and Analysis: From

Product Development Through Use. 

4th ed. New York: McGraw-Hill; 1993. 

44. Blumenthal D, Kilo CM. A report card 

on continuous quality improvement. 

Milbank Q 1998;76(4):625-48, 511. 

45. Gerowitz MB. Do TQM interventions 

change management culture? Findings 

and implications. Qual Manag Health 

Care 1998;6(3):1-11. 

46. Shortell SM, Bennett CL, Byck GR. 

Assessing the impact of continuous 

quality improvement on clinical 

practice: what it will take to accelerate 

progress. Milbank Q 1998;76(4):593-

624, 510. 

47. Lohr KN, Schroeder SA. A strategy for 

quality assurance in Medicare. N Engl 

J Med 1990;322(10):707-12. 

48. Oliveria SA, Lapuerta P, McCarthy 

BD, L'Italien GJ, Berlowitz DR, Asch 

SM. Physician-related barriers to the 

effective management of uncontrolled 

hypertension. Arch Intern Med 

2002;162(4):413-20. 

49. Roger VL, Jacobsen SJ, Weston SA, 

Goraya TY, Killian J, Reeder GS, et al. 

Trends in the incidence and survival of 

patients with hospitalized myocardial 

infarction, Olmsted County, 

Minnesota, 1979 to 1994. Ann Intern 

Med 2002;136(5):341-8. 

50. Ellis P, Robinson P, Ciliska D, Armour 

T, Raina P, Brouwers M, et al. 

Diffusion and Dissemination of 

Evidence-based Cancer Control 

Interventions. Evidence 

Report/Technology Assessment No. 79 

(Prepared by the McMaster University 

Evidence-based Practice Center under 

contract no. 290-97-0017). AHRQ 

Publication No. 03-E032. Rockville, 

MD: Agency for Healthcare Research

and Quality; 2003 May. 

51. Grimshaw JM, Shirran L, Thomas R, 

Mowatt G, Fraser C, Bero L, et al. 

Changing provider behavior: an 

overview of systematic reviews of 

43
interventions. Med Care 2001;39(8 

Suppl 2):II2-45. 

52. Stone EG, Morton SC, Hulscher ME, 

Maglione MA, Roth EA, Grimshaw 

JM, et al. Interventions that increase 

use of adult immunization and cancer 

screening services: a meta-analysis. 

Ann Intern Med 2002;136(9):641-51. 

53. Stafford RS. Alternative strategies for 

controlling rising cesarean section 

rates. JAMA 1990;263(5):683-7. 

54. Weingarten SR, Henning JM, 

Badamgarav E, Knight K, Hasselblad 

V, Gano A, Jr., et al. Interventions used 

in disease management programmes 

for patients with chronic

illness⎯which ones work? Metaanalysis of published reports. BMJ 

2002;325(7370):925. 

55. Thomson O'Brien MA, Oxman AD, 

Davis DA, Haynes RB, Freemantle N, 

Harvey EL. Educational outreach 

visits: effects on professional practice 

and health care outcomes. Cochrane 

Database Syst Rev 2000(2):CD000409. 

56. Thomson O'Brien MA, Oxman AD, 

Davis DA, Haynes RB, Freemantle N, 

Harvey EL. Audit and feedback versus 

alternative strategies: effects on 

professional practice and health care 

outcomes. Cochrane Database Syst Rev 

2000(2):CD000260. 

57. Thomson O'Brien MA, Oxman AD, 

Davis DA, Haynes RB, Freemantle N, 

Harvey EL. Audit and feedback: 

effects on professional practice and 

health care outcomes. Cochrane 

Database Syst Rev 2000(2):CD000259. 

58. Thomson O'Brien MA, Oxman AD, 

Haynes RB, Davis DA, Freemantle N, 

Harvey EL. Local opinion leaders: 

effects on professional practice and 

health care outcomes. Cochrane 

Database Syst Rev 2000(2):CD000125. 

59. Thomson O'Brien MA, Freemantle N, 

Oxman AD, Wolf F, Davis DA, Herrin 

J. Continuing education meetings and 

workshops: effects on professional 

practice and health care outcomes. 

Cochrane Database Syst Rev 

2001(2):CD003030. 

60. Alderson P, Bero L, Grilli R, 

Grimshaw J, McAuley L, Oxman A, et 

al., editors. Effective Practice and 

Organisation of Care Group. Oxford: 

Update Software; 2003. 

61. Grimshaw JM, Thomas RE, 

MacLennan G, Fraser C, Ramsay CR, 

Vale L, et al. Effectiveness and 

efficiency of guideline dissemination 

and implementation strategies. Health 

Technol Assess 2004 Feb;8(6):iii-iv, 1-

72. 

62. Kerry SM, Bland JM. Sample size in 

cluster randomisation. BMJ 

1998;316(7130):549. 

63. Kerry SM, Bland JM. The intracluster 

correlation coefficient in cluster 

randomisation. BMJ 

1998;316(7142):1455. 

64. Campbell MK, Mollison J, Grimshaw 

JM. Cluster trials in implementation 

research: estimation of intracluster 

correlation coefficients and sample 

size. Stat Med 2001;20(3):391-9. 

65. Torgerson DJ. Contamination in trials: 

is cluster randomisation the answer? 

BMJ 2001;322(7282):355-7. 

66. Donner A, Klar N. Issues in the metaanalysis of cluster randomized trials. 

Stat Med 2002;21(19):2971-80. 

67. Donner A, Piaggio G, Villar J. 

Statistical methods for the metaanalysis of cluster randomization trials. 

Stat Methods Med Res 

2001;10(5):325-38. 

68. Donner A, Piaggio G, Villar J. Metaanalyses of cluster randomization trials. 

Power considerations. Eval Health Prof 

2003;26(3):340-51. 

44
69. Localio AR, Berlin JA, Ten Have TR, 

Kimmel SE. Adjustments for center in 

multicenter studies: an overview. Ann 

Intern Med 2001;135(2):112-23. 

70. Frijling BD, Lobo CM, Hulscher ME, 

Akkermans RP, Braspenning JC, Prins 

A, et al. Multifaceted support to 

improve clinical decision making in 

diabetes care: a randomized controlled 

trial in general practice. Diabet Med 

2002;19(10):836-42. 

71. Kinmonth AL, Woodcock A, Griffin S, 

Spiegal N, Campbell MJ. Randomised 

controlled trial of patient centered care 

of diabetes in general practice: impact 

on current wellbeing and future disease 

risk. The Diabetes Care From

Diagnosis Research Team. BMJ 

1998;317(7167):1202-8. 

72. Meigs JB, Cagliero E, Dubey A, 

Murphy-Sheehy P, Gildesgame C, 

Chueh H, et al. A controlled trial of 

web-based diabetes disease 

management: the MGH diabetes 

primary care improvement project. 

Diabetes Care 2003;26(3):750-7. 

73. Foy R, Eccles M, Grimshaw J. Why 

does primary care need more 

implementation research? Fam Pract 

2001;18(4):353-5. 

74. Grol R, Grimshaw J. From best 

evidence to best practice: effective 

implementation of change in patients'

care. Lancet 2003;362:1225-30. 

75. Cabana MD, Rand CS, Powe NR, Wu 

AW, Wilson MH, Abboud PA, et al. 

Why don't physicians follow clinical 

practice guidelines? A framework for 

improvement.[comment]. JAMA 

1999;282(15):1458-65. 

76. Davies P, Walker A, Grimshaw J. 

Theories of behaviour change in 

studies of guideline implementation. 

Proceedings of the British

Psychological Society 2003;11:120. 

77. Logan J, Graham ID. The Ottawa 

Model of Research Use: revisions, 

reflections, prospects (unpublished). 

2003. 

78. Tiffany CR, Johnson Lutjens L. 

Planned change theories for nursing. 

Review, analysis and implications. 

Thousand Oaks, CA: Sage; 1997. 

79. Meleis A. Theoretical nursing. 

Development and progress. 3rd ed. 

New York: Lippincott; 1997. 

80. Merton R. Social theory and social 

structure. New York: Free Press; 1968. 

81. Fawcett J. Analysis and evaluation of 

contemporary nursing knowledge: 

Nursing models and theories. 

Philadelphia: FA Davis; 2000. 

82. Smith MJ, Liehre PR. Introduction: 

middle range theory and the ladder of 

abstraction. In: Smith MJ, Liehre PR, 

editors. Middle range theory for 

nursing. New York: Springer; 2003. p. 

1-24. 

83. Bacharach SB. Organizational theories: 

some criteria for evaluation. The 

Academy of Management Review 

1989;14(4):496-515. 

84. Rogers E. Diffusion of innovations. 3rd 

ed. New York: Free Press; 1983. 

85. Rogers E. Diffusion of innovations. 4th 

ed. New York: Free Press; 1995. 

86. Lomas J. Retailing research: increasing 

the role of evidence in clinical services 

for childbirth. Milbank Q 

1993;71(3):439-475. 

87. Lomas J. Teaching old (and not so old) 

docs new tricks: effective ways to 

implement research findings. In: Dunn 

E, Norton P, Stewart M, Tudiver F, 

Bass M, editors. Disseminating 

research/changing practice. Thousand 

Oaks, CA: Sage; 1994. p. 1-18. 

88. Lomas J. Medicine in context: a 

neglected perspective in medical 

45
education. Acad Med 1994;69(10 

Suppl):S95-101. 

89. Tiffany CR. Analysis of planned 

change theories. Nurs Manage 

1994;25:60-2. 

90. Tiffany CR, Cheatham AB, Doornbos 

D, Loudermelt L, Momadi GG. 

Planned change theory: survey of 

nursing periodical literature. Nurs 

Manage 1994;25(7):54-9. 

91. Green L, Kreuter M, Deeds S, Patridge, 

K. Health education planning: a 

diagnostic approach. Mountain View, 

CA: Mayfield Publishing; 1980. 

92. Green L, Kreuter M. Health promotion 

planning: an educational and 

environmental approach. Mountain 

View, CA: Mayfield Publishing; 1991. 

93. Berwick DM. Disseminating 

innovations in health care.[comment]. 

JAMA 2003;289(15):1969-75. 

94. Logan J, Graham ID. Toward a 

comprehensive interdisciplinary model 

of health care research use. Sci 

Commun 1998;20(2):227-46. 

95. Kotler P. Social marketing of health 

behavior. In: Frederiksen L, Solomon 

L, Brehony K, editors. Marketing 

health behavior: principles, techniques 

and applications. New York: Plenum

Press; 1983. 

96. Soumerai SB, Avorn J. Principles of

educational outreach ('academic 

detailing') to improve clinical decision 

making. JAMA 1990;263(4):549-56. 

97. Schroeder R, Van de Ven A, Scudder 

G, Rolley D. Managing innovation and 

change processes: findings form the 

Minnesota Innovation Research 

Program. Agribusiness Manage 

1986;2:501-523. 

98. Graham K, Logan J. Using the Ottawa 

Model of Research Use to implement a 

skin care program. 2003. 

99. Donabedian A. The quality of care. 

How can it be assessed? JAMA 

1988;260(12):1743-8. 

100. Ferlie EB, Shortell SM. Improving the 

quality of health care in the United 

Kingdom and the United States: a 

framework for change. Milbank Q 

2001;79(2):281-315. 

101. Mittman BS, Tonesk X, Jacobson PD. 

Implementing clinical practice 

guidelines: social influence strategies 

and practitioner behavior change. QRB 

1992;18(12):413-22. 

102. Bandura A. Self-efficacy: toward a 

unifying theory of behavioral 

change.[comment]. Psychol Rev 

1977;84(2):191-215. 

103. Ajzen I. The theory of planned 

behaviour. Organizational Behaviour 

and Human Decision Process 

1991;50:179-211. 

104. Wallston K. Hocus-pocus, the focus 

isn't strictly on locus: Rotter's social 

learning theory modified for health. 

Cognit Ther Res 1992;16:183-99. 

105. Conner M, Norman P. Health 

behaviour. In: Johnston D, Johnston M, 

editors. Comprehensive Clinical

Psychology, vol. 8: Health Psychology. 

Oxford: Pergamon; 1998. p.1-38. 

106. Godin G, Kok G. The theory of 

planned behavior: a review of its 

applications to health-related 

behaviors. Am J Health Promot 

1996;11(2):87-98. 

107. Godin G, Naccache H, Morel S, 

Ebacher MF. Determinants of nurses'

adherence to universal precautions for 

venipunctures. AJIC 2000;28(5):359-

64. 

108. Walker A, Grimshaw J, Armstrong E. 

Salient beliefs and intentions to 

prescribe antibiotics for patients with a 

sore throat. Br J Health Psychol 

2001;6:347-60. 

46
109. Blackman D. Operant conditioning: an

experimental analysis of behaviour. 

London: Methuen; 1974. 

110. Gollwitzer P. Goal achievement: the 

role of intentions. In: Stroebe W,

Hewstone M, editors. European review 

of social psychology. Chichester, UK: 

Wiley; 1993. p. 141-85. 

111. Orbell S, Hodgkins S, Sheeran P. 

Implementation intentions and the 

theory of planned behaviour. Pers Soc 

Psychol Bull 1997;23:945-54. 

112. Mazmanian PE, Daffron SR, Johnson 

RE, Davis DA, Kantrowitz MP. 

Information about barriers to planned 

change: a randomized controlled trial 

involving continuing medical education 

lectures and commitment to change. 

Acad Med 1998;73(8):882-6. 

113. Prochaska JO, DiClemente CC. Stages 

and processes of self-change of 

smoking: toward an integrative model 

of change. J Consul Clin Psychol 

1983;51(3):390-5. 

114. Prochaska JO, DiClemente CC, Velicer 

WF, Rossi JS. Standardized, 

individualized, interactive, and 

personalized self-help programs for 

smoking cessation. Health Psychol 

1993;12(5):399-405. 

115. Reimsma R, Pattenden J, Bridle C, 

Sowden A, Mather L, Walker A. A 

systematic review of the effectiveness 

of interventions based on a stages to 

change approach to promote individual 

behaviour change. Health Technol 

Assess 2002;6(24). 

116. Scott WR. Organizations: rational, 

natural and open systems. 4th ed. 

Upper Saddle River, NJ: Prentice Hill; 

1998. 

117. Shortell SM, Kaluzny AD. 

Organization theory and health care 

management. In: Shortell SM, Kaluzny 

AD, editors. Health care management: 

a text in organization theory and 

behavior. New York: John Wiley and 

Sons; 1988. 

118. Shortell SM, Kaluzny AD. Health care 

management: organization, design, and 

behavior. 3rd ed. Albany, NY: Delmar 

Publishers Inc.; 1994. 

119. Kaluzny AD, Hernandez SR. 

Organizational change and innovation. 

In: Shortell SM, Kaluzny AD, editors. 

Health care management: a text in 

organization theory and behavior. 2nd 

ed. New York: John Wiley and Sons; 

1988. 

120. Shortell SM, Kaluzny AD. Health care 

management: a text in organization 

theory and behavior. 2nd ed. New 

York: John Wiley and Sons; 1988. 

121. Westphal JD, Gulati R, Shortell SM. 

Customization or conformity? An 

institutional and network perspective 

on the content and consequences of 

TQM adoption. Adm Sci Q 

1997;42(2):366-94. 

122. Iles V, Sutherland K. Managing change 

in the NHS. Organisational change: a 

review for health care managers, 

professionals, and researchers. London: 

NCCSDO; 2001. 

123. Shortell SM, Jones RH, Rademaker 

AW, Gillies RR, Dranove DS, Hughes 

EF, et al. Assessing the impact of total 

quality management and organizational 

culture on multiple outcomes of care 

for coronary artery bypass graft surgery 

patients. Med Care 2000;38(2):207-17. 

124. Fox R, Mazmanian P, Putnam R. 

Changing and learning in the lives of 

physicians. New York: Praeger; 1989. 

125. Grol R. Personal paper. Beliefs and 

evidence in changing clinical practice. 

BMJ 1997;315(7105):418-21. 

126. Kotler P, Roberto E. Social marketing: 

Strategies for changing public 

behavior. New York: Free Press; 1989. 

47
127. Scanlon DP, Chernew M, Doty HE. 

Evaluating the impact of value-based 

purchasing: a guide for purchasers. 

Rockville, MD: Agency for Healthcare 

Research and Quality; 2002 May. 

AHRQ Publication No. 02-002

48