Department of Health and Human Services 

OFFICE OF


INSPECTOR GENERAL


CLINICAL PRACTICE GUIDELINES


SPONSORED BY THE


AGENCY FOR HEALTH CARE POLICY


EARLY


AND RESEARCH 

EXPERIENCESINCLINICALSETTINGS


‘eg JUNE GIBBS BROW 

< 

2 Inspector General 

‘6 

%++

> 

$ 

%’d~~ September 1995

OEI-01-94-00250 
TABLE OF CONTENTS


PAGE 

EXECUTIVE SUMMARY 

INTRODU(XION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1


FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...0. . . . . . . 5


The Extent ofGuideline Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5


Ways in Which the Guidelines Have Been Used . . . . . . . . . . . . . . . . . . . . . . . . 7


Organizations’ Assessments oftheir Guideline Use . . . . . . . . . . . . . . . . . . . . .


Obstacles to Guideline Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


RECOMMENDATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


COMMENTS ON THE DRAFW REPORT . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


APPENDICES


10 

11 

13 

15 

Z& 

B:


D:


Nonrespondent Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..A-l


Summary of Mail Survey Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..B-1


Complete Comments onthe Draft Report . . . . . . . . . . . . . . . . . . . . . . . . . . C-1


Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..D-1


c 
EXECUTIVE SUMMARY


PURPOSE 

This study describes the early experiences of healthcare organizations in using clinical 

practice guidelines sponsored by the Agency for Health Care Policy and Research. 

BACKGROUND 

Over 70 organizations, including Federal agencies, medical professional associations, 

health insurance associations, and health maintenance organizations, have sponsored 

approximately 1800 clinical practice guidelines. The Agency for Health Care Policy 

and Research (AHCPR) has released sixteen guidelines. Of the $154 million 

appropriated for AHCPR’S fiscal year 1994 programs, $5 million was spent on 

developing and updating guidelines, and $10 million on dissemination and evaluation. 

Of this, approximately $4 million was spent on printing and disseminating guidelines. 

The introduction of new technologies, including new clinical practice guidelines, can be 

a slow and difficult process. We prepared this report in response to AHCPR’S interest 

in gaining a better understanding of the extent to which, and the manner in which, 

healthcare organizations have been using its guidelines. 

We conducted a mail survey of 380 key U.S. healthcare organizations, including 150 

nursing homes; 150 small, nonteaching hospitals; and 80 health maintenance 

organizations that have staff-model components. We focused on these organizations 

because they offer prime opportunities for use of the AHCPR guidelines. Sixty-four 

percent responded. Our survey focused on the first six AHCPR guidelines released 

between March 1992 and April 1993: pain management, urinary incontinence, 

pressure ulcer prevention, cataracts, depression, and sickle cell disease. 

FINDINGS 

llventy pement of the survey respondents reprted that they have wed I or more of the 6 

AHCPR guidelines about which we inquii~ an addtionul 12 punt reprted that they 

plan to do so. 

�	 Impetus for using the guidelines has frequently come from the organizations’ internal, 

quality-improvement programs. External factors--such as directives from supervising 

organizations and State law requirements--have also played an important role. 

�	 The organizations’ expectations of guideline use have focused on improved 

clinical care. Many organizations have also viewed the guidelines as a means of 

reducing costs and providing better protection from malpractice litigation. 

i 
.	 The reasons most often cited for not having used the guidelines were that the 

organizations were not familiar with them (40 percent of all survey 

respondents) and the organizations have their own guidelines (34 percent of all 

sumey respondents). No organizations cited disagreement with the guidelines 

as a basis for not having used them. 

Ninety-sir pement of the su.rwy resyxmdknfi that have used 1 or more of the 6 AHCPR 

guidiliizes reprted that their #Torts to do so have focused on clinicians; 36percent 

reprted that they huve directed guideline-irnplementzition #orts towanik patients. 

Clinician-Directed Efforts 

.	 Clinician-education efforts--ranging from the distribution of guideline materials 

to the conduct of educational programs--were the most frequently reported 

means of implementing the guidelines. Such efforts were reported by most 

guideline users. 

.	 The development of an algorithm, policy and procedure, or other 

implementation tool to guide clinician practice was also reported by most 

guideline users. 

.	 Economic incentives to encourage clinicians to provide guideline-recommended 

care were reported by very few organizations. 

Patient-Directed Efforts 

.	 The distribution of guideline materials to patients was the most frequently 

reported patient-directed effort. 

� Educational sessions for patients were reported by some organizations. 

At this ear&point in the irnpkmentation process, only 8 percent of the respondents that 

have used 1 or more of the 6 AHCPR guidelines reported that they have measured the 

efects of their guideline use. 

.	 These measures have focused on patients’ clinical outcomes and clinicians’ 

performance. They have been based, primarily, on chart and record reviews. 

Sixty-three percent of the respoti that reptied huving used 1 or more of the 6 

AHCPR guidklinix also reprted having encountered obstacles to doing so. 

.	 Uncertainty about how to implement the guidelines was the most frequently 

cited obstacle to use. 

ii 
.	 Clinician resistance was the second most frequently cited obstacle. We learned 

that physicians have been concerned about what they have perceived to be 

cookbook medicine and that nurses have been concerned about what they have 

anticipated would result in increased administrative and patient care workloads. 

RECOMMENDATIONS 

% l%blic Health Service (PHS), through AHCP~ should &terrnine more efective ways 

to pmnote famdimity with and use oj the guidelines. 

The AHCPR reports that its guidelines were distributed to every hospital and nursing 

home in our sample population. Of all those in leadership positions that we surveyed 

in small, nonteaching hospitals; nursing homes; and HMOS with staff-model 

components; 80 percent reported that their organizations had not used the AHCPR 

clinical practice guidelines and 40 percent were unfamiliar with the guidelines. 

We recognize that introducing new clinical practice guidelines, like any effort to 

introduce systematic changes in the delivery of healthcare, poses challenges. A 

reconsideration of AHCPR’S efforts to promote familiarity with, and use of, the 

guidelines could yield more effective approaches to meeting these challenges. 

The PHS, through AHCP~ shouki make increased technical supp~ avaikzble to 

guideline Usem . 

Our findings demonstrate that most organizations that have used the AHCPR 

guidelines have encountered obstacles to doing so. Many organizations reported that 

uncertainty about how to use the guidelines has been an obstacle. Other frequently 

noted obstacles include resistance from physicians and nurses, an inability to measure 

compliance with the guidelines, and an inability to measure the effects of guideline 

use. To facilitate the use of the AHCPR guidelines, the PHS, through AHCPR, 

should strengthen its capacity to advise those organizations that are encountering 

obstacles. For example, the Agency could: 

�	 identi~ and disseminate benchmarking practices of those organizations that 

have overcome obstacles and made effective use of guidelines; and/or 

.	 build on its current programs and conduct training/technical assistance sessions 

in which representatives of healthcare organizations can learn more about the 

guidelines and how to use them effectively. 

l’he PHS, through AHCP~ shouki develop and irnpkment ystematik rnechaniwns for 

obtaining objective feedback about guideline use. As pat of tfi efo~ AHCPR should 

sponsor regular surveys of healkare oganuations. We surveys shouki be dksi@ed to 

p&e pmjectable indicatom that albw AHCPR to gauge changes over the k the 

tixtent of gukidine use. 

.,. 

111

Although AHCPR collects some information about the use of specific guidelines, it has 

not yet incorporated into its regularly scheduled activities efforts to gather information 

on the frequency with which AHCPR guidelines are used or user-feedback information 

from those organizations that have experience with the guidelines. To better support 

the Agency in its efforts to gauge the level of guideline use and to continuously 

improve the quality of its guidelines, the PHS, through AHCPR, should incorporate 

systematic efforts to gather objective information about guideline use into its ongoing 

responsibilities. 

To complement this important research, AHCPR could conduct other types of 

ongoing efforts designed to provide information about users’ experiences with the 

AHCPR guidelines and changes over time in the users’ experiences. For example, the 

Agency could: 

�	 conduct regular surveys of organizations that have requested the guidelines to 

obtain feedback about their experiences with, and assessments of, the guidelines; 

�	 conduct regular telephone interviews with select “key users” to obtain more indepth feedback; and/or 

� attach user-feedback forms to all guidelines that are disseminated. 

We hope that our report till provide information that is useful to AHCPR in the near 

term as it considers ways to facilitate the implementation of its guidelines. We also 

hope to provide useful information for organizations that are seeking ways to benefit 

from guideline use, and for Congress as it considers AHCPR’S future efforts. A 

companion report, Early Experiences with Clinical Practice Guidelines Sponsored by the 

Agenq for Health Care Policy and Research: Case Descriptions (OEI-01-94-O0251), 

presents short case descriptions that illustrate the experiences of several organizations 

with the AHCPR guidelines. 

COMMENTS ON THE DRAFT REPORT 

We solicited and received comments on our draft report from the Public Health 

Service (PHS). The full text of these comments is included in appendix C. 

The PHS offered technical comments on our draft report and these were considered 

as we developed the final report. Below we address the key elements of the PHS 

general comments on our findings, methodology, and recommendations. 

llndings: The PHS characterized our first finding--that 32 percent of the survey 

respondents reported that they have used, or plan to use, 1 or more of the AHCPR 

guidelines about which we inquired--as extremely encouraging. We recognize that 

introducing new clinical practice guidelines is a challenging endeavor, but found no 

guidance in the scientific literature for determining what level of use constitutes 

iv 
success. The AHCPR has not yet established benchmarks for guideline use or 

quantified expectations against which to measure its success. 

Methoddogy The PHS Stated that the types of organizations surveyed in our study 

are not representative of all healthcare institutions. We agree. We continue to 

believe, however, that these organizations are important components of this nation’s 

healthcare delivery system and that clinical practice guidelines are relevant to the 

provision of care in all three settings. We are considering additional research to 

examine the use of clinical practice guidelines in other healthcare settings and 

encourage AHCPR to do the same in our third recommendation. 

Recomrnendutins: In its comments on our recommendations, PHS cited various of its 

current efforts that it regards as relevant to the recommendations. We recognize the 

relevance and importance of a number of the activities cited by PHS. 

We are concerned, however, about our third recommendation, in which we urge PHS 

to develop and implement systematic mechanisms for obtaining objective feedback 

about guideline use. This recommendation proposes regular surveys as an important 

tool for continuous quality improvement. To clari~ the recommendation, we have 

included in the recommendation statement information that was formerly presented in 

the text supporting it. 

We continue to believe that systematically collected feedback, gathered at regular 

intervals, from current and potential guideline users, is a critical component of any 

continuous quality improvement effort. This feedback can be used as a valuable 

performance indicator addressing the degree of AHCPR’S success in facilitating the 

use of, and familiarity with, its guidelines. 

Our report finding--that 20 percent of survey respondents reported that they have 

used 1 or more of the 6 AHCPR guidelines about which we inquired--provides a 

baseline against which AHCPR can measure its progress in achieving greater 

familiarity with, and use of, its guidelines. 

v 
INTRODUCTION


PURPOSE 

This study describes the early experiences of healthcare organizations in using clinical 

practice guidelines sponsored by the Agency for Health Care Policy and Research. 

BACKGROUND 

Vatitions in Medical Bactice and Clinical hzctice Guidelines 

Variations in clinical practice among different physicians and hospitals, and in different 

geographic areas have long been observed .1 With the growing awareness of these 

variations, concern has developed about adverse patient outcomes and financial costs 

associated with inappropriate medical care. Clinical practice guidelines have emerged 

as part of a larger effort to reduce unwarranted variations in care and the costs 

associated with them. 

According to the Institute of Medicine, clinical practice guidelines are “systematically 

developed statements to assist practitioner and patient decisions about appropriate 

healthcare for specific clinical circumstances.”2 Such guidelines--and similar 

materials--have been sponsored by a multitude of organizations, including the Agency 

for Health Care Policy and Research (AHCPR), the Centers for Disease Control, and 

the National Institutes of Health, all within the Public Health Service (PHS) in the 

U.S. Department of Health and Human Services. The U.S. Preventive Services Task 

Force, the Rand Corporation, medical professional associations, health insurance 

associations, and health maintenance organizations have also developed guidelines.3 

Guidelines address a broad range of topics and vary along several dimensions, 

including the breadth of the topic addressed, the level of detail of care dealt with, the 

extent to which the guideline is directive, the extent to which scientific evidence forms 

the basis for guideline recommendations, the format in which the guideline is 

presented, and the method by which the guideline is developed. 

Guidelines have been used, or tested for use, in several different applications and to 

achieve several different goals. The Office of Technology Assessment has reported 

that clinical practice guidelines are being used in malpractice litigation and as a means 

of reducing defensive medicine. 4 Primary-care providers--such as hospitals, health 

maintenance organizations, and nursing homes--use guidelines in care provision and 

quality assurance reviews. At least one managed healthcare provider--with networks 

in 36 States--is testing the use of computer technology that allows physicians instant 

access to clinical practice guidelines, including those sponsored by AHCPR.5 At least 

one health maintenance organization has developed a typology of guideline 

implementation methods.b 

1

lhe Agency for Health Care Policy and Reseaxh 

The Omnibus Budget Reconciliation Act of 1989 (P.L. 101-239) established AHCPR 

as the successor to the former National Center for Health Services Research and 

Health Care Technology Assessment. Congress assigned the new Agency the 

responsibility to enhance the quality, appropriateness, and effectiveness of healthcare 

services and access to such services, through the promotion of improvements in clinical 

practice and in the organization, financing, and delivery of healthcare services.7 

As part of its activities to fulfill its mandate, the Agency arranges for the development 

and distribution of clinical practice guidelines.8 To date, the Agency has released 

sixteen guidelines--and each is produced in several formats, including the full-length 

clinical practice guideline, a quick reference guide for clinicians, and patient guides.g 

To develop the guidelines, the Agency convenes independent, multidisciplinary panels 

of private-sector clinicians and other experts that collaborate over an extended time 

period. The panels apply explicit, science-based methods to develop specific 

statements related to the diagnosis, treatment, and management of the clinical 

condition under consideration. In instances in which the scientific literature is 

incomplete or inconsistent, the professional judgment of the panel members and 

consultants is reflected in the guideline.l” 

The AHCPR employs a multi-pronged strategy to disseminate its guidelines. In 

addition to distributing all of its guidelines to the more than 10,000 organizations on its 

master mailing list, the Agency works with the medical, trade, and lay media to 

generate and place articles about guidelines. 11 It has established a toll-free, 

guideline-request telephone line through the AHCPR publications clearinghouse and 

has begun to make its guidelines electronically available through the National Library 

of Medicine and the Internet. It also works with professional societies and commercial 

organizations to reprint and further distribute guidelines. 12 Among its other 

dissemination activities, the Agency conducts conferences for those interested in 

learning more about guideline implementation and the experiences of those 

organizations that have used AHCPR’S guidelines.13 

Of the $154 million appropriated for AHCPR in 1994, $81 million was dedicated to 

the Agency’s Medical Treatment Effectiveness Program. Approximately $5 million of 

this sum was spent on the development and updating of guidelines and approximately 

$10 million was spent on dissemination and evaluation efforts. Of this, approximately 

$4 million was spent on printing and disseminating guidelines.14 

Concerns about Clinical Practice Guidelines 

An increasing number of organizations--including healthcare providers, health insurers, 

State governments, and medical professional associations--have become involved in the 

development of clinical practice guidelines; and an increasing number of clinicians 

have become aware of them. 
The introduction of new technologies, however--including new clinical practice 

guidelines--can be a slow and difficult process. 15 In research supported by AHCPR 

and other organizations, and in conferences about clinical practice guidelines, barriers 

have been identified to the greater acceptance and more extensive use of 

guidelines.lb Such barriers include confusion about how best to use guidelines; 

resistance from administrators, clinicians, and/or patients; administrative, bureaucratic, 

and/or technological problems in either preparing for or adhering to guidelines; the 

expense of guideline implementation; and limitations in organizations’ ability to 

measure compliance with guidelines and the effects of guideline implementation. 

THE OBJECTIVES OF THIS REPORT 

The AHCPR already supports many projects to evaluate the development and 

dissemination of its clinical practice guidelines. In addition, the Agency sponsors 

research on broader questions related to guidelines and their use.17 Many of these 

projects are multi-year, long-term efforts. We prepared this report in response to the 

agency’s interest in gaining a better understanding of the extent to which, and the 

manner in which, healthcare organizations have been using its guidelines. 

We hope that our study will yield information that is useful to AHCPR in the near 

term as it considers ways to facilitate the implementation of its guidelines. We also 

hope to provide useful information for organizations that are seeking ways to benefit 

from guideline use, and for Congress as it considers AHCPR’S future efforts. 

Accordingly, this report describes how organizations have used AHCPR clinical 

practice guidelines and illustrates ways that other organizations might use them. We 

do not evaluate the effectiveness of guideline use, test differences among various types 

of organizations’ guideline use, or distinguish the strengths and weaknesses of specific 

AHCPR guidelines, Furthermore, we do not evaluate AHCPR’S dissemination efforts, 

its internal processes for developing or disseminating guidelines, or its processes for 

evaluating guideline use. 

The report provides initial estimates of three dimensions of guideline use: (1) the extent 

to which healthcare organizations use the guidelines, (2) the ways in which organizations 

use the guidelines, and (3) the obstacles that organizations have encountered in their 

efforts to implement the guidelines. A companion report, Ear& Experiences with Clinical 

Practice Guidelines Sponsored by the Agency for Health Care Policy and Research: Case 

Descriptions (OEI-01-94-00251) presents a series of short case descriptions that illustrate 

the experiences of several organizations with the AHCPR guidelines. 

METHODOLOGY 

Our report is based on data from two primary sources. First, to develop initial 

estimates of guideline use, we conducted a mail survey of three types of healthcare 

provider organizations--150 randomly selected nursing homes, 150 randomly selected 

nonteaching hospitals with no more than 100 patient beds, and all 80 of the health 

maintenance organizations (HMOS) with staff-model components. The hospital and 

nursing home samples were drawn from the Health Care Financing Administration’s 

On-line Survey Certification and Report (OSCAR) System. The HMO sample was 

3 
taken from the report of the most recent HMO survey conducted by the American 

Managed Care and Review Association (AMCRA) Foundation. 

We included nonteaching hospitals in our sample because we anticipated that they 

would be more likely to use existing guidelines, such as those sponsored by AHCPR, 

than to develop their own.18 We further defined our hospital sample to include only 

those nonteaching hospitals with no more than 100 beds because we anticipated 

methodological difficulties in obtaining mail-survey responses from the appropriate 

staff in large hospitals with many, large, specialized departments. 

We included HMOS with staff-model components in our survey because HMOS 

represent a trend towards managed care. In addition, staff-model HMOS, because 

they employ physicians directly and exclusively, are probably better able than 

independent-practice/physician association HMOS, network-model HMOS, or groupmodel HMOS to influence the provision of care to their patients. Accordingly, we 

anticipate that staff-model HMOS, more than the other HMO models, might seek to 

use clinical practice guidelines. 

Sixty-four percent of our sample completed and returned the survey. Our analysis of 

those organizations that did not respond to the survey indicates that, for those 

variables considered, our results are unbiased with respect to differences between 

respondents and nonrespondents (see appendix A for more information on the 

nonrespondent analysis). The survey instrument focused on the organizations’ use of 

the first six AHCPR guidelines that were released between January 1, 1992 and 

January 1, 1994. Th~se B for a summary of 

the survey results). 

Second, to increase our understanding of the organizations’ experiences with the 

AHCPR guidelines, we conducted telephone interviews with half of the survey 

respondents that reported guideline use. 

We conducted our review in accordance with the Quali~ Standards for Inspections 

issued by the President’s Council on Integrity and Efficiency. 

4

FINDINGS


Twenty percent of the survey respondents reported that they have used 1 or more of


the 6 AHCPR guidelines about which we inquire~ an additional 12 pereent reported


that they plan to do so.


Of the 243 organizations that responded to our questionnaire, 49 (20 percent)


reported that they had used at least 1 of the 6 clinical practice guidelines that AHCPR


released between January 1, 1992 and January 1, 1994.19 An additional 28 (12


percent) indicated that, although they have not yet used any of the 6 AHCPR


guidelines, they intend to do so. Several of these organizations indicated that they are


in the early phases of developing guideline implementation plans and programs.n


Thirty-seven (76 percent) of the 49 AHCPR guideline users reported having used


more than 1 of the AHCPR guidelines. Pressure Ulcers In Adults (prevention), Urinary


Incontinence in Adults, and Acute Pain Management were the guidelines most


frequently used.21 Thirty-two (65 percent) of the AHCPR guideline users reported


that they had also used guidelines sponsored by other organizations to address


AHCPR guidelines topics.


�	 Imptus for using the guidelines has frequently come ~m the organizations’ 

hterna~ qua@y-hpnwment programs. External facton--such as directiws @m 

supervking o~anizations and State regukitory requirements--have also plkyed an 

important rule. 

Internal Factors: Quality-improvement initiatives were cited as having contributed to 

the decision to use ~CPR guidelines by 41 (84 percent) of the user organizations. 

Examples of the ways in which internal factors have contributed to the decision to use 

the AHCPR guidelines include the following: 

� One hospital incorporated two of the guidelines into a program for evaluating 

physicians’ clinical performance. 

� Another hospital decided to use a guideline after the death of a patient who had 

been admitted with severe pressure ulcers. 

� A nursing home adopted a guideline after a product sales representative provided 

three copies of the guideline and suggested that the home use it. 

External Factors: An external factor was cited as having contributed to the decision to 

use the AHCPR guidelines by 22 (45 percent) of the user organizations. Examples of 

the ways in which external factors have contributed to the decision to use the AHCPR 

guidelines include the following: 

5

F Eight nursing homes adopted one of the guidelines in response to deficiency 

citations from State nursing home surveyors or in response to the homes’ concerns 

about meeting survey standards. 

� An HMO began using a guideline because the Medicare Peer Review Organization 

(PRO) recommended adherence to the guideline as a means of resolving a patient 

complaint. 

E Two nursing homes decided to use two guidelines because their parent companies 

encouraged them to do so. 

�	 The o~anizations’ expectations of guidkline use have focused on iinpnnwd cllnical 

care. Many have aiko vikwed the guidelines as a means of reduciizg costs and 

providing better protection jiorn rnu@actice litigation 

Improved Clinical Care: Forty-seven (96 percent) of the users reported that they have 

expected the AHCPR guidelines to help them achieve at least one goal related to 

improved clinical care. Such goals ranged from improved clinical outcomes for 

patients, to reduced uncertainty among clinicians about appropriate care for patients, 

reduced variation in clinical decision-making, and increased patient satisfaction. 

Other Goals: Twenty-five (51 percent) of the user organizations reported having 

expected to reduce costs associated with clinical care. Twenty (41 percent) reported 

that they have expected to achieve better protection from malpractice litigation. 

�	 lhe reasom most often cited for not havhg used the *W were that the 

organizations were not familiar &h them and the oqynizations have their own 

guidelines. No organizations cited dikagreernent with the guideti as a bti for 

not having used them. 

One-hundred and ninety-four (80 percent) of all respondents to our survey indicated 

that they have not used any of the six AHCPR guidelines. Although 16 (8 percent) of 

them indicated that these guidelines are not clinically relevant for their institutions, 

none of the organizations indicated disagreement with the care recommended in the 

AHCPR guidelines. 

Lack of Familiarity: Ninety-seven respondents--50 percent of the nonusers (and 

40 percent of all respondents) --reported that they were not familiar with the 

guidelines.z2 In their comments on the mail survey, 27 (14 percent) of the nonusers 

requested copies of the AHCPR guidelines and/or expressed an interest in learning 

more about them. 

Other Guidelines: Eighty-two respondents--42 percent of the nonusers (and 34 percent 

of all respondents) --reported that they have their own guidelines. Thirty-four 

respondents--18 percent of the nonusers (and 14 percent of all respondents) --reported 

that they use guidelines developed by an organization other than AHCPR. 

6 
Ninety-six pereent of the survey respondents that have used 1 or more of the 6 

AHCPR guidelines reported that their efforts to do so have focused on clinici~, 

36 percent reported that they have directed @deline-implementation efforts towards 

patients. 

Clinician-Directed Efforts 

�	 Clinician-education @o*-ranging@m thq distribution of guidellne matetitzik to 

the condkct of t?ducatz%malprograms--werethe most frequently repited means of 

irnpkmentihg the guiiiklihes. Such #ens were reprted by most guideline usem 

Forty-two (86 percent) of the guideline users reported at least one educational effort


directed at clinicians. Most frequently cited was the distribution of AHCPR guideline


materials to clinicians, which was reported by 28 (57 percent) of the users.~


Examples of the ways in which organizations have distributed AHCPR guideline


materials include the following:


� One nursing home put up posters summarizing the guideline on each of its units. 

F An HMO distributed to physicians its own summary of the AHCPR guideline. 

E At another HMO, copies of the guideline materials were distributed at meetings of 

the medical and nursing staff. 

� A hospital distributed copies of the guideline to each of its nurses, physicians, and 

physician assistants, placed additional copies at the nursing station and in the hospital 

library, and circulated a hospital-wide memorandum announcing the availability of all 

of the guidelines. 

Educational programs have been offered by 24 (49 percent) of the users. Such 

programs have involved train-the-trainer efforts, in-service activities, and self-study 

video tapes. Examples of the ways in which organizations have used the guidelines in 

these efforts include the following: 

F A nursing home sent nurses to off-site classes on a guideline. These nurses then 

provided training for other staff on the recommended diagnostic and care techniques. 

� A hospital hosted a product-sales company’s full-day training session for nurses and 

luncheon lectures for physicians on guideline-recommended care. 

� Another hospital conducted a series of five, mandatory, in-service training sessions 

for its nurses. 

� A nursing home hired a consultant to train small groups of nurses about the 

guideline recommendations. 
b A hospital used a privately produced video series explaining the guideline to train 

its nurses and nursing assistants. 

In addition, 14 (29 percent) of the users reported that they provided feedback to 

clinicians about the extent to which their care conforms to guideline recommendations. 

At most organizations, this feedback has been provided informally, either to 

individuals or to staff-meeting groups. A few organizations, however, have provided 

more formal feedback to their clinicians: 

E A hospital has provided feedback to clinicians in reports that note variances 

between the guideline-recommended care and that provided by clinicians. 

* A nursing home has displayed weekly charts for each of its units documenting 

success rates in managing the guideline condition; the home also has run a weekly 

contest (with prizes) among the units to encourage staff to better manage the 

condition. 

�	 The dkveloprnent of an algorit~ polky and pmcedurq or other impbnentation 

tool to@& clinician practice was aiko reprted by most guidkline usem 

Thirty-six (73 percent) of the users reported having made at least one administrative 

effort to implement the guidelines. Most often, organizations adapted the guidelines 

for their own use by developing an implementation tool: 31 (63 percent) of the users 

reported such an effort. 

E At one hospital, a multidisciplinary task force worked over the course of six 

months to incorporate the guideline recommendations into existing policy and 

procedures and to develop an algorithm for patient care corresponding to the 

guideline. 

> At a nursing home, over the course of a month, the director and assistant director 

of nursing revised the home’s existing policy and procedures to reflect the guideline; 

the medical director approved the new draft. 

� A manager at another hospital developed a two-page summary of the guideline, 

which the hospital physicians and governing board then approved. 

� Another hospital collaborated with an area-wide group of care providers over the 

course of a year to adapt the guideline for a uniform standard of community care. 

Other administrative efforts to implement the guidelines have included the use of 

automated computer systems that reflect guideline recommendations, U changes to 

clinician-patient encounter forms, changes to lab- or test-order forms, and 

retrospective reminders for clinicians .X Twenty-five (51 percent) of the users 

reported having made at least one such effort. Examples include the following: 

8

� A hospital has used its computerized patient-care system to suggest appropriate 

care for treatment of the guideline condition and to allow monthly reviews of clinician 

compliance with guideline recommendations. 

� Another hospital implemented a five-point, self-reported pain-scale for adults and a 

five-point color scale, on which red represents the most severe pain and white 

represents the absence of pain. The color scale has been useful with children. 

� Two hospitals developed new clinician-patient encounter forms to be used on a 

daily basis for assessment of the guideline condition. 

� Another hospital developed a new culture-order form with expanded order options 

for the guideline condition. 

� Three organizations instituted charting systems to help nurses remember to adhere 

to guideline recommendations. 

�	 Ecommu”c incentives to encourage chicians to pmvkle guitkline—recommended 

care were reported by very fw o~anizations. 

Only three organizations (6 percent) reported that they have used economic incentives 

to encourage clinicians to provide guideline-recommended care. Two of these 

organizations implemented changes in reimbursement policy. In one HMO, clinicians 

who fail to provide the guideline recommended care risk losing their care provider 

contracts with the HMO. 

Patient-Directed Eftorts 

� l%e dWibution of guideline material to patierm was the mostjhquently repotied 

patientdi.rected ejfott and was reprted by 12 (24 pement) of the usem 

Educational s&ns for patients were reported by 9 (I8 penxmtj of the mers. 

Most users reported that they have not directed efforts to implement the guidelines 

towards patients. Some users explained that their patients are unable to help 

implement the guidelines because of physical and/or mental limitations. A few 

organizations anticipated that they would attempt patient-directed efforts in the future. 

Examples of ways in which the AHCPR guidelines have been used with patients 

include the following: 

� An HMO developed an article for its patients’ newsletter about the organization’s 

new policy for treatment of the guideline condition; the article invites patients to call 

for additional information. 

� A hospital implemented a new emergency room form that explains its policy for 

treatment of the guideline condition. 

9

� Another hospital has provided educational sessions on the guideline condition for 

its patients and for visitors to senior centers and health fairs. 

E A nursing home has explained the guideline recommendations to those of its at-risk 

residents who are able to participate in their own care. 

F A nursing home and a hospital have both distributed guidelines to patients and 

their families when the patients have been discharged for home care. 

� A hospital and a nursing home have used the guideline in home-health classes to 

educate both patients and their families about prevention and care of the guideline 

condition. 

At this early point in the implementation process, only 8 percent of the respondents 

that have used 1 or more of the 6 AHCPR guidelines reported that they have 

measured the effects of their guideline use. 

The first AHCPR guideline was issued in March 1992; two others were issued in that


year. The remaining three guidelines about which our survey inquired were issued in


1993. Although most of our survey respondents are in the early stages of


implementing the guidelines, four reported that they have evaluated the effects of


their guideline use. An additional four organizations that we interviewed reported that


they have measured or plan to measure the effects of their guideline use.


9 l%ese measures have f~ed on patients’ clinkal outcomes and clinicians’ 

pe@onnance. Thqv have been based prirn@v on chart and reconi revhvs. 

The organizations have assessed both clinicians’ performance and patients’ clinical 

outcomes. A few organizations have also measured patient satisfaction; a few have 

taken organization-wide measurements of the effects of their guideline 

implementation. Below are examples of the assessments that have been made. 

� A hospital has used its computerized patient-record system to assess how care was 

provided and patient outcomes. In addition, the hospital has conducted post-discharge 

patient-satisfaction surveys. These reviews documented increased compliance with the 

guidelines and increased patient satisfaction. 

� A nursing home has conducted weekly record reviews, which have demonstrated a 

decrease in the incidence and severity of pressure ulcers. 

� An HMO is planning to conduct an audit of clinician compliance with the guideline 

in the care of all patients treated for cataracts. The HMO also plans to collaborate 

with its PRO to review claims data for a random selection of patient records. 

10

E A coalition of hospitals and nursing homes took base-line measurements of the 

incidence of the guideline condition in several facilities. The coalition plans to take 

comparison measurements in these facilities six months after the implementation of 

the guideline and one year after that. 

� A nursing home’s quality assurance program has involved seven annual, in-depth 

surveys of care and patient status, and one annual, in-depth review by the parent 

company. Reports of all of the reviews, which address care for the guideline 

condition, are evaluated by the regional administrator and the corporate specialist on 

the guideline topic. 

> Another home has conducted quarterly assessments of patient outcomes for the 

guideline condition. This home’s parent company has also provided an annual peer 

review which addresses care provided for this condition. 

> Another hospital has conducted quarterly studies to assess how care is provided and 

patients’ clinical outcomes. 

Sixty-three percent of the respondents that reported having used one or more of the 

six AHCPR guidelines also reported having encountered obstacles to doing so. 

Although there are many examples of guideline implementation ((see Eady 

Experiences with Clinical Practice Guidelines Sponsored by the Agency for Health Care 

Policy and Research: Case Descriptions (OEI-01-94-00251)), introducing new clinical 

practice guidelines, like any effort to introduce systematic changes in the delivery of 

healthcare, poses challenges. 

�	 Unmtainty about how to implement the guidelines was the most jkquently cited 

obstacle to use. 

Twenty-one (43 percent) of the guideline users reported that such uncertainty was an 

obstacle to their use of the guidelines. Many of the organizations that we interviewed 

struggled over the development of a plan to implement the guidelines. Examples of 

organizations’ uncertainty about how to implement the guidelines are summarized 

below. 

� A nursing home’s staff has found it difficult to read and understand the entire 

guideline. The home has sought out seminars that address the guidelines to resolve 

this problem. 

E Uncertainty about how to use a diagnostic tool recommended by the guideline 

hampered one hospital’s implementation program. This hospital contacted AHCPR 

for further guidance. 

� For another hospital, confusion about the extent to which it need comply with the 

guideline slowed the implementation process. 
� Another hospital was initially confused about how to implement the guidelines and 

found that other hospitals were unable to provide assistance because they were 

unfamiliar with the guidelines. 

� Another hospital reported that the guidelines are “very general.” Adapting them for 

local use--by making them more specific--required a considerable investment of both 

time and money. 

E Developing a strategy for ensuring staff compliance with the guideline 

recommendations was a time-consuming challenge for another hospital. 

� Clinician resistance was the second most fhquently cited obstacle. 

Clinician resistance was reported by 16 (33 percent) of the guideline users. Physicians’ 

primary concern, according to those we interviewed, has been their perception that the 

guidelines are “cookbook medicine.” Nurses have been concerned that the guidelines 

would result in increased administrative and patient care workloads. Examples of 

clinician resistance to the guidelines are summarized below. 

� A nursing home and two hospitals reported that several of the physicians resented 

what they perceived to be government interference in their work. 

> A nursing home’s staff initially resisted the new policy and procedures because they 

anticipated that these would result in “too much work.” Some nurses continued to 

refuse to comply, but, according to the Director of Nursing, most learned that the new 

rules ultimately result in less work. 

> Three nursing homes and a hospital all noted that any change in prescribed practice 

is always difficult for clinicians and that staff training is a gradual and continuing 

project. 

Other obstacles to guideline use that were noted by survey respondents include an 

inability to measure compliance with the guidelines, an inability to measure the effects 

of guideline use, and the expense of guideline implementation. 

12

RECOMMENDATIONS


% RMc Heakh Service (PHS), through AHCP~ shouiii deternuke more @ective ways 

to promote farnilidy with and use 0$ the guidelines. 

The AHCPR reports that its guidelines were distributed to every hospital and nursing 

home in our sample population. Of all those in leadership positions that we surveyed 

in small, nonteaching hospitals; nursing homes; and HMOS with staff-model 

components; 80 percent reported that their organizations had not used the AHCPR 

clinical practice guidelines and 40 percent were unfamiliar with the guidelines. 

We recognize that introducing new clinical practice guidelines, like any effort to 

introduce systematic changes in the delive~ of healthcare, poses challenges. A 

reconsideration of AHCPR’S efforts to promote familiarity with, and use of, the 

guidelines could yield more effective approaches to meeting these challenges. As 

AHCPR itself has noted, “the term ‘effective dissemination’ includes the concept of 

diffusion of knowledge and information as well as the acceptance, inculcation, and 

utilization of disseminated information... distribution of information alone is 

insufficient to ensure adoption or use.”x 

The PHS, through AHCP~ should make increased technical support availizble to 

guliii?h%eUsem 

Our findings demonstrate that most organizations that have used the AHCPR 

guidelines have encountered obstacles to doing so. Many organizations reported that 

uncertainty about how to use the guidelines has been an obstacle. Other frequently 

noted obstacles include resistance from physicians and nurses, an inability to measure 

compliance with the guidelines, and an inability to measure the effects of guideline 

use. To facilitate the use of the AHCPR guidelines, the PHS, through AHCPR, 

should strengthen its capacity to advise those organizations that are encountering 

obstacles. For example, the Agency could: 

�	 identi~ and disseminate information on the benchmarking practices of those 

organizations that have overcome obstacles and made effective use of 

guidelines; and/or 

.	 build on its current programs and conduct training/technical assistance sessions 

in which representatives of healthcare organizations can learn more about the 

guidelines and how to use them effectively.27 

13

7ZWPHS, through AHCP~ shouki dkveibp and implement systematic mechati for 

obtahing objective feedback about guidklike use. AS part of this #ofl AHCPR should 

sponsor reguku surveys of heakhcare o~anizations. We SLUVeySshouki be desi@wd to 

=:f@?Z~WY that allbw AHCPR to gauge changes over time m the 

. 

Although AHCPR collects some information about the use of specific guidelines, it has 

not yet incorporated into its regularly scheduled activities efforts to gather information 

on the frequency with which AHCPR guidelines are used or user-feedback information 

from those organizations that have experience with the guidelines. To better support 

the Agency in its efforts to gauge the level of guideline use and to continuously 

improve the quality of its guidelines, the PHS, through AHCPR, should incorporate 

systematic efforts to gather objective information about guideline use into its ongoing 

responsibilities. 

To complement this important research, AHCPR could conduct other types of 

ongoing efforts designed to provide information about users’ experiences with the 

AHCPR guidelines and changes over time in the users’ experiences. For example, the 

Agency could: 

�	 conduct regular surveys of organizations that have requested the guidelines to 

obtain feedback concerning their experiences with and assessments of the 

guidelines; 

�	 conduct regular telephone interviews with select “key users” to obtain more indepth feedbaclq and/or 

� attach user-feedback forms to all guidelines that are disseminated. 

14

COMMENTS ON THE DRAFT REPORT


We solicited and received comments on our draft report from the Public Health 

Service (PHS). The full text of these comments is included in appendix C of this 

report, The PHS offered technical comments on our draft report, and these were 

considered in the development of our final report. Below, we respond to some of the 

key elements of the PHS general comments on our findings, methodology, and 

recommendations. 

FINDINGS 

The PHS characterized our first finding--that 20 percent of the sumey respondents 

reported that they have used 1 or more of the AHCPR guidelines about which we 

inquired and that an additional 12 percent reported that they plan to do so--as 

extremely encouraging, given the context of traditional rates of diffusion of new 

information into health care practice. 

We recognize that introducing new clinical practice guidelines is a challenging 

endeavor, but found no guidance in the scientific literature for determining what level 

of use constitutes success. The AHCPR has not yet established benchmarks for 

guideline use or quantified expectations against which to measure its success. 

METHODOIXIGY 

The PHS stated that the types of organizations surveyed in our study are not 

representative of all healthcare institutions. The PHS suggested that, to obtain a 

complete picture, additional types of providers should be surveyed. 

We continue to believe that small teaching hospitals, staff-model HMOS, and nursing 

homes are significant components of this nation’s healthcare delivery system and that 

clinical practice guidelines are relevant to the provision of care in all three settings. 

We agree that guidelines are relevant to healthcare delivery in other organizations. In 

response to PHS’S concern about obtaining a complete picture, we are considering 

additional research to examine the use of ~linical practi~e guidelines in additionalhealthcare settings and encourage AHCPR to do the same in our third 

recommendation. 

RECOMMENDATIONS 

The PHS concurred with each of the three recommendations presented in our draft 

report. In each case, PHS cited various efforts that it has underway that it regards as 

relevant to the recommendations. We recognize the relevance and importance of a 

number of the activities cited by PHS. 

15

We are concerned, however, about our third recommendation, in which we urge PHS 

to develop and implement systematic mechanisms for obtaining objective feedback 

about guideline use. This recommendation proposes regular surveys as an important 

tool for continuous quality improvement. To clari~ that recommendation, we have 

included in our recommendation statement information that was formerly presented in 

the text supporting it. 

We continue to believe that systematically collected feedback, gathered at regular 

intervals, from current and potential guideline users, is a critical component of any 

continuous quality improvement effort. This feedback can be used as a valuable 

performance indicator addressing the degree of AHCPR’S success in facilitating the 

use of, and familiarity with, its guidelines. 

Our report finding--that 20 percent of the survey respondents reported that they have 

used 1 or more of the 6 AHCPR guidelines about which we inquired--provides a 

baseline against which AHCPR can measure its progress in achieving greater 

familiarity with, and use of, its guidelines. 

16

APPENDIX A


NONRESPONDENT ANALYSIS 

An important consideration with research based on surveys of the type we have 

conducted is the bias that may be introduced into the results if the nonrespondents 

differ from survey respondents in systematic ways. To test for the presence of bias, we 

obtained information from AMCRA on the HMOS to which questionnaires were sent. 

In addition, we obtained information from the OSCAR file for all nursing homes and 

hospitals to which questionnaires were sent. 

Because the samples for HMOS, hospitals, and nursing homes were selected from 

different sampling frames, a separate nonrespondent analysis was performed for each 

of these groups. The variables used in the analysis for HMOS were the number of 

lives covered and the percentage of the HMO practice that was staff model. For both 

hospitals and nursing homes, the variables used were size (number of patient beds) 

and type of ownership (profit vs. nonprofit). 

For HMOS, the t-statistic was used to compare means for each of the two variables for 

respondents vs. nonrespondents. For hospitals and nursing homes, the t-statistic was 

used to compare the mean number of patient beds. The Chi-square statistic was used 

to test differences by type of ownership. 

The results are presented in the tables below. Our analysis of those organizations in 

the sample who did not respond indicates that our results are unbiased with respect to 

the variables considered. 

HMos 

T-TESTFORSIZEOFHMO 

N Mean S.E. of Mean 

Respondents 47 110,239 18,864 

Nonrespondents 33 66,804 16,251 

t= 1.649 prob > t= .103 

T-TESTFORPERCENTOFHMO PRACTICETHATWASSTAFFMODEL 

N Mean S.E. of Mean 

Respondents 47 61.80 4.747 

Nonrespondents 33 70.39 5.834 

t = 1.147 prob > t= .255 

A-1 
HOSPITALS 

T-TESTFORNUMBEROFPATIENTBEDSINHOSPITAL 

N Mean S.E. of Mean 

Respondents 101 51.62 2.595 

Nonrespondents 49 51.41 3.557 

t= 0.048 prob > t= .962 

CHI-SQUAREFORTYPEOFHOSPITALOWNERSHIP 

Respondents Nonrespondents Total % Respond 

Non-Profit 49 (49%) 25 (51%) 74 66% 

Proprietary 5 (5%) 3 (6%) 8 62% 

Government 47 (47%) 21 (43%) 68 69% 

Overall 101 49 150 67% 

CHI-SQ.=.22 DF=2 

NURSING HOMES 

T-TESTFORNUMBEROFPATIENTBEDSINNURSINGHOME 

1, 

N Mean S.E. of Mean 

I 

IIRespondents 95 126.95 8.725 

I 1 I 

Nonrespondents 55 114.73 8.488 

II 

t= 1.004 Prob > t= .317 

CHI-SQUAREFORTYPEOFNURSINGHOMEOWNERSHIP 

Respondents Nonrespondents Total % Respond 

Profit 62 (65%) 43 (78%) 105 5970 

Non-Profit 33 (35%) 12 (22%) 45 73% 

Overall 95 55 150 63% 

CHI-SQ= 2.768 DF= 1 

A-2

APPENDIX B 

SUMMARY OF SURVEY RESPONSES 

All data presented in this appendix are derived from an OIG mail survey that was conducted in 

December of 1994. 

B-1 

—_ 
T’abk..k AHcPR Maii %rvey and


Telephone k-s- GpF~m
 Dist@mw fhlrveyRespondents,


Ckideiine Users,md @I&~UH


thatwere~ ,“
 –	 ~~~ ~pqp~i

Surveys Distributed 

~fibafi 

Survey Respondents 101 67 95 ~~ ~~ 

Respondents that Reported Use of the 

AHCPR Guidelines* 

19 @ ~fl Q*

Guideline Users that we Intervkwed ml17 AZ 1~ 

* We are 90 percent confident that the percentage of small, nonteaching hospitals that have 

used the guidelines is between 12 and 25; that the percentage of nursing homes that have used 

the guidelines is between 14 and 28; and that the percentage of HMOS with staff-model 

components is between 11 and 31. For all organizations represented in our sample, we are 90 

percent confident that the percentage that have used the guidelin~ is between 16 and 24. 

B-2

Tabk 2 ‘llm oumbm and.pimxmtageof 

urganizkwionstit cited tkf@km@g:factoris 

iq havingcontri%uted:tatheir dixisi~ to 

m“the maw @idelines!

-+ 

a. State law requirements 

b. Insurance requirements 

c. A directive from a supervising or 

overseeing organization 

d. Your organization’s internal, qualityimprovement initiative 

e. OTHER factors 

Hiiidk


2 10 9 45 2 20 13 26’ 

15151 1036 

3 16 10 50 3 30 16 33 

17 9(I 14 ’70 10 100 41 w 

3 ’16 2 10 2 20 7 14 BIIE 

‘In this and in following tables, as noted: because respondents checked as many items as 

were appropriate in response to the question, the columns total more than 100 percent. 

‘In this and in following tables, as noted: total percentages are sometimes more or less 

than 100 due to rounding. 

B-3 
~L__L_—J~~


a. Increased patient satisfaction 17 90 11 55 6 60 34 69 

b. Improved clinical outcomes for patients 17 90 18 90 9 90 44 90 

c.	 Reduced uncertaintyamong clinicians 16 “84 13 65 9 9(3 38 ’78 

about appropriate care for patients 

d. Reduced variation in clinical decision- 16 “84 11 55 10 100 37 , 76


making


e. Reduced costs associated with clinical care 10 ‘ 53 9 45 6 60 25 51


f. Better protection from malpractice


litigation brought against the organization or 10 53 5 25 5 50 20 41


its providers


g. OTHER goals 15 0 Q 1 10 2 ,4


EKI WI


Number of Goals N 

B-4

i 

~====== 

a. Acute Pain Management


b Urinary Incontinence in Adults HEl?RE


Hospital Nursing Home HMO TOTAL 

N=19 N=20 N=1O N=49 

o II o 

1~1 t II 

II II 5 

TOTAL 19 W II 2a 

3This includes “full implementation” and “pilot project” use. 

B-5 

II 
toAmPRgtd tha~ -w. rld&e&*e devekqdby chid topkx$ other listed Or@miza UHML H~@al Table& me mnnber and perqemageof 

~tifiati

organizationsthai reported idaddition to -

bekd 

a. Acute Pain Management 7 37 7 35 2 20 16 33 

b. Urinary Incontinence in Adults 5 26 15 75 1 10 21 43 

c. Pressure Ulcers in Adults 6 32 12 60 2 Xl 20 41 

d. Cataract in Adults 2 10 1 5 2 20 5 10 

e. Depression in Primary Care o lo 7 M 4 40 11 22 

f.	 Sickle Cell Disease 0(1 $’ o 0 ~~~ IE 0000 

B-6

~abk 10. me number and percentage of 

organiza- that reported h@iig. @ 

A@2PR guideline in “&eMowing M’orts ‘LllM dirmted a@atie& 

~&6%JG~l 

a. Mailingpatients reminders-to seek


guideline-recornrnendedcare--either as part of 1 ~ 1 5 1 10 3 6
 a newsletter or in a more directed mailing


b. Distributingcopies of the Pa-s Guide 12 63 0 Q () l-J 12 24
 to patients


c. Conductingeducationalsessionsfor 

patients about the guideline-recommended 6 32 3 15 () () 9 18


care


d. ConductingOTHER efforts intendedto


encourage patients to seek guideline- 2 10 1 5 0 (-J 3 6
 recommended care 

/ 

B-7

~MHMM

a. Distributingcopies of the CZinicalRzxZice 

mmigmm 12 63 8 40 2 20 22 45 

GuMeilnes to clinicians 

b. Distributingcopies of the QuickR@mnce 14 “74. 5 z 2 20 21 ’43 

Guidefw CliniciarLSto clinicians 

c. Distributingcopies of the Patient%Guide to 15 W ~ 5 25 2 20 22 45 

clinicians 

d. Providingformal continuingeducation for 7 3’7: 11 55 ‘ o u 18 37 

clinicians on guideline-recommended care 

e. Conductingskillstrainingfor clinicianson 9 47 11 55 0 u 20 41 

guideline-recommended care 

f.	 Ccmducti.ngacademic detailing for clinicians 3 16 2 IQ o 0 5 10


about guideline-recommended care


g. Providing feedback to clinicians about the


extent to which their care conforms to 3 16 9 45 ~ 2 20 14 29


guideline recommendations


h. Conducting OTHER efforts intended to


encourage clinicians to provide guideline- 2 10 ~ 4 20 4 40 10 20


recommended care 1


B-8

Table 13. ‘I& ~umber ad pergatage of organ&@ium that reported having used 

AH~R @&l@es in the .i@w&d lmmllerof @ducatkmLde&hrt&dixee&&at.@ni@ms 

==lMmmm 

B-9 
r 

Table 14. The number and.perc@age of 

: c3rg@zatk?tlathat &ported hm@g:u$ed 

J%FXXXR:guidojjne(s).lmthe Mkiw&ig 

Z@ii&?mtite Z&W@e&?*giti.’eEm-ta -,+~ �BBL 

~fi= EZ 

a. Developing a clinical pa@ algori~ or


other guidelineimplementationtool has@ at 12 63 13 65 6 60 31 63


least in pm on AHCPR guidelines


b.Institutingautomated computer systems 2 10 3 15 0 0 5 : 10


that refleet guidelinerecommendations


c. Institutingretrospective remindmsfor 1 5 2 10, 1 10 4 8


Cliniciam+


d. Makingchanges to clinician-patient


encounter forms that encourage cliniciansto 9 47 6 30 1 10 16 33


provide the guideline-recommendedcare


e.Makingchanges to lab- or test-order forms


that encourage cliniciansto providethe 2 10 4 2Q o u 6 12


guideline-recommended care


f. Mandatingadherence to the guideline 1 5 3 15. 4 40 8 ~ 16


g. InstitutingOTHER administrativeor


technologicalchanges that encourage 1 5 3 15. 1 10 5 : 10


cliniciansto provide the guideline­


recommendedcare


4Examples of such automated computer systems include those that provide clinicians with 

on-line access to guidelines or guideline summaries; those for concurrent reminders that can be 

programmed to appear on the computerized charts of patients who have appointments; those 

for formularies; and those for prescription, lab- or test-orders. 

5An example of a retrospective reminder is a list of patients who have not yet received the 

guideline-recommended care. 

B-10 
B-n 
Talk 16. The munber aud pimxmtage of 

orgau@ticms thatrepwted bing U$@lhe 

~fo&nvinJjKxOn$lmicincentiwwtuEilcourage . . 

=-~y-y .. . ... ~~ ~e”m~ ~ ‘MHUL ~ ti Efi% G 

a. Financialbonuses for clinician provision of 0 0 () o () o () ‘f) 

the guideline-recommended care 

b. Financial penaltiesfor clinician failure to o 0 0 u’ o 0 0 0 

provide the guideline-recommended care 

1 

c. Changes in reimbumement policy that 

discourage care other than that recommended o 0 0 #. 2 2Q 2 4 

in the guideline 

d. OTHER economic incentives that


encourage clinician provision of the guideline- 0 0 0 u: 1 10 1 2


recommended care


B-12

B-13 
Resistance from: 

a. Physicians 7 37 5 25 2 20 14 , 29


b. Nurses 5 26 5 X5 o 0 10 20


c. Other Staff 2 10 4 Xl : 1 10 7 ‘ 14


d. Patients 1 5 3 15 0 q 4 $,


e. Lack of support from administration 1 5 0 0 0012


f. Uncertainty about how to implement 9 47 10 50


the guidelines


2 20’21 

g. Operational difficulties 2 10 3 15 2 20 7 14


h. Inability to measure compliance with 7 37 4 20 1 10 12 24


the guidelines


i. Inability to measure the effects of the 7 3’7 4 20 1 10 12 24


guideline use
 ~~ ,“: HE


j. Expense of implementation 5 26 2


k. OTHER factors o 0 2 IU ‘Imm


%xamples of operational difficulties include problems with computer systems, patientencounter forms, and medical-records administration. 

B-14 

43 
+1’’32” 

2 II 

6 6 

4 1 

3 2 

II 5 lU- 2 

2 

0 

0 

20 

B-15 
‘Eabk.2Z The qumberand.percentage.of ~mHMM

mmmm 

a. Distributing written materials explaining 

guidelines to clinicians and/or administrators 

b.	 Conducting educational sessions for 

clinicians and/or administrators about the 

guidelines 

c. Distributing written materials explaining 

guidelines to patients 

d. Conducting educational sessions for 

patients about the guidelines 

e. Using an existing quality assurance system 

to measure compliance with the guidelines 

f. Developing new measurements to assess 

compliance with the guidelines 

g. Seeking additional funding to cover the 

costs of guideline implementation 

h. OTHER stem to address the obstacles 

II Number of Steps II N I % II 

II 1 

II 2 

II 3 

4 2 10 3 

5 3 16 1 

6 1 52 

7 BE 1 5 0 

TOTAL 19 982 Irx 

B-16 
: Td)le 25, The mmher and pgx&ptage of 

uqg@z@ions *at rt=qxlrtedhasi@g@led:* Hospital 

: f@k!Uidg.~et@@ ‘w .meastl@@.’eE&% Qf 

th& ti ti$.th&.AHCl+Rguide~”“’: �RHH ~mmdmmid 

a. Measures of clinician performance 15 3 15 0 .@ Q ~


b. Measures of patient satisfaction 1 5 1 5 0 0 2 4 

c. Measures of patients’ clinical outcomes 15 57 257 Og 67 1$7 

d. Measures of cost 00 00 17 10 1 2“ 

e. Measures of organizational performance Otl 1.5 17 !0 2 4 

f. OTHER methods to measure the effects 00 0 0 1’ 10 1 2 

of the guidelines -I + : BE


7While three nursing homes indicated that they have measured the effects of their use of 

the AHCPR guidelines, five nursing homes indicated that they have taken measures of 

patients’ clinical outcomes to assess the effects of their use of the AHCPR guidelines. 

Similarly, while no HMOS indicated that they have measured the effects of their use of the 

AHCPR guidelines, one HMO reported that it has taken a measure of cost to assess the 

effects of its use of the AHCPR guidelines; one HMO reported that it has taken a measure of 

organizational performance, and one HMO reported that it has used another method to assess 

the effects of the guidelines. 

B-17 
B-18 
B-19 
Hospital Nursing Home HMO TOTAL 

N=82 N=75 N=37 N=194 

— 

B-20 
APPENDIX C 

COMPLETE coIm4EN’m ON THE D~ REPORT 

c-1 
--

Department OF HEALTH & HUJWY SERWCES	 PublicHealthService


Memorandum 

JUJ719S6 

Assistant Secretar”7 ~~ f Health 

offiCe Of InsDector General (OIG) Draft Reports on the


Utilization 02 Clinicai ?ractice ’Guidelines Sponsored by the


Agency for Health Care Policy and Research


Inspector General, OS


Attached are the public Health Service comments on the Subject 

OIG reports. We concur ‘witrhthe recommendations and have 

taken, and will continue to take actions to implement them. 

In addition to our comments on the specific recomendations~ 

we offer a series of technical comments for your 

consideration. 

Philip R. Lee, M.D.


Attachment


la—


Mm — 

PDIQ


DIGAS I 

DIGEI


DIGOl —
 DIGMP —
 AIGCFAA — /


4-

., 

.. .. L-1 

— 

J 

w 

,- m 

- , . ,..._ 

L“..


:.,.’


,,---”
 -1 

., 

. 

— 

w 
COMMENTS OF THE PUBLIC HEAL’T3 SERVICE ON THE OFFICE OF ~E


INSPECTOR GENERAL D~FT ?.
EpORT “CLINICAIJ p~CT~CE GUIDELI~S


SPONSORED BY THE AGENCY FOR HzALTH CARE POLICY .WD RESEARCH:


EARLY EXPERIENCES IN CLINICAL SETTINGS,” OEI-J1-?4-00250


General Comments


The Office of the Inspector Generai (OIG) has surveyed a sample 

of three types of providers regarding their experience with 

clinical practice guidelines deveioped under the auspices of the 

Agency for Health Care Policy and Research (AHCPR). AHCPR worked 

with the OIG in developing the suney and PHS is pleased to now 

receive useful feedback from it. The survey queried respondents 

regarding six guidelines released between March 1992 and April 

1993, and was conducted in December 1994. The findings indicate 

that in this very short time, almost one-third of the suneyed 

institutions were using or intended to use at least one of the 

guidelines to improve the effecci’~eness and WalitY Of clinical


praccice within tiheir indiviauai settings. viewed in the contexc


of traditional rates of diffusicn of new information into health


care practice, these findings are extremely encouraging.


The AHCPR-supported guidelines .have .
 been more rapidly and widely


adopted than many other guidelines, practice recommendations~ OF 

alerts issued by public agencies or professional OrgaIIIZatlOns. 

Further, the findings from the suney may understate the impact


of the guidelines if, as is noted in the draft report, additional


organizations may be indirectly relying on AHCPR-sponsored


guidelines. Many providers may draft or revise “internal”


guidelines based on guidelines developed by other organizations,


including AHCPR. Organizations may also incorporate into their


protocols recommendations that ~ia not come directly from AHCPR-


printed documents, but rather from reports or journal articies 

summarizing the guideline reccmmen.aations. If true, the survey


responses obtained bv the OIG r,avsubstantially under-report the 

full impact of the g~idelines. -

It is important to note that :he :hree t’YPes of o:ga~lzations


sampled by the OIG are not zepresen.tative of all ‘~ea~t-hcare


institutions. To Obtain a comDieCe Dlcture of the :ncenclves and
 . .
 obstacles to guideline use exper~en.ced across sett--­
 ‘-gs we will


need to survey additiona~ types =f providers (e.9.I ~pA ‘Oael


~,os, large hospitals, zeachinq hcsqltals, etc.) . It is also


important to note that differen.c ;’~ldelines are more re~evant ‘o


some :~es of providers than ozkers. We therefore “;:oula
expect,


and zma in the s’urvey z-esuits z=ported here) , thaz the


documentm~ ofdiffusion ormforma[lon carepracuce


1“ Anumberofstudies rates ofnew[echnologles mo health 

are cl[ed m the draft repor[. Morerecek mformauonre~araml -. [he AMA’sexpenencewt[h disseminating gwdellnes
 IS


d.$cusse dinour [echmcalcommems
 below.


C-5

particular types of facilities surveyed wouid be more likely to 

implement some guidelines than others. 

AHCPR has issued 16 clinical practice guidelines. Responding to


heavy demana, zhe agency has distributed close to 16 million


copies of these guidelines. We believe, as documented in the


draft report, that the validity and credibility of the AHCPR­


sponsorea guidelines have been major faccors in their growing use


for quality improvement. No organization surveyed cited


disagreement with the content of the guidelines as a reason for


not implementing them. Other evaluations, including the 

companion OIG report presenting case descriptions Of 

organizations that have implemented AHCPR-supported guidelines, 

confirm extensive ado”ptlon of AHCpR-sponsored guidelines. There 

is also substantial evidence that implementation of the 

guidelines by health. care organizations is resulting in


significant improvements in patient outcomes as well as


reductions in health care costs.


The OIG ~~Do~: ~dentifies barriers to guideiine use exper~enced 

by survey respondents. Acknowledging that the A.HCPR guideline 

program is very new, the report notes that introducing clinical


practice guidelines, as with other recommended changes in medical


practice, poses significant challenges. Research on the


implementation of clinical guidelines and, more generally, on the


assimilation into medical practice of new methods, technologies,


or other behavior changes, has consistently shown that change is


generally SICW and requires sustained, carefully-targeted efforts


that directiy involve practitioners and patients.


To achieve the full potential related to implementation of 

practice guidelines we need to find more effeccive ways tO 

promote aaopcion, adaptation, and implementation of the 

guidelines. The OIG’S study was not designed to evaluate AHCPR’S 

internal srccesses for aeveioping or disseminating guidelines, 

nor AHCPR’ s program for evacuating guideline dissemination and


implementazicz strategies. We believe that these evaluative


efforts are crucial if we are to promote greater use of clinical


practice ~Jitielines. As described in our responses to individual


recommenaaci C.-AS
below, AHCPR will continue ro focus and refine


its efforrs Z3 identify the most effecti-~e methods for promoting


guideline im~iementation. AHCPR is also embarked on a strategy


to improve t:.eefficienc~ of the guideli-ne development Process


and to bercer tailor gui~elines to the neeas of local users.


These changes should lead to development ~f guidelines and


guideline-reiated products and implementacicn tools that are more


easily assimil aced into practice by locai ixsci.tutions, and to


more effec:i-:e dissemlnat=Gn of the guiaeiines .


C-4

OIG Recommendation


The Public Health Service (PHS), throuah AHCPR, should determine 

more effective wavs to urornote familiarity with, and use of, the 

guidelines. 

PHS Comment


PHS concurs with this recommendation. As part of its ongoing


efforts to evaluate and improve the guideline development and


dissemination proc’ess, AHCPR sponsors a broad-based portfolio of


research and evaluation urojects that the OIG did noc examine as


part of the current scud;. These efforts include a variety of


projects to assess the effectiveness of alternative strategies to


encourage implementation of guidelines in different settings.


For example, AHCPR has a contract with the American Medical 

Review Research Center (AMRRC) that is comparing the 

effectiveness of alterna~i’ze educational outreach programs 

employed by peer review organizations to promote ado~cion of the 

AHCPR-sponsored guideiine on benign prostacic hyperp~asia. A 

cooperative agreement with the Department of Veterans Affairs 

(VA) will compare the effectiveness of two educational strategies 

for implementing the A.HCpR-sponsored pressure ulcer prevention 

guideline in VA facilities across the United States. Another 

cooperative agreement with the Centers for Disease Cantrol and 

Prevention will assess the effectiveness of targeted media 

campaign strategies in Oklahoma and Massachusetts to dlSSemlnate


the AHCPR-sponsored urinary incontinence guideline.


AHCpR has issued research solicitations to encourage studies Of


how guidelines can most effectively be implemented in large group


practices and in primary care settings. Several research


projects have been funaea in response to these solicitations and


are ongoing. AHC?R aiso,tias a contract through an i:ceragency


agreement with the Office of personnel Management to conduct


focus groups to elici: f=e~ack about guidelines fro~,potential


users, including primary care physicians, physicians in academic


settings, and nurses.


Basea on the findings cf these and other investigati=s, and


subject to the availabilic~y of resources, A.HCPR cent=.ues to


explore alternative scrace~ies for increasing the familiarity of


potential users with pracc~ce guidelines. At the same time, as


part of its plans to refine and improve the guideline development


process, AHCPR is placing greater emphasis on efforts ‘tO


facilitate guideline ::piementation, including devei~<ment of new


types of products (e.g.— , posters, laminated cards) =Za greater


use cf effective modes c: distribution suck as elecr=:n.lc


dissemination. AHCPR aiso plans to expand. partnerships to


deveiop guidelines in collaboration with ocher Feder=- and State


agencies and with pri~.-ateorganizations. These cont~=uing


efforts should help tc increase awareness, acceptance, and use of


the guidelines.


L-J

OIG Recommendation


The PHS, throuqh AHCPR, should make increased technical sup DOrt 

available to midelines users. 

PHS Comment


PHS concurs with this recommendation. Although not the focus of


this OIG study, AHCPR sponsors a number of prOJectS to provide


technical support to prospective users of practice guidelines.


We will continue to do so.


For example, AHCPR is sponsoring a series of reports on using


clinical practice guidelines. The first report, entitled Usinq


Clinical Practice Guidelines to Evaluate Oualitv of Care, was


issuea in March 1995. It discusses issues associated with 

implementing and rneasurlng conformance to practice guidelines and 

Earqecing opportunities for clinical quality improvement. More 

;mporcantly, it is a practical hanabook that lays out step by 

seep instructions for deve~oplng guideline-related quality 

evaluation tools (medical review criteria, performance measures,


and standards of quality) . This report has been widely


disseminated to a range of health care organizations, including


hospitals and medical review organizations. There is


considerable demand for the report from across the nation. For


example, the Group Health Association of America has requested


copies to send to all its member plans.


Other reports to be issued in the near future will provide


detailed descriptions of projects in which AHCPR-supported


guidelines, review criteria, and performance measures have been


used in quality improvement programs. A forthcoming report


focuses on a project in whiclh Peer Review Organizations appiied


neaicai review criteria derived from AHCPR-sponsored guidelines


cn accze ~ain, urinary incontinence, and benign prostatic


hype.rpiasla in Medicare quali~y of care reviews. Another Wiil


tiescribe the results of efforts by the VA to disseminate ana


~inpiem,entthe pressure ulcer prevention guideline.


:2c~~ ~:as supported the development of review Crlterla in


association with three Of the most recent AHCPR-sponsored


guidelines : heart failure, unstable angina, and otitis media.


Review criteria are aiso currentiy being developed for a


guideline on cardiac rehabilitation. These review criteria


snouid serve as practical tool,s :or organizations i.~tereste~ ~~­


impienenting these guidelines.


AiicJ?.R
 is sponsoring a series of efforts to co~~ect, evaluate, 32Ci


ziassi~’Y’performance measures to assist providers and other


keaith.care organizations in the~r quality improvement efforts.


parc ci this project involves li~king performance measures za


.J.HC2R-sponsoreapractice guidelines. This should assist


:.-.SEL
 ‘--Jcions .
 interested in implementing practice guidelines ~C


:nprc>-equality of care.


C-6

AHCPR also provides information and teckn=cal assistance tO state


and local health officials and policy makers through OUr user


Liaison Program. For exampie, a worksnc~ on medical


effectiveness and outcomes research, off=red semi-annuallYt has


included sessions on clinical practice ~::delines, review


criteria, and performance measurement. The sessions focused in


particular on applications in managed care, utilization and


quality review, medical malpractice, ana -;:orkerscompensation.


Subject to the availability of resources, .lHCPR continues to


explore options for providing effeccive :=cMical assistance tO


providers and other institutions interesc=d in implementing


clinical practice guidelines. Opportunities for technical


assistance should be expanded by the new collaborative


partnerships described in response to the previous


recommendation.


OIG Recommendation


The PHS, throuqh AHCPR, should develou and imulement systematic 

mechanisms for obtaininu obiective feedback about auideline use. 

PHS Comment


PHS concurs with this recommendation. Surveys can be an 

effective technique to assess the extent :3 which practitioners 

or patients generally are aware of, have accepted, and are using 

practice guidelines. AHCPR supports developmental work to design 

valid instruments to be used in surveys c: guideline awareness


and utilization. In an effort to leverags resources, AHCPR also


collaborates with a nu~er of outside organizations both in


developmental efforts as well as in enccu==ging these


organizations to include questions relate= za WCpR-sponsored


guidelines in their own routine surve’ys.


In addition to cross-sectional or perica~: surveys, we believe


that it is also important to conduct more zargeted evaluation


efforts, including:


o	 experimental or quasi-experimental s=’~cies of specific


dissemination and implementation stz=:egies;


o	 case studies of the logistics of gu~i=iine implementation in


specific organizational settings; =Zi


o	 fOCUS Gr dlSCUSSIOn grOUpS Gf pOteI?Z23i Users to obtain


feedback on ch,eeffectiveness of spe=~~ic guideline-related


produczs and suggestions for overcc:::g barriers tO


implementation.


Basea on developmental work, AHCPR is spcn=oring a surveY of


patient and practitioner attitudes regarti~ng the A.HCpR-sponsored


guideline on benign prostatic hyperplasia. Results of the survey


C-7

(conducted prior to che implementation of educational programs


designed to encouraae aaopcion of guideline recommendations),


alon~ with results ~f a post-intervention survey wiil be


publlshed later this year.


In response to a question included after consultation with AHCPR


staff, the Group Heaith Association of America’s 1994 Annual HMO


Performance Survey fauna E:hat of those HMOS encouraging guideline


use (82% of all HMOS sumeyed) , half (or 41~) had aaoP~ed Or 

adapted AHCPR-sponsoreci guidelines. Based on a surrey instrument 

deveioped in collaboration with the AHcPR-sponsored guideline 

panel and with AHCPR staff, the National Pressure Ulcer Advisory 

Panel surveyed nurses and other specialists in acute care


settings across the country regarding the AHCPR-sponsored


pressure ulcer prevention guideline. The New Hampshire Medicare


Peer Review Organization and the New Hampshire Foundation for


Medical Care have also sumeyed hospitals and skilled nursing


facilities in New Hampsniye and Vermont regarding the pressure


ulcer guideline.


AHCPR continues to assess the feasibility, costs, ~n~


methodological soundness cf various options for obt=nlng


systematic feedback cn a broad scale regarding the awareness and


use of AHCPR-sponsored practice guidelines. Subject to resource


availability, AHCPR aiso continues to explore options for


gathering additional information related to the most effective


methods to disseminate and implement practice guidelines.


C-8

Technical comments


Executive summary


Page Para Line: Comment


i 2 4 As of this date AHCPR has released 16 clinical 

practice guidelines. 

i 2 6 The paragraph states that in Fy 1994, +fJmilliOn 

was spent on dissemination and evaluat~on of 

guidelines, of which $4 million was spent on 

printing and dissemination. 

During FY 1994, approximately $6 million WaS SPent 

on grants and contracts which focused on the 

evaluation of guidelines. It should be noted that 

FY 1994 was an atypical year for spending on


evaluations. Durng this yearf fourl~r9e scalef


multi-year evaiuacion projects were lnltlatea, SO


that spending reflected projects covering more


than one fiscal year. 

i 3 1 “The introduction of new technologies, including 

new clinical practice guidelines, can be a S1OW 

and difficult process. “ 

AHCPR-sponsored research shows that the


introduction and adoption of new technologies and


medical knowledge is in fact complex, influenced


by factors such as consistency with current


opinions and practices, skills and resources,


necessary to implement new knowledge, flnanc+al


reimbursement incentives, perceived impllcatzons


for litigation, and patient preference. Patient


factors play a large role in the aiff”us~on as well


as the effectiveness of health and ciinical


information, particularly in areas such as


compliance with. tueatment and medication


recommendations .anaadoption of prevention


behaviors.


In addition to publications cited in footnote 14


of the report, AHCPR has sponsored the following


work related to the effective dissemination cf


health and clinical information:


Agency for Heaith Care Policy and Research.


Effective Dissemination of Clinicai and Health


Information: Conference Summary. Secrest L. ,


Baker TE, Rogers EM, Campbell TF, and Grady ML,


editors. AHCPR Pub. NO. 95-0015. Rockville, 14D:


Agency for Health Care Policy and Research, Public

------- --- --

F-------

Health Service, U.S. Department of Health and


Human Services. December 1594.


Amenrv , fn~ w-alth care Polic’y and Research.. . -- - - . . -——­ lcal
 E~fec~ive Dissemination of Heaith and clin,


ourauhv.
 Information to Consumer:s: -notated Bibli~


AHCPR. Pub. —— No. 95-0055. — Rockville, MD: Agency--’ILL. 

=a.1. L-11 for 4ealth rare ___
 policy and Research, Public H{


Service, U.S. -_=----------

Services. May 1995. 

i 4 1 i!weconducted a mall su~ey of 380 key U.S. health 

care ....” 

nenartment of Health and Human


While the OIG survey addressed three important


segments of the health care market, it is


important to note that the particular types of


institutions surveyed are not necessarily


representative oi all health care institutions.


For example, according to data from the American


Hospital Association, less than half of all


hospitals are small non-teaching hospitals. More 

important, these institutions account for less 

than 10% of all inpatient admissions in the U.S. 

Furthermore, according to InterStudy, the majority


of HMO enrollees are served by IPA or network


models rather than staff model HMOS. Staff model


HMOS account for the smallest proportion of


enrollees of all model types. 

5 6 (First bullet on the page) : “Impetus for using 

guidelines has frequently come from organizations’ 

internal quality improvement programs. “ 

In fact the impetus is over’wheirninglyfromTable 2 internal quality improvement interests. 

(Page B-3) notes that 84% (41 of 49) of those 

using the guidelines cited ~his as their reason. 

“External factors--such as directives from


supervising organizations and State law


requirements--have also played an important role. “


Table 2 on page B-3 shows that external ‘actors


play a much smaller role (26 cr 33%), and,among


the reporting organizati~ns, i= is primarl~Y


,nursinghomes clhatreport e.xte~rAalfactors as


playing a role in guideline use. 

ii 45 (Fourth builet on the page) : “~conom+c in~entives 

to encourage clinicians were =Portea bY ‘ew 

organizations. “ 

c-lo

iii


iii


In face economic incentives were reported by only


HMos , and only two (or perhaps 3, depending on


whetker che question asked providea a muitiple


response option) .


(Firsr bullet on the page) : “Clinician resistance 

was the second most frequently cited obstacle. ” 

Clinician resistance is closely followea by


inability to measure guideline use and izability


to measure guideline effect. In total 24


respondents reported physician resistance, while


12 reported these other two difficulties. When


the full table on page B-14 is examined, the


message that seems clear is that tools and 

technical assistance are needed to facilitate the 

implementation of guidelines, the measurement of 

guideline conformance, and examination of the 

effecc ct guideline conformance on eutccmes. 

Sentences 2 and 3 in the paragraph: These two


sentences, which are repeated in other sections ot


the report, are unclear.


One seems to state that 46% were “unfamiliar with 

the guidelines, the other that 40% were 

unfamiliar. 

In aadition, sentence 3 should clarify chat the


focus of the study was only on small non-teaching 

hospitals and HMOS with staff model components. 

Without this distinction, the sentence =Ppears to 

generalize to all hospitals, nursing homes and


HMOS .


iii Z 3 Seccnd,~aragraph under first recommenaac~on: “a 

reconslaeration” of A.HCPR efforts cc prc-,ote 

fami’.iarity with the guidelines. 

We agree that AHCPR should continue ZO sponsor 

research to identify the most effeccive strategies 

for dissemination, lmplementationf ~~oP~~~n and 

diff.~slcn of innovation. However, basec on the


follcwing reasons, we do not believe ZhaZ the OIG


study suDDorts a cOnclUSIOn that AHCF?.’s


dissernin~~ion methods have been ineif=cci-ze.


1) 3.HCP?.’Sdissemination efforts do not stress


Agenc~,’name recognition. The Agency name does not


featcye prominently on the cover of the


guideii~es , and they are frequently described in


the ~eaia and published sources as “federallY


sponsored” guidelines (as o:posed CO “~CPR


guidelines”) . They are produced by paneis of


C-11

private indiviauais convened under the auspices of


the Federal government. Consequently, failure to


achieve name recocrnition should not be equated


with inadequate d~ssemination.


2) The terms l!fami~iarity wi~h guidelines: ad


“guideline use” from the su~ey questionnaire may


have different meanings to different individuals.


Individuals may have seen a guideline, but not


looked at it in detail, or not internalized its


content. Conversely, an organization can have


internalized the recommendations so thoroughly


that they become standard practice, rather than a


guideline produced by an external entity.


3) It should be noted that the rate of guideline 

penetration reported by organizations responding 

to the survey (20% use and 32% including those 

reporting behavioral intent to use the guidelines) 

is consistent with the diffusion of other medicai 

effectiveness information and technologies. For 

example, after distributing its smoking cessation


guideline to 200,000 physicians the American


Medical Association found that only 20% recalled


having received the guidelines. Only 25% of


physicians sumeyed by the AMA reported using the


HIV guidelines issued by AMA.


Hearn, Wayne. “On the Road to Prevention”.


American Medical News. June 5, 1995. P. 9- (A coPY


of this article is provided with this report)


4) Dissemination of clinical information does not


occur rapidly. Antman et. al. report that despite


evidence from randomized controlled clinical


trials of the eifecciveness of therapies and


interventions, it is often many years later that


findings become standard recommendations in


textbooks and ciinical expert reviews appearing in


published literature. For example, findings from


multiple randomized controlled triais indicace


that significance =sductions in,hospital mortality


could be achieved by administration of


thrombolytic drugs ~or patients with acute


myocardial infarczian. However, widespread


recommendation of ~his therapy by clinical experts


did not occur until 13 years after zhe appearance


of data from these zrials. since 1985, ‘wh:n ~~ 

approximate 20% ~eduction in the risk of aeatn. was 

established at ~he ?<.001 ~evel, ~4 of the 43


review articles examined did not mention the


treatment or felt Lz was still experimental.


c-i2 
Antman, Elliott M. et a~. “A Comparison of Results


of Mets-analyses of Randomized Control Trials and


Recommendations of Clinical Experts. ” Journal of


the American Medical A,SsOclatiOn 1992; VO1.268,


No. 2. P. 240. (A copy of the article is


provided. )


We believe that this contextual information is


essential in interpreting the data and the


recommendations presented in the Inspector


General’s report. 

iv 1 1 Systematic mechanisms for obtaining feedback on 

guideline use. 

The suggested strategies for accomplishing this 

recommendation all focus on su~eys to examme 

familiarity with the guidelines. While we agree


that some cross sectional suney work is desirable


to assess penetration of guidelines~ other


possibilities should be considered, e.g.


targeted evaluation efforts, in the form of quasiexperimental or experimental design, to look at 

implementation using specific dissemination and 

implementation strategies. 

case study examination of the logistics Of


guideline implementation in various orgamzational


settings.


user groups to discussion methods for implementing


guidelines and effective methods resolving


barriers to implementation.


targeted examination of guideline products to


assess what factors contribute to their


usefulness .


Clearly a case study approach, such as thatoused by the


OIG can augment survey data and provide addlt~onal 

valuable “feedback” on detailed aspects of the


guideline.


One of the problems with cross sectional surveys is


that instrumentation problems may bias results; without


additional qualitative or quasi experimental researcht
 For
 it may not be possible to detect the biases.


example, the term “’~se”of guidelines may mean


different things to different individuals and


organizations . The i~formation in footnote 19 of the


OIG report specificailv discusses ~he different


interpretations Of “au~deline .
 use”, and how differences


C-13

in the wording of questions may result in different


reported rates of guideline penetration.


- With respect to user feedback forms attached to each


guideline, AHCPR has considered this method for


obtaining feedback. Mailback cards or feedback forms


may be used to offer qualitative suggestions to the


agency via open ended questions. However, their


reliability ana validity for evaluating guideline


penetration or attitudes about the guidelines is


questionable because of sample based response bias. It


is not possible to identify with any certainty the


sample frame, the characteristics of the respondents 

versus nonrespondents, or whether the individuals


respond more than once to a mailback card.


Introduction


Page Para Line


2 1 2 AHCPR was the successor to (rather than the 

replacement for) the National Center for Health 

Semites Research. 

Although AHCPR activities related to health 

senices research are consistent with the mandate 

of NCHSR (including the conduct of the National 

Medicai Expenditure .$u~~eY), AHCpR’s mandate ‘s 

substantially broader than that of NCHSR, 

encompassing: the conduct and sponsorship of 

research, demonstration projects, eva~uations~ 

training, guideline development and d~ssem~nation 

of


ciinical practice guide~ines~


medical effectiver.ess research \inciud~ng


Patient outcomes Research Teams and other


outcomes research projects) ,


dissemination research


and a variety of other =reas. 

2 2 2 As of this date, AHCpR has released 16 clinical 

practic e guidelines. 

2 2 3 “The,~anels apply science based methccs to develop 

specl~i c statements on -management for z.heclinical 

condition under consideration. “ 

It is more accurate to state that 

“The panels use an explicit approach and


meth.oaolog for evaluac inq scientific evidence to


deveicp recommendations reiated to the diagnosis


C-14

2 4 ‘4 

2 4 8 

2 5 5 

2 7 1 

treatment, and management cf the clinical


condition under consideration. In instances. ...”


In addition to establishing a toll-free guideline


request telephone line and availability of


guidelines through the National Librav of


Medicine and the Internet, the Agency has issued


the text of clinical.practice guidelines,


including quick reference guides and patient


booklets, in CD-ROM format. This product is


currently available for use by medical libraries


and will be available to the publlc through the


Government Printing Office in Fall 1995.


AHCPR both conducts and sponsors conferences on


guideline implementation. In addition, the Agency


sponsors targeted and investigator initiated


research on guideline implementation and effective 

methods of disseminating health information.


During FY 1994, four large scale, multi-year


evaluation ”projects were initiated, so that


spending reflected projects covering more than one


fiscal year.


This sentence suggests that the body of research


supported by the Agency has resultea @y in the


identification of barriers to use of guldelmes.


The paragraph implies that the barriers listed in


the third sentence were identified from AHCPR


sponsored research, when they are items from the


OIG survey.


Whiie it is certainly true that barriers to


guideline implementation exist, and that AHCPR


sponsored research has identified some of these


barriers, it is important to note ‘hat ‘CPR’S


research portfolio focuses on a broaa program of


research on the effectiveness of clinical


interventions as well as methods and tools for


transiting medicai effectiveness information into


clinical decision making.


Barriers to guideline implementation are


ider.tifiea in the cantext of research studies to


evaiuate guidelines, to determine effective


clir.ical strategies, :0 identify salient quality


improvement strate~lesf and to disccver ways to


improve the commun~cation of clinicai and health


information.


The statement found in our commencs above (i 3 1)


more correctly characterizes the nature of the


Agency’s research.


C-15

4 1 1&3 This sentences is not entirely clear. At first it 

might appear to say non-teaching hospitals seine 

at the forefront of medical education. Perhaps it 

would be more direct to say: 

I!wesampled non-teaching hospitals because ‘e 

anticipated that they would be more likely to use 

existing guidelines such as those supported by 

AHCPR than to develop their own. In contrast, we


assumed that teaching hospitals, because they


serve at the forefront of medical education, might


be more likely to develop their own guidelines. ” 

4 l&2 1 The sampling strategy employed by OIG may have 

resulted in an underestimation of guideline use. 

We do not find evidence to support the assumptions 

underlying the sampling strate9Y. For example, 

the assumption is that small, non-teaching 

hospitals would be more likely to use nationally


developed guidelines. One might assume the


converse: because teaching hospitals are at the


forefront of medical education, they,would be more 

familiar with AHCPR-supported gu~del~nes, and more 

likely to use them. Many guideline Panel.me~ers


are prominent researchers in academic medical


centers throughout the U.S. ; conse~ently, Such 

centers may be more likely to use nationally 

developed guidelines such as those developed bY


AHCPR .


The second paragraph on this page states that HMOS


with staff model components were selected because


they show a trend toward managed care, and because


staff model plans may be more able to influence


providers. As indicated in a previous comment,


staff model :<MOS are the smallest segment of the 

managed care market; data from InterStudy show 

that IPA/network models are the fastest growing


segment of the market place.


The Group Health Association of America study


cited in footnote 18 notes that in surveying HMOS


of all model types, approximately 33% used AHCPR-


supported guidelines or adapted these guidelines


for use. Consequently, the assumption that -~os


with staff model components would be more likely


to use guidelines developed by AHCPR may not be


correct.


C-16

5 3 2


6 8 1


6 8 1


6 9 3


13 1 2


13 1 2


Parenthetical for pressure ulcers guideline should


be “ (prediction and prevention of pressure


ulcers) “.


This figure does not include organizations that,


in developing their own guidelines, used AHCPR (or


other national guidelines) as a basis.


This finding should be qualified to reflect the 

wording of the OIG questionnaire. As noted in 

footnote 18, the term “use” of,gu+dellnes may mean 

different things to different lndlvlduals and 

organizations. The information in footnote 18 of 

the OIG report specifically discusses the 

different interpretations of “~ideline use”, -d 

how different wording of questions may result in 

different reported rates of use of the gu~delines. 

This information should be presented within the


text of the report.


Footnote 20 should be associated with this


sentence, rather than the first sentence in


paragraph 8.


The information in footnote 24 provides a number 

of examples of AHCPR di.ssem~natlon aCtlVltieS 

including a publications clearinghouse, electronic


dissemination of the guideline, medla,outreacht


and organizational outreach through direct


mailings, public-private partnerships, and


intermediaries (such as opinion leaders, medical


professional associations, continuing education


programs and consumer organizations) . The report


recommends a reconsideration of dissemination


efforrs, but does not adequately recognize ongoing


dissemination activities.


As indicated in the executive summary, sentences 2


and 3 in the paragraph are unclear.


One states that 46% were unfamiliar with the


guidelines, the other that 40% were unfamiliar.


Sentence 3 should ciarify that the survey focused


on smail non-teaching hospitals and ~Os ‘=th


staff model components. The sentence could be


constr.~ed to apply to all hospitals, :~rsing homes


and W40S.


C-17

APPENDIX D


NOTES 

1. One of the first reports on geographic variations in medical practice was published 

in 1938 and focussed on the variation in rates of tonsillectomy among school children 

in England (JA Glover, “The Incidence of Tonsillectomy in School Children,” 

Proceedings of the Royal Society of Medicine, 1938; pp. 1219 -1236.) 

In the past two decades, John Wennberg has published several articles that document 

great geographic variation in medical practice in the United States. In a 1988 article, 

he found that “A resident of New Haven, Connecticut is about twice as likely to 

undergo a coronary bypass operation as is a resident of Boston; for carotid 

endarterectomy, the risks are the other way around, The numbers of knee and hip 

replacements per capita are much more common among Bostonians, while New 

Havenites experience substantially higher risks for hysterectomy and back surgery.” 

(John E. Wennberg, “Commentary: Improving the Medical Decision-Making Process,” 

Health Affairs, Spring 1988; pp. 99- 106.) 

Robert H. Brook and his associates from the Rand Corporation’s Health Program 

have also published on this subject. In a 1986 article, they report that “Of 123 

procedures studies, 67 showed at least threefold differences between sites with the 

highest and lowest rates of use.” (Mark R. Chassin, Robert H. Brook, et al, “Variations 

in the Use of Medical and Surgical Services by the Medicare Population,” New 

England Journal of Medicine; January 30, 1986; pp. 285- 290.) 

2. Institute of Medicine, Committee to Advise the Public Health Service on Clinical 

Practice Guidelines, Marilyn J. Field and Kathleen N. Lohr, editors, Clinical Practice 

Guidelines: Directions for a New Program (Washington DC: National Academy Press, 

1990) p. 8. 

Clinical practice guidelines are also known by other names, including “practice 

parameters, “ “clinical indicators, “ “medical necessity guidelines,” “medical standards,” 

and “practice policies.” (David M. Eddy, “Clinical Decision Making: From Theory to 

Practice, Practice Policies--What Are They?” Journal of the American Medical 

Association; February 9, 1990; pp. 877- 880.) 

The American Medical Association definition of practice parameters refers to 

“strategies for patient management developed to assist physicians in clinical decision 

making.” (Directo~ of Practice Parameters: Titles, Sources, and Updates, 1994 Edition, 

p. 189, Office of Quality Assurance and Medical Review, American Medical 

Association, Chicago, IL.) 

D-1

3.The 1995 Edition of the American Medical Association’s (AMA) Directoty of 

Practice Pararneten lists approximately 1800 practice parameters developed by more 

than 70 physician associations and other organizations. 

4. U.S. Congress, Office of Technology Assessment, Defensive Medicine and Medical 

Malpractice, OTA-H-602, July 1994, pp. 140-148. 

5. “CAPPCARE Developing Wireless Hardware for Instant Guidelines Access,” 

Report on Medical Guidelines& Outcomes Research, 5 May 1994,pp. 4-5. 

6.According to Dr. James Cooley of the Harvard Community Health Plan (a staffmodel HMO based in Boston, MA), guidelines can be implemented through methods 

that are patient directed and include educational sessions (in which either individual 

patients or groups of patients are educated about the guideline) and reminder mailings 

(in either newsletters or focussed mailings on single topics); methods that are directed 

at clinicians and include guideline publication and distribution, continuing medical 

education (CME) sessions, CME take-home quizzes, skills-training programs, and 

academic detailing; administrative methods that include concurrent reminders (such as 

automatic reminders that can be programmed to appear on the computerized charts 

of those patients who have appointments), retrospective reminders (such as lists of 

patients who have not yet received the prescribed intervention), encounter-form 

changes that prompt physicians to ask certain questions or to provide certain services, 

order-form changes that either encourage or discourage the use of specific laboratory 

tests, the designation of personnel to manage and/or provide the specific care, and 

feedback to providers about the extent to which they are complying with the guideline; 

and methods that involve economic incentives, such as bonuses or penalties, and 

changes in reimbursement policies (Telephone conversations with Dr. James Cooley 

on September 14 and 21, 1994). 

7.	 The Agency for Health Care Policy and Research Reauthorization Act of 1992 

(P.L. 102-410) extended the authorization of AHCPR and amended some provisions 

for the development of clinical practice guidelines. 

8. According to P.L. 102-410, the guidelines must be based on the best available 

research and professional judgement; presented in formats appropriate for use by 

physicians, other healthcare practitioners, medical educators, medical review 

organizations, and consumers; presented in treatment-specific or condition-specific 

forms appropriate for use in clinical practice, educational programs, and reviewing 

quality and appropriateness of medical care; include information on the risks and 

benefits of alternative strategies for prevention, diagnosis, treatment, and management 

of the particular health condition(s); and include information on the costs of 

alternative strategies for prevention, diagnosis, treatment, and management of the 

particular health condition(s) where cost information is available and reliable. 

The AHCPR chooses its guideline topics on the basis of several factors: the adequacy 

of scientific-based evidence from which to develop guidelines; the number of people 

affected by a condition; the condition’s amenability to prevention; expected potential 

D-2 
for reducing inappropriate variations in the prevention, diagnosis, management, or 

outcome of the condition or disease; specific needs of the Medicare and Medicaid 

populations; and the costs of the condition to all payers, including consumers 

(AHCPR, Program Note: Clinical Guideline Development, August 1990). 

9.	 Guidelines released to date include Acute Pain Management (issued in 3/92), 

Urina~ Incontinence in Adults (3/92), Pressure Ulcers in Adults--Prediction and 

Prevention (5/92), Cataract in Adults (2/93), Depression in Primary Care (4/93), and 

Sickle Cell Dtkease (4/93), Early HIV Infection (1/94), Benign Prostatic Hyperplasia 

(2/94), Management of Cancer Pain (3/94), Unstable Angina (3/94) Heart Failure (6/94), 

Otitk Media with E~ion (7/94), Quality Determinants of Mammography (10/94), Acute 

Low Back Pain in Adults (12/94], Treatment of Pressure Ulcers (12/94), and Post-stroke 

Rehabilitation (5/95). 

10. See “Guideline Development and Use,” included in each guideline published by 

the Agency. 

11. Among the organizations on AHCPR’S master mailing list are healthcare 

education and credentialing organizations, hospitals, insurance companies, and 

associations and societies for healthcare professionals. 

According to the March 27, 1995 AHCPR Guideline Marketing Summary Reports, 

7,288 articles have been published about the guidelines in newspapers, magazines, and 

trade publications. In addition, 107 guideline reprints, excerpts, or summaries have 

appeared in professional journals. 

12. Among the companies that have issued reprints of the AHCPR guidelines are 

Abbott Laboratories, Aetna, Bristol-Myers Squibb, Burroughs Wellcome, 

Communicore, Dupont Pharmaceuticals, Eli Lilly, Forest Labs, Janssen, Key 

Pharmaceuticals, Knoll Pharmaceuticals, Mead Johnson, Medco, Merck, Pfizer, Purdue 

Frederick, Roche Laboratories, Roxane Pharmaceuticals, SmithKline Beecham, Syntex 

Laboratories, and Zeneca Pharmaceuticals. 

13. As examples of its dissemination efforts, AHCPR reports that it conducts direct 

mailing of its guidelines to Health Resources and Services Administration grantees, 

including community, rural, and migrant health centers and Area Health Education 

Centers, and to Indian Health Service healthcare providers. In addition, the agency 

has collaborated with the National Cancer Institute (NCI) to distribute its guidelines 

on the management of cancer pain and mammography through the NCI clearing 

house. Similarly, it has collaborated with the Centers for Disease Control to distribute 

the HIV guideline through that agency’s clearing house. The AHCPR has also worked 

with the Food and Drug Administration to distribute the mammography guideline to 

mammography facilities and with the Health Care Financing Administration to 

distribute the guidelines on acute pain management, urinary incontinence, and 

pressure ulcer prevention to hospitals and nursing homes. 

The AHCPR research projects also affect dissemination. Among the disseminationD-3 
related projects is one that involves the Medicare Peer Review Organizations in an 

effort to develop, implement, and evaluate quality and utilization review criteria for 

three guidelines. Another research project involves an interagency agreement with the 

Department of Veterans Affairs to test various dissemination strategies for introducing 

the pressure ulcer prevention guideline. 

14. Fiscal Year 1996 Budget, p. 49, U.S. Department of Health and Human Services; 

and Fzkcal Year 1996 Budget Justification to OMB, p. 25, U.S. Department of Health 

and Human Services. Additional budget information was provided by AHCPR to OIG 

in a memorandum from Jill Bernstein (OPD/AHCPR), dated April 25, 1995. 

15.Please see the following articles: 

Ann L. Greer, “The State of the Art Versus the State of the Science: The Diffusion 

of New Medical Technologies into Practice,” International Journal of Technology 

Assessment in Health Care, v. 4, 1988, pp. 5-26. 

Jonathan Lomas et al., “Opinion Leaders vs Audit and Feedback to Implement 

Practice Guidelines: Delivery After Previous Cesarean Section,” Journal of the 

American Medical Association, May 1, 1991, pp. 2202-2207. 

Peter J. Greco and John M. Eisenberg, “Changing Physicians’ Practices,” New England 

Journal of Medicine, October 21, 1993, pp. 1271-1274. 

Roberto Grilli et al., “The Impact of Patient Management Guidelines on the Care of 

Breast, Colorectal, and Ovarian Cancer Patients in Italy,” Medical Care, January 1991, 

pp. 50-63. 

Jacqueline Kosecoff et al., “Effects of the National Institutes of Health Consensus 

Development Program on Physician Practice,” Journal of the American Medical 

Association, November 20, 1987, pp. 2708-2013. 

Margaret VanAmringe and Terry E. Shannon, “Awareness, Assimilation, and 

Adoption: The Challenge of Effective Dissemination and the First AHCPRSponsored Guidelines” Quality Review Bulletin, December 1992, pp. 397-404. 

16.Institute of Medicine, Committee on Clinical Practice Guidelines, Marilyn J. Field 

and Kathleen N. Lohr, editors, Guidelines for Clinical Practice: from Development to 

Use, (Washington DC: National Academy Press, 1992). A discussion of the ways in 

which different factors affect guideline use can be found in chapter four, beginning on 

page 83. 

17.Among the projects currently funded by AHCPR is one to assess whether aspects 

of the guideline-development process are associated with user perceptions of the 

validity, reliability, cost, and utility of the guidelines; another to model the cost impact 

of various guideline recommendations; and others that focus on the development of 

D-4

medical review criteria based on guideline recommendations. Additional projects 

examine variables related to the acceptance and implementation of specific guidelines. 

Other projects seek to document changes in practice and patient outcomes stemming 

from guideline use in primary care. 

18. In contrast, we anticipated that teaching hospitals, because they serve at the 

forefront of medical education, might be more likely to develop their own. 

19. We are 90 percent confident that the percentage of organizations that have used 

the guidelines is between 16 and 24. 

20.	 In 1994 HMO Pe~orrnance Repoti, the Group Health Association of America 

(GHAA) summarizes the results of its recent survey of a representative sample of 100 

of its member HMOS (the Association achieved a 71 percent response rate to this 

survey). The GHAA found that 82 percent of its respondents (approximately 58 

HMOS) encourage their providers to follow specific clinical practice guidelines and 

that 40.8 percent of this subset (approximately 24 HMOS) have adapted guidelines 

from those published by AHCPR. Thus, GHAA found that approximately 33 percent 

(.82 x .408) of its respondents have adapted guidelines from those published by 

AHCPR. 

By contrast, our survey of all 80 HMOS with staff-model components found that 20 

percent have used one or more of the AHCPR guidelines about which we inquired. 

The difference between the findings may be accounted for by several factors: 

1. The GHAA surveyed a sample of all of its member HMOS, while we 

surveyed only the 80 HMOS that have staff-model components. 

2. The GHAA inquired about the use of AHCPR guidelines in general, while 

our survey focused on the use of six specific AHCPR guidelines. 

3. The GHAA posed three broad questions about guideline use, while we 

asked twelve pages of detailed questions. 

4. The GHAA’s survey question asked whether or not the organizations had 

developed their own guidelines and, if so, whether or not these were based on the 

AHCPR guidelines; while we asked if organizations had “used’ the AHCPR guidelines 

(respondents may have interpreted “used” as meaning direct application of the 

AHCPR materials and as excluding the adaptation of these materials for local use). 

21. Certain guidelines may be of greater relevance than others to certain types of


healthcare provider organizations. One would not expect that all guidelines would be ‘


used by all organizations. For example, one would not expect the guideline on sickle


cell disease to be used by many nursing homes for the aged.


D-5

22. The AHCPR reports that its guidelines have been distributed to every hospital 

and nursing home in our survey sample. Substantial evidence demonstrates that the 

introduction of new technologies, including new clinical practice guidelines, is a slow 

and difficult process. See end note # 15 for a partial listing of relevant articles. 

23. Such materials include the Clinical Practice Guideline, the Patient’s Guide, and/or 

the Quick Reference Guide for Clinicians. 

24.Examples of automated computer systems include those that provide clinicians 

with on-line access to guidelines or guideline summaries; those for concurrent 

reminders that can be programmed to appear on the computerized charts of patients 

who have appointments; those for formularies; and those for prescription, lab- or testorders. 

25.An example of a retrospective reminder is a list of patients who have not yet 

received the guideline-recommended care. 

26.Information Diweminahon to Health Care Practitioners and Policymakers: 

Annotated Bibliography, Agency for Health Care Policy and Research (AHCPR), 

Public Health Service, U.S. Department of Health and Human Services, April 1992, 

p.1. 

27.The AHCPR’S efforts in this domain include both its User Liaison Program 

meetings and a recently released guide for organizations entitled “Using Clinical 

Practice Guideline to Evaluate Quality of Care.” 

D-6