Department of Health and Human Services 
OFFICE OF

INSPECTOR GENERAL

CLINICAL PRACTICE  GUIDELINES

SPONSORED  BY  THE

AGENCY  FOR  HEALTH  CARE  POLICY

AND  RESEARCH 
EARLY
 EXPERIENCESINCLINICALSETTINGS

e $ 
g  JUNE  GIBBS  BROW 
< 
2  Inspector  General 
6 
%++
%d~~  September  1995
> 
OEI-01-94-00250  TABLE  OF  CONTENTS

PAGE 
EXECUTIVE  SUMMARY 
INTRODU(XION  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1

FINDINGS  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  5

...0.  .  .  .  .  .  . 
The  Extent  ofGuideline  Use  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  5

Ways in Which  the  Guidelines  Have  Been  Used  . . . . . . . . . . . . . . . . . . . . . . . .  7

Organizations  Assessments  oftheir  Guideline  Use  . . . . . . . . . . . . . . . . . . . . .
 10 
Obstacles  to Guideline  Use  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
 11 
RECOMMENDATIONS  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
 13 
COMMENTS  ON THE  DRAFW REPORT  . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
 15 
APPENDICES

Z& Nonrespondent  Analysis  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..A-l

B:
 Summary  of Mail Survey Results  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..B-1

c  Complete  Comments  onthe  Draft  Report  . . . . . . . . . . . . . . . . . . . . . . . . . .  C-1

D:
 Notes  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..D-1
 EXECUTIVE  SUMMARY

PURPOSE 
This  study  describes  the  early  experiences  of healthcare  organizations  in using  clinical 
practice  guidelines  sponsored  by the  Agency  for Health  Care  Policy and  Research. 
BACKGROUND 
Over  70 organizations,  including  Federal  agencies,  medical  professional  associations, 
health  insurance  associations,  and  health  maintenance  organizations,  have  sponsored 
approximately  1800 clinical  practice  guidelines.  The  Agency  for Health  Care  Policy 
and  Research  (AHCPR)  has released  sixteen  guidelines.  Of the  $154 million 
appropriated  for AHCPRS  fiscal year  1994 programs,  $5 million  was spent  on 
developing  and  updating  guidelines,  and  $10 million  on dissemination  and  evaluation. 
Of this,  approximately  $4 million was spent  on printing  and  disseminating  guidelines. 
The  introduction  of new technologies,  including  new clinical  practice  guidelines,  can be 
a slow and  difficult  process.  We  prepared  this report  in response  to AHCPRS  interest 
in gaining  a better  understanding  of the  extent  to which,  and  the  manner  in which, 
healthcare  organizations  have  been  using its guidelines. 
We  conducted  a mail  survey of 380 key U.S. healthcare  organizations,  including  150 
nursing  homes;  150 small, nonteaching  hospitals;  and  80 health  maintenance 
organizations  that  have  staff-model  components.  We  focused  on these  organizations 
because  they  offer  prime  opportunities  for use of the  AHCPR  guidelines.  Sixty-four 
percent  responded.  Our  survey focused  on the  first six AHCPR  guidelines  released 
between  March  1992 and  April  1993:  pain  management,  urinary  incontinence, 
pressure  ulcer  prevention,  cataracts,  depression,  and  sickle cell disease. 
FINDINGS 
llventy pement  of the survey respondents  reprted  that  they have  wed  I or more of the 6 
AHCPR  guidelines about which we inquii~  an addtionul  12 punt  reprted  that they 
plan  to do so. 
	 Impetus  for using the  guidelines  has frequently  come  from  the  organizations  internal, 
quality-improvement  programs.  External  factors--such  as directives  from  supervising 
organizations  and  State  law requirements--have  also played  an important  role. 
	 The  organizations  expectations  of guideline  use have  focused  on improved 
clinical  care.  Many  organizations  have  also viewed  the  guidelines  as a means  of 
reducing  costs and  providing  better  protection  from  malpractice  litigation. 
i  .	 The  reasons  most  often  cited  for not  having  used  the  guidelines  were  that  the 
organizations  were  not  familiar  with them  (40 percent  of all survey 
respondents)  and  the  organizations  have  their  own guidelines  (34 percent  of all 
sumey  respondents).  No organizations  cited  disagreement  with the  guidelines 
as a basis  for not  having  used  them. 
Ninety-sir pement  of the su.rwy resyxmdknfi that  have used 1 or more of the 6 AHCPR 
guidiliizes reprted  that their #Torts to do so have focused  on clinicians; 36percent 
reprted  that they huve directed guideline-irnplementzition #orts  towanik patients. 
Clinician-Directed  Efforts 
.	 Clinician-education  efforts--ranging  from  the  distribution  of guideline  materials 
to the  conduct  of educational  programs--were  the  most  frequently  reported 
means  of implementing  the  guidelines.  Such efforts  were  reported  by most 
guideline  users. 
.	 The  development  of an algorithm,  policy and  procedure,  or other 
implementation  tool  to guide  clinician  practice  was also reported  by most 
guideline  users. 
.	 Economic  incentives  to encourage  clinicians  to provide  guideline-recommended 
care  were  reported  by very few organizations. 
Patient-Directed  Efforts 
.	 The  distribution  of guideline  materials  to patients  was the  most  frequently 
reported  patient-directed  effort. 
  Educational  sessions  for patients  were  reported  by some  organizations. 
At  this ear&point  in the irnpkmentation  process, only 8 percent  of the respondents that 
have used 1 or more of the 6 AHCPR  guidelines reported that they have measured  the 
efects  of their guideline use. 
.	 These  measures  have  focused  on patients  clinical  outcomes  and  clinicians 
performance.  They  have  been  based,  primarily,  on chart  and  record  reviews. 
Sixty-three percent  of the respoti  that reptied  huving used 1 or more of the 6 
AHCPR  guidklinix also reprted  having encountered obstacles to doing so. 
.	 Uncertainty  about  how to implement  the  guidelines  was the  most  frequently 
cited  obstacle  to use. 
ii  .	 Clinician  resistance  was the  second  most  frequently  cited  obstacle.  We  learned 
that  physicians  have  been  concerned  about  what  they  have  perceived  to be 
cookbook  medicine  and  that  nurses  have  been  concerned  about  what  they  have 
anticipated  would  result  in increased  administrative  and  patient  care  workloads. 
RECOMMENDATIONS 
%  l%blic Health  Service (PHS), through AHCP~  should &terrnine more efective  ways 
to pmnote  famdimity  with  and use oj  the guidelines. 
The  AHCPR  reports  that  its guidelines  were  distributed  to every hospital  and  nursing 
home  in our  sample  population.  Of all those  in leadership  positions  that  we surveyed 
in small,  nonteaching  hospitals;  nursing  homes;  and  HMOS with staff-model 
components;  80 percent  reported  that  their  organizations  had  not  used  the  AHCPR 
clinical  practice  guidelines  and  40 percent  were  unfamiliar  with the  guidelines. 
We recognize  that  introducing  new clinical  practice  guidelines,  like any effort  to 
introduce  systematic  changes  in the  delivery  of healthcare,  poses  challenges.  A 
reconsideration  of AHCPRS  efforts  to promote  familiarity  with, and  use  of, the 
guidelines  could  yield more  effective  approaches  to meeting  these  challenges. 
The PHS, through AHCP~  shouki make  increased technical supp~  avaikzble to 
guideline Usem  . 
Our  findings  demonstrate  that  most  organizations  that  have  used  the  AHCPR 
guidelines  have  encountered  obstacles  to doing  so.  Many  organizations  reported  that 
uncertainty  about  how to use the  guidelines  has been  an obstacle.  Other  frequently 
noted  obstacles  include  resistance  from  physicians  and  nurses,  an inability  to measure 
compliance  with the  guidelines,  and  an inability  to measure  the  effects  of guideline 
use.  To  facilitate  the  use  of the  AHCPR  guidelines,  the  PHS,  through  AHCPR, 
should  strengthen  its capacity  to advise  those  organizations  that  are  encountering 
obstacles.  For  example,  the  Agency  could: 
	 identi~  and  disseminate  benchmarking  practices  of those  organizations  that 
have  overcome  obstacles  and  made  effective  use of guidelines;  and/or 
.	 build  on its current  programs  and  conduct  training/technical  assistance  sessions 
in which  representatives  of healthcare  organizations  can learn  more  about  the 
guidelines  and  how to use them  effectively. 
lhe PHS, through AHCP~  shouki develop and irnpkment  ystematik  rnechaniwns for 
obtaining objective feedback  about guideline use.  As pat  of tfi  efo~  AHCPR  should 
sponsor regular surveys of healkare  oganuations.  We  surveys shouki be dksi@ed  to 
p&e  pmjectable  indicatom that albw  AHCPR  to gauge changes over the  k  the 
tixtent of gukidine  use. 
.,. 
111
 Although  AHCPR  collects  some  information  about  the  use  of specific  guidelines,  it has 
not yet incorporated  into  its regularly  scheduled  activities  efforts  to gather  information 
on the  frequency  with which AHCPR  guidelines  are  used  or user-feedback  information 
from  those  organizations  that  have  experience  with the  guidelines.  To better  support 
the  Agency  in its efforts  to gauge  the  level of guideline  use and  to continuously 
improve  the  quality  of its guidelines,  the  PHS,  through  AHCPR,  should  incorporate 
systematic  efforts  to gather  objective  information  about  guideline  use into  its ongoing 
responsibilities. 
To  complement  this important  research,  AHCPR  could  conduct  other  types  of 
ongoing  efforts  designed  to provide  information  about  users experiences  with  the 
AHCPR  guidelines  and  changes  over  time  in the  users experiences.  For  example,  the 
Agency  could: 
	 conduct  regular  surveys of organizations  that  have  requested  the  guidelines  to 
obtain  feedback  about  their  experiences  with, and  assessments  of, the  guidelines; 
	 conduct  regular  telephone  interviews  with select  key users to obtain  more  in-
depth  feedback;  and/or 
  attach  user-feedback  forms  to all guidelines  that  are  disseminated. 
We  hope  that  our  report  till  provide  information  that  is useful  to AHCPR  in the  near 
term  as it considers  ways to facilitate  the  implementation  of its guidelines.  We  also 
hope  to provide  useful  information  for organizations  that  are  seeking  ways to benefit 
from  guideline  use,  and  for Congress  as it considers  AHCPRS  future  efforts.  A 
companion  report,  Early Experiences with Clinical Practice Guidelines Sponsored by the 
Agenq  for Health  Care Policy and Research:  Case Descriptions (OEI-01-94-O0251), 
presents  short  case  descriptions  that  illustrate  the  experiences  of several  organizations 
with  the  AHCPR  guidelines. 
COMMENTS  ON THE  DRAFT  REPORT 
We  solicited  and  received  comments  on our  draft  report  from  the  Public  Health 
Service  (PHS).  The  full text  of these  comments  is included  in appendix  C. 
The  PHS  offered  technical  comments  on our  draft  report  and  these  were  considered 
as we developed  the  final report.  Below we address  the  key elements  of the  PHS 
general  comments  on our findings,  methodology,  and  recommendations. 
llndings:  The  PHS  characterized  our  first finding--that  32 percent  of the  survey 
respondents  reported  that  they have  used,  or plan  to use,  1 or more  of the  AHCPR 
guidelines  about  which we inquired--as  extremely  encouraging.  We recognize  that 
introducing  new clinical  practice  guidelines  is a challenging  endeavor,  but  found  no 
guidance  in the  scientific  literature  for determining  what  level of use  constitutes 
iv  success.  The  AHCPR  has not yet established  benchmarks  for guideline  use  or 
quantified  expectations  against  which to measure  its success. 
Methoddogy  The  PHS  Stated  that  the  types  of organizations  surveyed  in our  study 
are  not  representative  of all healthcare  institutions.  We  agree.  We  continue  to 
believe,  however,  that  these  organizations  are  important  components  of this nations 
healthcare  delivery  system  and  that  clinical  practice  guidelines  are  relevant  to the 
provision  of care  in all three  settings.  We  are  considering  additional  research  to 
examine  the  use of clinical  practice  guidelines  in other  healthcare  settings  and 
encourage  AHCPR  to do the  same  in our third  recommendation. 
Recomrnendutins:  In its comments  on our  recommendations,  PHS  cited  various  of its 
current  efforts  that  it regards  as relevant  to the  recommendations.  We recognize  the 
relevance  and  importance  of a number  of the  activities  cited  by PHS. 
We  are  concerned,  however,  about  our  third  recommendation,  in which we urge  PHS 
to develop  and  implement  systematic  mechanisms  for obtaining  objective  feedback 
about  guideline  use.  This recommendation  proposes  regular  surveys as an important 
tool  for  continuous  quality  improvement.  To  clari~  the  recommendation,  we have 
included  in the  recommendation  statement  information  that  was formerly  presented  in 
the  text  supporting  it. 
We  continue  to believe  that  systematically  collected  feedback,  gathered  at regular 
intervals,  from  current  and  potential  guideline  users,  is a critical  component  of any 
continuous  quality  improvement  effort.  This feedback  can be  used  as a valuable 
performance  indicator  addressing  the  degree  of AHCPRS  success  in facilitating  the 
use  of, and  familiarity  with, its guidelines. 
Our  report  finding--that  20 percent  of survey respondents  reported  that  they  have 
used  1 or more  of the  6 AHCPR  guidelines  about  which we inquired--provides  a 
baseline  against  which AHCPR  can measure  its progress  in achieving  greater 
familiarity  with,  and  use  of, its guidelines. 
v  INTRODUCTION

PURPOSE 
This  study  describes  the  early  experiences  of healthcare  organizations  in using clinical 
practice  guidelines  sponsored  by the  Agency  for Health  Care  Policy and  Research. 
BACKGROUND 
Vatitions  in Medical  Bactice  and Clinical hzctice  Guidelines 
Variations  in clinical  practice  among  different  physicians  and  hospitals,  and  in different 
geographic  areas  have  long been  observed .1 With  the  growing  awareness  of these 
variations,  concern  has  developed  about  adverse  patient  outcomes  and  financial  costs 
associated  with inappropriate  medical  care.  Clinical  practice  guidelines  have  emerged 
as part  of a larger  effort  to reduce  unwarranted  variations  in care  and  the  costs 
associated  with them. 
According  to the  Institute  of Medicine,  clinical practice  guidelines  are  systematically 
developed  statements  to  assist practitioner  and  patient  decisions  about  appropriate 
healthcare  for specific  clinical  circumstances.2  Such guidelines--and  similar 
materials--have  been  sponsored  by a multitude  of organizations,  including  the  Agency 
for Health  Care  Policy and  Research  (AHCPR),  the  Centers  for Disease  Control,  and 
the  National  Institutes  of Health,  all within  the  Public  Health  Service  (PHS)  in the 
U.S. Department  of Health  and  Human  Services.  The  U.S. Preventive  Services  Task 
Force,  the  Rand  Corporation,  medical  professional  associations,  health  insurance 
associations,  and  health  maintenance  organizations  have  also developed  guidelines.3 
Guidelines  address  a broad  range  of topics  and vary along  several  dimensions, 
including  the  breadth  of the  topic  addressed,  the  level of detail  of care  dealt  with, the 
extent  to which  the  guideline  is directive,  the  extent  to which  scientific  evidence  forms 
the  basis  for  guideline  recommendations,  the  format  in which the  guideline  is 
presented,  and  the  method  by which the  guideline  is developed. 
Guidelines  have  been  used,  or tested  for use, in several  different  applications  and  to 
achieve  several  different  goals.  The  Office  of Technology  Assessment  has reported 
that  clinical  practice  guidelines  are  being  used  in malpractice  litigation  and  as a means 
of reducing  defensive  medicine. 4  Primary-care  providers--such  as hospitals,  health 
maintenance  organizations,  and  nursing  homes--use  guidelines  in care  provision  and 
quality  assurance  reviews.  At least  one  managed  healthcare  provider--with  networks 
in 36 States--is  testing  the  use  of computer  technology  that  allows physicians  instant 
access  to clinical  practice  guidelines,  including  those  sponsored  by AHCPR.5  At least 
one  health  maintenance  organization  has  developed  a typology  of guideline 
implementation  methods.b 
1
 lhe  Agency for Health  Care Policy and Reseaxh 
The  Omnibus  Budget  Reconciliation  Act  of 1989 (P.L.  101-239) established  AHCPR 
as the  successor  to the  former  National  Center  for Health  Services  Research  and 
Health  Care  Technology  Assessment.  Congress  assigned  the  new Agency  the 
responsibility  to enhance  the  quality,  appropriateness,  and  effectiveness  of healthcare 
services  and  access  to such services,  through  the  promotion  of improvements  in clinical 
practice  and  in the  organization,  financing,  and  delivery  of healthcare  services.7 
As part  of its activities  to fulfill its mandate,  the  Agency  arranges  for the  development 
and  distribution  of clinical practice  guidelines.8  To  date,  the  Agency  has  released 
sixteen  guidelines--and  each  is produced  in several  formats,  including  the  full-length 
clinical  practice  guideline,  a quick reference  guide  for clinicians,  and  patient  guides.g 
To  develop  the  guidelines,  the  Agency  convenes  independent,  multidisciplinary  panels 
of private-sector  clinicians  and  other  experts  that  collaborate  over  an extended  time 
period.  The  panels  apply  explicit,  science-based  methods  to develop  specific 
statements  related  to the  diagnosis,  treatment,  and  management  of the  clinical 
condition  under  consideration.  In instances  in which the  scientific  literature  is 
incomplete  or inconsistent,  the  professional  judgment  of the  panel  members  and 
consultants  is reflected  in the  guideline.l 
The  AHCPR  employs  a multi-pronged  strategy  to disseminate  its guidelines.  In 
addition  to distributing  all of its guidelines  to the  more  than  10,000 organizations  on its 
master  mailing  list, the  Agency works with the  medical,  trade,  and  lay media  to 
generate  and  place  articles  about  guidelines. 11 It has  established  a toll-free, 
guideline-request  telephone  line through  the  AHCPR  publications  clearinghouse  and 
has begun  to make  its guidelines  electronically  available  through  the  National  Library 
of Medicine  and  the  Internet.  It also works with professional  societies  and  commercial 
organizations  to reprint  and  further  distribute  guidelines. 12 Among  its other 
dissemination  activities,  the  Agency  conducts  conferences  for those  interested  in 
learning  more  about  guideline  implementation  and  the  experiences  of those 
organizations  that  have  used  AHCPRS  guidelines.13 
Of the  $154 million  appropriated  for AHCPR  in 1994, $81 million  was dedicated  to 
the  Agencys Medical  Treatment  Effectiveness  Program.  Approximately  $5 million  of 
this sum was spent  on the  development  and  updating  of guidelines  and  approximately 
$10 million  was spent  on dissemination  and  evaluation  efforts.  Of this, approximately 
$4 million  was spent  on printing  and  disseminating  guidelines.14 
Concerns about  Clinical Practice Guidelines 
An increasing  number  of organizations--including  healthcare  providers,  health  insurers, 
State  governments,  and  medical  professional  associations--have  become  involved  in the 
development  of clinical  practice  guidelines;  and  an increasing  number  of clinicians 
have  become  aware  of them.  The  introduction  of new technologies,  however--including  new clinical  practice 
guidelines--can  be  a slow and  difficult  process. 15 In research  supported  by AHCPR 
and  other  organizations,  and  in conferences  about  clinical practice  guidelines,  barriers 
have  been  identified  to the  greater  acceptance  and  more  extensive  use  of 
guidelines.lb  Such barriers  include  confusion  about  how best  to use guidelines; 
resistance  from  administrators,  clinicians,  and/or  patients;  administrative,  bureaucratic, 
and/or  technological  problems  in either  preparing  for or adhering  to guidelines;  the 
expense  of guideline  implementation;  and  limitations  in organizations  ability  to 
measure  compliance  with guidelines  and  the  effects  of guideline  implementation. 
THE  OBJECTIVES  OF  THIS  REPORT 
The  AHCPR  already  supports  many  projects  to evaluate  the  development  and 
dissemination  of its clinical  practice  guidelines.  In addition,  the  Agency  sponsors 
research  on broader  questions  related  to guidelines  and  their  use.17  Many  of these 
projects  are  multi-year,  long-term  efforts.  We prepared  this report  in response  to the 
agencys interest  in gaining  a better  understanding  of the  extent  to which,  and  the 
manner  in which,  healthcare  organizations  have  been  using its guidelines. 
We  hope  that  our  study will yield information  that  is useful  to AHCPR  in the  near 
term  as it considers  ways to facilitate  the  implementation  of its guidelines.  We  also 
hope  to provide  useful  information  for organizations  that  are  seeking  ways to benefit 
from  guideline  use,  and  for Congress  as it considers  AHCPRS  future  efforts. 
Accordingly,  this report  describes  how organizations  have  used  AHCPR  clinical 
practice  guidelines  and  illustrates  ways that  other  organizations  might  use  them.  We 
do not  evaluate  the  effectiveness  of guideline  use, test  differences  among  various  types 
of organizations  guideline  use, or distinguish  the  strengths  and weaknesses  of specific 
AHCPR  guidelines,  Furthermore,  we do not  evaluate  AHCPRS  dissemination  efforts, 
its internal  processes  for developing  or disseminating  guidelines,  or its processes  for 
evaluating  guideline  use. 
The  report  provides  initial  estimates  of three  dimensions  of guideline  use:  (1) the  extent 
to which  healthcare  organizations  use the  guidelines,  (2) the ways in which  organizations 
use the  guidelines,  and  (3) the  obstacles  that  organizations  have  encountered  in their 
efforts  to implement  the  guidelines.  A companion  report,  Ear& Experiences  with Clinical 
Practice  Guidelines Sponsored by the Agency for Health  Care Policy and Research:  Case 
Descriptions  (OEI-01-94-00251)  presents  a series  of short  case  descriptions  that  illustrate 
the  experiences  of several  organizations  with the  AHCPR  guidelines. 
METHODOLOGY 
Our  report  is based  on data  from  two primary  sources.  First,  to develop  initial 
estimates  of guideline  use, we conducted  a mail survey of three  types  of healthcare 
provider  organizations--150  randomly  selected  nursing  homes,  150 randomly  selected 
nonteaching  hospitals  with no more  than  100 patient  beds,  and  all 80 of the  health 
maintenance  organizations  (HMOS)  with staff-model  components.  The  hospital  and 
nursing  home  samples  were  drawn  from  the  Health  Care  Financing  Administrations 
On-line  Survey  Certification  and  Report  (OSCAR)  System.  The  HMO  sample  was 
3  taken  from  the  report  of the  most  recent  HMO  survey conducted  by the  American 
Managed  Care  and  Review  Association  (AMCRA)  Foundation. 
We included  nonteaching  hospitals  in our  sample  because  we anticipated  that  they 
would  be  more  likely to use existing guidelines,  such as those  sponsored  by AHCPR, 
than  to  develop  their  own.18 We further  defined  our hospital  sample  to include  only 
those  nonteaching  hospitals  with no more  than  100 beds  because  we anticipated 
methodological  difficulties  in obtaining  mail-survey  responses  from  the  appropriate 
staff in large  hospitals  with many,  large,  specialized  departments. 
We  included  HMOS with staff-model  components  in our  survey because  HMOS 
represent  a trend  towards  managed  care.  In addition,  staff-model  HMOS, because 
they  employ  physicians  directly  and  exclusively, are  probably  better  able  than 
independent-practice/physician  association  HMOS, network-model  HMOS,  or group-
model  HMOS  to influence  the  provision  of care  to their  patients.  Accordingly,  we 
anticipate  that  staff-model  HMOS, more  than  the  other  HMO  models,  might  seek  to 
use clinical  practice  guidelines. 
Sixty-four  percent  of our  sample  completed  and  returned  the  survey.  Our  analysis  of 
those  organizations  that  did not  respond  to the  survey indicates  that,  for those 
variables  considered,  our  results  are  unbiased  with respect  to differences  between 
respondents  and  nonrespondents  (see  appendix  A for more  information  on the 
nonrespondent  analysis).  The  survey instrument  focused  on the  organizations  use  of 
the  first six AHCPR  guidelines  that  were  released  between  January  1, 1992 and 
January  1, 1994.  Th~se  B for a summary  of 
the  survey results). 
Second,  to increase  our  understanding  of the  organizations  experiences  with  the 
AHCPR  guidelines,  we conducted  telephone  interviews  with half of the  survey 
respondents  that  reported  guideline  use. 
We  conducted  our  review  in accordance  with the  Quali~  Standards for Inspections 
issued  by the  Presidents  Council  on Integrity  and  Efficiency. 
4
 FINDINGS

Twenty  percent  of the  survey respondents  reported  that  they  have  used  1 or more  of

the  6 AHCPR  guidelines  about  which we inquire~  an additional  12 pereent  reported

that  they  plan  to do so.

Of the  243 organizations  that  responded  to our  questionnaire,  49 (20 percent)

reported  that  they  had  used  at least  1 of the  6 clinical practice  guidelines  that  AHCPR

released  between  January  1, 1992 and  January  1, 1994.19 An additional  28 (12

percent)  indicated  that,  although  they have  not yet used  any of the  6 AHCPR

guidelines,  they  intend  to do so.  Several  of these  organizations  indicated  that  they  are

in the  early  phases  of developing  guideline  implementation  plans  and  programs.n

Thirty-seven  (76 percent)  of the  49 AHCPR  guideline  users  reported  having  used

more  than  1 of the  AHCPR  guidelines.  Pressure Ulcers In Adults  (prevention),  Urinary

Incontinence  in Adults,  and Acute  Pain Management  were  the  guidelines  most

frequently  used.21  Thirty-two  (65 percent)  of the  AHCPR  guideline  users  reported

that  they  had  also used  guidelines  sponsored  by other  organizations  to address

AHCPR  guidelines  topics.

	 Imptus  for using the guidelines has frequently come ~m  the organizations 
hterna~  qua@y-hpnwment  programs.  External facton--such  as directiws @m 
supervking o~anizations  and State regukitory requirements--have also plkyed  an 
important  rule. 
Internal Factors:  Quality-improvement  initiatives  were  cited  as having  contributed  to 
the  decision  to use ~CPR  guidelines  by 41 (84 percent)  of the  user  organizations. 
Examples  of the  ways in which internal  factors  have  contributed  to the  decision  to use 
the  AHCPR  guidelines  include  the  following: 
  One  hospital  incorporated  two of the  guidelines  into  a program  for evaluating 
physicians  clinical  performance. 
  Another  hospital  decided  to use  a guideline  after  the  death  of a patient  who  had 
been  admitted  with severe  pressure  ulcers. 
  A nursing  home  adopted  a guideline  after  a product  sales representative  provided 
three  copies  of the  guideline  and  suggested  that  the  home  use it. 
External Factors:  An  external  factor  was cited  as having  contributed  to the  decision  to 
use the  AHCPR  guidelines  by 22 (45 percent)  of the  user  organizations.  Examples  of 
the  ways in which  external  factors  have  contributed  to the  decision  to use  the  AHCPR 
guidelines  include  the  following: 
5
 F  Eight  nursing  homes  adopted  one  of the  guidelines  in response  to deficiency 
citations  from  State  nursing  home  surveyors  or in response  to the  homes  concerns 
about  meeting  survey standards. 
  An HMO  began  using a guideline  because  the  Medicare  Peer  Review  Organization 
(PRO)  recommended  adherence  to the  guideline  as a means  of resolving  a patient 
complaint. 
E  Two nursing  homes  decided  to use two guidelines  because  their  parent  companies 
encouraged  them  to do so. 
	 The o~anizations  expectations of guidkline use have focused  on iinpnnwd cllnical 
care.  Many have aiko vikwed the guidelines as a means of reduciizg costs and 
providing better protection jiorn  rnu@actice  litigation 
Improved  Clinical Care:  Forty-seven  (96 percent)  of the  users  reported  that  they have 
expected  the  AHCPR  guidelines  to help  them  achieve  at least  one  goal  related  to 
improved  clinical  care.  Such goals ranged  from  improved  clinical  outcomes  for 
patients,  to reduced  uncertainty  among  clinicians  about  appropriate  care  for patients, 
reduced  variation  in clinical  decision-making,  and  increased  patient  satisfaction. 
Other Goals:  Twenty-five  (51 percent)  of the  user  organizations  reported  having 
expected  to reduce  costs  associated  with  clinical care.  Twenty  (41 percent)  reported 
that  they  have  expected  to achieve  better  protection  from  malpractice  litigation. 
	 lhe  reasom  most often cited for not havhg  used the *W  were that the 
organizations were not familiar  &h  them and the oqynizations  have their own 
guidelines.  No organizations cited dikagreernent with the guideti  as a bti  for 
not having used them. 
One-hundred  and  ninety-four  (80 percent)  of all respondents  to our  survey indicated 
that  they  have  not  used  any of the  six AHCPR  guidelines.  Although  16 (8 percent)  of 
them  indicated  that  these  guidelines  are  not  clinically relevant  for their  institutions, 
none  of the  organizations  indicated  disagreement  with the  care  recommended  in the 
AHCPR  guidelines. 
Lack  of Familiarity:  Ninety-seven  respondents--50  percent  of the  nonusers  (and 
40 percent  of all respondents)  --reported  that  they were  not  familiar  with the 
guidelines.z2  In their  comments  on the  mail  survey, 27 (14 percent)  of the  nonusers 
requested  copies  of the  AHCPR  guidelines  and/or  expressed  an interest  in learning 
more  about  them. 
Other Guidelines:  Eighty-two  respondents--42  percent  of the  nonusers  (and  34 percent 
of all respondents)  --reported  that  they have  their  own guidelines.  Thirty-four 
respondents--18  percent  of the  nonusers  (and  14 percent  of all respondents)  --reported 
that  they  use  guidelines  developed  by an organization  other  than  AHCPR. 
6  Ninety-six pereent  of the survey respondents  that have used 1 or more  of the 6 
AHCPR  guidelines reported  that their efforts to do so have focused on clinici~, 
36 percent  reported  that they have directed  @deline-implementation  efforts towards 
patients. 
Clinician-Directed  Efforts 
	 Clinician-education  @o*-ranging@m  thq distribution of guidellne matetitzik to 
the condkct  of t?ducatz%malprograms--werethe most frequently repited  means of 
irnpkmentihg  the guiiiklihes.  Such #ens  were reprted  by most guideline usem 
Forty-two  (86 percent)  of the  guideline  users  reported  at least  one  educational  effort

directed  at clinicians.  Most  frequently  cited was the  distribution  of AHCPR  guideline

materials  to clinicians,  which was reported  by 28 (57 percent)  of the  users.~

Examples  of the  ways in which organizations  have  distributed  AHCPR  guideline

materials  include  the  following:

  One  nursing  home  put  up posters  summarizing  the  guideline  on each  of its units. 
F  An HMO  distributed  to physicians  its own summary  of the  AHCPR  guideline. 
E  At  another  HMO,  copies  of the  guideline  materials  were  distributed  at meetings  of 
the  medical  and  nursing  staff. 
  A hospital  distributed  copies  of the  guideline  to each  of its nurses,  physicians,  and 
physician  assistants,  placed  additional  copies  at the  nursing  station  and  in the  hospital 
library,  and  circulated  a hospital-wide  memorandum  announcing  the  availability  of all 
of the  guidelines. 
Educational  programs  have  been  offered  by 24 (49 percent)  of the  users.  Such 
programs  have  involved  train-the-trainer  efforts,  in-service  activities,  and  self-study 
video  tapes.  Examples  of the  ways in which  organizations  have  used  the  guidelines  in 
these  efforts  include  the  following: 
F  A nursing  home  sent  nurses  to off-site  classes on a guideline.  These  nurses  then 
provided  training  for other  staff on the  recommended  diagnostic  and  care  techniques. 
  A hospital  hosted  a product-sales  companys  full-day training  session  for nurses  and 
luncheon  lectures  for physicians  on guideline-recommended  care. 
  Another  hospital  conducted  a series  of five, mandatory,  in-service  training  sessions 
for its nurses. 
  A nursing  home  hired  a consultant  to train  small groups  of nurses  about  the 
guideline  recommendations.  b  A hospital  used  a privately  produced  video  series  explaining  the  guideline  to train 
its nurses  and  nursing  assistants. 
In addition,  14 (29 percent)  of the  users  reported  that  they  provided  feedback  to 
clinicians  about  the  extent  to which their  care  conforms  to guideline  recommendations. 
At most  organizations,  this feedback  has been  provided  informally,  either  to 
individuals  or to staff-meeting  groups.  A few organizations,  however,  have  provided 
more  formal  feedback  to their  clinicians: 
E  A hospital  has provided  feedback  to clinicians  in reports  that  note  variances 
between  the  guideline-recommended  care  and  that  provided  by clinicians. 
*  A nursing  home  has displayed  weekly  charts  for each  of its units  documenting 
success  rates  in managing  the  guideline  condition;  the  home  also has run  a weekly 
contest  (with prizes)  among  the  units  to encourage  staff to better  manage  the 
condition. 
	 The dkveloprnent of an algorit~  polky  and pmcedurq  or other impbnentation 
tool to@&  clinician practice  was aiko reprted  by most guidkline usem 
Thirty-six  (73 percent)  of the  users  reported  having  made  at least  one  administrative 
effort  to implement  the  guidelines.  Most  often,  organizations  adapted  the  guidelines 
for their  own use by developing  an implementation  tool:  31 (63 percent)  of the  users 
reported  such an effort. 
E  At  one  hospital,  a multidisciplinary  task  force  worked  over  the  course  of six 
months  to incorporate  the  guideline  recommendations  into  existing policy and 
procedures  and  to develop  an algorithm  for patient  care  corresponding  to the 
guideline. 
>  At  a nursing  home,  over the  course  of a month,  the  director  and  assistant  director 
of nursing  revised  the  homes  existing policy and  procedures  to reflect  the  guideline; 
the  medical  director  approved  the  new draft. 
  A manager  at another  hospital  developed  a two-page  summary  of the  guideline, 
which  the  hospital  physicians  and  governing  board  then  approved. 
  Another  hospital  collaborated  with an area-wide  group  of care  providers  over  the 
course  of a year  to adapt  the  guideline  for a uniform  standard  of community  care. 
Other  administrative  efforts  to implement  the  guidelines  have  included  the  use  of 
automated  computer  systems  that  reflect  guideline  recommendations,  U changes  to 
clinician-patient  encounter  forms,  changes  to lab- or test-order  forms,  and 
retrospective  reminders  for clinicians .X  Twenty-five  (51 percent)  of the  users 
reported  having  made  at least  one  such effort.  Examples  include  the  following: 
8
   A hospital  has  used  its computerized  patient-care  system  to suggest  appropriate 
care  for treatment  of the  guideline  condition  and  to allow monthly  reviews  of clinician 
compliance  with guideline  recommendations. 
  Another  hospital  implemented  a five-point,  self-reported  pain-scale  for adults  and  a 
five-point  color  scale,  on which red  represents  the  most  severe  pain  and white 
represents  the  absence  of pain.  The  color  scale has been  useful  with children. 
  Two hospitals  developed  new clinician-patient  encounter  forms  to be  used  on a 
daily basis  for assessment  of the  guideline  condition. 
  Another  hospital  developed  a new culture-order  form  with expanded  order  options 
for the  guideline  condition. 
  Three  organizations  instituted  charting  systems to help  nurses  remember  to  adhere 
to guideline  recommendations. 
	 Ecommuc incentives to encourage chicians  to pmvkle  guitklinerecommended 
care were reported by very fw  o~anizations. 
Only three  organizations  (6 percent)  reported  that  they have  used  economic  incentives 
to encourage  clinicians  to provide  guideline-recommended  care.  Two  of these 
organizations  implemented  changes  in reimbursement  policy.  In one  HMO,  clinicians 
who fail to provide  the  guideline  recommended  care  risk losing their  care  provider 
contracts  with the  HMO. 
Patient-Directed  Eftorts 
  l%e dWibution  of guideline material  to patierm  was the mostjhquently  repotied 
patientdi.rected  ejfott and was reprted  by 12 (24 pement)  of the usem 
Educational  s&ns  for patients  were reported by 9 (I8 penxmtj  of the mers. 
Most  users  reported  that  they have  not  directed  efforts  to implement  the  guidelines 
towards  patients.  Some  users  explained  that  their  patients  are  unable  to help 
implement  the  guidelines  because  of physical  and/or  mental  limitations.  A few 
organizations  anticipated  that  they would  attempt  patient-directed  efforts  in the  future. 
Examples  of ways in which the  AHCPR  guidelines  have  been  used with patients 
include  the  following: 
  An HMO  developed  an article  for its patients  newsletter  about  the  organizations 
new policy for treatment  of the  guideline  condition;  the  article  invites  patients  to  call 
for additional  information. 
  A hospital  implemented  a new emergency  room  form  that  explains  its policy for 
treatment  of the  guideline  condition. 
9
   Another  hospital  has provided  educational  sessions  on the  guideline  condition  for 
its patients  and  for visitors  to senior  centers  and  health  fairs. 
E  A nursing  home  has explained  the  guideline  recommendations  to those  of its at-risk 
residents  who are  able  to participate  in their  own care. 
F  A nursing  home  and  a hospital  have  both  distributed  guidelines  to patients  and 
their  families  when  the  patients  have  been  discharged  for home  care. 
  A hospital  and  a nursing  home  have  used  the  guideline  in home-health  classes  to 
educate  both  patients  and  their  families  about  prevention  and  care  of the  guideline 
condition. 
At this early  point  in the  implementation  process,  only 8 percent  of the respondents 
that have used 1 or more  of the 6 AHCPR  guidelines reported  that they have 
measured  the effects of their guideline use. 
The  first AHCPR  guideline  was issued  in March  1992; two others  were  issued  in that

year.  The  remaining  three  guidelines  about  which  our  survey inquired  were  issued  in

1993.  Although  most  of our  survey respondents  are  in the  early  stages  of

implementing  the  guidelines,  four  reported  that  they have  evaluated  the  effects  of

their  guideline  use.  An additional  four  organizations  that  we interviewed  reported  that

they  have  measured  or plan  to measure  the  effects  of their  guideline  use.

9  l%ese measures have f~ed  on patients clinkal  outcomes  and clinicians 
pe@onnance.  Thqv have been based prirn@v  on chart and reconi revhvs. 
The  organizations  have  assessed  both  clinicians performance  and  patients  clinical 
outcomes.  A few organizations  have  also measured  patient  satisfaction;  a few have 
taken  organization-wide  measurements  of the  effects  of their  guideline 
implementation.  Below  are  examples  of the  assessments  that  have  been  made. 
  A hospital  has used  its computerized  patient-record  system to assess  how care  was 
provided  and  patient  outcomes.  In addition,  the  hospital  has conducted  post-discharge 
patient-satisfaction  surveys.  These  reviews  documented  increased  compliance  with the 
guidelines  and  increased  patient  satisfaction. 
  A nursing  home  has  conducted  weekly  record  reviews, which have  demonstrated  a 
decrease  in the  incidence  and  severity  of pressure  ulcers. 
  An HMO  is planning  to conduct  an audit  of clinician  compliance  with the  guideline 
in the  care  of all patients  treated  for cataracts.  The  HMO  also plans  to collaborate 
with its PRO  to review  claims  data  for a random  selection  of patient  records. 
10
 E  A  coalition  of hospitals  and  nursing  homes  took  base-line  measurements  of the 
incidence  of the  guideline  condition  in several  facilities.  The  coalition  plans  to take 
comparison  measurements  in these  facilities  six months  after  the  implementation  of 
the  guideline  and  one  year  after  that. 
  A nursing  homes  quality  assurance  program  has involved  seven  annual,  in-depth 
surveys  of care  and  patient  status,  and  one  annual,  in-depth  review by the  parent 
company.  Reports  of all of the  reviews, which  address  care  for the  guideline 
condition,  are  evaluated  by the  regional  administrator  and  the  corporate  specialist  on 
the  guideline  topic. 
>  Another  home  has  conducted  quarterly  assessments  of patient  outcomes  for the 
guideline  condition.  This homes  parent  company  has  also provided  an  annual  peer 
review  which  addresses  care  provided  for this condition. 
>  Another  hospital  has conducted  quarterly  studies  to assess how care  is provided  and 
patients  clinical  outcomes. 
Sixty-three  percent  of the respondents  that reported  having used one or more  of the 
six AHCPR  guidelines also reported  having encountered  obstacles  to doing so. 
Although  there  are  many  examples  of guideline  implementation  ((see  Eady 
Experiences with Clinical Practice Guidelines Sponsored by the Agency for Health  Care 
Policy and Research:  Case Descriptions (OEI-01-94-00251)),  introducing  new  clinical 
practice  guidelines,  like any effort  to introduce  systematic  changes  in the  delivery  of 
healthcare,  poses  challenges. 
	 Unmtainty  about how to implement  the guidelines was the most jkquently  cited 
obstacle  to use. 
Twenty-one  (43 percent)  of the  guideline  users  reported  that  such uncertainty  was an 
obstacle  to their  use of the  guidelines.  Many  of the  organizations  that  we interviewed 
struggled  over  the  development  of a plan  to implement  the  guidelines.  Examples  of 
organizations  uncertainty  about  how to implement  the  guidelines  are  summarized 
below. 
  A nursing  homes  staff has found  it difficult  to read  and  understand  the  entire 
guideline.  The  home  has  sought  out  seminars  that  address  the  guidelines  to resolve 
this problem. 
E  Uncertainty  about  how to use a diagnostic  tool  recommended  by the  guideline 
hampered  one  hospitals  implementation  program.  This hospital  contacted  AHCPR 
for further  guidance. 
  For  another  hospital,  confusion  about  the  extent  to which it need  comply  with the 
guideline  slowed  the  implementation  process.    Another  hospital  was initially  confused  about  how to implement  the  guidelines  and 
found  that  other  hospitals  were  unable  to provide  assistance  because  they were 
unfamiliar  with the  guidelines. 
  Another  hospital  reported  that  the  guidelines  are  very general.  Adapting  them  for 
local  use--by  making  them  more  specific--required  a considerable  investment  of both 
time  and  money. 
E  Developing  a strategy  for ensuring  staff compliance  with the  guideline 
recommendations  was a time-consuming  challenge  for another  hospital. 
  Clinician resistance was the second most fhquently  cited obstacle. 
Clinician  resistance  was reported  by 16 (33 percent)  of the  guideline  users.  Physicians 
primary  concern,  according  to those  we interviewed,  has been  their  perception  that  the 
guidelines  are  cookbook  medicine.  Nurses  have  been  concerned  that  the  guidelines 
would  result  in increased  administrative  and  patient  care  workloads.  Examples  of 
clinician  resistance  to the  guidelines  are  summarized  below. 
  A nursing  home  and  two hospitals  reported  that  several  of the  physicians  resented 
what  they  perceived  to be  government  interference  in their  work. 
>  A nursing  homes  staff initially resisted  the  new policy and  procedures  because  they 
anticipated  that  these  would  result  in too much  work.  Some  nurses  continued  to 
refuse  to comply,  but,  according  to the  Director  of Nursing,  most  learned  that  the  new 
rules  ultimately  result  in less work. 
>  Three  nursing  homes  and  a hospital  all noted  that  any change  in prescribed  practice 
is always difficult  for clinicians  and  that  staff training  is a gradual  and  continuing 
project. 
Other  obstacles  to guideline  use that  were  noted  by survey respondents  include  an 
inability  to measure  compliance  with the  guidelines,  an inability  to measure  the  effects 
of guideline  use, and  the  expense  of guideline  implementation. 
12
 RECOMMENDATIONS

%  RMc  Heakh  Service (PHS), through AHCP~  shouiii deternuke more @ective  ways 
to promote  farnilidy  with  and use 0$ the guidelines. 
The  AHCPR  reports  that  its guidelines  were  distributed  to  every hospital  and  nursing 
home  in our  sample  population.  Of all those  in leadership  positions  that  we surveyed 
in small,  nonteaching  hospitals;  nursing  homes;  and  HMOS with staff-model 
components;  80 percent  reported  that  their  organizations  had  not  used  the  AHCPR 
clinical  practice  guidelines  and  40 percent  were  unfamiliar  with the  guidelines. 
We  recognize  that  introducing  new clinical practice  guidelines,  like any effort  to 
introduce  systematic  changes  in the  delive~  of healthcare,  poses  challenges.  A 
reconsideration  of AHCPRS  efforts  to promote  familiarity  with, and  use  of, the 
guidelines  could  yield more  effective  approaches  to meeting  these  challenges.  As 
AHCPR  itself  has noted,  the term  effective  dissemination  includes  the  concept  of 
diffusion  of knowledge  and  information  as well as the  acceptance,  inculcation,  and 
utilization  of disseminated  information...  distribution  of information  alone  is 
insufficient  to ensure  adoption  or use.x 
The PHS, through AHCP~  should make  increased technical support availizble to 
guliii?h%eUsem 
Our  findings  demonstrate  that  most  organizations  that  have  used  the  AHCPR 
guidelines  have  encountered  obstacles  to doing  so.  Many  organizations  reported  that 
uncertainty  about  how to use the  guidelines  has been  an obstacle.  Other  frequently 
noted  obstacles  include  resistance  from  physicians  and  nurses,  an inability  to measure 
compliance  with  the  guidelines,  and  an inability  to measure  the  effects  of guideline 
use.  To  facilitate  the  use of the  AHCPR  guidelines,  the  PHS,  through  AHCPR, 
should  strengthen  its capacity  to advise  those  organizations  that  are  encountering 
obstacles.  For  example,  the  Agency  could: 
	 identi~  and  disseminate  information  on the  benchmarking  practices  of those 
organizations  that  have  overcome  obstacles  and  made  effective  use  of 
guidelines;  and/or 
.	 build  on its current  programs  and  conduct  training/technical  assistance  sessions 
in which  representatives  of healthcare  organizations  can learn  more  about  the 
guidelines  and  how to use them  effectively.27 
13
 7ZWPHS, through AHCP~  shouki dkveibp and implement  systematic mechati  for 
obtahing  objective feedback  about guidklike use.  AS part of this #ofl  AHCPR  should 
sponsor reguku surveys of heakhcare o~anizations.  We  SLUVeysShouki be desi@wd to 
=:f@?Z~WY  that allbw AHCPR  to gauge changes over time m the 
. 
Although  AHCPR  collects  some  information  about  the  use of specific  guidelines,  it has 
not  yet incorporated  into  its regularly  scheduled  activities  efforts  to gather  information 
on the  frequency  with which AHCPR  guidelines  are  used  or user-feedback  information 
from  those  organizations  that  have  experience  with the  guidelines.  To better  support 
the  Agency  in its efforts  to gauge  the  level of guideline  use and  to continuously 
improve  the  quality  of its guidelines,  the  PHS,  through  AHCPR,  should  incorporate 
systematic  efforts  to gather  objective  information  about  guideline  use into  its ongoing 
responsibilities. 
To  complement  this important  research,  AHCPR  could  conduct  other  types  of 
ongoing  efforts  designed  to provide  information  about  users experiences  with  the 
AHCPR  guidelines  and  changes  over  time  in the  users experiences.  For  example,  the 
Agency  could: 
	 conduct  regular  surveys of organizations  that  have  requested  the  guidelines  to 
obtain  feedback  concerning  their  experiences  with  and  assessments  of the 
guidelines; 
	 conduct  regular  telephone  interviews  with select  key users to obtain  more  in-
depth  feedbaclq  and/or 
  attach  user-feedback  forms  to all guidelines  that  are  disseminated. 
14
 COMMENTS  ON  THE  DRAFT  REPORT

We  solicited  and  received  comments  on our  draft  report  from  the  Public  Health 
Service  (PHS).  The  full text of these  comments  is included  in appendix  C of this 
report,  The  PHS  offered  technical  comments  on our  draft  report,  and  these  were 
considered  in the  development  of our final report.  Below, we respond  to some  of the 
key elements  of the  PHS  general  comments  on our findings,  methodology,  and 
recommendations. 
FINDINGS 
The  PHS  characterized  our first finding--that  20 percent  of the  sumey  respondents 
reported  that  they  have  used  1 or more  of the  AHCPR  guidelines  about  which we 
inquired  and  that  an additional  12 percent  reported  that  they  plan  to do so--as 
extremely  encouraging,  given the  context  of traditional  rates  of diffusion  of new 
information  into  health  care  practice. 
We recognize  that  introducing  new clinical practice  guidelines  is a challenging 
endeavor,  but  found  no guidance  in the  scientific  literature  for determining  what  level 
of use  constitutes  success.  The  AHCPR  has not yet established  benchmarks  for 
guideline  use  or quantified  expectations  against  which to measure  its success. 
METHODOIXIGY 
The  PHS  stated  that  the  types  of organizations  surveyed  in our  study are  not 
representative  of all healthcare  institutions.  The  PHS  suggested  that,  to obtain  a 
complete  picture,  additional  types  of providers  should  be  surveyed. 
We  continue  to believe  that  small teaching  hospitals,  staff-model  HMOS,  and  nursing 
homes  are  significant  components  of this nations  healthcare  delivery  system  and  that 
clinical  practice  guidelines  are  relevant  to the  provision  of care  in all three  settings. 
We  agree  that  guidelines  are  relevant  to healthcare  delivery  in other  organizations.  In 
response  to PHSS concern  about  obtaining  a complete  picture,  we are  considering 
additional  research  to examine  the  use of ~linical practi~e  guidelines  in additional
healthcare  settings  and  encourage  AHCPR  to do the  same  in our  third 
recommendation. 
RECOMMENDATIONS 
The  PHS  concurred  with each  of the  three  recommendations  presented  in our  draft 
report.  In each  case,  PHS  cited various  efforts  that  it has underway  that  it regards  as 
relevant  to the  recommendations.  We recognize  the  relevance  and  importance  of a 
number  of the  activities  cited  by PHS. 
15
 We  are  concerned,  however,  about  our  third  recommendation,  in which we urge  PHS 
to develop  and  implement  systematic  mechanisms  for obtaining  objective  feedback 
about  guideline  use.  This recommendation  proposes  regular  surveys as an important 
tool  for  continuous  quality  improvement.  To clari~  that  recommendation,  we have 
included  in our  recommendation  statement  information  that  was formerly  presented  in 
the  text  supporting  it. 
We  continue  to believe  that  systematically  collected  feedback,  gathered  at regular 
intervals,  from  current  and  potential  guideline  users,  is a critical  component  of any 
continuous  quality  improvement  effort.  This feedback  can be  used  as a valuable 
performance  indicator  addressing  the  degree  of AHCPRS  success  in facilitating  the 
use  of, and  familiarity  with, its guidelines. 
Our  report  finding--that  20 percent  of the  survey respondents  reported  that  they  have 
used  1 or more  of the  6 AHCPR  guidelines  about  which we inquired--provides  a 
baseline  against  which AHCPR  can measure  its progress  in achieving  greater 
familiarity  with,  and  use  of, its guidelines. 
16
 APPENDIX  A

NONRESPONDENT  ANALYSIS 
An important  consideration  with research  based  on surveys of the  type we have 
conducted  is the  bias  that  may be introduced  into  the  results  if the  nonrespondents 
differ  from  survey respondents  in systematic  ways.  To test  for the  presence  of bias, we 
obtained  information  from  AMCRA  on the  HMOS to which  questionnaires  were  sent. 
In addition,  we obtained  information  from  the  OSCAR  file for all nursing  homes  and 
hospitals  to which  questionnaires  were  sent. 
Because  the  samples  for HMOS, hospitals,  and  nursing  homes  were  selected  from 
different  sampling  frames,  a separate  nonrespondent  analysis was performed  for  each 
of these  groups.  The  variables  used  in the  analysis for HMOS were  the  number  of 
lives covered  and  the  percentage  of the  HMO  practice  that  was staff model.  For  both 
hospitals  and  nursing  homes,  the  variables  used were  size (number  of patient  beds) 
and  type  of ownership  (profit  vs. nonprofit). 
For  HMOS,  the  t-statistic  was used  to compare  means  for each  of the  two variables  for 
respondents  vs. nonrespondents.  For  hospitals  and  nursing  homes,  the  t-statistic  was 
used  to compare  the  mean  number  of patient  beds.  The  Chi-square  statistic  was used 
to test  differences  by type  of ownership. 
The  results  are  presented  in the  tables  below.  Our  analysis  of those  organizations  in 
the  sample  who did not  respond  indicates  that  our results  are  unbiased  with respect  to 
the  variables  considered. 
HMos 
T-TESTFORSIZEOFHMO 
N  Mean  S.E. of Mean 
Respondents  47  110,239  18,864 
Nonrespondents  33  66,804  16,251 
t=  1.649  prob > t=  .103 
T-TESTFORPERCENTOFHMO PRACTICETHATWASSTAFFMODEL 
N  Mean  S.E. of Mean 
Respondents  47  61.80  4.747 
Nonrespondents  33  70.39  5.834 
t =  1.147  prob > t=  .255 
A-1  HOSPITALS 
T-TESTFORNUMBEROFPATIENTBEDSINHOSPITAL 
N  Mean  S.E. of Mean 
Respondents  101  51.62  2.595 
Nonrespondents  49  51.41  3.557 
t=  0.048 prob > t=  .962 
CHI-SQUAREFORTYPEOFHOSPITALOWNERSHIP 
Respondents  Nonrespondents  Total  % Respond 
Non-Profit  49 (49%)  25 (51%)  74  66% 
Proprietary  5  (5%)  3  (6%)  8  62% 
Government  47 (47%)  21 (43%)  68  69% 
Overall  101  49  150  67% 
CHI-SQ.=.22  DF=2 
NURSING HOMES 
T-TESTFORNUMBEROFPATIENTBEDSINNURSINGHOME 
1, 
II 
N  Mean  S.E. of Mean 
I 
IIRespondents  95  126.95  8.725 
I 1  I 
Nonrespondents  55  114.73  8.488 
t=  1.004  Prob > t=  .317 
CHI-SQUAREFORTYPEOFNURSINGHOMEOWNERSHIP 
Respondents  Nonrespondents  Total  % Respond 
Profit  62 (65%)  43 (78%)  105  5970 
Non-Profit  33 (35%)  12 (22%)  45  73% 
Overall  95  55  150  63% 
CHI-SQ=  2.768  DF= 1 
A-2
 APPENDIX  B 
SUMMARY  OF  SURVEY  RESPONSES 
All data presented  in this appendix are derived from an OIG mail survey that was conducted in 
December of 1994. 
B-1 
_  Tabk..k  AHcPR  Maii %rvey and

Telephone k-s-
GpF~m

Dist@mw  fhlrveyRespondents,

Ckideiine Users,md  @I&~UH

thatwere~  ,

	 ~~~ 
~pqp~i
Surveys Distributed 
~fibafi 
Survey Respondents 
101  67  95  ~~  ~~ 
Respondents  that Reported  Use of the  19  @  ~fl  Q*
AHCPR Guidelines* 
Guideline Users that we Intervkwed  ml
17  AZ  1~ 
*  We are 90 percent confident that the percentage of small, nonteaching hospitals that have 
used the guidelines is between  12 and 25; that  the percentage  of nursing  homes  that  have used 
the  guidelines  is between  14 and 28; and that  the percentage  of HMOS with staff-model 
components is between 11 and 31.  For all organizations represented  in our sample, we are 90 
percent confident that the percentage that have used the guidelin~  is between 16 and 24. 
B-2
 Hiiidk

Tabk 2  llm oumbm and.pimxmtageof 
urganizkwionstit  cited tkf@km@g:factoris 
iq havingcontri%uted:tatheir dixisi~  to 
mthe  maw  @idelines!
-+ 
a.  State law requirements  2  10 9 45 2 20 13 26 
b.  Insurance requirements  15151  1036 
c.  A directive from a supervising or  3  16  10 50  3  30  16 33 
overseeing organization 
d.  Your organizations internal, quality- 17  9(I  14  70 10  100  41  w 
improvement initiative 
e.  OTHER  factors  3 16  2 10 2  20  7 14
BIIE 
In this and in following tables, as noted:  because respondents checked as many items as 
were appropriate  in response to the question, the columns total more than 100 percent. 
In this and in following tables, as noted:  total percentages are sometimes more or less 
than 100 due to rounding. 
B-3  ~L__L_J~~

a.  Increased  patient  satisfaction  17  90  11  55  6  60  34  69 
b.  Improved  clinical outcomes  for patients  17  90  18  90 9  90 44 90 
c.	 Reduced uncertaintyamongclinicians  16  84  13  65  9  9(3  38  78 
about appropriate  care for patients 
d.  Reduced variation in clinical decision- 16  84  11  55  10  100  37  , 76

making

e.  Reduced costs associated with clinical care  10   53  9  45  6  60  25  51

f.  Better protection from malpractice

litigation brought against the organization or  10  53  5  25  5  50  20  41

its providers

g. OTHER  goals  15  0  Q  1  10  2  ,4

EKI  WI

Number of Goals  N 
B-4
 i 
~====== 
a.  Acute  Pain  Management

b  Urinary  Incontinence  in Adults  HEl?RE

Hospital  Nursing  Home  HMO  TOTAL 
N=19  N=20  N=1O  N=49 
II  o  II  o 
1~1  t  II 
II 
II  5 
TOTAL  19  W  II  2a 
3This includes  full implementation  and pilot project use. 
B-5  tiahddad~ition
toAmPRgtd
-wde.oOvthreH@ek~mrq@diazblay
trolcd thl&obiispdeetk&kexdd*$ e
- UHML
Table&  me  mnnberandperqemageof 
organizationsthai reported 
~tifiati
a.  Acute Pain Management  7  37  7  35  2  20 16 33 
b.  Urinary Incontinence in Adults  5  26  15  75  1  10  21  43 
c.  Pressure Ulcers in Adults  6  32  12  60  2  Xl  20  41 
d.  Cataract in Adults  2  10  1  5  2  20  5  10 
e.  Depression in Primary Care  o  lo  7  M  4  40  11  22 
f.	 Sickle Cell Disease  0(1  o  0  0000 
$  ~~~  IE 
B-6
 ~abk 10.  me number  and percentage of 
organiza- that reported h@iig. @ 
A@2PR  guideline  in &eMowing  Morts
LllM
dirmted a@atie& 
~&6%JG~l 
a.  Mailingpatients reminders-to seek

guideline-recornrnendedcare--either as part of 
1  ~  1  5  1  10  3 6

a newsletter  or in a more  directed  mailing

b.  Distributingcopies of the Pa-s  Guide 
12  63  0  Q  ()  l-J 12 24

to patients

c.  Conductingeducationalsessionsfor 
patients about the guideline-recommended 
6  32 3 15 () ()  9 18

care

d.  ConductingOTHER efforts intendedto

encourage patients to seek guideline-
2  10 1 5  0 (-J 3  6

recommended  care 
/ 
B-7
 ~MHMM
mmigmm
a.  Distributingcopies of the CZinicaRl zxZice  12 63  8  40 2 20 22 45 
GuMeilnes  to clinicians 
b.  Distributingcopies of the QuickR@mnce  14  74.  5  z  2  20  21  43 
Guidefw  CliniciarLSto clinicians 
c.  Distributingcopies of the Patient%Guide to  15  W ~ 5  25  2  20 22 45 
clinicians 
d.  Providingformal continuingeducation for  7  37:  11  55    o  u  18 37 
clinicians  on guideline-recommended  care 
e.  Conductingskillstrainingfor clinicianson  9  47  11  55  0  u  20 41 
guideline-recommended care 
f.	 Ccmducti.ngacademic detailing for clinicians  3  16  2  IQ  o  0  5  10

about guideline-recommended care

g.  Providing feedback to clinicians about the

extent to which their care conforms to  3  16  9  45 ~ 2  20  14  29

guideline recommendations

h.  Conducting OTHER  efforts intended to

encourage clinicians to provide guideline- 2  10 ~  4  20  4  40  10  20

recommended care 
1

B-8
 Table 13.  I&  ~umber ad  pergatage  of organ&@ium that reported having used 
AH~R  @&l@es  in the .i@w&d  lmmllerof @ducatkmLde&hrt&dixee&&at.@ni@ms 
==lMmmm 
B-9  
r 
Table 14.  The number and.perc@age  of 
: c3rg@zatk?tlathat &ported hm@g:u$ed 
J%FXXXR:guidojjne(s).lmthe Mkiw&ig 
Z@ii&?mtite  Z&W@e&?*giti.eEm-ta
-,+~ 
BBL 
~fi=  EZ 
a.  Developing a clinical pa@ algori~  or

other guidelineimplementationtool has@  at  12  63  13  65  6  60  31  63

least in pm  on AHCPR guidelines

10

b.(cid:1)Institutingautomated computer systems  2  10  3  15  0 0  5 : 
that refleet guidelinerecommendations

c.  Institutingretrospective remindmsfor  1  5  2  10,  1  10  4  8

Cliniciam+

d.  Makingchangesto clinician-patient

encounter forms that encourage cliniciansto  9  47  6  30  1  10  16  33

provide the guideline-recommendedcare

e.(cid:1)Makingchangesto lab-or test-orderforms

that encourage cliniciansto providethe  2  10  4  2Q  o u  6 12

guideline-recommended  care

f.  Mandatingadherence to the guideline  1  5  3  15.  4  40  8  ~ 16

g.  InstitutingOTHER  administrativeor

technologicalchangesthat encourage  1  5  3  15.  1  10  5  :  10

cliniciansto provide the guideline

recommendedcare

4Examples  of such automated  computer  systems include  those  that  provide  clinicians with 
on-line  access to guidelines  or guideline  summaries;  those  for concurrent  reminders  that  can be 
programmed  to appear  on the computerized  charts  of patients  who have appointments;  those 
for formularies;  and those  for prescription,  lab- or test-orders. 
5An example  of a retrospective  reminder  is a list of patients  who have not yet received  the 
guideline-recommended  care. 
B-10  B-n  MHUL 
Talk 16. The munber aud pimxmtage of 
orgau@ticms thatrepwted bing  U$@lhe 
~fo&nvinJjKxOn$lmiicncentiwwtuEilcourage 
. . 
=-~y-y  ~~  ~em~ 
..  . ...  ~ 
~  ti  Efi%  G 
a.  Financialbonuses for clinician provision  of  0  0  ()  o  ()  o  ()  f) 
the  guideline-recommended  care 
b.  Financialpenaltiesfor clinician failure  to  o  0  0  u  o  0  0  0 
provide the guideline-recommended care 
1 
c.  Changes in reimbumement policy that 
discourage care other than that recommended  o  0  0  #.  2  2Q  2  4 
in the guideline 
d.  OTHER  economic incentives that

encourage clinician provision of the guideline- 0  0  0  u:  1  10  1  2

recommended care

B-12
 B-13  Resistance  from: 
a.  Physicians  7  37  5  25  2  20  14 ,  29

b.  Nurses  5  26  5  X5 o  0  10 20

c.  Other Staff  2  10  4  Xl :  1  10  7   14

d. Patients  1  5  3  15 0  q  4  $,

e.  Lack of support from administration  1  5  0  0  0012

f.  Uncertainty about how to implement  9  47  10  50
 2  2021  43 
the guidelines

g.  Operational  difficulties  2  10  3  15  2  20  7  14

h.  Inability to measure compliance with  7  37  4  20  1  10  12  24

the guidelines

i.  Inability to measure the effects of the  7  37  4  20  1  10  12  24

guideline use
 ~~  ,:  HE

j.  Expense of implementation  5  26  2

k.  OTHER  factors  o  0  2  IU Imm

%xamples  of operational  difficulties  include  problems  with computer  systems, patient-
encounter  forms, and medical-records  administration. 
B-14  +132  6  6 
4  1 
II 
2  3  2 
II  5  lU- 2 
2 
0 
0 
20 
B-15  ~mHMM
Eabk.2Z The qumberand.percentage.of
mmmm 
a.  Distributing  written  materials  explaining 
guidelines  to clinicians and/or  administrators 
b.	 Conducting  educational  sessions for 
clinicians  and/or  administrators  about  the 
guidelines 
c.  Distributing  written  materials  explaining 
guidelines  to patients 
d.  Conducting  educational  sessions  for 
patients  about  the guidelines 
e.  Using  an existing quality assurance  system 
to measure  compliance  with the guidelines 
f.  Developing  new measurements  to assess 
compliance  with the guidelines 
g.  Seeking  additional  funding  to cover the 
costs of guideline  implementation 
h.  OTHER  stem  to address  the  obstacles 
%
II  Number  of Steps  II  N  I  II 
II  1 
II 
2 
II  3 
4  2  10  3 
5  3  16  1 
6  1  52 
7  1  5  0 
BE 
TOTAL  19  982  Irx 
B-16  
: Td)le 25,  The mmher and pgx&ptage of 
uqg@z@ions *at  rt=qxlrtedhasi@g@led:*  Hospital 
: f@k!Uidg.~et@@  w .meastl@@.eE&%  Qf 
th&  ti  ti$.th&.AHCl+Rguide~
~mmdmm:  id  RHH
.@ Q ~

a.  Measures  of clinician performance  15  3  15 0
b.  Measures of patient satisfaction  1  5  1  5  0  0  2  4 
c.  Measures of patients clinical outcomes  15  57  257  Og  67  1$7 
d.  Measures of cost  00  00  17  10  1  2 
e.  Measures of organizational performance  Otl  1.5  17  !0  2  4 
f.  OTHER  methods to measure the effects  00  0  0  1  10  1  2 
of the guidelines 
-I  + : BE

7While three  nursing  homes  indicated  that  they have measured  the effects  of their  use of 
the AHCPR  guidelines,  five nursing  homes  indicated  that  they have taken  measures  of 
patients  clinical outcomes  to assess the effects of their  use of the AHCPR  guidelines. 
Similarly, while no HMOS indicated  that  they have measured  the  effects of their  use of the 
AHCPR  guidelines,  one  HMO  reported  that  it has taken  a measure  of cost to assess the 
effects  of its use of the AHCPR  guidelines;  one  HMO  reported  that  it has taken  a measure  of 
organizational  performance,  and one HMO  reported  that  it has used  another  method  to assess 
the effects  of the guidelines. 
B-17  B-18  B-19  Hospital  Nursing  Home  HMO  TOTAL 
N=82  N=75  N=37  N=194 
B-20 
  APPENDIX  C 
COMPLETE  coIm4ENm ON THE  D~  REPORT 
c-1  Department  OF HEALTH  & HUJWY  SERWCES	 PublicHealtShervice

Memorandum 
JUJ719S6 
Assistant  Secretar7 f ~~ Health 
offiCe  Of InsDector  General  (OIG) Draft Reports  on the

Utilization  02 Clinicai ?ractice Guidelines  Sponsored by the

Agency for Health Care Policy  and Research

Inspector General,  OS

Attached  are the public Health  Service comments  on  the Subject 
OIG reports.  We concur witrhthe  recommendations  and  have 
taken,  and  will  continue  to take actions  to implement  them. 
In addition  to our comments on the specific  recomendations~ 
we offer a series of technical  comments  for your 
consideration. 
Philip R. Lee, M.D.

Attachment

la

Mm   
PDIQ

DIGAS  I 
DIGEI

DIGOl  

DIGMP  

/

AIGCFAA  
4
-
., 
.... 
L-1 
., 
,- m 
-- -  
, . 
,..._ 
J 
L..
 . 
:.,.

,,---

-1 
w 
 
w  COMMENTS  OF THE PUBLIC HEALT3SERVICE ON THE OFFICE OF ~E

INSPECTOR GENERAL D~FT  ?.E
pORT  CLINICAIJp~CT~CE  GUIDELI~S

SPONSORED  BY THE AGENCY  FOR HzALTH CARE POLICY .WD RESEARCH:

EARLY EXPERIENCES  IN CLINICAL SETTINGS,  OEI-J1-?4-00250

General Comments

The Office  of the Inspector  Generai  (OIG) has surveyed a sample 
of three types of providers  regarding their experience with 
clinical practice  guidelines  deveioped under  the auspices of the 
Agency for Health  Care Policy and Research  (AHCPR).  AHCPR worked 
with the OIG  in developing  the suney  and PHS is pleased to now 
receive useful  feedback  from it.  The survey queried respondents 
regarding six guidelines  released  between March  1992 and April 
1993, and was  conducted  in December 1994.  The findings indicate 
that in this very short time, almost one-third  of the suneyed 
institutions  were using or intended to use at least one of the 
guidelines  to improve the effecci~eness and WalitY  Of clinical

praccice within  tiheirindiviauai  settings.  viewed in the contexc

of traditional  rates of diffusicn  of new information  into health

care practice,  these findings  are extremely  encouraging.

have been more rapidly and widely

The AHCPR-supported  guidelines  ..

adopted than many  other guidelines,  practice  recommendations~  OF 
alerts issued by public  agencies or professional  OrgaIIIZatlOns. 
Further, the findings  from the suney  may understate  the impact

of the guidelines  if, as is noted in the draft report, additional

organizations  may be indirectly  relying on AHCPR-sponsored

guidelines.  Many providers  may draft or revise  internal

guidelines  based on guidelines  developed by other organizations,

including AHCPR.  Organizations  may also incorporate  into their

protocols  recommendations  that ~ia not come directly from AHCPR-

printed documents,  but rather  from reports or  journal articies 
summarizing  the guideline  reccmmen.aations.  If true, the survey

responses obtained  bv  OIG r,avsubstantially  under-report  the 
the 
full impact of the g~idelines.  -
It is important  to note that :he :hree tYPesof o:ga~lzations

sampled by the OIG are not zepresen.tative of all ~ea~t-hcare

institutions.  To Obtain a comD.ieCe .D
lcture of the :ncenclves  and

obstacles  to guideline  use exper~en.ced across  sett---
gs we will

need to survey  additiona~  types =f providers  (e.9.I ~pA Oael

~,os, large hospitals,  zeachinq hcsqltals,  etc.) .  It is also

important to note that differen.c ;~ldelines are more re~evant  o

some :~es  of providers  than ozkers.  We therefore ;:oulae
xpect,

and  zma  in the surveyz-esuitsz=ported here) , thaz the

1 Anumberofstuddioecsu mentrma~t eosfd iffusoifnoenw [echnoloogrlmefso rma[mloo nhe altcha repracuce

arec l[medt hed rafrte porM[o.r erecemkf ormauorne-~.a ra[mhleA MAsexpenencewdti[shs emingawtdienlgl nIeSs
 

d.$cusdsien o[uerc hmcalcommbeemlso
 w.

C-5
 particular  types of facilities  surveyed wouid be more  likely to 
implement  some guidelines  than others. 
AHCPR has issued 16 clinical practice guidelines.  Responding  to

heavy demana,  zhe agency has distributed  close to 16 million

copies of these guidelines.  We believe,  as documented  in the

draft report,  that the validity  and credibility  of the AHCPR

sponsorea  guidelines have been major faccors  in their growing use

for quality  improvement.  No organization  surveyed  cited

disagreement  with the content of the guidelines  as a reason for

not implementing  them.  Other evaluations,  including  the 
companion  OIG report presenting  case descriptions  Of 
organizations  that have implemented AHCPR-supported  guidelines, 
confirm extensive  adoptlon  of  AHCpR-sponsored  guidelines.  There 
is also substantial  evidence  that implementation  of the 
guidelines  by health.care organizations  is resulting  in

significant  improvements  in patient outcomes  as well as

reductions  in health care costs.

The 
OIG  ~~Do~:  ~dentifies barriers  to guideiine  use exper~enced 
by survey respondents.  Acknowledging  that the A.HCPRguideline 
program is very new, the report notes that introducing  clinical

practice guidelines,  as with other recommended  changes in medical

practice,  poses significant  challenges.  Research  on the

implementation  of clinical guidelines and, more generally,  on the

assimilation  into medical practice of new methods,  technologies,

or other behavior  changes,  has consistently  shown that change is

generally  SICW and requires  sustained, carefully-targeted  efforts

that directiy  involve practitioners  and patients.

To achieve  the full potential  related to implementation  of 
practice guidelines  we need to find more effeccive  ways  tO 
promote aaopcion,  adaptation,  and implementation  of the 
guidelines.  The OIGS study was  not designed  to evaluate AHCPRS 
internal srccesses  for aeveioping  or disseminating  guidelines, 
nor AHCPR s program for evacuating guideline  dissemination  and

implementazicz  strategies.  We believe that these evaluative

efforts are crucial if we are to promote greater  use of clinical

practice ~Jitielines.  As described  in our responses  to individual

recommenaaci C.-ASb
elow, AHCPR will continue  ro focus and refine

its efforrs  Z3 identify the most effecti-~e methods  for promoting

guideline  im~iementation.  AHCPR is also embarked  on a strategy

to improve  t:.eefficienc~  of the guideli-ne development  Process

and to bercer  tailor gui~elines  to the neeas of local users.

These changes should lead to development  ~f guidelines  and

guideline-reiated  products  and implementacicn  tools that are more

easily assimilaced into practice by locai ixsci.tutions, and to

more effec:i-:edissemlnat=Gn  of the guiaeiines .

C-4
 OIG Recommendation

The Public Health Service  (PHS), throuah AHCPR,  should determine 
more effective wavs  to urornote  familiarity  with,  and use of, the 
guidelines. 
PHS Comment

PHS concurs with this recommendation.  As part of its ongoing

efforts to evaluate  and improve  the guideline  development  and

dissemination  process,AHCPR  sponsors  a broad-based  portfolio of

research and evaluation urojects  that the OIG did noc examine as

part of the current scud;.  These efforts  include a variety of

projects to assess the effectiveness  of alternative  strategies to

encourage  implementation  of  guidelines  in different  settings.

For example, AHCPR has a contract  with the American  Medical 
Review Research  Center  (AMRRC) that is comparing  the 
effectiveness  of alterna~ize educational  outreach programs 
employed by peer review organizations  to promote  ado~cion of the 
AHCPR-sponsored  guideiine  on benign prostacic  hyperp~asia.  A 
cooperative  agreement  with  the  Department  of Veterans Affairs 
(VA) will compare the effectiveness  of two educational  strategies 
for implementing  the A.HCpR-sponsored pressure  ulcer prevention 
guideline  in VA  facilities  across the United States.  Another 
cooperative  for Disease  Cantrol and 
agreement  with  the  Centers 
Prevention will  assess  the  effectiveness  of  targeted media 
campaign strategies  in Oklahoma  and Massachusetts  to dlSSemlnate

the AHCPR-sponsored  urinary  incontinence  guideline.

AHCpR has issued research  solicitations  to encourage  studies Of

how guidelines  can most effectively  be implemented  in large group

practices and in primary  care  settings.  Several research

projects have been funaea in response  to these solicitations  and

are ongoing.  AHC?R aiso,tias a contract  through an i:ceragency

agreement with the Office of personnel Management  to conduct

focus groups  to elici: f=e~ack  about guidelines  fro~,potential

users, including primary  care physicians,  physicians  in academic

settings, and nurses.

Basea on the findings cf these and other investigati=s,  and

subject to the availabilic~y of resources,  A.HCPRcent=.ues to

explore alternative  scrace~ies  for increasing  the familiarity of

potential users with pracc~ce  guidelines.  At the same time, as

part of its plans to refine and improve the guideline development

process, AHCPR  is placing  greater  emphasis  on efforts tO

facilitate guideline  ::piementation,  including devei~<ment  of new

types of products  (e.g., posters,  laminated  cards) =Za greater

 
use cf effective  modes c: distribution  suck as elecr=:n.lc

dissemination.  AHCPR aiso plans  to expand.partnerships  to

deveiop guidelines  in collaboration  with ocher Feder=- and State

agencies and with pri~.-ateorganizations.  These cont~=uing

efforts should help tc increase  awareness,  acceptance,  and use of

the guidelines.

L-J
 OIG Recommendation

The PHS, throuqh AHCPR,  should make increased  technical  sup DOrt 
available  to midelines  users. 
PHS Comment

PHS concurs with this recommendation.  Although  not the focus of

this OIG study, AHCPR sponsors  a number of prOJectS  to provide

technical support  to prospective  users  of practice guidelines.

We will continue  to do so.

For example, AHCPR  is sponsoring  a series of reports on using

clinical practice  guidelines.  The first report, entitled Usinq

Clinical Practice  Guidelines  to Evaluate Oualitv  of Care, was

issuea in March  1995.  It discusses  issues associated with 
implementing  conformance  to  practice  guidelines  and 
and  rneasurlng 
Earqecing opportunities  for clinical quality  improvement.  More 
;mporcantly,  it is a practical  hanabook  that lays out step by 
seep instructions  for deve~oplng  guideline-related  quality 
evaluation  tools  (medical  review criteria, performance  measures,

and standards  of quality) .  This report has been widely

disseminated  to a range of health care organizations,  including

hospitals  and medical  review organizations.  There is

considerable  demand  for the report  from across  the nation.  For

example, the Group Health Association  of America  has requested

copies to send to all its member plans.

Other reports  to be issued in the near future will provide

detailed descriptions  of projects  in which AHCPR-supported

guidelines,  review criteria,  and performance  measures have been

used in quality  improvement  programs.  A forthcoming  report

focuses on a project  in whiclh Peer Review Organizations  appiied

neaicai review  criteria derived  from AHCPR-sponsored  guidelines

cn accze ~ain,  urinary  incontinence,  and benign prostatic

hype.rpiasla in Medicare  quali~y of care reviews.  Another Wiil

tiescribethe results of efforts by the VA to disseminate  ana

~inpiem,entthe pressure  ulcer prevention guideline.

:2c~~  ~:as  supported  the  development  of review Crlterla in

association  with three Of  the  most  recent  AHCPR-sponsored

guidelines : heart failure, unstable  angina, and otitis media.

Review criteria  are aiso currentiy being developed  for a

guideline  on cardiac rehabilitation.  These review criteria

snouid serve as practical  organizations  i.~tereste~ ~~

tool,s  :or 
impienenting  these guidelines.

AiicJ?.iRs
 sponsoring  a series of efforts to co~~ect, evaluate,  32Ci

ziassi~Yperformance  measures  to assist providers  and other

keaith.care organizations  in the~r quality  improvement efforts.

parc ci this project  involves  li~king performance  measures  za

.J.HC2R-sponsoreapractice  guidelines.  This should assist

:.-.SE-.L-
 
Jcions interested  in implementing practice  guidelines  ~C

:nprc>-equality  of care.

C-6
 AHCPR also provides  information  and teckn=cal  assistance  tO state

and local health officials  and policy makers  through OUr user

Liaison Program.  For exampie, a worksnc~  on medical

effectiveness  and outcomes  research,  off=red  semi-annuallYt  has

included sessions on clinical practice  ~::delines,  review

criteria, and performance  measurement.  The sessions  focused in

particular  on applications  in managed  care, utilization  and

quality review, medical malpractice,  ana -;:orkerscompensation.

Subject to the availability  of resources,  .lHCPR continues  to

explore options  for providing  effeccive  :=cMical  assistance  tO

providers  and other institutions  interesc=d  in implementing

clinical practice  guidelines.  Opportunities  for technical

assistance  should be expanded by the new collaborative

partnerships  described  in response  to the previous

recommendation.

OIG Recommendation

The PHS, throuqh AHCPR,  should develou  and imulement  systematic 
mechanisms  for obtaininu obiective  feedback  about  auideline use. 
PHS Comment

PHS concurs with this recommendation.  Surveys  can be an 
effective  technique  to assess the extent  :3 which practitioners 
or  patients  generally  are aware of, have  accepted,  and are using 
practice guidelines.  AHCPR  supports developmental  work to design 
valid instruments  to be used in surveys  c: guideline  awareness

and utilization.  In an effort to leverags  resources,  AHCPR also

collaborates  with a nu~er  of outside organizations  both in

developmental  efforts as well as in enccu==ging  these

organizations  to include questions  relate=  za WCpR-sponsored

guidelines  in their own routine  surveys.

In addition  to cross-sectional  or perica~:  surveys,  we believe

that it is also important to conduct more  zargeted  evaluation

efforts, including:

o	 experimental  or quasi-experimental  s=~ciesof specific

dissemination  and implementation  stz=:egies;

o	 case studies of the logistics of gu~i=iine  implementation  in

specific  organizational  settings;  =Zi

o	 fOCUS Gr dlSCUSSIOn grOUpS Gf pOteI?Z23i Users to obtain

feedback on ch,eeffectiveness  of spe=~~ic  guideline-related

produczs  and suggestions  for overcc:::g  barriers  tO

implementation.

Basea on developmental  work, AHCPR is spcn=oring  a surveY of

patient and practitioner  attitudes  regarti~ng the A.HCpR-sponsored

guideline on benign prostatic  hyperplasia.  Results of the survey

C-7
 (conducted prior  to che implementation  of educational  programs

designed to encouraae  aaopcion  of guideline  recommendations),

alon~ with results ~f a post-intervention  survey wiil be

publlshed  later this year.

In response to a question  included  after consultation  with AHCPR

staff, the Group Heaith Association  of Americas  1994 Annual HMO

Performance Survey  fauna E:hatof those HMOS encouraging guideline

use  (82% of all HMOS sumeyed)  , half  (or 41~) had aaoP~ed Or 
adapted AHCPR-sponsoreci guidelines.  Based on a surrey instrument 
deveioped  in collaboration  with the AHcPR-sponsored  guideline 
panel and with AHCPR  staff, the National  Pressure  Ulcer Advisory 
Panel surveyed  nurses and other  specialists  in acute care

settings across  the country  regarding  the AHCPR-sponsored

pressure ulcer prevention  guideline.  The New Hampshire Medicare

Peer Review Organization  and the New Hampshire  Foundation for

Medical Care have also sumeyed  hospitals  and skilled nursing

facilities in New Hampsniye  and Vermont regarding  the pressure

ulcer guideline.

AHCPR continues  to assess  the feasibility,  costs, ~n~

methodological  soundness cf various options  for obt=nlng

systematic feedback  cn a broad  scale regarding  the awareness and

Subject to resource

use of AHCPR-sponsored  practice  guidelines. 
availability,  AHCPR aiso continues  to explore  options for

gathering additional  information  related to the most effective

methods to disseminate  and implement practice  guidelines.

C-8
 Technical comments

Executive  summary

Page Para Line:  Comment

i  2  4  As of this date AHCPR has released  16 clinical 
practice guidelines. 
i  2  6  The paragraph  states that in Fy 1994, +fJmilliOn 
was spent on dissemination  and evaluat~on of 
guidelines,  of which $4 million  was spent on 
printing  and dissemination. 
During FY 1994, approximately  $6 million WaS SPent 
on grants and contracts which  focused on the 
evaluation  of guidelines.  It should be noted  that 
FY 1994 was an atypical year for spending on

evaluations.  Durng  this yearf  fourl~r9e  scalef

multi-year  evaiuacion projects  were lnltlatea,  SO

that spending  reflected projects  covering more

than one fiscal year. 
i  3  1  The introduction  of new technologies,  including 
new clinical practice guidelines,  can  be  a  S1OW 
and  difficult  process.   
AHCPR-sponsored  research shows that the

introduction  and adoption of new technologies  and

medical knowledge  is in fact complex,  influenced

by factors such as consistency  with current

opinions and practices,  skills and resources,

necessary  to implement new knowledge,  flnanc+al

reimbursement  incentives, perceived  impllcatzons

Patient

for litigation,  and patient preference. 
factors play a large role in the aiffus~onas well

as the effectiveness  of health  and ciinical

information, particularly  in areas such as

compliance with.tueatment and medication

recommendations  .anaadoption  of prevention

behaviors.

In addition to publications  cited in footnote  14

of the report, AHCPR has sponsored  the following

work related to the effective  dissemination  cf

health and clinical information:

Agency for Heaith Care Policy and Research.

Effective Dissemination  of Clinicai and Health

Information:  Conference Summary.  Secrest L. ,

Baker TE, Rogers EM, Campbell TF, and Grady ML,

Rockville,  14D:

editors.  AHCPR  Pub. NO. 95-0015. 
Agency for Health Care Policy and Research,  Public
 Health Service,  U.S. Department  of Health and

Human Services.  December  1594.

-A--m--e--nrv  ,  -f--n~  -w----al-t-h. .  c-are  Policy and Research.. . 
E~fec~ive Dissemination  of Heaith  and clin,l
cal

Information  to Consumer:s:  -notated  Bibli~o
urauhv.

AHCPR.  Pub.  No. 95-0055.  Rockville,  MD:  Agency--ILL. 
for F4-e-al-t-h- --r_a_r_e
  policy  and  Research,  Public H{=a
 .1.L-11
Service, U.S. -n_e=n-a-r-t-m-e-n-t- -o-f Health  and Human

Services.  May  1995. 
i!weconducted  a mall su~ey  of 380 key U.S. health 
i  4  1 
care .... 
While the OIG survey addressed  three important

segments of the health care market,  it is

important to note that the particular  types of

institutions  surveyed are not necessarily

representative  oi all health  care institutions.

For example,  according  to data  from the American

Hospital Association,  less than half of all

More 
hospitals are small non-teaching  hospitals. 
important,  these institutions  account  for less 
than  10%  of  all  inpatient  admissions  in  the U.S. 
Furthermore,  according  to InterStudy,  the majority

of HMO enrollees  are served by IPA or network

Staff model

models rather than staff model HMOS. 
HMOS account  for the smallest  proportion  of

enrollees of all model types. 
5  6  (First bullet  on the page) :  Impetus for using 
guidelines  has frequently  come from organizations 
internal quality  improvement  programs.  
In fact the impetus is overwheirninglyfrom
Table  2
internal quality  improvement  interests. 
(Page B-3) notes that 84%  (41 of 49) of those 
using the guidelines  cited  ~his as their reason. 
External factors--such  as directives  from

supervising  organizations  and State law

requirements--have  also played  an important role.

Table 2 on page B-3 shows that external actors

play  a much smaller  role  (26 cr 33%), and,among

the reporting  organizati~ns,  i= is primarl~Y

,nursinghomes clhatreport e.xte~rAalfactors as

playing a role in guideline  use. 
ii  45  (Fourth builet  on the page) : ~conom+c in~entives 
to encourage  clinicians  were =Portea  bY ew 
organizations.  
c-lo
 reported by only

In face economic  incentives  were 
HMos , and only two (or perhaps  3, depending on

whetker  che question  asked providea a muitiple

response option) .

iii
 (Firsr bullet  on the page) :  Clinician resistance 
was the second most frequently  cited obstacle.   
Clinician  resistance  is closely  followea by

inability  to measure guideline  use and izability

to measure  guideline  effect.  In total 24

respondents  reported physician  resistance, while

When

12 reported  these other two difficulties. 
the full table on page B-14 is examined, the

message  that seems  clear  is  that  tools and 
technical  assistance  are needed  to facilitate the 
implementation  of guidelines,  the measurement  of 
guideline  conformance,  and examination  of the 
effecc ct guideline  conformance  on eutccmes. 
iii
 Sentences  2 and 3 in the paragraph:  These two

sentences,  which are repeated  in other sections ot

the report,  are unclear.

One seems to state that 46% were unfamiliar with 
the  guidelines,  the other  that 40% were 
unfamiliar. 
In aadition,  sentence 3 should clarify chat the

focus of the study was  only on small non-teaching 
hospitals  and HMOS with staff model components. 
Without  this distinction,  the sentence =Ppears to 
generalize  to all hospitals,  nursing homes and

HMOS .

a 
iii  Z  3  Seccnd,~aragraph  under first recommenaac~on: 
reconslaeration  of A.HCPR efforts  cc prc-,ote 
fami.iarity with the guidelines. 
We agree that AHCPR should continue  ZO sponsor 
research  to identify the most effeccive strategies 
for dissemination,  lmplementationf  ~~oP~~~n and 
diff.~slcn of innovation.  However,  basec on the

follcwing  reasons,  we do not believe ZhaZ the OIG

study suDDorts  a cOnclUSIOn  that AHCF?.s

dissernin~~ion methods have been ineif=cci-ze.

1)  3.HCP?.Sdissemination  efforts  do not stress

Agenc~,name recognition.  The Agency name does not

featcye prominently  on the cover of the

guideii~es , and they are frequently described  in

the ~eaia and published  sources as federallY

sponsored  guidelines  (as o:posed  CO ~CPR

guidelines) .  They are produced  by paneis of

C-11
 private  indiviauais  convened  under the auspices  of

the Federal government.  Consequently,  failure  to

achieve name recocrnition should  not be equated

with inadequate  d~ssemination.

l!fami~iaritywi~h guidelines:  ad

2)  The terms 
guideline use  from the su~ey  questionnaire  may

have different  meanings to different  individuals.

Individuals may have seen a guideline,  but not

looked at it in detail, or not internalized  its

content.  Conversely,  an organization  can have

internalized  the recommendations  so thoroughly

that they become  standard practice,  rather than a

guideline produced by an external  entity.

3)  It should be noted that the rate of guideline 
penetration  reported by organizations  responding 
to the survey  (20% use  and 32% including  those 
reporting behavioral  intent  to use the guidelines) 
is consistent  with the diffusion  of other medicai 
effectiveness  information  and  technologies.  For 
example, after distributing  its smoking cessation

guideline  to 200,000 physicians  the American

Medical Association  found that only 20% recalled

having received  the guidelines.  Only 25% of

physicians  sumeyed  by the AMA reported using  the

HIV guidelines  issued by AMA.

Hearn, Wayne.  On the Road  to Prevention.

American Medical  News. June  5, 1995. P. 9- (A coPY

of this article  is provided  with this report)

4)  Dissemination  of clinical  information  does not

occur rapidly.  Antman et. al. report that despite

evidence  from randomized controlled  clinical

trials of the eifecciveness  of therapies  and

interventions,  it is often many years later that

findings become  standard recommendations  in

textbooks and ciinical expert  reviews appearing  in

published  literature.  For example,  findings  from

multiple  randomized controlled  triais indicace

that significance =sductions  in,hospital  mortality

could be achieved by administration  of

thrombolytic  drugs ~or patients  with acute

myocardial  infarczian.  However,  widespread

recommendation  of ~his therapy by clinical experts

did not occur until 13 years  after zhe appearance

of data from these zrials.  since  1985, wh:n~~ 
approximate  20% ~eduction in the risk of aeatn. was 
established  at ~he ?<.001 ~evel,  ~4 of the 43

review articles  examined did not mention  the

treatment or felt Lz was still experimental.

c-i2  Antman, Elliott  M. et a~.  A Comparison of Results

of Mets-analyses  of Randomized  Control Trials and

Recommendations  of Clinical  Experts.  Journal of

the American  Medical A,SsOclatiOn 1992; VO1.268,

No. 2. P. 240.  (A copy of the article is

provided. )

is

We believe  that this contextual  information 
essential  in interpreting  the data and the

recommendations  presented  in the Inspector

Generals  report. 
iv  1  1  Systematic  mechanisms  for obtaining  feedback on 
guideline  use. 
The suggested  strategies  for accomplishing  this 
recommendation  all focus on su~eys  to examme 
While we agree

familiarity with the guidelines. 
that some cross  sectional  suney  work is desirable

to assess penetration  of guidelines~  other

possibilities  should be considered,  e.g.

targeted evaluation  efforts,  in the form of quasi-
experimental  or experimental  design, to look at 
implementation  using specific  dissemination  and 
implementation  strategies. 
case study examination  of the logistics Of

in various orgamzational

guideline  implementation 
settings.

user groups  to discussion  methods  for implementing

guidelines  and effective  methods  resolving

barriers  to implementation.

targeted examination  of guideline  products  to

assess what  factors contribute  to their

usefulness .

Clearly a case study approach,  such as thatoused by the

OIG  can augment  survey data and provide addlt~onal 
valuable  feedback  on detailed  aspects of the

guideline.

One of the problems  with cross sectional surveys is

that instrumentation  problems  may bias results; without

additional  experimental  researcht

qualitative  or  quasi  For

it may not be possible  to detect  the biases.

example, the term  ~seof guidelines  may mean

different  things to different  individuals and

organizations  .  The  i~formation  in footnote 19 of the

OIG report specificailv  discusses  ~he different

interpretations  Of  .a
u~deline use, and how differences

C-13
 in the wording  of questions  may result  in different

reported  rates of guideline  penetration.

- With respect  to user feedback  forms attached to each

guideline,  AHCPR has considered  this method for

obtaining  feedback.  Mailback  cards or feedback forms

may be used to offer qualitative  suggestions to the

However, their

agency via open ended questions. 
reliability  ana validity  for evaluating  guideline

penetration  or  attitudes about the guidelines  is

questionable  because  of sample based response bias.  It

is not possible  to identify with any certainty the

sample  frame, the characteristics  of the respondents 
versus nonrespondents,  or whether  the individuals

respond  more than once to a mailback  card.

Introduction

Page Para Line

2  1  2  AHCPR was the successor  to  (rather than the 
replacement  for) the National  Center for Health 
Semites  Research. 
Although  AHCPR activities  related  to health 
senices  research are consistent  with the mandate 
of NCHSR  (including the conduct of the National 
Medicai Expenditure  .$u~~eY), AHCpRs mandate  s 
substantially  broader than that of NCHSR, 
encompassing:  the conduct and sponsorship of 
research,  demonstration  projects,  eva~uations~ 
training,  guideline development  and d~ssem~nation 
of

ciinical practice  guide~ines~

medical effectiver.ess research  \inciud~ng

Patient  outcomes  Research  Teams and other

outcomes  research projects) ,

dissemination  research

and a variety of other =reas. 
2  2  2  As of this date, AHCpR has released  16 clinical 
practic e guidelines. 
2  2  3  The,~anels apply science based methccs to develop 
specl~ic statements on -management for z.heclinical 
condition  under consideration.  
It is more accurate to state that 
The panels  explicit  approach and

use  an 
meth.oaolog  for evaluacinq scientific  evidence  to

deveicp recommendations  reiated  to the diagnosis

C-14
 treatment,  and management  cf the clinical

In instances. ...

condition  under consideration. 
2  4  4  In addition  to establishing  a toll-free guideline

request telephone  line and availability  of

guidelines  through the National  Librav  of

Medicine  and the Internet,  the Agency has issued

the text of clinical.practice  guidelines,

including  quick reference guides  and patient

booklets,  in CD-ROM format.  This product  is

currently  available  for use by medical  libraries

and will be available  to the publlc  through the

Government  Printing Office  in Fall 1995.

2  4 8  AHCPR both conducts and sponsors  conferences  on

In addition,  the Agency

guideline  implementation. 
sponsors targeted and investigator  initiated

research on guideline  implementation  and effective 
methods of disseminating  health  information.

2  5 5  During FY 1994, four large scale,  multi-year

evaluation projects were  initiated,  so that

spending reflected projects  covering more  than one

fiscal year.

2  7 1  This sentence  suggests that the body of research

supported by the Agency has resultea @y  in the

identification  of barriers  to use of guldelmes.

The paragraph  implies that the barriers  listed in

the third sentence were identified  from AHCPR

sponsored  research, when they are items from the

OIG survey.

Whiie  it  is  certainly  true  that barriers  to

guideline  implementation  exist,  and that AHCPR

sponsored  research has identified  some of these

barriers,  it is important to note  hat CPRS

research portfolio  focuses on a broaa program  of

research on the effectiveness  of clinical

interventions  as well as methods  and tools for

transiting  medicai effectiveness  information  into

clinical decision making.

Barriers  to  guideline  implementation  are

ider.tifiea in the  cantext of research studies  to

evaiuate guidelines,  to determine  effective

clir.ical strategies,  :0 identify  salient quality

improvement  strate~lesf  and to disccver ways to

improve the commun~cation  of clinicai and health

information.

The statement  found in our commencs  above  (i 3 1)

more correctly  characterizes  the nature of the

Agencys research.

C-15
 At  first it 
4  1  1&3  This  sentences  is  not  entirely  clear. 
might appear  to say non-teaching  hospitals seine 
Perhaps it 
at the forefront  of medical  education. 
would be more direct to say: 
I!wesampled  non-teaching  hospitals  because e 
anticipated  that  they would  be more likely to use 
existing guidelines  such as those supported by 
In contrast, we

AHCPR than to develop  their own. 
assumed that teaching hospitals,  because they

serve at the forefront of medical  education, might

be more likely  to develop  their own guidelines.  
4  l&2  1  The sampling  strategy employed by OIG may have 
resulted  in an underestimation  of guideline use. 
We  do  not  find  evidence  to  support  the  assumptions 
For example, 
underlying  the sampling  strate9Y. 
the assumption  is that small, non-teaching 
hospitals  would be more likely to use nationally

developed  guidelines.  One might assume the

converse:  because  teaching hospitals are at the

forefront of medical education,  they,would be more 
familiar  with AHCPR-supported  gu~del~nes,  and more 
likely to use them.  Many guideline Panel.me~ers

are prominent  researchers  in academic medical

centers throughout  the U.S. ; conse~ently,  Such 
centers  may  be more likely to use nationally 
developed  guidelines  such as those developed bY

AHCPR .

The second paragraph  on this page states that HMOS

with staff model components  were selected because

they show a trend toward managed  care, and because

staff model  plans may be more able to influence

providers.  As indicated  in a previous  comment,

staff model  :<MOSare the smallest  segment of the 
managed care  market; data from InterStudy show 
that IPA/network  models are the fastest growing

segment of the market place.

The Group Health Association  of America  study

cited in footnote  18 notes that in surveying HMOS

of all model  types, approximately  33% used AHCPR-

supported guidelines  or adapted these guidelines

for use.  Consequently,  the assumption  that -~os

with staff model  components  would be more likely

to  use guidelines  developed  by AHCPR  may not be

correct.

C-16
 5 3 2
 Parenthetical  for pressure  ulcers guideline  should

be (prediction and prevention  of pressure

ulcers) .

6 8 1
 This figure does not include organizations  that,

used AHCPR  (or

in developing  their own guidelines, 
other national guidelines)  as a basis.

6 8 1
 This  finding  should  be  qualified  to  reflect  the 
As  noted  in 
wording  of  the  OIG  questionnaire. 
footnote  18,  the  term  use  of,gu+dellnes  may mean 
different  things to different  lndlvlduals and 
organizations.  The information  in footnote 18 of 
the OIG report  specifically  discusses  the 
different  interpretations  of  ~ideline  use,  -d 
how different wording of questions  may result  in 
different reported  rates of use of the gu~delines. 
This information  should be presented  within the

text of the report.

6 9 3
 Footnote 20 should be associated  with this

sentence, rather than the first sentence in

paragraph  8.

13 1  2
 The information  in footnote 24 provides a number 
of  examples  of  AHCPR  di.ssem~natlon  aCtlVltieS 
including a publications  clearinghouse,  electronic

dissemination  of the guideline,  medla,outreacht

and organizational  outreach  through direct

mailings, public-private  partnerships,  and

intermediaries  (such as opinion  leaders, medical

professional  associations,  continuing  education

The report

programs and consumer organizations) . 
recommends  a reconsideration  of dissemination

efforrs, but does not adequately  recognize ongoing

dissemination  activities.

13 1  2
 As indicated  in the executive  summary, sentences  2

and 3 in the paragraph  are unclear.

One states that 46% were unfamiliar  with the

guidelines,  the other that  40%  were  unfamiliar.

Sentence 3 should ciarify that the survey focused

on smail non-teaching  hospitals  and ~Os  =th

The sentence could be

staff model components. 
constr.~ed to  apply  to all  hospitals,  :~rsing  homes

and W40S.

C-17
 APPENDIX  D

NOTES 
1.  One  of the  first reports  on geographic  variations  in medical  practice  was published 
in 1938 and  focussed  on the  variation  in rates  of tonsillectomy  among  school  children 
in England  (JA  Glover,  The Incidence  of Tonsillectomy  in School  Children, 
Proceedings of the Royal Society of Medicine,  1938; pp.  1219 -1236.) 
In the  past  two decades,  John  Wennberg  has published  several  articles  that  document 
great  geographic  variation  in medical  practice  in the  United  States.  In a 1988 article, 
he found  that  A resident  of New Haven,  Connecticut  is about  twice  as likely to 
undergo  a coronary  bypass  operation  as is a resident  of Boston;  for carotid 
endarterectomy,  the  risks are  the  other  way around,  The  numbers  of knee  and  hip 
replacements  per  capita  are  much  more  common  among  Bostonians,  while  New 
Havenites  experience  substantially  higher  risks for hysterectomy  and  back  surgery. 
(John  E. Wennberg,  Commentary:  Improving  the  Medical  Decision-Making  Process, 
Health Affairs, Spring  1988; pp.  99- 106.) 
Robert  H. Brook  and  his associates  from  the  Rand  Corporations  Health  Program 
have  also published  on this subject.  In a 1986 article,  they  report  that  Of 123 
procedures  studies,  67 showed  at least  threefold  differences  between  sites with  the 
highest  and  lowest  rates  of use. (Mark  R. Chassin,  Robert  H. Brook,  et  al, Variations 
in the  Use  of Medical  and  Surgical  Services  by the  Medicare  Population,  New 
England Journal of Medicine; January  30, 1986; pp.  285- 290.) 
2.Institute  of Medicine,  Committee  to Advise  the  Public  Health  Service  on Clinical 
Practice  Guidelines,  Marilyn  J. Field  and  Kathleen  N. Lohr,  editors,  Clinical Practice 
Guidelines:  Directions for a New Program (Washington  DC:  National  Academy  Press, 
1990) p. 8. 
Clinical  practice  guidelines  are  also known  by other  names,  including  practice 
parameters,   clinical indicators,  medical  necessity  guidelines,  medical  standards, 
and  practice  policies.  (David  M. Eddy,  Clinical Decision  Making:  From  Theory  to 
Practice,  Practice  Policies--What  Are  They? Journal of the American  Medical 
Association;  February  9, 1990; pp.  877- 880.) 
The  American  Medical  Association  definition  of practice  parameters  refers  to 
strategies  for patient  management  developed  to assist physicians  in clinical  decision 
making.  (Directo~  of Practice Parameters: Titles, Sources, and  Updates, 1994 Edition, 
p.  189, Office  of Quality  Assurance  and  Medical  Review,  American  Medical 
Association,  Chicago,  IL.) 
D-1
 3.The  1995 Edition  of the  American  Medical  Associations  (AMA)  Directoty  of 
Practice  Pararneten  lists approximately  1800 practice  parameters  developed  by more 
than  70 physician  associations  and  other  organizations. 
4.  U.S.  Congress,  Office  of Technology  Assessment,  Defensive Medicine  and Medical 
Malpractice,  OTA-H-602,  July 1994, pp.  140-148. 
5.CAPPCARE  Developing  Wireless  Hardware  for Instant  Guidelines  Access, 
Report on Medical  Guidelines&  Outcomes Research, 5 May  1994,pp.  4-5. 
6.According  to Dr.  James  Cooley  of the  Harvard  Community  Health  Plan  (a staff-
model  HMO  based  in Boston,  MA),  guidelines  can be  implemented  through  methods 
that  are  patient  directed  and  include  educational  sessions  (in which  either  individual 
patients  or groups  of patients  are  educated  about  the  guideline)  and  reminder  mailings 
(in either  newsletters  or focussed  mailings  on single topics);  methods  that  are  directed 
at clinicians  and  include  guideline  publication  and  distribution,  continuing  medical 
education  (CME)  sessions,  CME  take-home  quizzes,  skills-training  programs,  and 
academic  detailing;  administrative  methods  that  include  concurrent  reminders  (such  as 
automatic  reminders  that  can be programmed  to appear  on the  computerized  charts 
of those  patients  who have  appointments),  retrospective  reminders  (such  as lists of 
patients  who  have  not yet received  the  prescribed  intervention),  encounter-form 
changes  that  prompt  physicians  to ask certain  questions  or to provide  certain  services, 
order-form  changes  that  either  encourage  or discourage  the  use  of specific  laboratory 
tests,  the  designation  of personnel  to manage  and/or  provide  the  specific  care,  and 
feedback  to providers  about  the  extent  to which they  are  complying  with the  guideline; 
and  methods  that  involve  economic  incentives,  such as bonuses  or penalties,  and 
changes  in reimbursement  policies  (Telephone  conversations  with Dr.  James  Cooley 
on September  14 and  21, 1994). 
7.	 The  Agency  for Health  Care  Policy and  Research  Reauthorization  Act  of 1992 
(P.L.  102-410) extended  the  authorization  of AHCPR  and  amended  some  provisions 
for the  development  of clinical practice  guidelines. 
8.  According  to P.L.  102-410, the  guidelines  must  be based  on the  best  available 
research  and  professional  judgement;  presented  in formats  appropriate  for use by 
physicians,  other  healthcare  practitioners,  medical  educators,  medical  review 
organizations,  and  consumers;  presented  in treatment-specific  or condition-specific 
forms  appropriate  for use in clinical practice,  educational  programs,  and  reviewing 
quality  and  appropriateness  of medical  care;  include  information  on the  risks and 
benefits  of alternative  strategies  for prevention,  diagnosis,  treatment,  and  management 
of the  particular  health  condition(s);  and  include  information  on the  costs of 
alternative  strategies  for prevention,  diagnosis,  treatment,  and  management  of the 
particular  health  condition(s)  where  cost information  is available  and  reliable. 
The  AHCPR  chooses  its guideline  topics  on the basis  of several  factors:  the  adequacy 
of scientific-based  evidence  from which to develop  guidelines;  the  number  of people 
affected  by a condition;  the  conditions  amenability  to prevention;  expected  potential 
D-2  for reducing  inappropriate  variations  in the  prevention,  diagnosis,  management,  or 
outcome  of the  condition  or disease;  specific  needs  of the  Medicare  and  Medicaid 
populations;  and  the  costs of the  condition  to all payers,  including  consumers 
(AHCPR,  Program Note:  Clinical Guideline Development,  August  1990). 
9.	 Guidelines  released  to date  include Acute  Pain Management  (issued  in 3/92), 
Urina~ Incontinence  in Adults  (3/92), Pressure Ulcers in Adults--Prediction  and 
Prevention  (5/92),  Cataract in Adults  (2/93), Depression in Primary Care (4/93),  and 
Sickle Cell Dtkease (4/93), Early HIV Infection (1/94), Benign Prostatic Hyperplasia 
(2/94), Management  of Cancer Pain (3/94),  Unstable Angina  (3/94) Heart Failure (6/94), 
Otitk Media  with E~ion  (7/94),  Quality Determinants  of Mammography  (10/94), Acute 
Low Back Pain in Adults  (12/94],  Treatment of Pressure Ulcers (12/94),  and Post-stroke 
Rehabilitation  (5/95). 
10.  See  Guideline  Development  and  Use, included  in each  guideline  published  by 
the  Agency. 
11.  Among  the  organizations  on AHCPRS  master  mailing  list are  healthcare 
education  and  credentialing  organizations,  hospitals,  insurance  companies,  and 
associations  and  societies  for healthcare  professionals. 
According  to the  March  27, 1995 AHCPR  Guideline  Marketing  Summary  Reports, 
7,288 articles  have  been  published  about  the  guidelines  in newspapers,  magazines,  and 
trade  publications.  In addition,  107 guideline  reprints,  excerpts,  or summaries  have 
appeared  in professional  journals. 
12.  Among  the  companies  that  have  issued  reprints  of the  AHCPR  guidelines  are 
Abbott  Laboratories,  Aetna,  Bristol-Myers  Squibb,  Burroughs  Wellcome, 
Communicore,  Dupont  Pharmaceuticals,  Eli Lilly, Forest  Labs,  Janssen,  Key 
Pharmaceuticals,  Knoll Pharmaceuticals,  Mead  Johnson,  Medco,  Merck,  Pfizer,  Purdue 
Frederick,  Roche  Laboratories,  Roxane  Pharmaceuticals,  SmithKline  Beecham,  Syntex 
Laboratories,  and  Zeneca  Pharmaceuticals. 
13.  As examples  of its dissemination  efforts,  AHCPR  reports  that  it conducts  direct 
mailing  of its guidelines  to Health  Resources  and  Services  Administration  grantees, 
including  community,  rural,  and  migrant  health  centers  and Area  Health  Education 
Centers,  and  to Indian  Health  Service  healthcare  providers.  In addition,  the  agency 
has  collaborated  with the  National  Cancer  Institute  (NCI)  to distribute  its guidelines 
on the  management  of cancer  pain  and  mammography  through  the  NCI  clearing 
house.  Similarly,  it has  collaborated  with the  Centers  for Disease  Control  to distribute 
the  HIV  guideline  through  that  agencys clearing  house.  The  AHCPR  has  also worked 
with the  Food  and  Drug  Administration  to distribute  the  mammography  guideline  to 
mammography  facilities  and with the  Health  Care  Financing  Administration  to 
distribute  the  guidelines  on acute  pain  management,  urinary  incontinence,  and 
pressure  ulcer  prevention  to hospitals  and  nursing  homes. 
The  AHCPR  research  projects  also affect  dissemination.  Among  the  dissemination-
D-3  related  projects  is one  that  involves the  Medicare  Peer  Review  Organizations  in an 
effort  to develop,  implement,  and  evaluate  quality  and  utilization  review  criteria  for 
three  guidelines.  Another  research  project  involves an interagency  agreement  with the 
Department  of Veterans  Affairs  to test  various  dissemination  strategies  for introducing 
the  pressure  ulcer  prevention  guideline. 
14.  Fiscal  Year 1996 Budget, p. 49, U.S. Department  of Health  and  Human  Services; 
and Fzkcal Year 1996 Budget Justification to OMB, p. 25, U.S.  Department  of Health 
and Human  Services.  Additional budget information was provided by AHCPR  to OIG 
in a memorandum  from Jill Bernstein  (OPD/AHCPR),  dated April 25,  1995. 
15.Please  see  the  following  articles: 
Ann L. Greer,  The State  of the  Art  Versus  the  State  of the  Science:  The  Diffusion 
of New Medical  Technologies  into  Practice,  International Journal of Technology 
Assessment  in Health  Care, v. 4, 1988, pp.  5-26. 
Jonathan  Lomas  et al., Opinion  Leaders  vs Audit  and  Feedback  to Implement 
Practice  Guidelines:  Delivery  After  Previous  Cesarean  Section, Journal of the 
American  Medical Association,  May  1, 1991, pp.  2202-2207. 
Peter  J. Greco  and  John  M. Eisenberg,  Changing  Physicians Practices,  New England 
Journal of Medicine,  October  21, 1993, pp.  1271-1274. 
Roberto  Grilli  et al., The Impact  of Patient  Management  Guidelines  on the  Care  of 
Breast,  Colorectal,  and  Ovarian  Cancer  Patients  in Italy, Medical  Care, January  1991, 
pp.  50-63. 
Jacqueline  Kosecoff  et al., Effects  of the  National  Institutes  of Health  Consensus 
Development  Program  on Physician  Practice,  Journal of the American  Medical 
Association,  November  20, 1987, pp.  2708-2013. 
Margaret  VanAmringe  and  Terry  E. Shannon,  Awareness,  Assimilation,  and 
Adoption:  The  Challenge  of Effective  Dissemination  and  the  First  AHCPR-
Sponsored  Guidelines  Quality Review Bulletin, December  1992, pp.  397-404. 
16.Institute  of Medicine,  Committee  on Clinical  Practice  Guidelines,  Marilyn  J. Field 
and  Kathleen  N. Lohr,  editors,  Guidelines for Clinical Practice: from Development  to 
Use, (Washington  DC:  National  Academy  Press,  1992).  A discussion  of the  ways in 
which  different  factors  affect  guideline  use can be found  in chapter  four,  beginning  on 
page  83. 
17.Among  the  projects  currently  funded  by AHCPR  is one  to assess whether  aspects 
of the  guideline-development  process  are  associated  with user  perceptions  of the 
validity,  reliability,  cost, and  utility of the  guidelines;  another  to model  the  cost impact 
of various  guideline  recommendations;  and  others  that  focus  on the  development  of 
D-4
 medical  review  criteria  based  on guideline  recommendations.  Additional  projects 
examine  variables  related  to the  acceptance  and  implementation  of specific  guidelines. 
Other  projects  seek  to document  changes  in practice  and  patient  outcomes  stemming 
from  guideline  use in primary  care. 
18.  In contrast,  we anticipated  that  teaching  hospitals,  because  they serve  at the 
forefront  of medical  education,  might  be more  likely to develop  their  own. 
19.  We  are  90 percent  confident  that  the  percentage  of organizations  that  have  used 
the  guidelines  is between  16 and  24. 
20.	 In 1994 HMO  Pe~orrnance Repoti,  the  Group  Health  Association  of America 
(GHAA)  summarizes  the  results  of its recent  survey of a representative  sample  of 100 
of its member  HMOS  (the  Association  achieved  a 71 percent  response  rate  to this 
survey).  The  GHAA  found  that  82 percent  of its respondents  (approximately  58 
HMOS)  encourage  their  providers  to follow specific  clinical  practice  guidelines  and 
that  40.8 percent  of this subset  (approximately  24 HMOS)  have  adapted  guidelines 
from  those  published  by AHCPR.  Thus,  GHAA  found  that  approximately  33 percent 
(.82 x .408) of its respondents  have  adapted  guidelines  from  those  published  by 
AHCPR. 
By contrast,  our  survey of all 80 HMOS with staff-model  components  found  that  20 
percent  have  used  one  or more  of the  AHCPR  guidelines  about  which we inquired. 
The  difference  between  the  findings may be  accounted  for by several  factors: 
1.  The  GHAA  surveyed  a sample  of all of its member  HMOS, while we 
surveyed  only the  80 HMOS that  have  staff-model  components. 
2.  The  GHAA  inquired  about  the  use of AHCPR  guidelines  in general,  while 
our  survey  focused  on the  use of six specific  AHCPR  guidelines. 
3.  The  GHAA  posed  three  broad  questions  about  guideline  use, while we 
asked  twelve  pages  of detailed  questions. 
4.  The  GHAAs  survey question  asked  whether  or not  the  organizations  had 
developed  their  own guidelines  and,  if so, whether  or not  these  were  based  on the 
AHCPR  guidelines;  while we asked  if organizations  had  used the  AHCPR  guidelines 
(respondents  may have  interpreted  used as meaning  direct  application  of the 
AHCPR  materials  and  as excluding  the  adaptation  of these  materials  for local  use). 
21.  Certain  guidelines  may be  of greater  relevance  than  others  to certain  types  of

healthcare  provider  organizations.  One  would  not  expect  that  all guidelines  would  be  

used  by all organizations.  For  example,  one would  not  expect  the  guideline  on sickle

cell disease  to be  used  by many nursing  homes  for the  aged.

D-5
 22.  The  AHCPR  reports  that  its guidelines  have  been  distributed  to every  hospital 
and  nursing  home  in our  survey sample.  Substantial  evidence  demonstrates  that  the 
introduction  of new technologies,  including  new clinical  practice  guidelines,  is a slow 
and  difficult  process.  See  end  note  #  15 for a partial  listing of relevant  articles. 
23.  Such materials  include  the  Clinical Practice Guideline, the Patients Guide,  and/or 
the  Quick Reference Guide for Clinicians. 
24.Examples  of automated  computer  systems include  those  that  provide  clinicians 
with  on-line  access  to guidelines  or guideline  summaries;  those  for  concurrent 
reminders  that  can be  programmed  to appear  on the  computerized  charts  of patients 
who  have  appointments;  those  for formularies;  and  those  for prescription,  lab- or test-
orders. 
25.An example  of a retrospective  reminder  is a list of patients  who have  not yet 
received  the  guideline-recommended  care. 
26.Information Diweminahon  to Health  Care Practitioners and Policymakers: 
Annotated  Bibliography, Agency  for Health  Care  Policy and  Research  (AHCPR), 
Public  Health  Service,  U.S. Department  of Health  and  Human  Services,  April  1992, 
p.1. 
27.The  AHCPRS  efforts  in this domain  include  both  its User  Liaison  Program 
meetings  and  a recently  released  guide  for  organizations  entitled  Using Clinical 
Practice  Guideline  to Evaluate  Quality  of Care. 
D-6