Department of Health  and Human  Sefices 
OFFICE OF

INSPECTOR GENEML

The NCH Consensus Development Program 
Dissetiation  of Findin~  through Medical School 
ContinuingEducation Activities 
L 
Februiq  MM  OFFICE OF INSPECrOR  GENERAL 
The mission of the Office of Inspector General  (OIG), as mandated by Public Law 95-452, as 
amended, is to protect  the integrity of the Department  of Health and Human Setices  (HHS) 
programs as well as the health and welfare of beneficiaries served by those programs.  This 
statutory mission is carried out through a nationwide network of audits, investigations, and 
inspections conducted by three  OIG operating components  the Office of Audit Services, the 
Office of Investigations, and the Office of Evaluation and Inspections.  The OIG also informs 
the Secretary of HHS of pro~am,  and management problems, and recommends courses to 
correct them. 
OFFICE OF AUDIT  SERVICES 
The OIGS Office of Audit Services (OAS) provides all auditing setices  for HHS, either by 
conducting audits with its own audit resources or by overseeing audit work done by others. 
.%dits CXwninethe Ferformancc of HHS programs and/or its grantees am! contrr ton  h 
carrying out their respective responsibilities and are intended to provide inciepermmt 
assessments of HHS programs and operations in order to reduce waste, abuse, and 
mismanagement and to promote economy and efficiency throughout  the Department. 
OFFICE OF INVESTIGATIONS 
The OIGS Offkx of Investigations (01) conducts criminal, civd, and administrative 
investigations of allegations of wrongdoing in HHS programs or to HHS beneficiaries and of 
unjust enrichment  by providers.  The investigative efforts of 01 lead to criminal convictions, 
administrative sanctions, or civil money penalties.  The 01 also oversees State Medicaid fraud 
control units which investigate and prosecute fraud and patient  abuse in the Medicaid program. 
OFFICE OF EVALUATION  AND  INSPECIIONS 
The OIGS Office of Evaluation and Inspections (OEI) conducts short-term management and 
program evaluations (called inspections) that focus on issues of concern to the Department, 
the Congress, and the public.  The findings and recommendations contained in these inspection 
reports generate  rapid, accurate, and up-to-date information on the efficiency, vulnerability, 
and effectiveness of departmental  programs.  This report was prepared  in the Boston regional 
office under the direction of Mark R. Yessian, Ph.D., Regional Inspector General and Martha 
B. Kvaal, Deputy Regional Inspector  General.  Project stati. 
REGION  HEADQUARTERS 
Dana Miller, I+oject Leader  Mary Beth Clarke, ProgramSpecialkt 
Timothy Corbett, Lead Anatyst 
Fordiitiod  *oftimptie  contadduB@on-o@e  Qvte-at 
(617) 565-1050, or by Fax at (617) 565-3751.  Department of Health  and Human  Services 
OFFICE OF

INSPECTOR GENERAL

The NE-I Consensus Development 
Pro~ram 
w 
Dissetiation  of Findings through Medical School 
tintinutig Education Activities 
I 
I  February  l= 
mI-ol-91-o1760 
I 
   EXECUTIVE  SUMMARY 
PURPOSE 
To  evaluate  the  impact  of the  efforts  of the  National  Institutes  of Health  to 
disseminate  Consensus  Development  Program  information  through  medical  school 
continuing  education  activities. 
BACKGROUND 
In response  to  growing  concerns  about  the  effectiveness  of some  medical  procedures 
and variations  in medical  practice,  the  National  Institutes  of Health  (NIH)  created  the 
Consensus  Development  Program  (CDP)  in 1977.  The  Program  brings  together  non-
governmental  pane!s  to assess  the  safety> efficacy, and  appropriateness  of e+sting  and 
emerging  medical  tecnrmlogies.  ISach panei  presents  its con~i~si~il~  in the  i~ii~i  ui  a 
Consensus  Statement,  which is widely disseminated  by NIH  to medical. professionals 
and  the  public  through  direct  mailings,  publication  in journals,  and  other  methods. 
The  potential  importance  of the  Consensus  Program  has increased  as concerns  about 
cost  and  quality  in health  care  have  intensified.  Recent  evaluations,  however,  have 
pointed  out  potential  vulnerabilities  in the  Program--including  questions  about  its 
success  in disseminating  Conference  findings.  A 1989 Rand  report  suggested  that 
continuing  medical  education  (CME)  activities  could  be  a critical  dissemination 
vehicle,  and  recommended  that  a survey of directors  of continuing  education 
regarding  the  use  of information  from  the  Consensus  Program  would  be valuable. 
We  conducted  such  an  examination  of continuing  education  programs  at  U.S. medical 
schools, which  represent  a major  resource  for CME.  This report  is based  primarily 
upon  a survey  of continuing  education  directors  and  chairs  of departments  of family 
medicine,  neurology,  and  oncology  at all U.S. medical  schools.  We inquired  about 
general  familiarity  with  the  Consensus  Program,  awareness  of specific  Statements,  use 
of Consensus  information  in continuing  education  activities,  and  opinions  about  the 
Program.  We  did not  attempt  to assess  changes  in medical  practice  resulting  from 
either  continuing  education  activities  or the  Consensus  Program  itself. 
FINDINGS 
Fifly-two pexent  of medical  school &pa~nt  chaih  reported having sponsored a 
continuing education  activity thut addkrsed  the @dings  of a recent NIH Consensus 
Ck3nference. 
Chairs  of family  medicine  reported  more  limited  use of CDP  information  in 
	
continuing  education  activities  than  did chairs  of neurology  or oncology:  30 
percent  of family  medicine  chairs,  47 percent  of neurology  chairs,  and  77 
percent  of oncology  chairs  reported  such use. 
i  NIH  Consensus  findings  have  been  addressed  in a variety  of medical  school 
	
continuing  education  formats;  the  most  common  have  been  grand  rounds. 
Of those  department  chairs  who reported  use  of NIH  Consensus  information  in 
	
their  continuing  education  activities,  81 percent  reported  that  these  were 
targeted  to in-house  faculty,  and  72 percent  that  these  were  targeted  to 
practicing  physicians  horn  outside  the  medical  school  setting. 
Several factons hhder  &  dirsemindion  and acceptance  of CDP biifonnation and 
incorpomtion  of that information into conhhuing education  acdvities. 
Limited  familiarity  with the  Consensus  Development  Promam.  Sixteen  percent 
	
of department  chairs  knew  nothing  about  the  CDP;  an additional  17 percent 
were  aware  of the  Program  but  knew  little  about  it.  Family  medicine  chairs 
reuorted  particularly  limited  familiarity  with the  Program:  27 percent  had 
rwwr  ne~xi  :~fi: and  another  L5 percent  were  aware  Lu, .-.~i~  ,- ,,,.  ./Jt  it. 
Unsuitable  format  of the  Consensus  Statements  for  continuing  education 
	
activities.  Only 33 percent  of those  department  chairs  who were  familiar  with 
the  CDP  rated  the  format  in which Conference  findings  are  presented  as very 
appropriate  for continuing  education.  Respondents  suggested  several  types  of 
materials  that  would  more  readily  facilitate  the  use  of Conference  findings  in 
their  continuing  education  activities,  including  slides,  overheads,  and  curriculum 
materials  for short,  small-group  activities. 
Concerns  about  the  Consensus  Development  Program  itself.  Most  medical 
	
school  department  chairs  voiced  general  respect  for  the  Program.  Nevertheless, 
a number  of specific  concerns  were  voiced  by respondents  familiar  with the 
Program  about  limitations  to its effectiveness  and  usefulness,  including 
questions  about  the  appropriateness  of panelists  and  speakers,  the  practicality 
and  directiveness  of the  Statements,  and  the  adequacy  of CDP  procedures.  A 
few respondents  expressed  fundamental  objections  to the  use  of consensus 
methods  to address  controversies  in medicine.  These  concerns,  whether 
reflective  of perceived  or actual  weaknesses  in the  CDP,  represent  a serious 
barrier  to wider  dissemination  and  acceptance  of NIH  Consensus  findings. 
RECOMMENDATIONS 
The  NIH  has  demonstrated  a firm  commitment  to ongoing  evaluation  and 
improvement  of the  Consensus  Development  Program.  As NIH  continues  such 
efforts,  it should  aim  to maximize  the  potential  for medical  school  continuing 
education  as a vehicle  for disseminating  Consensus  findings.  In particular: 
ii 
  l%eNH  shoukl  tdx  step  to tiease  warerless  of the CDP among those responsible for 
continuing medical  hation.  IIw NH  should  make  qxxid  @om  to reach tie  who 
qumsor  continuing education  actiwkies for generul and f-physicians. 
In this effort,  NIH  should  maximize  the  effectiveness  of direct-mailing  efforts  by 
tailoring  such  mailings  to department  chairs  and  continuing  education  directors  in their 
capacity  as educators,  and  by encouraging  them  to cover  Consensus  findings  in their 
continuing  education  activities.  The  NIH  should  work  with medical  school  continuing 
education  organizers  to plan  appropriate  dissemination  strategies.  In this effort,  NIH 
could  also work  with  organizations  other  than  medical  schools,  including  relevant 
specialty  societies  and  academic  groups. 
The NIH shouki  Ment#J more @ective  ways of encouraging the ihcoqoratibn  of 
ConseIUUSjindihgs  into continuing educatidn  activdk 
The  NIH  should  expiore  ways of packaging  and  disseminating  Comenstis  i.,.~ings  that 
would  more  readily  facilitate  their  use in continuing  education  activities,  including  the 
production  of slides,  overheads,  or other  curriculum  materials.  The  NIH  should  also 
conduct  focus-group  discussions  with key audiences  of Consensus  findings,  including 
medical  school  department  chairs  and  continuing  education  directors,  to learn  their 
perspectives  on format  issues.  The  NIH  could  use its future  research  efforts  as 
opportunities  to examine  the  attitudes  and  practices  of other  types  of continuing 
education  sponsors. 
171eIWHshouMstrengthe?l  h#omto  un&m@ul  and  &es  basic concerns about the 
Conwruus Devekpnent  l+vgram 
Because  no  dissemination  efforts  will be  successful  in reaching  and  influencing  people 
with basic  concerns  about  the  effectiveness  and  usefulness  of the  Programs  methods, 
NIH  should  strengthen  its efforts  to understand  and  address  such  concerns.  The  NIH 
could  use focus  groups  or informal  working  groups  of department  chairs  and 
continuing  education  directors,  and  other  key audiences,  to learn  about  and  address 
these  concerns.  As first steps,  NIH  should  identi$  and  test  better  ways of involving  a 
wider  range  of people  in the  planning  stage  of upcoming  Consensus  Conferences  to 
maximize  acceptance  of the  outcomes,  and  should  address  concerns  about  panel  and 
speaker  selection. 
AGENCY  COMMENTS 
We solicited  and  received  comments  on a draft  of this report  from  the  Public  Health 
Service  (PHS).  The  PHS  concurred  with  our  recommendations  and  identified  several 
activities  that  NIH  currently  has  underway  or has  planned  that  address  these  issues. 
These  include  the  publication  of a revised  statement  of operating  guidelines  and 
procedures,  efforts  to identi~  new ways of exploiting  CME  as an  avenue  for 
dissemination,  and  continuing  evaluation  of the  procedures  and  impact  of the  CDP. 
... 
Ill  The  PHS  also  suggested  that  criticisms  of the  Consensus  Program  by some  survey 
respondents  may  be based  on incomplete  or out-of-date  information.  We  continue  to 
believe  that  these  concerns  merit  serious  attention  by NIH,  and  have  modified  the  text 
to put  these  issues  into  clearer  perspective.  The  complete  text  of the  PHS  comments 
appears  in appendix  E. 
iv 
-  TABLE  OF  CONTENTS

PAGE 
m~SUMMARY  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..i

INTRODUCITON  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...1

FINDINGS 
. Use  of NIHConsensus  findingsin  medical  school

continuing  education  activities  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...5

e Limi[s ?0 the  widerd~~emfr+{nn and  acceptance  ofCDP 
information  and  iis incorporation  into  continuing  education  ~Cth7itit5S: 
Limited  familiarity  wth the  Consensus  Development  Program  . . . . . . . . .  8

 
Unsuitable  format  of the  Consensus  Statements  for

	
continuing  education  activities  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...10

Concerns  about  the  Programs  effectiveness  andusefilness  . . . . . . . . . . .  11

 
RECOMMENDATIONS  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...15

AGENCY  COMMENm  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...19

APPENDICES

A  Methodology  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..  A-l

B  Continuing  Medical  Education:  Background  . . . . . . . . . . . . . . . . . . . . . .  B-1

c  Summary  of Department  Chairs  Survey Responses  , . . . . . . . . . . . . . . . .  C-1

D  Summary  of CMEDirectors  Sumey  Responses  . . . . . . . . . . . . . . . . . . . .  D-1

E  PHSComments  onthe  Draft  Repofi  . . . . . . . . . . . . . . . . . . . . . . . . . . . .  E-1

F  Notes  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  F-1
  INTRODUCTION

PURPOSE 
To  evaluate  the  impact  of the  efforts  of the  National  Institutes  of Health  to 
disseminate  Consensus  Development  Program  information  through  medical  school 
continuing  education  activities. 
BACKGROUND 
Health-care  technology assessment and N~ 
In recent  years,  health-care  providers,  policy makers,  and  the  public  have  expressed 
incl~asin~  ccncern  about  rising ccsts.  v~riaticms in medical  practice.  and  overuse  ~f 
certain  mechcal  procedures.  These  concerns  have  spurred  greater  interest  in the 
evaluation  of the  appropriateness  and  effectiveness  of existing  and  emerging  health-
care  technologies,  including  drugs,  devices,  and  procedures. 
The  Congress  determined  in the  late  1970s that  the  National  Institutes  of Health 
(NIH),  as the  principal  biomedical  research  institution  in the  nation,  should  play a role 
in this arena.  The  NIH  director  at  the  time  testified  that  NIH  and  the  rest  of the 
scientific  community  must  assume  more  responsibility  for the  effect  of research  on the 
quality  of health  care  delivered.  The  need  for accelerating  the  transfer  of new 
technology  across  the  interface  between  biomedical  research  and  the  health  care 
community  and  systems  is a major  issue.1 
The Consensus Development  Program 
The  Consensus  Development  Program  (CDP)  was created  as the  focus  of NIHs 
technology  assessment  efforts.  The  CDP  aims to promote  the  timely  incorporation  of 
beneficial  medical  innovations  into  clinical  practice,  encourage  the  abandonment  of 
obsolete  technologies  in favor  of ones  that  are  more  efficacious  or safe,  discourage  the 
adoption  of technologies  that  have  little value,  and  form  public  policy choices  that 
encourage  or discourage  the  use of certain  medical  technologies.2 
The  Consensus  Program  employs  a group-judgment  process  that  brings  together  non-
governmental  panels  to address  the  safety  and  efficacy  of controversial  existing  or 
emerging  medical  technologies  .3  At the  end  of each  2-1/2 day Consensus  Conference, 
the  panel  crafts  a Consens~s  Statement  presenting  its finding;  and  conclusions. 
Reaching dktors:  Dksemihation  of CDP information 
The  NIH  aims  to disseminate  each  Consensus  Statement  widely in order  to achieve 
the  maximum  impact  of the  Statement  on health  care  practice.4  The  NIHs  efforts 
in this area  reflect  a recognition  that  the term  effective  dissemination  includes  the 
1  concept  of diffusion  of knowledge  and  information  as well as the  acceptance, 
inculcation,  and  utilization  of disseminated  information;  the  distribution  of 
information  alone  is insufficient  to ensure  adoption  or use.s 
Several  recent  evaluations  of the  Program  have  raised  questions  about  the  NIHs 
success  in disseminating  its findings,  and  have  suggested  the  need  for further  research 
in this area.6  These  reports  have  noted  in particular  that  NIH  has  had  especially 
limited  success  in reaching  general  and  family physicians--an  important  target  audience 
because  they  both  use many  of the  technologies  evaluated  by the  Program  and, 
through  the  referrals  they make,  serve  as gatekeepers  who  often  determine  the  type  of 
care  their  patients  will receive. 
Wdical  school confining	 medical  educalion  advities 
asavehiclefor  thediw emmation  of CDPfindings 
In a  1989 study  of me  ?UH  Consensus  Program,  the  Rand  Curpurauwi  ~m..+  t.ui: 
continuing  medical  education  (CME)  could  be  a critical  dissemination  vehicle  for the 
Program,  and  recommended  that  it play a more  prominent  role  in the  Programs 
dissemination  strategy.s  Focus-group  discussions  with  physicians  recently  conducted 
by NIH  likewise  found  that  most  physicians  prefer  face-to-face  contacts  as a method  of 
learning,  and  cited  continuing  education  programs  as a key source  of information.g 
Continuing  medical  education  activities  are  sponsored  by a wide variety  of 
organizations,  and  sexve as a major  means  for practicing  doctors  to keep  up  to date 
with current  knowledge  about  medical  procedures  and  technologies.  According  to the 
Accreditation  Council  for Continuing  Medical  Education,  medical  schools,  through 
their  faculty  members,  provide  the  major  resource  for  CME  in this country.lo  (See 
appendix  B for more  information.) 
Doctors  in positions  of leadership  in medical  schools  and  those  who  plan  and  conduct 
continuing  education  activities  seine  as switching points  for new  medical  information 
and  can be  particularly  influential  in spreading  CDP  findings.11  In addition  to their 
role  in setting  the  agenda  for undergraduate,  graduate,  and  continuing  education  at 
their  own institutions,  department  chairs  often  conduct  continuing  education  activities 
at their  respective  specialty  society  conferences,  and  are  invited  to speak  at  community 
hospitals.12  As noted  in the  Rand  report,  information  may well follow a two-step 
flow from  opinion leader to follower.  . . .  Major  communication  sources  may alert 
specialists  and  academic  physicians  to the  existence  of new medical  information 
(eg., Consensus  Conference  recommendations),  and  this information  may then  filter 
down to primary  care  practitioners  and  other  physicians  in private  practice.13 
m  n?ptm 
The  Rand  report  suggested  that  a survey  of directors  of CME  regarding  actual  and 
potential  uses  of NIH  Consensus  material  would  be valuable.14  This  report  presents 
an examination  of the  current  and  potential  future  utilization  of NIH  Consensus 
2  findings  by medical  school  continuing  education  programs.  We provide  information  on

NIHs efforts  to reach  medical  school  department  chairs  and  CME  directors,  how

these  people  view the  Program  and  its products,  and  the  extent  to which  the

continuing  education  activities  they sponsor  serve  as vehicles  for the  wider

dissemination  of NIH  Consensus  information.  We do not  attempt  to assess  changes  in

medical  practice  resuhing  from  either  CME  activities  or the  Consensus  Program 
itself.15 
.

METHODO~GY 
Our  primary  means  of data  collection  was a set  of mail  surveys  of the  directors  of 
continuing  education  and  of the  chairs  of the  departments  or divisions  of family 
medicine,  neurology,  and  oncology  at all U.S. medical  schools.  We  inquired  about 
general  familiarity  with  the  Consensus  Program,  awareness  and  use  of seven  recent 
Consensus  Statements.  and  opinions  about  the  Pro~ram.  In the  findings,  we focus  on 
the  responses  of department  chairs,  wiio are  generally  key m~di~tii  i)pi.ilti,.  ,.-..i:-i  Uiid 
who are  more  closely  involved  than  continuing  education  directors  in the  subject 
matter  of their  departments  continuing  education  activities  (see  appendices  C and  D 
for a summary  of survey  responses).  We supplemented  this information  with 
telephone  interviews  with a sample  of survey respondents,  discussions  with NIH  staff, 
a review  of NIH  dissemination  materials,  a review  of the  literature,  and  discussions 
with experts  in the  field.  (See  appendix  A for detailed  methodology.) 
We  conducted  our  review  in accordance  with the  Quality  Standards  for  Inspections 
issued  by the  Presidents  Council  on Integrity  and  Efficiency. 
3   FINDINGS

FIFTY-TWO  PERCENT  OF  MEDICAL  SCHOOL  DEPARTMENT  CHAIRS

REPORTED  HAVING  SPONSORED  A CONTINUING  EDUCATION  ACITVITY

THAT  ADDRESSED  THE  FINDINGS  OF A RECENT  NIH  CONSENSUS

CONFERENCE.

We  asked  medical  school  department  chairs  if they  had  sponsored  CME  activities

during  the  previous  three  years  on at least  one  of a sample  of seven  topic  areas,  and  if

such activities  had  addressed  the  findings  of the  recent  NIH  Consensus  Conference  on

each  topic.16

Fifty-two  percent  had  sponsored  a CME  activity  that  addressed  the  NIH  Consensus

findinw  on one  of the  sample  topics.  Seventeen  percent  had  sponsored  a mlmmnt

continuing  education  activity,  but  had  not  addressed  the  NIH  Conseusus  rimmgs  on

the  topic.17  Sixteen  percent  had  not  sponsored  a relevant  CME  activity  (see  table  1).

Table 1:  Use  of  CDP  Fmdinm  in CME  Activities  Percentage  of  department  chairs who 
sponsmed  a recent  continuing  education  activity  on  at least  one  of the  sample  topics  in 
their  field  and  the  percentage  of  these  that  addreased  the  findings  of  the  NIH  Consensus 
C2mkxence  on that  topic  I 
Conducted  a CME  Addressed  the 
activity  on  at least  N-m  Cklsensus 
one  of the  sample  Fmdiqp  on that 
topkx  in their  field  topic 
FamiIy  Neurology  oncology  Average 
Medicine  Chaim  chairs 
chain 
(n=S4)  (n=SS)  (n=95)  (n=267) 
Yes  Yes  30%  47%  77%  52% 
No*  22%  20%  9%  17% 
Dont know  26%  16%  4%  I  15% 
No  23%  17%  9%  16% 
Total

SoumxO*I
G swvey of medical school department  chairs, November 1992

Nom  Includes those who responded definite~  and prubab~  yea or no.

Differences are statistically significant at the .05 level.

5
 We  also inquired  as to whether  department  chairs  were  aware  of continuing  education

activities  sponsored  by their  departments  that  had  addressed  the  findings  of additional

Consensus  Conferences,  other  than  those  in our  sample.  Twelve  percent  were  aware

of such additional  CME  activities.  Of these,  however,  most  had  also  addressed  at least

one  of the  Statements  in the  sample;  only 6 (2 percent)  had  not  also addressed  at

least  one  of the  Statements  in the  sample.

Chuim of family  medkine  reprted  mom lhnkd  use of CDP tiformation  in conlbuukg 
educalion  activdies  than did chaiis of neurobgy  or oncology. 
We  found  significant  variation  in the  reported  use of Consensus  information  in CME

activities  among  the  three  specialty  groups:  30 percent  of family medicine  chairs,  47

percent  of neurology  chairs,  and  77 percent  of oncology  chairs  reported  such use  (see

table  1).

We  also found  that  neurology  and  oncology  chairs  were  more  definitive  about  their

departments  use  of CDP  information  in continuing  education  activities  than  were

family medicine  chairs:  59 percent  of oncology  chairs  reported  that  they  had

sponsored  relevant  CME  activities  that  had dej_initeZyaddressed  CDP  findings;  30

percent  of neurology  chairs and only 12 percent  of family medicine  chairs reported  the

same.  This  may reflect  both  less intensive  involvement  in continuing  education  on the

part  of family  medicine  chairs  (see  appendix  C, table  C-17),  and  less use  of CDP

information  in their  continuing  education  activities.

We  address  some  of the  reasons  for the variation  in the  use of Consensus  information

both  within  and  among  specialty  groups  in our  second  finding,  beginning  on page  8.

NH  Consensus fhdings  have  been adlikssed  k  a varie~  of medical  school  continuing 
educatim  formuts;  the mart cornrnon have been grand mwnd.r. 
In-house  programs,  such as grand  rounds,  and  large-group  activities  were  the  most

common  forums  for  CME  activities  that  addressed  NIH  Consensus  findings.  Oncology

chairs  reported  significantly  greater  use of small-group  activities,  such  as discussion

groups,  than  did neurology  or family medicine  chairs  (see  appendix  C, table  C-10).

A number  of respondents  explained  how they  had  made  use  of Consensus  findings  in

their  continuing  education  activities.  One  department  chair  reported  that  the  CDP

findings  had  been  the  central  topic  of a regularly  scheduled  department  session  on

current  topics  in the  field, while several  noted  that  the  findings  had  been  addressed

only peripherally,  as part  of larger  programs.  Several  indicated  that  Consensus

Statements  have  been  useful  in subspecialty  conferences,  departmental  meetings,  and

formal  lectures.  Some  respondents  noted  that  the  concise  nature  of the  Consensus

Statements  makes  them  more  appropriate  for shorter,  small-group  discussions;  they

are  already  distilled  beyond  usefulness  for longer  activities.  A CME  director

6
 proposed  that  they  are  best used  in small groups  with  an  expert  moderator  or in

self-instruction  format.

iUedkal  school continuing ea!ucation activitk  addkwzn g NEl  Conwnsus jidkgs  have 
been tmgeted  to both in-houses@  and facully,  and pructicingphysicians  frvm  outside 
the medkal  school setting. 
Of those  department  chairs  who reported  addressing  CDP  information  in their  CME

activities,  89 percent  noted  that  these  activities  had  been  targeted  to residents,  81

percent  to in-house  faculty,  and  68 percent  to in-house  hospital  staff.

These  activities  also served  to reach  practicing  doctors  beyond  the  medical  school

setting:  72 percent  of department  chairs reported  that  their  activities  had  been

targeted  to local  or nonlocal  practicing  physicians--35  percent  to practicing  physicians

from  outside  the  iucai  ar~~ (see  appenuix  C, tabie  C-11).18

7
 SEVERAL  FACIORS  HINDER  WIDER  DISSEMINATION  AND  ACCEPTANCE

OF  CDP  INFORMATION  AND  INCORPORATION  OF  THAT  INFORMATION

INTO  CONTINUING  EDUCATION  ACHWTI13.

Lhnited  f~  with the Ctmsensus Development  ??vgmm 
Sixteen  percent  of department  chairs  knew  nothing  about  the  CDP;  an additional  17

percent  were  aware  of the  Program  but  knew  little  about  it.19

Family  medicine  chairs  reported  much  more  limited  familiarity  with  the  Program  than

did oncology  and  neurology  chairs:  27 percent  of family  medicine  chairs,  18 percent  of

neurology  chairs,  and  only 3 percent  of oncology  chairs  reported  that  they  had  never

heard  of the  Program.  There  was similar  variation  among  the  groups  with  regard  to

those  who were  aware  of the  Program  but  knew  little  about  it (see  table  2).

Table  2  Awareness  of the  CDP  Percentage  of medieal  school  CME  directors  and 
department  chairs who  were  aware  of the  NIH  Consensus  Development  program 
. . 
EXii?i5 
:  -F 
Very familiar
 11%  14%  31%  47%  31%

Somewhat familiar,

but not sure of all

the details
 48%  33%  34%  41%  36%

Aware, but didnt

know much about it
 23%  25%  17%  8%  17%

Not at all aware
 18$Z0  27%  18%  3%  16%

Sourax OIGsuneyofmedicalschoolCMEdwectorasnddepartmencthairsN, memher1992

Note Differenceasm  ongdepartmencthairsarestadsticallsyig nificanatt
 the.05level.

Awareness  of s~ecific  Consensus  Statements  was more  limited  than  ~eneral  awareness 
of the CDP.n  Again, chairs of family medicine were less likely to b: aware of 
Consensus  Statements  relevant  to  their  field than  were  other  department  chairs;  for 
example,  89 percent  of oncology  chairs  but  only 57 percent  of family medicine  chairs 
were  aware  of the  Statement  on EarZyStage Breast Cancer.  Even  many  department 
chairs  who had  sponsored  continuing  education  activities  on topics  that  had  been 
addressed  by Consensus  Conferences  were  not  aware  of the  relevant  Consensus 
8
 Statements  on those  topics.  For  example,  only 44 percent  of the  chairs  of family

medicine  who  had  sponsored  a CME  activity on gastrointestinal  surgery  for  severe

obesity  were  aware  of the  Consensus  Statement  on that  subject  (see  table  3).21

Table  3:  Awareness  of Sued  c Consensus  Statement  Percentage  of  all responden~  and 
of  tkxe  who  sponsored  a continuing  education  activity  that  mhkssed  each  topiq  who  wxxe 
aware  of each  Statement 
. 
B  ~  i;::  ~~  o;;; 
i, 
I    =61  a~~::~:    =69  at:~=    =  ~  kt::~:z: 
Suqety  for  .  85%  87%  -
Epilepsy  (n=54) 
Clinical use of  18%  100%  67%  79%  -
Botulinum  Toxzn  (n=2)  (n=43) 
Intravenous  19%  45% 
Immunoglobulin  (n=m) 
Adjuvant  - 79%  * 88% 
* 
Therapy for 
Colon and 
 
Rectum  Cancer  (tt=dll]

Early Stage  57%  60%  - - 89%  92% 
Breast Cancer  (n=35)  (n=79)

Gastrointestinal  31%  44%  -

Sugery for 
Severe Obesity  (n=9]

Treatmentof  56%  55%  -
Panic Disorder  (n=42]

SOH  OIG suNey of medical school department  chaixy Nwemher  1992

Note  Adaah (-) indicatea that the group waa not aaked about that Statement.

In their  written  comments,  a number  of respondents  demonstrated  this limited

familiarity  with  the  Program.  One  neurolo~  chair  commented  that  he  was not  aware

of the  dimension  of the  Program;  these  are  worthwhile  goods  that  are  insufficiently

known.  A family  medicine  chair  noted  that  the  CDP  findings  may go unnoticed  at

times.  One  CME  director  noted  that  we have  never  received  anything  from  NIH,

while  another  suggested  that  this Program  is not well recognized.  As one  oncology

9
 chair  remarked,  I dont know all of the  Consensus  Conferences--how  could  a

practicing  doctor?

Department  chairs  reported  that  their  most  common  sources  of information  about  the

CDP  during  the  previous  three  years  had  been  direct-mail  copies  of Consensus

Statements  and journal  publications.  Nonetheless:

25 percent  reported  that  they  had  not  received  a Consensus  Statement  in the 

mail  (or  could  not  recall  having  received  one)  during  the  previous  3 years.  This 
ranged  from  8 percent  of oncologists,  to 36 percent  of family physicians,  to 39 
percent  of neurologists.~ 
40 percent  had  not  read  a Statement  in either  the Journal  of the American 

Medical  Association  (JA.MM) or a specialty  journal  in the  previous  3 years.  This 
ranged  from  33 nercent  of family physicians,  to 38 percent  of neurologists.  to 48 
percwt  ui oncoiog~stb.:4 
Other  sources  of information,  including  colleagues,  the  popular  media,  and  continuing

education  activities  sponsored  by outside  organizations,  were  reported  by department

chairs  as less important  ways of hearing  about  the  Program  or specific  Conferences

(see  appendix  C, table  C-3).X

Unsuitable format  of the Consensus Statements for contbuing  education  activitia 
Only 33 percent  of those  department  chairs  who were  familiar  with the  Consensus

Program  rated  the  format  in which  Conference  findings  are  presented  as very

appropriate  for  CME.  This  figure  ranged  from  12 percent  of family  medicine  chairs

to 40 percent  of neurology  and  oncology  chairs.  Only  18 percent  of continuing

education  directors  gave this rating.

Respondents  criticized  the  Consensus  Statements  as bland,  lacking  in adequate

background  information,  and  lacking  in sufficient  data  to support  the  findings  and

conclusions.  One  family medicine  chair  noted  that  the  format  is pretty  dry and  the

findings  are  not  easily recalled.  Might  be helpful  if CDP  topics  were  contrasted  with

current  prevailing  professional  practices.

Respondents  also  noted  that  it would  require  work  on their  part  to translate  a

Consensus  Statement  into  a format  appropriate  for a continuing  education  activity.

Many  suggested  that  they would  be  more  likely to make  use  of Consensus  information

if they  had  access  to materials  that  could  readily  be incorporated  into  their  ongoing

continuing  education  activities.  Our  survey respondents  rated  the  potential  usefulness

of 10 types  of materials  for their  CME  activities;  the  materials  rated  most  highly were

CME  curriculum  materials  for short  sessions  (such  as grand  rounds),  reference

10
 bibliographies,  and  conference  proceedings.  Video  tapes  of the  Conferences  received

the  lowest  ratings  for usefulness  in continuing  education. X  (See  appendix  C,

table  C-16.)

A number  of respondents  offered  the  suggestion  that  slides  or overheads  outlining  the

findings  and  recommendations  of Conferences  would  be very useful  for CME

activities,  lectures,  talks,  and  meetings.  Several  people  with whom  we spoke  urged

that  such materials  could  best  be  used  in regularly  scheduled  activities,  such  as

monthly  departmental  update  meetings  or grand  rounds.  They  suggested  that  NIH

might  make  such  materials  available,  or announce  their  availability,  through  targeted

mailings  to department  chairs  and  others  responsible  for continuing  education.  The

NIH  could  suggest  in a cover  letter  that  such people  discuss  Conference  findings  at  an

upcoming  continuing  education  activity.27

Ldtwms  aDoul  iw  Frogrurns  ejfecdventw  and  usef 
Most  medical  school  department  chairs  voiced  general  respect  for the  Consensus

Program.n  Nevertheless,  a number  of specific  concerns  were  voiced  by respondents

familiar  with  the  Program  about  limitations  to its effectiveness  and  usefulness.  The

most  prominent  of these  concerns  were  related  to the  appropriateness  of Consensus

panelists  and  speakers,  the  practicality  and  directiveness  of Consensus  Statements,  and

the  adequacy  of CDP  procedures.  A few respondents  expressed  fundamental

objections  to the  use  of consensus  methods  to address  controversies  in medicine.

Many  of these  comments  parallel  criticisms  raised  in previous  evaluations  of the  NIH

Consensus  Program,  including  NIH-commissioned  studies  by Rand,  the  University  of

Michigan,  and  the  Institute  of Medicine.

The  groups  we su~eyed  represent  some  of the  best  informed  and  most  up-to-date

members  of the  medical  community.  Their  concerns,  whether  reflective  of actual  or

perceived  weaknesses  in the  CDP,  represent  a serious  barrier  to wider  dissemination

and  acceptance  of NIH  Consensus  findings.

Choice of panelists  and speakens:  In unprompted  written  comments,  10 percent  of 
those  department  chairs  who were  familiar  with the  Consensus  Program  voiced 
concerns  about  the  credentials  and  intellectual  independence  of Conference  panelists 
and  speakers. 29 As one  summed  up these  concerns,  the CDP  is highly dependent 
on the  panel  selected  to make  the  assessment.  The  stature,  knowledge,  and  respect  of 
a single individual  can  alter  the  conclusions  in an inappropriate  manner.  The 
predominant  criticism  among  these  respondents  was that  the  process  is biased  by the 
selection  of NIH  insiders, both  to present  evidence  and  to sit on the  panels.  As one 
neurologist  put  it, attendees  appear  limited  and  the  panel  of experts  seems  hand-
picked;  it doesnt  represent  a true  broad  sampling  of opinion  to develop  consensus, 
but just  reflects  a preconceived  opinion  of NIH  or a small  panel.  A CME  director, 
him~elf a physician,  described  pa~icipants  as the  same insider  crowd--experts  who 
often  have  their  own conflicts  of interest. 
11
 Family  physicians  were  particularly  critical  of what  they  regarded  as the  subspecialty

focus  and  research-orientation  of the  panelists  and  speakers.w  In a typical  criticism,

one  strongly  urged  that  NIH  balance  the  committees  towards  practical  users  of the

information  away from  so-called  experts.  Another  voiced  the  widely  noted  comment

that  the panels  should  include  family physicians  since we are  likely to be  the

consumers  of much  of the  Consensus  findings.

Practicality  and  directiveness  of Statements:  Only 40 percent  of those  department  chairs 
who were  familiar  with the  Program  strongly  agreed  that  Consensus  recommendations 
are  practical--ranging  from  26 percent  of family physicians  to 52 percent  of 
neurologists.  Likewise,  only 38 percent  of department  chairs  strongly  agreed  that 
Consensus  recommendations  are  sufficiently  directive--ranging  from  26 percent  of 
family physicians  to  46 percent  of neurologists  (see  appendix  C, table  C-14). 
While  there  w~re  few strongly  negative  ratings  with regard  to these  characteristics  of

Consemus  atawnems,  we ~elieve  mat  the  fact  that  60 percem  u.  QJpi.LL.---..A-i~lairs

do not  strongly  regard  the  Statements  as sufficiently  practical  or directive  may well

serve  as a barrier  to greater  use  of NIH  Consensus  findings  in their  CME  activities.

Family  physicians  were  particularly  critical  of the  usefulness  of Consensus  Statements

for  their  practices.  In a typical,  unprompted  comment,  one  argued  that  Consensus

Conferences  are biased  by a subspecialty  focus.  The  underlying  assumption  that  the

Conference  results  are  relevant  for application  in general  medical  practice  is, at best,

partially  correct.  The  Program  would  benefit  from  a vast  increase  in practice-based

research.  Another  similarly  noted  that  the requisite  knowledge  for frontline  practice

has not  and  will not  come  from  subspeciaky,  highly selected  patients.  Thus  Consensus

guidelines  are  often  unuseful  and  sometimes  irrelevant  to generalists.  Another

argued  that  the  panels  should  make  the  recommendations  &  more  practical  and

applicable.  Consensus  Statements  are  written  by research  experts  not  knowledgeable

about  the  patients  and  settings  for which they  are  intended.31

Adequa~  of CDP procedures:  Respondents  identified  two areas  of concern  regarding 
the  mechanics  of the  Consensus  process:  Conference  planning  and  length. 
In unprompted  comments,  14 percent  of those  department  chairs  who were  familiar

with  the  Program  criticized  the  process  used  for planning  Consensus  Conferences.

They  urged  that  there  be broader  participation  in the  planning  stages  of Conferences,

including  topic-,  panelist-,  and  speaker  selection.  Several  noted,  in particular,  that

there  should  be  greater  medical  school  involvement  in the  plaming  process,  and  urged

that  all medical  school  department  chairs  in a relevant  specialty  should  be  invited  to

attend  the  actual  Conferences.  Also in unsolicited  comments,  18 percent  of those

family medicine  chairs  who were  familiar  with the  Program  urged  greater  involvement

of primary  care  specialists  in the  planning  of Conferences.

Only 31 percent  of respondents  strongly  agreed  that  the  2-1/2 day length  of the

Conferences  was sufficient  to adequately  consider  the  issues?2  Their  concerns  were

12
 illustrated  by 2 respondents  who had  particular  familiarity  with the  Program:  one

department  chair,  who had  chaired  an NIH  Consensus  Conference,  noted  that

minority  opinions  get lost in the  rush--sometime  after  midnight--to  complete  the

report  and  achieve  consensus.  I find the  format  a bit  too  rushed--even  a few extra

hours  would  help.  A second  respondent,  who had  participated  on two Consensus

panels,  likewise  rated  the  length  of the  Conferences  has  highly inadequate.

Objections  to  consensus  methods:  In unprompted  comments,  6 percent  of those 
respondents  who were  familiar  with the  CDP  expressed  a philosophical  problem  with 
the  conce  t of consensus  development  as a means  of resolving  controversies  in 
f
medicine.  3  Some  of these  respondents  objected  to what  they  regarded  as a 
cookbook  approach  to medicine,  arguing  that  this type  of activity  could  be  harmful  to 
the  independent  judgment  of practicing  doctors.  Many  of these  respondents  asserted 
that  Consensus  recommendations  are  the  result  of compromise,  and  argued  for a 
more  rigorous  method  of technolo~  assessment.  One  urged  that  NIH  use  an  explicit, 
evidence-based,  pauem  pretercnce-kwussed  approach. 
One  neurology  chair  summed  up what  several  believed  is the  fundamental  paradox  of

the  consensus  process:  The extent  of information  and  studies  that  need  to be

reviewed  for the  consensus  cannot  be  mastered  unless  the  panelists  are  already  expert.

Yet  if they  are,  they will bring  their  own biases  to the  panel.~

13
  RECOMMENDATIONS 
Our  findings  indicate  that  medical  school  continuing  education  activities  can  play  an 
important  part  in the  dissemination  of NIH  Consensus  findings.  Indeed,  medical 
school  department  chairs  and  others  responsible  for planning  and  conducting 
continuing  education  activities  are  key opinion  leaders  in their  fields  and  can  serve  as 
major  conduits  for transmitting  such  information  more  broadly. 
These  findings  also suggest,  however,  that  this dissemination  channel  is not  being  fully 
utilized,  in part  because  (1) a significant  minority  of those  people  who  play  a role  in 
medical  school  continuing  education  know little  or nothing  about  the  Program;  (2) 
many  of those  who  are  aware  of the  CDP  feel  that  its products  camot  be  readily 
incorporated  into  their  activities;  and  (3) some  voice  basic  concerns  about  the  current 
effectiveness  and  usefulnws  of the  Program. 
To  maximize  the  potential  for medical  school  continuing  education  as a vehicle  for 
disseminating  Consensus  findings,  we offer  the  following  recommendations: 
l%e IWHshoukii  takestepf  tohcreaseawaren.4m  of the CDP among tlume rqponsible  for 
conthubag medical  katim  lh  w  shouki make special  ~orts  m reach those who 
sponsor Contbluing education for gerund  and fmphysiciiwm 
Considering  the  important  role  of department  chairs  and  CME  directors  in planning 
and  conducting  medical  school  continuing  education  activities,  it is important  that  NIH 
reach  them  and  ensure  that  they  are  familiar  with the  CDP  and  its products.  To  do 
SO,NIH  should: 
.	 Maximize  the  effectiveness  of direct-mailing  efforts  by tailoring  such mailings  to 
department  chairs  and  CME  directors  as educatois.  While  some  currently 
receive  direct-mailings  in their  capacity  as members  of specialty  societies,  we 
believe  that  specialized  mailings  would  be more  effective.  The  NIH  could  use 
such mailings  as an opportunity  to encourage  recipients  to cover  Consensus 
findings  in their  continuing  education  activities. 
.	 Work  with individual  department  chairs,  members  of CME  adviso~ 
committees,  deans,  and  continuing  education  directors  at key institutions--or 
gToups of such individuals--to  plan  appropriate  dissemination  strategies. 
In addition,  NIH  could: 
.	 Work  with  organizations  other  than  medical  schools--including  relevant  specialty 
societies,  associations  of medical  school  professors  in particular  specialty  areas 
(such  as the  Society  of University  Professors  of Neurology  and  the  Society  of 
Teachers  of Family  Medicine),  the  Society  of Medical  College  Directors  of 
CME,  the  Association  of American  Medical  Colleges,  and  the  Accreditation 
15  Council  for  Graduate  Medical  Education--to  plan  appropriate  dissemination 
strategies. 
.	 Explore  ways of identifying  those  people  in individual  departments  who  are 
most  closely involved  in the  planning  and  conduct  of continuing  education 
activities  in particular  subject  areas. 
In each  of these  areas,  NIH  should  make  special  efforts  to reach  general  and  family 
physicians.  They  are  an important  target  audience  for the  CDP  because  they  both  use 
the  technologies  addressed  in Consensus  Conferences,  and  serve  as gatekeepers  who 
determine  the  type  of specialty  care  their  patients  receive.  Their  role  is likely to grow 
in importance  as a result  of expected  reforms  in the  health  care  system  and  an 
increasing  emphasis  on managed  care  systems. 
2%e NIH should _  more +ective  wavs of  encoumging  the  zkcorpomthn  of 
Corawnsw jnamgs  inw ~waruung  txiucuu.onacntiia. 
The  NIH  should  explore  ways of packaging  and  disseminating  Consensus  findings  that 
would  facilitate  their  use in CME  activities.  In this effort,  NIH  should: 
.	 Produce--or  work with  other  organizations  to produce--slides,  overheads,  or 
other  curriculum  materials.  The  NIH  should  build  upon  its experience  with 
prior  efforts  in this area. 
.	 Conduct  focus-group  discussions  with key audiences  of Consensus  findings, 
including  medical  school  department  chairs  and  CME  directors,  to learn  further 
ideas  on format  issues. 
The  NIH  could  also  examine  the  attitudes  and  practices  of other  types  of continuing 
education  sponsors.  In its fiture  research,  NIH  could  consider  the  following  ideas: 
.	 Survey  CME  directors  at community  hospitals,  State  medical  societies,  State 
and  local  medical  and  specialty  societies,  and  local  chapters  of national  societies 
regarding  their  current  and  potential  uses  of Consensus  information. 
.	 Conduct  focus group  discussions  with representatives  of national  specialty 
societies,  including  the  American  Academy  of Family  Practice  and  other 
prima~-care  groups,  regarding  their  current  and  potential  uses  of Consensus 
information. 
nae  NIHshouus@en@en  ihefomto  Wdmklnd  and  &ld&?ssbasic CO?lCtY7LS  (ZbOld  the 
~rwmus  Devekpnent  l%ogmm 
The  NIH  has  demonstrated  a firm  commitment  to ongoing  evaluation  and 
improvement  of the  Consensus  Development  Program,  including  Conference  planning, 
conduct,  and  information  dissemination.  Because  no dissemination  efforts  will be 
16
 successful  in reaching  and  influencing  people  who have  basic  concerns  about  the 
effectiveness  and  usefulness  of the  Programs  methods,  NIH  should  intensify  its efforts 
to understand  and  address  such  concerns.  Some  of these  concerns  may be  based  on 
limited  familiarity  with  or understanding  of the  CDP.  This  possibility  suggests  the 
need  for intensified  efforts  to clari~  the  Programs  procedures,  targeting  in particular 
key groups  of opinion  leaders,  such as those  we surveyed.  At the  same  time,  as noted 
in the  report,  the  fact  that  some  of these  concerns  have  been  raised  repeatedly  in 
evaluations  of the  CDP  may suggest  a genuine  need  to reexamine  some  of the 
Programs  methods. 
The  NIH  could  use  focus groups  or informal  working  groups  of department  chairs  and 
CME  directors,  and  other  key audiences,  to learn  about  and  address  these  concerns. 
As initial  steps,  NIH  should: 
Identify  ana  test  better  ways of involving a wider  range  of pc o~M ,fi L... 
planning  stage  of upcoming  Consensus  Conferences  to maximize  acceptance  of 
the  outcomes. 
Address  concerns  about  panel  and  speaker  selection  by (1) strengthening  its 
efforts  to increase  primary-care  representation,  and  (2) clearly  stating  in each 
Consensus  Statement  how panelists  and  speakers  were  selected  and  reporting 
their  credentials. 
17
  AGENCY  COMMENTS 
We solicited  and  received  comments  on a draft  of this report  from  the  Public  Health 
Service  (PHS).  The  PHS  concurred  with our  recommendations  and  identified  several 
activities  that  NIH  currently  has underway  or has  planned  that  address  these  issues. 
In particular,  PHS  noted  that  the  NIH  Office  of Medical  Applications  of Research  is 
in the  process  of publishing  a revised  statement  of operating  guidelines  and 
procedures  for the  CDP.  In addition,  NIH  will explore  new ways of exploiting  CME 
as an avenue  for  dissemination  of Consensus  findings.  The  NIH  will also  continue  its 
ongoing  efforts  to evaluate  the  procedures  and  impact  of the  CDP. 
The  PHS  also  suggested  that  the  report  be revised  to make  clear  that  respondents 
criticisms  of Ihc  Ccm~sm~MPro&grammav he  based  on limited  famil;atitv  -~tfi the 
Program  or out-of-date  iniormatlon;  PHS  suggested  that  these  opiruons  may rcxlect 
perceptions  of problems,  rather  than  actual  problems.  We  continue-to  believe  that  the 
concerns  reported  to us by survey  respondents  merit  serious  attention  by NIH.  First, 
our  respondents  represent  some  of the  best  informed  and  most  up-to-date  members  of 
the  medical  community,  and  are  in key positions  to seine  as conduits  for the 
dissemination  of Consensus  findings;  their  opinions  are  important.  Second,  the 
criticisms  raised  by this critical  minority  of respondents  have  been  identified  in prior 
studies,  some  of which were  commissioned  by NIH  itsel~  this suggests  either 
continuing  problems  or a need  for better  communication.  We have  modified  the  text 
to put  these  issues  into clearer  perspective. 
The  complete  text  of the  PHS  comments  appears  in appendix  E. 
19
  APPENDIX  A

METHODOIX3GY 
Our  findings  and  recommendations  are  based  upon  five main  sources  of information: 
1.  Mail surveys:  We  conducted  two sets of mail  surveys  in November  1992.  First,  we 
selected  three  specialty  areas  in which  recent  Consensus  Conferences  had  been  held 
--family medicine,  neurology,  and  oncology--and  surveyed  the  chairs  of those 
departments  or divisions  at all U.S. medical  schools,  as listed  in the  Association  of 
American  Medical  Colleges  Directorv  of American  Medical  Education  (see  appendix 
C for a summary  of responses).  Second,  we surveyed  the  directors  of CME  at all U.S. 
medical  schools;  the  Accreditation  Council  for Continuing  Medical  Education  provided 
us with its mailinz  list (see  appendix  D for a summary  of responses). 
In our  sutveys,  we inquired  about  overall  familiarity  with  the  Consensus  Development 
Program,  awareness  of a sample  of specific  Consensus  Statements,  use  of Consensus 
findings  in continuing  education  activities,  and  opinions  about  the  Program  and  its 
products.  We  asked  department  chairs  to consult  appropriate  colleagues  regarding 
specific  subject  areas,  as necessary. 
Of the  12 Consensus  Conferences  held  in 1990 and  1991, we selected  7 for our 
sample,  targeted  to a variety  of audiences  (see  table  A-l). 
TableA-1: Targetaudiencesforthe7sampleConsensusStatements 
Target audiences includti 
Clmsensus Statement 
~ ~ 
Swgq  for Epilep~  (March 1990)  x 
Chkicaf use of Botulinum Toxin  (November 1990)  x  x 
Intravenous Immunog[obulin  (May 1990)  x 
Adjuvant Therapyfor Colon and Rectum Cancer  x 
(April 1990) 
Earfy Stage Breast Cancer  (June 1990)  x  x 
II  I  ( 
Gastrointestinal Suqey  for Severe Obesity  x 
(March 1991) 
Treaonent of Panic D&order (September 1991)  x 
Our  sample  selection  was based  on discussions  with NIH  staff and  respondents  to our 
survey pretest,  and  was designed  to produce  a sample  that  included  more  than  one 
A-1  Statement  addressed  toeachof  thethree  specialty  groups.  Twoof  the  Statements 
were  addressed  to more  than  one  of the  groups;  the  other  five were  addressed  to only 
one  of the  groups  each.  We  asked  family medicine  chairs  about  4 of the  7, neurology 
chairs  about  2 of the  7, and  oncology  chairs  about  3 of the  7. 
We received  a response  rate  of 77 percent  among  department  chairs  and  66 percent 
among  CME  directors.  Our  respondents  represented  medical  schools  of all sizes, 
types  of sponsorship,  and  geographic  location.  We  do not  believe  that  nonrespondents 
differed  in any significant  way from  respondents  (see  appendices  C and  D for 
breakdowns  of our  response  rates). 
In the  findings,  we have  focussed  on the  responses  of department  chairs,  who  are 
generally  key medical  opinion  leaders  and who are  more  closely involved  than  CME 
directors  in the  subject  matter  of their  departments  continuing  education  activities. 
2.  In-person  and  telephone  discussions  with medical  school  dep~rmien~  cnati ~ ad 
continuing  education  directors:  We  spoke  with a large  number  of CME  directors  and 
department  chairs  both  before  and  after  the  mail survey.  In particular,  we telephoned 
all those  department  chairs  who  reported  in their  surveys  that  their  departments 
relevant  continuing  education  activities  had definitely  not  addressed  Consensus  findings, 
in order  to learn  better  why not.  We  also followed  up with a sample  of those  who 
responded  that  their  departments  CME  activities  dej7niteZy had  addressed  CDP 
findings,  in order  to learn  more  of their  thoughts  about  the  Program  and  its 
dissemination  activities. 
3.  Contacts  with  NIH:  We  conducted  numerous  formal  and  informal  discussions  with 
the  staff of NIHs  Office  of Medical  Applications  of Research  and  a review  of NIHs 
dissemination  materials.  We  also  attended  a July  1992 meeting  of the  NIH 
Coordination  Committee  on the  Assessment  and  Transfer  of Technology,  and  both  the 
public  and  executive  sessions  of the  September  1992 NIH  Consensus  Conference  on 
Gallstones  and  Laparoscopic  Cholecystectomy. 
4.  Other  discussions:  We spoke  with directors  of CME  at several  medical  specialty 
societies,  including  the  American  Academy  of Neurology,  the  American  Society  of 
Clinical  Oncology,  the  American  College  of Cardiology,  and  the  American  Psychiatric 
Association;  and  representatives  of the  Alliance  for Continuing  Medical  Education,  the 
Accreditation  Council  for Continuing  Medical  Education,  the  Society  of Medical 
College  Directors  of Continuing  Medical  Education,  the  Association  of American 
Medical  Colleges,  the  American  Medical  Association,  the  Rand  Corporation,  the 
Institute  of Medicine,  the  Public  Health  Services Agency  for Health  Care  Policy and 
Research,  and  the  Office  of the  Assistant  Secretary  for Planning  and  Evaluation. 
5.  Review  of the  literature:  We  examined  the  literature  on medical  technology 
assessment,  the  diffusion  of medical  innovations,  the  theory  and  practice  of medical 
education,  and  physician  information  habits. 
A-2
 APPENDIX  B

CONTINUING  MEDICAL  EDUCATION: 
BACKGROUND 
The  American  Medical  Association  and  the  Accreditation  Council  for Continuing

Medical  Education  (ACCME)  define  continuing  education  as consisting  of

educational  activities  which  serve  to maintain,  develop,  or increase  the  knowledge,

skills, and  professional  performance  and  relationships  that  a physician  uses  to provide

services  for patients,  the  public,  or the  profession,  and  its content  as that  body  of

knowledge  and  skills generally  recognized  by the  profession  as within  the  basic  medical

sciences,  the  discipline  of clinical  medicine,  and  the  provision  of health  care  to the

public.35  Such  information  can be  conveyed  in a variety  of settings  and  formats,

including  periodic  conferences  and  formal  lectures:  ongoing  hospital  grand  vlnds,

seminars,  and  discussion  groups;  ancl seii-instructional  materlais.

To  ensure  some  degree  of consistency,  the  ACCME  accredits  most  sponsors  of

continuing  education  for physicians,  including  most  medical  schools,  national  medical

specialty  societies,  teaching  and  community  hospitals,  State  and  local  specialty

societies,  voluntary  health  organizations,  pharmaceutical  and  device  firms,  and  for-

profit  educational  companies.  State  medical  societies  can  be  accredited  both  to grant

credit  for their  own programs,  and  to accredit  community  hospitals  as sponsors  of in-

house  and  locally  marketed  activities.  The  ACCME  also  accredits  the  NIH  Office  of

Education,  which  grants  credit  to physicians  who attend  the  public  sessions  of NIH

Consensus  Conferences.  In addition,  NIH  has begun  to explore  the  possibility  of

granting  credit  for self-study  of printed  Consensus  Statements.

Physicians  may be  required  or encouraged  to complete  a certain  number  of hours  of

CME  by their  respective  State  medical  societies;  national,  State,  or local  professional

organizations;  State  licensing  boards;  malpractice  insurers;  and  hospitals.  To  further

encourage  participation  in continuing  education,  the  AMA  grants  the  Physicians

Recognition  Award  to doctors  upon  completion  of certain  requirements.

The  primary  continuing  education  activities  of national  specialty  societies  are  large

annual  meetings  that  often  draw  more  than  10,000 attendees  and  involve  hundreds  of

different  educational  offerings.  A societys planning  committee  or education  office

usually  selects  such  offerings  from  among  proposals  submitted  by members.  Some

societies  also conduct  smaller  educational  offerings  throughout  the  year,  either  at their

headquarters  or regionally.

Medical  schools  and  teaching  hospitals  offer  a wide range  of continuing  education

programs.  Some  of these  are  conducted  in-house  at the  school  or hospital,  and  are

intended  for in-house  faculty  and  staff.  Others  are  marketed  to a broader  audience,

require  a fee,  and  are  held  at hotels,  conference  facilities,  or other  hospitals.

B-1
 At many  institutions,  including  most  medical  schools  and  hospitals,  continuing

education  activities  are  coordinated  by an  office  of continuing  education,  which

provides  educational  and  administrative  support.  The  structure  of these  offices  varies

greatly  from  one  institution  to  another:  Some  CME  offices  are  headed  by physicians, 
but  many  are  headed  by either  educational  professionals  or administrators. 
The  CME  directors  expertise  in curriculum  development,  familiarity  with  physicians 
educational  activities,  and  position  in the  medical  school  all suggest  that  they  could 
play a useful  role  in the  dissemination  of Consensus  findings.  Many  of those  who 
responded  to our  survey urged  that  they  receive  Consensus  information  directly.  One 
assistant  dean  for medical  education  noted  that  the formal  CME  activities  of medical 
schools  comprise  a major  system  for disseminating  Consensus  Statements. 
At the  same  time,  these  CME  directors  generally  have  a limited  role  in deciding  what 
information  is presented  in the  activities  for which  they  grant  credit.  They  generally 
rely on ~Ld~dh~  Ur  ii~-h~  ~~i~mitt~~s  d  redid  facuit~  or  stali  LL)rLL)~i~..:  . . .Aun 
and  program  development.  The  chairs  of individual  medical  school  departments  are 
key players  in setting  the  agenda  for continuing  education  at their  institutions.%  The 
specific  content  of most  courses  is determined  by the  individual  faculty  or staffi  as one 
director  noted,  the vast majority  of our  programs  are  initiated,  planned,  and  executed 
by individual  departments.37 
B-2
 APPENDIX  C

SUMMARY  OF  DEPARTMENT  CHAIR  SURVEY  RESPONSES 
Tables  Page C-2 
c-1  OIG  survey sample  sizes and response  rates 
c-2  Familiarity with the CDP 
c-3  Sources of information about the CDP 
c-4  Awareness of each of 7 sample Consensus Statements 
C-5  ,~wareness  of :lkiitional  Consensus  Statements 
C-6  Percentage  whose  departments  address  each type of topic  in CME  activities 
c-7 Percentage  of department  chairs who sponsored  CME  activities  in each of 7 sample 
topic areas 
C-8 Percentage  of those  department  chairs who sponsored  CME  activities  on each  of 7 
sample  topic  areas who were  aware of a relevant  Consensus  Statement 
c-9 Percentage  of department  chairs reporting  that  a relevant  CME  activity had addressed 
the  findings  of at least one  of 7 recent  Consensus  Conferences 
c-lo  Formats  of CME  activities  that  addressed  CDP  information 
C-n  Target  audiences  for medical  school  CME  activities  that  addressed  CDP  information 
C-12 Additional  use of CDP  information  in CME 
C-13 Of those  28 percent  of department  chairs that  attended  a CME  activity sponsored  by 
another  organization,  the  percentage  reporting  that  this activity had been  sponsored  by 
each type of organization 
C-14  Comments  on the CDP 
C-15  Degree  of respect  for the  findings of the  CDP 
C-16 Percentage  of department  chairs describing  each type of material  as potentially  useful 
for their  CME  activities 
C-17 Reported  degree  of department-chair  involvement  in the selection  of topics  for CME 
activities in their department 
Blank SmVeyhstrumen~  Page C-15 
c-1  1 
I Table  C-1:  OIG  survey  sample  sizes  and response  rates  I 
Family 
Medieine 
II 
chairs 
Sample  Size  110  118  119  347  ii 
I 
Number  of  84  WI  95  267 ~ 
Respondents 
Response  Rate  76%  75%  I  80%  77%  j] 
r
Family  Neurology  Oncology  Average 
Medieine  Chairs  Chairs 
Chairs
(n=&l)  (n=88)  (n=95)  (n=267) 
Very familiar  II  14%  31%  47%  3170 
Somewhat familiar, 
but not sure of all 
the details  33%  34V0  41%  36V0 
Aware, but didnt 
know much about it  25%  17%  8%  1770 
Not at all aware  II  27%  18%  3%  16570 
Source  OIG smwy of medieal school department  chairs, November 1992 
Note  Differences are statistically significant at the .05 level. 
c-2  Table  C-3:  sources  of information  about  the  CDP 
-C 
Received  a CDP  Statement  by mail 
E  ~~    ~~  ~  ~~ 
Read  a CDP  Statement  in JAMA    62%  3170  32%  40% 
* 
Read about the CDP in JAMA  61%  22%  2270  * 32$Z0 
Read a CDP Statement in a specialty 
I 
journal  13%  47%  299Z0 * 3170 
.  ~ E 
Read  about  the LDP  in a speclaity  I 
journal  189Z0  3597(0  30%  28% 
Received  a CDP  Statement  at a 
CME  activity, professional  society 
meeting,  or conference  26%  22%  33%  28% 
Heard  about  the  CDP  from 
colleagues  within  own institution  15%  25%  35%  * 26% 
Heard  about  the  CDP  from 
colleagues  at outside  institutions  1570  21%  3370  * 24% 
Read  or saw something  about  the 
CDP  in the  popular  media  1890  4%  23%  * 16% 
Source  OIG survey of medical school department  chaim, November  1992

Note  F.xciud~ those 42 department  chaim (16%) who described themselves as Not at all aware of the CDP; an

asterisk (*) indicates that differences among department  chaira are statistically significant at the .05 level.

c-3  Table  CA  Awareness  of each  of 7 sanmle  Consensus  Statements  ~~ 
Statement 
Su&ety for  Epilep~ 
Clinical  Use of Botulinum  Toxin 
Intravenous  Immunoglobulin 
Adjuvant  Theropy for  Colon  and  79% 
Rectum  Cancer  (n=92) 
il  - I  (n=61)  I  (11=!/1) ,, 
Gastrointestinal  Sutgety for  ~  3170  - I 
Severe Obesity  (n=61)  :! 
I 
Treatment  ofPanic Disorder  ~  56%  -1 
(n=61) 
I 
1

Source  OIG sutvcy of medical school department  chairs, November 1992

Note:  Excludes those 42 department  chairs (16%) who described themselves as Not at all

aware of the CDP; a dash (-) indicates that the group was not asked about a given statement. 
c-4  Table  C-5:  Awareness  of  additional  Gmsensus 
Statements,  not  included  in our sample 
Number  of Statements  Percentage  of 
respondents 
(n=225) 
None  72% 
1  19% 
2  7% 
3  2% 
4  1% 
Source  OIG survey of medical school department  chairs, November 
1992 
..-   _.  -
Table  C*.  Percentage  reporting  that  their  departments  address  each  type  of 
topic  in their  CME  activities 
r~  n%lr 
c. Specialty  society  77%  5570  71%  *  68%

practice  guidelines/

practice  parameters  (n=82)  (n=87)  (n=94)  (n=263)

Soureex OIG survey of medical school department  chairs, November 1992

Note:  An asterisk (*) indicates that the differences among department  chairs are statistically significant at

the .05 level.

c-5  Table  C-7:  Percentage  of department  chairs who  sponsored  CME 
activities  on each  of  7 sample  topic  areas 
Topic  Family  Neurology  Oncology 
Medicine  chairs  chairs 
chairs 
(n=S4)  (n=88) 
Surgery for Epilepsy  78% 
Clinical use of Botulinum Toxin  2.4%   57% 
Intravenous Immunoglobulin  23% 
(n=94) 
Adjuvant Therapy for Colon  7570 
(n=94) 
and Rectum Cancer 
Early Slagc Breast Cancer  ~  56?ZG  .:  ,. 
~t]=~~j 
Gastrointestinal  Surgery  for  12% 
Severe  Obesity 
Treatment  of Panic  Disorder  69% 
Source  OIG survey of medical school department  chain,  November 1992 
Note  A dash (-) indicates that the group was not asked about a given topic. 
C-6
 Table  C-&  Percentage  of those  department  chairs  who  sponsored 
CME  activities  on  each  of  7 sample  topic  areas who  were  aware  of  a 
relevant  COnsensus  Statement 
Topic  Family  Neurology  0nu310gy 
Medicine  chairs  chairs 
Chaim 
Surge~  for Epilepsy  87% 
(n=54) 
Clinical  use of Botulinum  Toxin  100%  79% 
(n=2)  (n=43) 
Intravenous  Immunoglobulin  45% 
(n=20) 
Adjuvant  Therapy  for Colon  w% 
and l<QUILJiil  LtiL;..J~ 
Early Stage  Breast  Cancer  I  60%  929io 
I  (n=35)  (n=79)  I 
Gastrointestinal  Surgery  for  44% 
Severe Obesity  (n=9) 
Treatment  of Panic Disorder  55% 
(n=42) 
Source:  OIG survey of medieal school department  chairs, November 1992

Note  Excludes those who were not at all aware of the program; a dash (-) indicates that the

group was not asked about a given topic.

c-7  Table  C-9  Percentage  of  department  chairs  reporting  that  a relevant

CME  activity  had addressed  the  findings  of  at least  one  of seven 
I

recent  Consensus  Conferences  durim  the  urevious  3 Years

e.	 .
 I

Family  Neurology  Oncology  Average  ! 
Medicine  chairs  chairs 
Chairs 
(n=65)  (n=73)  (n=85)  (n=223)  ~ 
Definitely  yes
 15%  36970  65% 
Probably  yes
 23%  2190  20%  21%  1 
Definitely  not
 12%  1490  3%  9%  ~ 
Probably  not
 15%  11%  7%  11% jl

I

Dont  know
 3470  19%  5%  1870  I

  - 
Source  OIG survey 0[ medical school department  chairs, Novemoer  1992

Note  Inchsdea only those 223 who did addreas at least one of the 7 sample topics in a CME  I

activi~  differences are statistically significant at the .05 level.  1

C-8
 Table  C-l&  Formats  of  CME  activities  that  addressed  CDP  information 
Family  Neurology  OncOlogy  Average 
Medicine  chairs  chairs 
chairs 
(n=25)  (n=41)  (n=73)  (n=139) 
Large  group  activity  6090
 56%  44%  50% 
Small group  activity (eg. discussion

grouphvorkshop)  24%
 34%  53%  * 42% 
In-house  activity (eg. grand  rounds)  60%
 73%  71%  70% 
Activity open  to outside  audiences  4090
 44%  34%  38% 
Self-directed  study  4%
 5%  5%  5% 
 ,LIrcc ~!G  sumtv of medical schooi department  c!,.aim.November 1992 
II Note:  includes only those 139 (62 percent) wno reported  that their @lE  achwties on the relmant tovics naa u.. hlLUYor  ,[ 
probabty addressed-CDP findin&; &terisk (*) indie&s  that the differences noted are statistically sign~fieant at the .05 level. 
Table  C-II:  Target  audiences  for medical  school  CME  activities  that addressed  CDP 
information 
=EqE!EElr 
In-house  hospital  staff  44%  I  76%  71%  * 68940 
In-house  medical  school  faculty  48%  I  98%  84%  * 81% 
Residents  80%  ~  95%  89%  89% 
Researchers  from  other  institutions  o%  5%  16%  10% 
Practicing  physicians:  local or nonlocal 
68%  76%  71%  72% 
From the local area 
6s%  66%  7070  6a% 
From outaide the local area 
40%  qlqo  30%  35% 
Source  OIG sutvey of medieal school department  chairs, November 1992

Note  Includes only those 139 (62 pereent) who reported  that their CME activities on the relevant topies had definitely or

probably addreased CDP findingq an asterisk (*) indieatea that the differences noted are statistically significant at the .05

level.

c-9  Table  C-12  Additional  use  of  CDP  information  in CME 
Family  Neurology  Oncology  Average 
Me&ine  Chahx  Chairs 
chairs  n=264 
Percentage  aware of CME  activities  that  7%  970  18%  *  12% 
have been  conducted  at their  institution  that 
addressed  information  from a Consensus 
Conference  other  than  those  in our sample  (n=83)  (n=87)  (n=94)  (n=264) 
Percentage  that  have attended  a CME  14%  22%  44%  * 2870 
activity sponsored  by an organization  other 
than  their  own that  addressed  information 
from a Consensus  Conference  (n=83)  (n=87)  (n=95)  (n=263) 
Source:  (3JG survev of medical school department  chaim. November 1992 
Table  C-13:  Of  those  73 (28  percent)  that  attended  a CME  activity  sponsored  by another 
organizitio~  the  percentage  reporting  that this activity  had been  sponsored  by each  type  of 
organization 
ME 
a. Another  medical  school/teaching 
=b. A national  specialty  society  7  :  :  : 
c. A state  or local  specialty  society  17%  5%  12%  11% 
d.	 A for-profit  firm (eg.,  a  25%  11%  7%  11% 
pharmaceutical  company,  etc.) 
Souros  OIG sumq of medical school department  chain,  November 1992 
c-lo  Table  C-14  Comments  on  the  CDP 
Very  Some  A  Not  1 N/A 
much  what  little  at all  ~ 
a. the  topics  addressed  in NIH  ~Average  62  32  4  01  2

Consensus  Conferences  are

timely  , Family Medicine  36  51  11  o~  2

I ~n::oBB 77  17  1  0  4 
68 30 1  0 0 
b. the issues addressed  by the  Average  38  47 12  11 3

CDP  are appropriate  for

resolution  through  the  group  Family Medicine  39  41  15  0  5

judgment  method  of consensus  Neurology  46  45  6  o  3

development  Oncolo~  30  52  14  2  1

. 
c. the  Consensus  Statements  Average  *  48 41 9  1 2 
usefully distill large bodies  of 
information Family Medicine  49  31  16  0  3 
Neurology  52  38  4  1  4 
Oncology  45  49  7  0,  0 
d. the  2-1/2 day format  of the  Average  31 39 9  4  18

NIH Consensus  Conferences

allows sufficient  time to examine  Family Medicine  24  32  14  71  24

and consider  the  issues in  Neurology  35  42  7 
01  16

question  Oncology  33  40  7  4  16

e. the  recommendations  of the  Average  *  40 50 7  1 3 
consensus panels  are practical 
Family Medicine  26  54  13  3  3 
Neurology  52  42  3  0  3 
Oncology  39  52  7  0,  2 
L the Consensus  Statements  are  Average  *  38  47  10  1  4 
sufficiently directive 
Family Medicine  26  51  15  3  5 
Neurology  46  46  4  0  4 
Oncology  41  46  10  1  2 
1. the  recommendations  are  Average  55  40  3  0  2

iirected towards clinical, not just

esearch audiences  Family Medicine  40  45  12  0  3

Neurolo~  58  39  0  0  3 
Oncology  62  37  0  0  1 
1.NIH sponsorship of the CDP  Average  63  24  9  2  3

mhances the credibility of the

?rogramsfindings  Family Medicine  56  23  11  21  8

Neurology  71  22  3  3  1 
Oncology  61  26  11  1  1 
C-n  Table  C-14:  Comments  on  the  CDP 
Very  Some  A  Not  NIA 
,  much  what  little  at all 
I 
i. the  topics addressed  by the  Average  52
 38 9  1  1 
CDP  are appropriate  for

incorporation  into  CME  Family Medicine  38
 48 13  0  2 
activities  Neurology  58
 35 4  1 1 
Oncology  57  34 9  1 0 
j. the  format  in which the  Average  33  45  16  2 4 
findings of Consensus  panels  are 
presented  are  useful  for CME  Family Medicine  12  60  24  2 3 
activities  Neurology  42  39  12  3 4 
Oncology  40  40  14  2 3 
.- .    . 
. 
!1 
Note  Excluding these who were not at all aware of the CDP; n=222 (Family Medicine n=61;  Neurology n=6q  Oncology   
n=92);  an asterisk (*) indicates lhat the differences are statistically significant at the .05 level; numbers are in percxmt. 
Table  C-15:  Degree  of  respect  for the  findings  of  the  ~ 
I 
A little  No 
respect  respect 
( I 
Average  49  44  8 0 
Family  Medicine  38  52  10  0 
Neurology  64  32  31  0 
Oncology  44  46  10  01 
1 
Source  OIG survey of medical school department  chairs, November 1992 
Note:  Excluding those who were not at all aware of the CDP; n=220  (Family 
Medicine n=61;  Neurology n=68;  Oncology n=91);  the differences among 
department  chairs are statistically significant at the .05 level. 
C-12
 Table  C-16  Percentage  of department  chairs describing each type of material as potentially 
useful for their  CME  activities 
Family  Neurology  ~ Oncology  ~  Average 
Medicine  Chairs  ;  Chairs  I 
chairs 
~ 
CME  curriculum  materials  for small-group  I  II 
settings  89%  76910  80%  81~o 
CME curriculum materials for large-group 
settings  78%  69$Z0I  62%  70% 
CME  curriculum  materials  for self-directed 
study  80%  64%  ~~  61%  ;F,  * 6870 
CME  curriculum  materials  for longer  CME  ~  II 
activities  (1 day conferences/  II 
courses)  II  45?70  4570  39%  I  4370 
Videotapes of Conference highlights  II  42%  I  49%  35%  II  42% 
[
Videotapes of Conference press  I 
conferences  9%  11%  6%  9% 
Conference proceedings  I 
in booklet form  62%  82%  81%  * 75% 
Reference  bibliographies  86%  85%  II  * 81% 
Complete  listing of past CDP  topics  83%  83%  ~  74%  I  80% 
Source  OIG suIvey of medical school department  chairs, November 1992

Note  N is based on all 267 department  chair respondents,  regardleas of their level of awareness of the CDP; the response

rate for individual questions ranged from 235 to 250 an asterisk (*) indicates that the differences among department  chaim

are statisticaifv simificant  at the .05 level.

C-13
 Table  C-17  Reported  degree  of  department-chair  involvement  in the  selection  of topics  for 
~=F 
, Very involved  44%  78V0  62% 
Somewhat involved  !  47%  16%  33%  32% 
Not at all involved  9%  6%  2%  6% 
Source  OIG sumey of medical school department  chain,  November 1992 
C-14
 SURVEY  INSTRUMENT  MEDICAL  SCHOOL  DEPARTMENT  CHAIRS

The  following  survey  was  sent  to  chairs  of  family medicine.  Similar  surveys  were  sent 
to chairs  of  neurology  and  oncdo~,  in  those  surveys,  questions  A-3 and  B-2 addressed 
Casensus  topics  relevant  to  those  fields. 
C-15
 SECIION  k  AWARENESS  OF  THE  NH  CONSENSUS  DEVELOPMENT  PROGRAM 
1.  Prior  to receiving  this survey, how familiar were you with the  National  Institutes  of Health 
Consensus  Development  Program?  Please  check one: 
~El 
Very familiar

Somewhat  familiar, but not sure of all the details

Aware,  but didnt  know much about  it

Not  at all aware 
I

If Not  at all aware,  then  please  go  to Section  B on  Page  2.  Otherwise,  please  continue. 
2.  From  what  sources  have  you  heard  about  the  CDP  during  the  past  three  years?  Please  check all 
that  apply: 
a. Received  advance  notice  of at least one  upcoming  NIH  Consensus  Conference  in 
the  mail 
~1----1 
b. Received  at least one  NIH  Consensus  Statement  in the  mail  IL----I 
IIc. Received  at least one  NIH  Consensus  Statement  at a CME  activity, professional  II  II 
IIsociety meeting, or conference  L-4 
I d. Heard about the CDP from colleagues within my institution 
IHI 
e. Heard  about  the  CDP  from colleagues  at other  institutions  I-----4 
II
f. Read about the CDP in .MM4  IL--! 
g. Read about the CDP in a specialty journal.  Please specify: 
h. Read or saw something about the CDP in the popular media 
IMI

i. Read  at least one  NIH  Consensus  Statement  in L0L4 
j. Read  at least one  NIH  Consensus  Statement  in a specialty journal.  Please  speci~: 
H 
k. Other.  Please speciijc 
C-16
 3.  Prior  to receiving  this  survey,  were  you  aware  of the  fo]]owing specific Consensus  Statements? 
IITOPIC 
IIa. Clinical  Use of Botuhzwn  Toxin  (November  1990) 
IL-U 
b. Earfy Stage Breast Cancer  (June  1990) 
c. Gastrointestinal  Suqgety for Severe Obesity  (March  1991)  l--+-+ 
d. Treatment  of Panic Disorder  (September  1991) 
Iul 
4.  Please  list any other  specific Consensus  Statements  of which you are aware: 
SECTION  B.  USE  OF  NH-I CONSENSUS  CONFERENCE  INFORMATION 
1.  Does your department  address  the  following  types of topics in CME  activities? 
~EIEl 
a. The results  of medical  technology 
assessments 
b. Clinical  applications  of research 
c. Specialty  society  practice  guidelines/ 
practice  parameters 
.B 
2.  Has your department  conducted  a CME  activity in the  past three  years  (either  in-house  or for 
outside  audiences)  that  addressed,  at least  in part,  any of the following  topics: 
lUPIC 
a. Clinical Use of Botulinum  Toxin  ~a  0 
b. Early Stage Breast Cancer 
II 1  1 
c. Gastrointestinal Surgery  for Severe  Obesity 
II  I I 
d. Treatment  of Panic Disorder  II  I  I 
C-17
 3.  If ~  for at least  one  of these  topics, were the  findings of the  NIH  Consensus  Conference  on that 
topic addressed  in the  CME  activity (ies)?  Please  check one: 
d

Definitely  yes

Probably,  but  not sure

Dont  know

Probably  not,  but not sure

Definitely  not

3b. If YES,  what was the format(s)  of the activity(ies)?  Please  check  all that  apply: 
a. Large  group  activity  II  II 
b. Small group  activity (eg. discussion  group/ 
workshop) 
c. In-house  activity (eg. grand  rounds) 
d. Activity open  to outside  audiences  l----+ 
e. Self-directed  study 
L

f. Other:  Please  describe: 
3C. To whom  was the  activity(ies)  targeted?  Please  check  all that  apply: 
~m 
a. In-house  hospital  staff 
b. In-house  medical school  faculty  II  II 
II  II 
c. Residents 
d. Researchers  from outside  your institution  II  II 
e. Practicing physicians from the local area  II  II 
f. Practicing physicians from outside the local 
area 
g. Other:  Please  describe: 
!_____l 
C-18
 4.	 Are  you aware of any CME  activities sponsored  by your  institution  that  have addressed  information 
from other  NIH  Consensus  Conferences? 
Yes  No 
4b.  If YES,  please  list which Consensus  Conference(s): 
5. Have you ATIENDED  any CME  activity sponsored  by an organization  other  than  your own that 
addressed  the  findings of one  of the  NIH  Consensus  Conferences  noted  above? 
Yes  No 
5b. If YES, what sort  of organization  sponsored  this activity (ies)?  Please  check  aii that apply: 
IIa. Another  medical school/teaching  hospital  Iul 
b. A national  specialty  society 
I-----! 
c. A state  or local specialty  society 
d. A for-profit  firm (eg., a pharmaceutical 
company,  etc.) 
e. Other  (please  specify): 
H 
IL  II Ii 
C-19
 SECITON  C  COMMENIS  ON  THE  CDP 
1. The  opinions  of  CME  and clinical  profSion&  tow~ds  the  ~p  and towards  group  judgment 
efforts  in general  may affect  the  way in which  specific  r~mmendatiom  from  the  Program  are received 
In the  following  table,  we  ask you  to answer  several  questions  by circling  a choice  on  a four-point  scale, 
where  1 =  Verv  Muck  2  =  somewha~  3  =  A Little,  and 4  =  Not  at All.  If you  are unable  to answer 
a queatio~  please  circle  Not  Applicable,  LA 
! Please  rate  the  degree  to which	 Vay  Some- A  Notat  1NA 
Much  what  Little  Au 
1 a. the  topics  addressed  in NIH  Consensus  1234  NA1 
], Conferences  are timelv 
I b. the issues addressed  by the  CDP  are appropriate  1234  ~NA 
~ for resolution  through  the  group judgment  method  of  I 
 consensus  development 
c. the  Consensus  Statements  usefully distill large  1234  NA 
~bodies of infoma?ion 
..  .. 
1d. the  2 lfl-day  format  of the  NIH  Cons&us  1234  I NA  ~: 
I Conferences  allows sufficient  time to examine  and  I 
I 
~1consider  the issues in question 
e. the recommendations  of the consensus  panels  are  1234  ~NA  II 
i practical 
1 
 f. the  Consensus  Statements  are sufficiently  directive  1234  lNA\ 
I 
IIg. the recommendations  are directed  towards  clinical,  I  1  2  3  4 I NA  II 
InIot just  research audiences  I
) 
1 
h. NIH  sponsorship  of the  CDP  enhances  the  1234  INA1 
, credibility  of the  Programs  findings  (
I  \ , 
i. the topics addressed by the CDP are appropriate  1234  ~NA   
~ 
! for incorporation into CME activities  ! 
4 
i	 j. the  format in which the findings of Consensus  1234  ~NA 
panels are presented  are useful for CME activities 
2.  Please rate your overall degree of respect for the findings of NIH Consensus Development 
Conferences (circle one): 
Great  Some  A little  No 
Respect  Respect  Respect  Respect 
1 2 3 4 
C-20
 SECIfON  D:  CME  ACTIWIES 
In this sectio~  we  ask how  NIH  might  make  information  from  the  NIH  bnsensus  Development 
Program  more  useful  for CME  activities. 
1.  Would  it be useful  if NIH  made  the  following  materials  related  to Consensus  Development  Program 
topics available  for your  use in CME  activities?  Please  check Yes or No: 
I

IF=F=

i  MATERIALS 
I 
a. CME  curriculum  materials  for small group  settings 
I 
~ 
b. CME  curriculum  materials  for large group  settings 
] c. CME  curriculum  materials  for self-directed  study 
d. CME  curriculum  materials  for short  sessions

(eg. 1 hour  grand  rounds) 
A  E

c. Ch4E curruuurn  mawri~is  luI  iol]gcr CJIE  a-uv]ties 
II(eg. 1 day conferences/courses)  ~,1 
I f. Videotapes  of conference  highlights 
g. Videotapes  of conference  press conferences 
h. Conference  proceedings  in booklet  form 
i. Reference  bibliographies 
j. Complete  listing of past  CDP  topics 
k. Other:  Please  speci~:  I 
2.  How else might NIH  work with medical school  faculty to facilitate  the  use of Consensus 
Development  information  in CME  activities? 
3.  Please  add any additional  comments: 
C-21
 SECIION  E  RESPONDENT  INFORMATION

In order  to better  understand  the  responses  to our survey  questions,  we  ask a few  questions  about  your

professional  background  This  information  will be  used  solely  in aggregate,  and will be  kept

confidential.

Your  name: 
Your  title: 
1.  Please  note  your  area of specialty: 
2. How long have you been  head of your department 
or section? 
3. Please  rate  the degree  to which you are 
{~~~1.~~din ~~:: ~ele::~on U! t.,pi~s (or 
CME  activities  m your department  (circle one): 
Very  Somewhat  Not  at all 
Involved  Involved  Involved 
Thank  you  for taking  time  to complete  this surv~  we  appreciate  your  assistance.  Please  return  the 
sumey  in the  enclosed  postage-paid  envelope  to 
HHS/OIG/OEI 
Tel: 
FAX: 
c-22  APPENDIX  D

SUMMARY  OF  CME  DIRECTORS  SURVEY  RESPONSES 
Tabl~  Page  D-2 
D-1  OIG survey sample  size and response  rate

D-2  Familiarity  with  the  CDP

D-3  Sources  of information  about  the  CDP

D-4  Awareness  of each of 7 sample  Consensus  Statements

D-5  )waremxs  of additicm:]! Cmsensus  Stltements

D-6  Percentage  reporting  that  their  institutions  address  each type of topic in their  CME  activities 
D-7  Use of CDP  information  by CME  offices 
D-8  Comments  on  the  CDP 
D-9  Degree  of respect  for the  findings of the  CDP 
D-10 Percentage  of CME  directors  describing  each  type of material  as potentially  useful  for their 
CME  activities 
Blank SWey  Instrument  Page  D-9 
D-1
 Table  D-1:  OIG  survey  sample  size  and 
remonse  rate 
Number  of Respondents  I  79 
Response  Rate  66.4% 
1 
Table  D-Z  Awareness  of  the  CDP  II 
Very familiar  -I 
L----l
Somewhat  familiar, but not sure  of all  ~ 
the details  48%
Ii
Aware,  but didnt know much about  it  I,I41
Not  at all aware  II  1890 
I 
Source  OIG survey of medical school CME directo~  November  1992  II 
D-2
 Table  D-3:  Sources  of  information  about  the  CDP  II 
1 
_7iiE 
Directors 
(n=65) 
------
Received  a CDP  Statement  by mail  82% 
Received  advanced  notice  of a conference  bv mail  59?70 
Read a CDP Statement  in JAMA  25% 
Read about the CDP in JAMA  31% 
Read a CDP Statement  in a specialty journal  6% 
Read about the CDP in a specialty journal  2% 
Received  a CDP  Statement  at a CME  activity, 
proiesswna:  society meeting,  or conicrence  ?gy~  
Heard  about  the  CDP  from colleagues  within  own 
institution  17% 
Heard  about  the  CDP  from colleagues  at outside 
institutions  12% 
Read  or saw something  about  the  CDP  in the 
popular  media  15%
; 
Source:  OIG sumeyof medical school CME directors;  November 1992

Note  Excludes those 14 CME directom (18%) who described themselves as Not at all

aware of the CDP

D-3
 Table  D-4  Awareness  of each  of 7 sample  Consensus 
Statements  II 
Statement 
Direetors 
(n=62) 
Surgey for Epi[ep~  32% 
Clinical  Use of Botulinum  Toxin  29% 
I/Intravenous Immuno@obulin 
Adjuvant  Therapy for Colon and Rectum  44% 
Cancer 
 
Source:  OIG sutvev of medical school CME directow  November 1992 
Note  Excludes [hose 14 CME directom (187.) who described Themselves 
~
Table  D-5:  Awareness  of  additional  Consensus 
Statements:  Percentage  reporting  that  they  were 
aware  of Statements  not  in our sample 
Number  of  Statements  Percentage  of 
respondents 
(n=65) 
none  7970 
1  9% 
2  5% 
3  3% 
4  3% 
6  1.5% 
Source  OIG survey of medieal schwl  CME directo~  November 1992 
D-4
 Table  D-&  Percentage  reporting  that  their  institutions  ~ 
address  each  tvue  of  to~ic  in their  CME  activities  I 
======+ 
Directors 
(n=78)  I 
The results of medical technology assessments 
Clinical applications  of research  d94%  I 
Specialty  society practice  guidelines/  60% 
practice  parameters 
Source:  OIG survey of medical school CME directors;  November  1992  j 
D-5
 -r 
Table  D-7  Use  of  CDP  information  by CME  offices
[ 
a. Percentage  reporting  that  their  office plays a role in topic selection  82%

for CME  activities  (n=78)

[b. Percentage  reporting  that  they are familiar with the  content  of the  96%

CME  activities  for which  they grant  credit  (n=79)

I
c. Percentage  of those  familiar with the content  of their  CME  23%

activities who were  aware of any CME  activities  sponsored  by their

institution  during  the  past 3 years that  had addressed  information

from an NIH  Consensus  Conference  I (n=75)

d. Percentage  of those  CME  directors  who had received  a 
Consensus  Statement  by mail during  the  past 3 years who had
forwarded  it to a member  of the  CME  committee  or faculty: 
never  67% 
once  4% 
2-4 times  19% 
5 or more  times  170 
not sure  9% 
(n=79) 
e. Percentage  reporting  that  a faculty or staff member  had 
approached  them  during  the  previous  3 years with a recommendation 
to offer  a CME  activity addressing  a specific NIH  Consensus 
Conference: 
never  84% 
once  6% 
2-4 times  5% 
5 or more  times  090 
not sure  5% 
(n=79) 
Source  OIG sutvey of medical school CME directo~  November 1992 
D-6
 Table  D-&  Comments  on  the  CDP 
YLui_lE

Very  some 
much  what 
m 
a. the  topics  addressed  in NIH  Consensus  46  33 
Conferences  are timely 
b. the  issues addressed  by the  CDP  are appropriate  20  51 
for resolution  through  the group judgment  method 
of consensus  development 
c. the  Consensus  Statements  usefully distill large  37  35 
bodies  of information 
d. the 2-1/2 day format  of the  NIH  Consensus  18  24 
Conferences  allows sufficient  time to examine  and 
.,
~:~nsl[~~~t~l:. :  es  in am.x:icn 
e. the  recommendations  of the consensus  panels  are  28  30 
practical 
f. the  Consensus  Statements  are sufficiently  directive  25  33  7  0  36  ~ 
18 ~
g. the  recommendations  are directed  towards  clinical,  38  28  2  0 
! 
not just  research  audiences 
h. NIH  sponsorship  of the  CDP  enhances  the  61  20  2  0  18 ~ 
credibility  of the  Programs  findings 
i. the  topics  addressed  by the  CDP  are appropriate  50  31  3  0  16 ~ 
for incorporation  into  CME  activities 
, 
j. the  format  in which  the  findings of Consensus  18  44  16  0 21 i 
panels  are presented  are useful  for CME  activities 
1 
sour~  OIG survey of medical school CME directors, November 1992 
Note  Excluding thase who were not at all aware of the CDP; n varies from 61 to 63 for individual questions; numbers are in 
I 
percent 
D-7
 Table  D-9  Degree  of respeet  for the  findings  of  the  CDP  II 
I  Great  Some  1 A little  No  I 
~1Respeet  ~ respeet  respect  respeet 
i 
CME  directors  59 I 38  3  0 
Source  OIG survey of medical school CME directo~  November  1992

Note  Excludes those who reported  that they were Not at all aware of the CDP; n=58

Table  D-l&  Percentage  of CME  direetors  deseriiing  each 
type  of  material  as potentially  useful  for their  CME 
activities 
IICME curriculum materials for small-group settings  II  89% II 
IICME curriculum materials for large-group settings  88% 
II  1 
IICME curriculum materials for self-directed study  II  80% II 
CME curriculum materials for short sessions 
(eg., 1 hour grand rounds) 
CME curriculum materials for longer  CME 
activities (1 day conferenceshxmrses) 
II IHI 
Videotapes of Conference highlights 
IIVideotapes of Conference press conferences  IL---=-I 
II II  I 
Conferenceproceedhtgsin booldet form  80% 
IIReference bibliographies  II  81%  II 
Complete listing of past CDP topics  80% 
Source  OIG survey of medieal school CME directo~  November  1992

Note  N is based on all 79 CME director respondents,  regardless of their level of

awareness of the CDP; it ranged from 70 to 75 for individual questions

D-8
 SURVEY  INSTRI.JIWEITC  MEDICAL  SCHOOL  CME  DIRE~ORS 
D-9  SECTION A  AWARENESSOF THE NIH CONSIWSW3I)IWELOPMENTPROGRAM 
1. Priorto receivingthissurvey,howfamiliarwereyouwiththeNationalInstitutesofHealthConsensus 
DevelopmentProgram(CDP)? Pleasecheckone: 
ii

ml 
!II
 Ul 
~ Very familiar

II  II 
1 Somewhat familiar, but not sure of all the details

II 
I Aware, but didnt know much about it
 II 
il Not  at all aware

lL__J 
If Not at all aware, then please go to section  B on Page 2  Otherwise, please continue. 
2.  From what sources have you heard  about  the CDP (iurmg the past three  years?  Please  check all that  apply: 
SOURCE	 ,:  4

1

a. Received advance notice of at least one upcoming NIH Consensus Conference in the mail  II

b. Received at least one NIH Consensus Statement in the mail 
i

c. Received at least one NIH Consensus Statement at a CME activity, professional society  r
II 
meeting, or conference  ~ 
d. Heard about the CDP from colleagues within my institution 
E 
!1 
e. Heard about the CDP from colleagues at other institutions 
f. Read about the CDP in l-4Wl  1--
,---
g. Read  about  the  CDP  in a specialty journal.  Please specitjc

I

h. Read  or saw something  about  the CDP  in the popular  media  ,1 
y 
i.  Read  at least one  NIH  Consensus  Statement  in .L4M4 
;-

II

j. Read  at least one  NIH  Consensus  Statement  in a specialty journal.  Please  speci@:  I

k. Other.  Please  specify:  IIrI

D-10
 3.  Prior  to receiving  this survey, were you aware  of the following  specific  Consensus  Statements?  Please check 
Yes or No: 
IIa. Surgeryfor Epilep~  (March  1990) 
II  I 
b. Clinical Use of Botzdinum Toxin  (November  1990) 
IL--L 
c. IntravenousImrnunoglobulin (May 1990)
II  II  I 
i  d. Adjuvant Thera~ for Colon and Rectum Cancer (April  1990) 
I  II  I 
IIe. Early Stage Breast Cancer (June  1990)  II  I 
f. GastrointestinalSurgeyfor SevereObesity (March  1991) 
g. Treatment of Panic Dtiorder  (September  1991) 
H 
4.  Please  list any other  specific Consensus  Statements  of which you are aware: 
SECTION B:  USE OF NIX-ICONSENSUS  DEVELOPMENT  PROGRAM  INFORMATION 
Medicai school  CME offices differ in the degree to which they are involved in the seleetion  of topics and 
development  of eurrieula for the CME activities for which they grant credit.  In this section, we ask some 
questions  about how your program operates, and about the use of CDP information  at your institution. 
1.  Does your institution address the following types of topics in CME activities?  Please check Yes or No: 
a. The results  of medical  technology 
assessments 
b. Clinical  applications  of research  I 
II  I II 
c. Specialty  society practice  guidelines/ 
practiceparameters 
2.  Does your CME office play a role in the choice of topics for CME activities for which you grant credit? 
Yes  No 
D-II
 3. Are you familiar  with  the content  of the CME  activities  for which YOU grant  credit? 
Yes  No 
3b.  If YES,  are you aware of any CME  activities  sponsored  by your  institution  during  the past three 
years that  have addressed  information  from an NIH  Consensus  Conference? 
Yes  No 
3c.  If YES,  please  list which  Consensus  Conference  (s): 
4.  How many times  in the  past  three  years have you forwarded  an NIH  consensus  Statement  to a member  of 
your CME  committee  or a member  of your  faculty for review? 
None  one  2-4 times  5 or more times  Not Sure 
Which Statement  (s)? 
4.  How many times  in the  past three  years has a member  of your  institutions  faculty or staff approached  you 
with a recommendation  to offer a CME  activity addressing  a specific NIH  Consensus  Conference? 
None  One  2-4 times  5 or more  times  Not  Sure 
Which  Statement  (s)? 
D-12
 SECIION C  COMMENTS  ON THE CDP 
1. The opinions  of CME and clinical professionals  towards the CDP and towards group judgment  efforts in 
general may affect the way in which specific recommendations  from the Program are received  In the following 
tabl~  we ask you to answer scweral questions by circling a choice on a four-point  scale, where 1 =  Vew  MUCIL2 
=  Somewha& 3 = A Utt.le, and 4 = Not at All.  If you are unable to answer a question,  please circle Not 
Applicable  &A 
Please rate the degree to which  VW  ~- A  Notat  NA 
MudI  what  little  Au 
a. the topicsaddressedhtNIHConsensus(lmferencesare  1234  NA

timely

b. the issuesaddressedby the CDP are appropriate for  1  234  NA

resolutionthroughthe groupjudgmentmethodof

consensusdevelopment

c. the ConsensusStatementsusefullydistill large bodies  of  1  234  NA 
intorm.llloll 
- _
d. the 2 1/2 day format of the NIH Consensus Conferences  1  234  NA	
allows sufficient  time to examine and consider the issues	
in question	
e. the recommendations  of the consensus panels are  1  234  NA	
practical	
f. the Consensus Statements  are sufficiently directive  1234  NA	
g. the recommendations  are directed towards clinical, not  1  234  NA	
just research audiences	
h. NIH sponsorship  of the CDP enhances the credibility of  1234  NA	
the Programs findings	
i. the topics addressed by the CDP are appropriate for  1  234  NA	
incorporation  into CME activities	
j. the format in which the findings of Consensus panels  1234  NA	
are presented are useful for CME activities	
2. Please  rate your  overall  degree  of respect  for the findings  of NIH  Consensus  Development  Conferences 
(circle  one): 
Great  Some  A little  No 
Respect  Respect  Respect  Respect 
1 2 3 4 
D-13
 SECIION D  CME ACmTITES 
IIIthis seetio~  we ask how NIH might make tio~tion  from the NIH COIMXMUDSevelopment  Program more 
useful for CME activities. 
1.  Would it be useful if NIH made the following  matefials  related to consensus  Development  Program topics 
available for your use in CME  activities?  Please  check Yes or No: 
a. CME  curriculum  materials  for small group  settings 
II  r  I  I 
b. CME  curriculum  materials  for large group  settings 
II c IL>II 
II CME curriculum materials for self-directed study  I 
II  I 
IId. Videotapes  of conference  highlights  I 
i 
e.
Videotapesof conferencepress conferences 
II
,,  I  I 
g.
II
Referencx  bibliographies 
(1  I 
h. Complete  listing of past  CDP  topics 
II i. r 1 
II Other:  Please  specify:  II  I 
2.  How else might  NIH  work with medical  school  CME  offices or facul~.  to facilitate  the use of Consensus 
Development  in~ormation  in CME  activities? 
3.  Please  add any additional  comments: 
D-14
 SECTION E2 RESPONDENT  INFORMATION 
In order to better understand the responses to our survey questions, we ask a few questions  about your 
educational  and professional  background.  This information will be used solely in aggregate, and will be kept 
confidential. 
Your  name: 
Your  title: 
1.  Please  note  your  educational  degree  (s): 
2.  If you are an M.D., please  note your  area 
of specialty 
2. How long.  have vou been  in the field of C,ME?
3.  How long have you been  in your  current  position? 
I%ankyou for taking time to complete  this sunq  we appreciate your assistance.  Please return the survey in 
the enclosed  postage+aid  envelope  to 
HHS/OIG/OEI 
Tel: 
FAX: 
D-15
  APPENDIX  E 
PUBLIC  HEALTH  SERVICE 
COMMENTS  ON THE  DW  REPORT 
E-1  ,, 
...,, 
/+  $ 
:~  DEPARTMENT OF HEALTH & HUJ}lW  SERVICES
;
5  Public  Health  Sewi< 
-g 
*, 
+>%,,
RockvilMleD
 20857~

DEC
27  1~ 
From:  Deputy  Assistant  Secretag  for Health Management 
 Operations 
Subject:	 Office  of  Inspector  General  (OIG)  Draft  Report 
NIH  Consensus  Development  Progrm:  The 
Dissemination  of 
Findings  Through  Medical  School  Continuing  Education 
Activities,    OEI-01-91-01760 
To:  lnspeczor  General,  OS 
Attached  are  the  Public  Health  Service  comments  on  the  subject 
OIG  draft  report. 
We  concur  with  the  recommendations  and  the 
National  Institutes  of Health has taken or plans to take 
actions  to  implement  them. 
Attachment 
E-2
 PUBLIC HEALTH SERVICE  (PHS~  COMMENTSON THE  OFFICE  OF

INSPECTOR  GENERAL  (OIG~ DRAFT REPORT  THE NIH

CONSENSUS  DEVELOPMENT  PROGRAM:  DISSEMINATION

OF FINDINGS THROUGH MEDICAL SCHOOL CONTINUING

EDUCATION ACTIVITIES,  OEI-01-91-01760

General Comments

The National  Institutes of Health  (NIH) appreciates  the study

performed  by  OIG  of  an  important  avenue  for information 
dissemination  that could enhance the effectiveness  of the 
Consensus  Development  Program  (CDP).  Continuing Medical 
Education  (WE)  was  identified  in previous  evaluation  studies 
as an  important  channel  wi.thi.n  American  medicine  that  could  be 
better  utilized  by  the  CDP.  The OIG report reaffirms  this 
observation  and  suggests  several  important  steps  that  could  be 
-.]<en.  ~~-,?:7T::f  ~  of-:-:- -(-~ of  ::2~~:.-=-1-  ApplLcat~mM  of Research  GO 
clarifymedical treatment issues and inform the community of 
practitioners. 
OIG Recommendation

1.	 The NIH should take steps to increase awareness  of the 
CDP among  those  responsible  for  C14E.  NIH  should  make 
special  efforts to reach those who sponsor continuing 
education  for general and family physicians. 
PHS Comment

We concur.  To promote awareness  of the  CDP,  the  Office of 
Medical Applications  of Research  is in the process of 
publishing  its recently revised  and clarified  statement of 
operating  guidelines  and procedures.  The updated procedures 
will serve to share the results of  experience 
NIH~s  growing 
with  key  persons  in  the  international  community  of  health 
technology  professionals.  Special  efforts  will  be  made  to 
i.dentj.fy  leaders  in  the  CM13community  and  make  them  aware  of 
the  cDP  and  of  lllHs  procedures  for  dissemination  of  findings. 
OIG Recommendation

2.	 The  NIH  should  identify  more effective ways ~of 
encouraging%,the incorporation  of consensus  findings into 
continuing  education activities. 
PHS Comment

We concur.  NIH will attempt to icient~fy  or  create  new  ways  to 
adapt their materials  and exploit the CME activities  to reach 
more practitioners.  In this-process,  NIH will solicit ideas

from CME leaders and other professionals,  and will consider

E-3
 2,

developing  slides,  overheads, and other curriculum materials

as suggested  in the report.

OIG  Recommendation 
3.	 The  NIH  should  strengthen  its  efforts  to  understand  and 
address  basic  concerns  about  the  CDP. 
PHS Comment  

We  agree to continue  evaluating the procedures used to 
initiate a conference,  develop a consensus statement,  and

disseminate  the results.  NIH will also continue its efforts 
to evaluate  the impact of conferences  on medical practice  to 
&scertair& whic1-Lt~~..esc.f.:cmfesence  :G.LCS,  zcGt2ment  .t~rmats, 
and dissemination  efforts have the greatest impact.  The 
concerns  expressed  by the critical  of  respondents  in 
minority 
this OIG study will be considered and assessed seriously.

Technical  Comment

Paqes 10  to  12.  We  suggest  that  the  report  indicate  that  the 
negative  views  of  the  CDP that were volunteered  by a minority 
of respondents,  while  sincerely held, may be inaccurate  or 
based on out-of-date  information. 
E-4
 APPENDIX  F 
NOTES 
1.  National  Journal,  January  22, 1977, p.  142. 
2.	 An NH-I Overview  of  a Report  of the  Council  on Health  Care  Technology  of 
the  Institute  of Medicine  on NIH  Consensus  Conferences,  NIH  Coordinating 
Committee  on Assessment  and  Transfer  of Technology,  February  10, 1992, p. 3. 
The  CDP  is administered  by the  Office  of Medical  Applications  of Research 
(OMAR),  within  the  Office  of the  NIH  Director;  each  Conference  is 
cosponsored  by OMAR  and  one  or more  of NIHs  institutes,  centers,  or 
diwsions. Since  its inception,  OM~  has conducted  almost  nine~  Consensus 
Conferences  on topics  across  the  spectrum  oi medicai  science  and  practice. 
3.	 The  founders  of the  Program  stressed  that  NIHs  role  was to facilitate 
technical  consensus  on medical  technologies,  which was defined  as 
scientific/medical  agreement  on the  scientific  facts  that  a given  innovation  is 
deemed  optimal  and  potentially  feasible  for introduction  into  practice.  The 
NIH  was not  to become  directly  involved  in what  was termed  interface 
consensus  on issues such  as cost-effectiveness,  patient  preference,  or ethics. 
Indeed,  NIH  administrators  expressed  a critical  assumption,  that  %roadened 
responsibilities  [would] ~  draw  NIH  into  activities  inappropriate  to its primary 
research  mission.  Responsibilities  of NIH  at the Health  Research/Health  Care 
lnte~ace,  Seymour  Perry,  MD,  Office  of the  Director,  NIH,  February  1977, p. 
A-4, A-2. 
In  1989, Congress  also created  the  Agency  for Health  Care  Policy  and 
Research  (AHCPR),  within  the  Public  Health  Service,  to evaluate  medical 
technologies  and  treatments  and  to develop  practice  guidelines. 
4.	 Guidelines  for  the Selection  and  Management  of Consensus Development 
Conferences,  OffIce  of Medical  Applications  of Research,  National  Institutes  of 
Health,  September  1988, p. 3. 
The  NIH  develops  specialized  dissemination  strategies  for individual 
Conferences.  Dissemination  activities  include  press  conferences;  press  releases; 
distribution  of video  highlights  of Conferences;  coverage  in the  popular  press; 
publication  of Statements  in both  general  and  specialty  medical  journals;  and 
direct  mailing  of Statements  to physicians,  hospitals,  medical  schools,  third-party 
payers,  policy makers,  continuing  medical  education  programs,  and  others.  In 
addition  to maintaining  a mailing  list of standard  contacts,  NIH  also  purchases 
targeted  mailing  lists from  relevant  specialty  societies  and  other  groups  for 
individual  Conferences.  The  NIH  does  not  target  medical  school  department 
F-1
 chairass a discretcea tegor(yO MAR  I)kec,toofr CommunicationsP,e
 rsonal

CommunicationwiththeOIG,October14,1992).

5.  Information  Dissemination  to Health  Care  Practitioner  and  Policymakens: 
Annotated Bibliography,  Agency  for Health  Care  Policy and  Research 
(AHCPR),  Public  Health  Service,  U.S. Department  of Health  and  Human 
Services,  April  1992, p.  1. 
6.  The  most  prominent  of these  have  been  funded  by NIH  itself,  and  include: 
Evaluation  of the  NIH  Consensus  Development  Process,  Paul  Wortman,  Center 
for Research  on Utilization  of Scientific  Knowledge,  Institute  for Social 
Research,  University  of Michigan,  1982. 
Changing  Medical  Practice  through  Technolow  Assessment:  An Evaluation  of 
,.le  i tii.  _Gil>~tCSUS  ~evei:jpment  Program,  Dawd  km~  JL3, et  w.,  kind 
Corporation,  March  1989. 
Consensus  Development  at the  NIH:  Imuroving  the  Pronam,  Committee  to 
Improve  the  NIH  Consensus  Development  Program,  Council  on Health  Care 
Technology,  Institute  of Medicine  (IOM),  1990. 
The  OMAR  has  also conducted  several  smaller  evaluation  projects,  including 
an examination  of the  role  of medical  opinion  leaders  in the  diffusion  of 
medical  innovations  (Expert  Panel  Meeting  Summay  The Role  of Medical 
Opinion  Leaden  in Disseminating  Consensus  Development  Conference 
Recommendations,  January  31, 1989) and  a series  of focus  groups  with 
practicing  physicians  on their  attitudes  towards  the  CDP  (Physician  Focus 
Groups  10 Identifi  Strategies  for  Dikremination  of  Consensus  Development 
Statements,  June  1990).  The  focus  group  report  noted  that  more  information 
concerning  the  specific  topic  of physician  awareness  and  perception  of 
[Consensus]  statements  is needed.  NIH  Physicitirz  Focus  Groups,  p.  3. 
7.	 Rand,  p. 244.  The  report  oversight  committee  further  urged  that  reaching 
these  primary  care  physicians  should  have  high priority  (p. x). 
The  NIH  plans  to target  generalist  physicians  and  internists  in an upcoming 
series  of surveys regarding  practice  patterns  and  awareness  of Consensus 
findings.  The  NIH  will suxvey them  regarding  every  Conference,  because  they 
are  more  likely to be  first in contact  with  patients  with  the  disorder.  OiWIR 
Quick-Launch  Physician  Practice  SurvqY, NIH  Clearance  Package  submitted  to 
the  Office  of Management  and  Budget  (OMB),  September  1991, p. 32. 
8.	 Rand  p. 245, p. 233.  The  study found  that  CME  courses,  conferences,  and 
meetings  were  rated  by responding  physicians  as the  most  important 
information  sources  for both finst hearing  about  and deciding  to use  a new 
F-2
 procedure.Italsonotedt hattheetiento fpafiicipatiionnC
MEwas'tthe

single  most  important  predictor  of whether  physicians  are  aware  of specific 
conference  recommendations  (p. xiii, p. 97). 
9.  NIH  Physician  Focus  Groups,  p. 9. 
10.  Continuing  Medical  Education:  A Primer,  Alliance  for Continuing  Medical 
Education,  1992, p. 96. 
Administrators  at NIH  noted  at the  time  of the  CDPS founding  that  medical 
school  teaching  hospitals  and  main  disease  research  centers  currently  represent 
the  most  effective  transfer  points  for the  movement  of research  knowledge  into 
health  practice,  and  that  any proposed  solutions  to the  dissemination  problem 
will have  to utilize  these  strengths  already  in place.  Perry,  appendix  A-1, A-3. 
l%e  Rand  study  also  determmed  that  medical  schcmis were  one  of the  wo  n:cm 
important  sources  of CME  for doctors  who were  aware  of the  Program;  local 
hospitals  were  the  other  most  important  source  (p. 95). 
11.  David  Kanouse,  Rand  Corporation,  personal  communication  with  the  OIG, 
October  9, 1992. 
12.  Van  Harrison,  Ph.D.,  Director  of CME,  University  of Michigan,  personal 
communication  with  the  OIG,  October  16, 1992. 
The  OMAR  has identified  opinion  leaders  as people  holding  positions  of 
leadership,  or having  high status  title,  rank,  or office.  The Role  Of Medical 
Opinion  Leaden,  p. 4, 6. 
Rand  also  found  that  medical  school  facuhy  members  were  more  likely than  the 
average  physician  to be  aware  of the  Program  and  were  among  those  most 
likely to first hear  about  CDP  information  (p.  72). 
13.  Rand,  p.  101. 
Everett  M. Rogers,  Diffusion  of Innovations,  3rd  cd., Free  Press,  New Yorlq 
1983, p. 331. 
14.  Rand,  p. 245. 
15.  The  impact  of CME  on physician  behavior  is a matter  of considerable 
controversy.  Many  researchers  have  suggested  that  reimbursement,  discipline, 
quality-assurance,  and  peer-review  activities  have  greater  effect  on physician 
behavior.  This  report  does  not  address  such questions. 
Furthermore,  this report  does  not  explore  the  issue  of getting  doctors  into 
CME.  As an NIH  administrator  noted  at the  time  of the  founding  of the 
-.
 Consensus  Program,  while the  results  of consensus  building  should  be  useful  in

meeting  continuing  education  needs  of physicians  and  other  health

professionals,  this process  will not  resolve  the  problem  of individuals  who  are

unable  or unwilling to involve  themselves  in such  activities.  Perry,  p. 9.

16.  We asked  family physicians  about  four  topics,  oncologists  about  three  topics,

and  neurologists  about  two topics.  The  topics  were  surgery  for  epilepsy,  clinical

use of botulinum  toxin, intravenous  immunoglobulin,  adjuvant  therapy  for colon

and  rectum  cancer,  early  stage  breast  cancer,  gastrointestinal  surgery  for  severe

obesity,  and  treatment  of panic  disorder.  Each  of these  had  been  the  subject  of

an NIH  Consensus  Conference  held  in  1990 or  1991.  Members  of each  group

were  asked  only about  the  Statements  that  were  relevant  to  their  field  (see

appendix  A for a listing of the  Statements  and  their  audiences).

17.  Of those  who reported  that  their  departments  relevant  CME  activities  had

we~irulay iitit addressed  the  Consensus  findings,  one  informed  us that  the

activity  had  been  conducted  prior  to the  given  Conference;  two informed  us

that  they were  not  aware  of the  relevant  Consensus  Statement  at the  time  of

the  activity.  Several  others  were  unable  to say exactly when  their  relevant

CME  activity  had  been  held,  but  told  us that  they  would  not  have  incorporated

the  Consensus  findings  into  the  activity  at any time  because  they were  unaware

of it, did not  agree  with it, or did not  consider  it appropriate  for CME.

18.  We  did not  ask respondents  how many  people  had  been  reached  in these  CME

activities;  it would  not  have  been  feasible  for them  to access  such  information.

19.  These  figures  are  considerably  higher  than  those  reported  by Rand  researchers

for  a sumey  they  conducted  of randomly  chosen  physicians  in all settings  (Rand,

p. 71).  The  groups  that  we sumeyed,  however,  represent  key opinion  leaders  in 
their  respective  fields.  Considering  their  importance,  we regard  their  level  of 
familiarity  with the  Program  as limited.  Furthermore,  we expect  that  the  level 
of awareness  of the  Program  among  physicians  in community  settings  is likely to 
be  considerably  lower  than  that  of these  academic  groups. 
20.  We  also  asked  department  chairs  if they were  familiar  with  any additional 
Consensus  Statements,  other  than  the  7 in our  sample;  72 percent  of all 
department  chairs  (including  72 percent  of each  specialty  group)  reported  that 
they were  not.  Only 9 percent  were  aware  of 2 or more  additional  statements. 
21.  These  figures  may somewhat  underestimate  total  awareness  of specific 
Statements  and  specific  recommendations  in those  Statements,  as people 
generally  separate  the  message from  the  messenger  and  forget  the  latter  over 
time.  We  did not  measure  awareness  of--or  conformity  to--the  specific 
recommendations  made  by NIH  Consensus  Panels. 
One  family medicine  chair  with whom  we spoke  pointed  out  that  NIH  may 
have  a difficult  time  gaining  name  recognition  for  the  CDP  because  of the 
F-4  recentproliferatoifomne dicaltechnologays sessmenetf forthse; deseribetdh
 e

resulotf thesedifferepnrto gramsasa cacophonyo
fparameters.

22.  We found  wide-ranging  opinions  regarding  the  relative  value  of direct-mailings.

and journals  as vehicles  for  disseminating  CDP  information;  these  reflect,  in

part,  differing  information  habits  among  doctors.  In our  interviews,  some

department  chairs  told  us that  direct  mail was the  only way to reach  them;  they

miss things  published  in journals.  Others  assured  us, however,  that  publishing  is

the  only way to get  to them;  as one  put  it, I receive  three  feet  of mail  a week

and  throw  most  of it away unread.

23.  Of the  82 percent  of CME  directors  who had  received  at least  one  Consensus

Statement  in the  mail  during  the  previous  3 years,  57 percent  had  never

forwarded  a Statement  to a member  of their  CME  committee  or other  faculty

for review.

24.  As would  be  expected,  family physicians  were  much  more  likely to have  heard

about  the  Program  or read  a Statement  in JAMA  than  were  o-ther specialists;

the  reverse  was also true  with regard  to specialty  journals  (see  appendix  C,

table  C-3).

25.  Other  sources  of information  about  the  CDP  and  specific  Conferences

included:

Colleagues:  The  NIH  working  group  on the  role  of medical  opinion  leaders  in

information  dissemination  predicted  that  opinion  leaders  are  more  likely than

the  average  doctor  to hear  about  Consensus  recommendations  from  colleagues

outside  their  own institutions.  This  appears  to be  more  true  of oncologists  and

neurologists  than  of family  physicians:  Only  15 percent  of family medicine

chairs,  but  21 percent  of neurology  and  33 percent  of oncology  chairs  had

heard  about  the  CDP  from  colleagues  at  outside  institutions.

Popular  media;  The  popular  media  was not  a major  source  of information

about  the  Program:  Only  16 percent  of department  chairs  had  heard  about  the

Program  through  the  written  or print  popular  media--ranging  from  4 percent  of

neurology  chairs,  to  18 percent  of family medicine  chairs,  to 23 percent  of

oncology  chairs.

Continuing  Medical  Education:  Twenty-eight  percent  of department  chairs

reported  that  they  had  attended  a CME  activity  sponsored  by an organization

other  than  their  own medical  school  that  had  addressed  the  findings  of an NIH

Consensus  Conference--ranging  from  14 percent  of family medicine  chairs  to 44

percent  of oncology  chairs.  Of these,  85 percent  had  attended  such  activities

sponsored  by national  specialty  societies,  and  30 percent  had  attended  such

activities  sponsored  by other  medical  schools  or teaching  hospitals  (see

appendix  C, tables  C-12 and  C-13).

F-5
 26.	 The  Rand  study suggested  that  video  coverage  of Consensus  Conferences, 
particularly  of the  press  conferences,  could  be valuable  for  CME  activities 
(p. 245).  Our  findings  do not  bear  this out  for medical  school  CME  programs. 
Video  tapes  of the  press  conferences  received  the  lowest  rating  for  usefulness 
in CME,  9 percent.  Video  tapes  of conference  highlights  received  the  second-
lowest  rating,  42 percent.  A number  of respondents  noted  that  they  do not 
have  the  time  or desire  to sit and watch  a video;  they  prefer  either  a quick 
summa~  to read  alone,  or interactive  materials  to use  in a group. 
27.	 Several  respondents  also  recommended  more  ambitious--and  expensive--efforts 
on the  part  of NIH.  In particular,  a number  recommended  that  NIH  directly 
sponsor,  and  finance,  local  CME  activities  and/or  provide  visiting  speakers. 
One  respondent  recommended  the  preparation  of case  examples  and  pre- and 
post-tests  to facilitate  the  development  of CME  programs. 
-~10-.  ~;i~iety-tpherrecee nto fthosedepartmentc hairwsh o werefamiliawriththe

CDP expressesdo me degreeofrespecfto ritsfindings--p4e9r centv
oicedgreat

respectr,a ngingf rom38percento ffamilym edicinechairst,o
 44percentof

oncologychairst,o 65percentofneurologcyh airsS.i xty-thrpeeer
 centalso

stronglayg reedthatNIH sponsorshiopf
theprogramenhancesthecredibility

ofitsfinding(ss
eeappendixC, table  C-14, C-15). 
29.	 University  of Michigan  researchers  also identified  the  problem  of selection 
bias in the  choice  of Conference  topics  and panelists  as a potential  threat  to 
the  credibility  of the  Program.  The  comments  we received  from  department 
chairs  suggest  that  their  concerns  regarding  this issue  persist. 
Wortman,  Paul,  et. al., Do Consensus  Conferences  Work?  A Process 
Evaluation  of the  NIH  Consensus  Development  Program,  Journal  of Health 
Politics,  Policy,  and  Law,  vol.  13, no. 3, Fall  1988. 
30.	 Even  among  some  specialists  in oncology  and  neurology,  there  was concern 
about  too  great  a @specialty  focus.  One  radiation  oncolo~-st,  for  example, 
objected  to the  overrepresentation  of medical  oncologists  vs. surgical  and 
radiation  oncologists. 
31.	 One  family  physician  argued  that  NIH has to get  into  the  trenches  with  the 
people  who are  going to use this information,  rather  than  bypassing  them.  NIH 
has  traditionally  been  antagonistic  towards  family physicians,  and  then  it 
wonders  why it cant reach  them. 
32.	 Again,  we believe  that  the  fact  that  69 percent  of department  chairs  do not 
strongly  regard  the  length  of Consensus  Conferences  as adequate  may well 
serve  as a barrier  to greater  use NIH  Consensus  findings  in their  CME 
activities. 
F-6
 33.	 Typical  comments  were:  consensus is not  the  same  as correct,  and  consensus 
is not  necessarily  equal  to state  of the  art  and  not  necessarily  wdely 
applicable.  As one  respondent  described  this view, you have  one  consensus 
and  then  three  months  later  you  have  another  consensus;  medicine  is always in 
a state  of flux. 
34.	 An oncologist  also asked,  can individuals  not  knowledgeable  about  the  area 
under  study  truly contribute  to consensus  development?  Another  argued,  the 
information  is generally  available.  Most  conferences  and  clinicians  review  the 
original  data  and journal  reviews  and  draw  their  own  conclusions;  why should 
an academic  clinician  care  what  these  non-experts  say? 
Some  respondents  expressed  doubts  that  any panel  could  objectively  arrive  at  a

consensus  of opinion  that  would  be  adequately  grounded  in fact.  As one

expressed  this sentiment,  the process  can  be self-serving  and  represent

primariiy  the  vmvpoint  of a few individuals  with vested  interests  rather  lhan  a

consensus  of the  scientific  community  after  it has  had  an opportunity  to review

the  data  in the  scientific,  peer-reviewed  literature.

35.	 7%e Physician  h Recognition  Award:  1991  Information  Booklet,  American 
Medical  Association,  1991. 
36.	 In our  survey,  62 percent  of department  chairs  reported  that  they  are  vey 
involved  in the  selection  of topics  for CME  activities;  another  32 percent 
reported  that  they were somewhat  involved.  In  addition,  52 percent  of the 
department  chairs  who were  familiar  with the  CDP  rated  the  topics  addressed 
in Consensus  Conferences  as very appropriate  for incorporation  into  CME 
activities--ranging  horn  38 percent  of family  medicine  chairs  to  about  57 
percent  of both  neurology  and  oncology  chairs. 
37.	 Although  83 percent  of CME  directors  reported  in our  survey  that  their  ofllces 
play a role  in the  selection  of topics  for CME  activities,  this role  tends  to be 
limited.  For  example,  96 percent  of CME  directors  reported  that  they were 
familiar  with  the  content  of the  CME  activities  for which  they  grant  credit,  but 
only 23 percent  of these  were  aware  of a CME  activity  at  their  institution  that 
had  addressed  findings  from  a Consensus  Conference.  This number  is much 
lower  than  that  reported  by the  chairs  of individual  departments. 
F-7
  .