Department of Health and Human Sefices 

OFFICE OF


INSPECTOR GENEML


The NCH Consensus Development Program 

Dissetiation of Findin~ through Medical School 

ContinuingEducation Activities 

L Februiq MM 
OFFICE OF INSPECrOR GENERAL 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, as 

amended, is to protect the integrity of the Department of Health and Human Setices’ (HHS) 

programs as well as the health and welfare of beneficiaries served by those programs. This 

statutory mission is carried out through a nationwide network of audits, investigations, and 

inspections conducted by three OIG operating components the Office of Audit Services, the 

Office of Investigations, and the Office of Evaluation and Inspections. The OIG also informs 

the Secretary of HHS of pro~am, and management problems, and recommends courses to 

correct them. 

OFFICE OF AUDIT SERVICES 

The OIGS Office of Audit Services (OAS) provides all auditing setices for HHS, either by 

conducting audits with its own audit resources or by overseeing audit work done by others. 

.%dits CXwninethe Ferformancc of HHS programs and/or its grantees am! contrr ‘ton h 

carrying out their respective responsibilities and are intended to provide inciepermmt 

assessments of HHS programs and operations in order to reduce waste, abuse, and 

mismanagement and to promote economy and efficiency throughout the Department. 

OFFICE OF INVESTIGATIONS 

The OIGS Offkx of Investigations (01) conducts criminal, civd, and administrative 

investigations of allegations of wrongdoing in HHS programs or to HHS beneficiaries and of 

unjust enrichment by providers. The investigative efforts of 01 lead to criminal convictions, 

administrative sanctions, or civil money penalties. The 01 also oversees State Medicaid fraud 

control units which investigate and prosecute fraud and patient abuse in the Medicaid program. 

OFFICE OF EVALUATION AND INSPECI’IONS 

The OIGS Office of Evaluation and Inspections (OEI) conducts short-term management and 

program evaluations (called inspections) that focus on issues of concern to the Department, 

the Congress, and the public. The findings and recommendations contained in these inspection 

reports generate rapid, accurate, and up-to-date information on the efficiency, vulnerability, 

and effectiveness of departmental programs. This report was prepared in the Boston regional 

office under the direction of Mark R. Yessian, Ph.D., Regional Inspector General and Martha 

B. Kvaal, Deputy Regional Inspector General. Project stati. 

REGION HEADQUARTERS 

Dana Miller, I+oject Leader Mary Beth Clarke, ProgramSpecialkt 

Timothy Corbett, Lead Anatyst 

For diitiod *oftimptie contadduB@on-o@e Qvte-at 

(617) 565-1050, or by Fax at (617) 565-3751. 
Department of Health and Human Services 

OFFICE OF


INSPECTOR GENERAL


The NE-I Consensus Development Pro~ram w 

Dissetiation of Findings through Medical School 

tintinutig Education Activities 

I 

I February l= mI-ol-91-o1760 

I 

–—— 

EXECUTIVE SUMMARY 

PURPOSE 

To evaluate the impact of the efforts of the National Institutes of Health to 

disseminate Consensus Development Program information through medical school 

continuing education activities. 

BACKGROUND 

In response to growing concerns about the effectiveness of some medical procedures 

and variations in medical practice, the National Institutes of Health (NIH) created the 

Consensus Development Program (CDP) in 1977. The Program brings together nongovernmental pane!s to assess the safety> efficacy, and appropriateness of e+sting and 

emerging medical tecnrmlogies. ISach panei presents its con~i~si~il~ in the i~ii~i ui a 

Consensus Statement, which is widely disseminated by NIH to medical. professionals 

and the public through direct mailings, publication in journals, and other methods. 

The potential importance of the Consensus Program has increased as concerns about 

cost and quality in health care have intensified. Recent evaluations, however, have 

pointed out potential vulnerabilities in the Program--including questions about its 

success in disseminating Conference findings. A 1989 Rand report suggested that 

continuing medical education (CME) activities could be a “critical dissemination 

vehicle,” and recommended that a survey of directors of continuing education 

regarding the use of information from the Consensus Program would be valuable. 

We conducted such an examination of continuing education programs at U.S. medical 

schools, which represent a major resource for CME. This report is based primarily 

upon a survey of continuing education directors and chairs of departments of family 

medicine, neurology, and oncology at all U.S. medical schools. We inquired about 

general familiarity with the Consensus Program, awareness of specific Statements, use 

of Consensus information in continuing education activities, and opinions about the 

Program. We did not attempt to assess changes in medical practice resulting from 

either continuing education activities or the Consensus Program itself. 

FINDINGS 

Fifly-two pexent of medical school &pa~nt chaih reported having sponsored a 

continuing education activity thut addkrsed the @dings of a recent NIH Consensus 

Ck3nference. 

�	 Chairs of family medicine reported more limited use of CDP information in 

continuing education activities than did chairs of neurology or oncology: 30 

percent of family medicine chairs, 47 percent of neurology chairs, and 77 

percent of oncology chairs reported such use. 

i 
�	 NIH Consensus findings have been addressed in a variety of medical school 

continuing education formats; the most common have been grand rounds. 

�	 Of those department chairs who reported use of NIH Consensus information in 

their continuing education activities, 81 percent reported that these were 

targeted to in-house faculty, and 72 percent that these were targeted to 

practicing physicians horn outside the medical school setting. 

Several factons hhder & dirsemindion and acceptance of CDP biifonnation and 

incorpomtion of that information into conhhuing education acdvities. 

�	 Limited familiarity with the Consensus Development Promam. Sixteen percent 

of department chairs knew nothing about the CDP; an additional 17 percent 

were aware of the Program but knew little about it. Family medicine chairs 

reuorted particularly limited familiarity with the Program: 27 percent had 

rwwr ne~xi :~fi: and another L5 percent were aware Lu, .-.~i~ ,- ,,,. ./Jt it. “ 

�	 Unsuitable format of the Consensus Statements for continuing education 

activities. Only 33 percent of those department chairs who were familiar with 

the CDP rated the format in which Conference findings are presented as very 

appropriate for continuing education. Respondents suggested several types of 

materials that would more readily facilitate the use of Conference findings in 

their continuing education activities, including slides, overheads, and curriculum 

materials for short, small-group activities. 

�	 Concerns about the Consensus Development Program itself. Most medical 

school department chairs voiced general respect for the Program. Nevertheless, 

a number of specific concerns were voiced by respondents familiar with the 

Program about limitations to its effectiveness and usefulness, including 

questions about the appropriateness of panelists and speakers, the practicality 

and directiveness of the Statements, and the adequacy of CDP procedures. A 

few respondents expressed fundamental objections to the use of consensus 

methods to address controversies in medicine. These concerns, whether 

reflective of perceived or actual weaknesses in the CDP, represent a serious 

barrier to wider dissemination and acceptance of NIH Consensus findings. 

RECOMMENDATIONS 

The NIH has demonstrated a firm commitment to ongoing evaluation and 

improvement of the Consensus Development Program. As NIH continues such 

efforts, it should aim to maximize the potential for medical school continuing 

education as a vehicle for disseminating Consensus findings. In particular: 

ii 

— 
l’%eNH shoukl tdx step to tiease warerless of the CDP among those responsible for 

continuing medical hation. IIw NH should make qxxid @om to reach tie who 

qumsor continuing education actiwkies for generul and f-physicians. 

In this effort, NIH should maximize the effectiveness of direct-mailing efforts by 

tailoring such mailings to department chairs and continuing education directors in their 

capacity as educators, and by encouraging them to cover Consensus findings in their 

continuing education activities. The NIH should work with medical school continuing 

education organizers to plan appropriate dissemination strategies. In this effort, NIH 

could also work with organizations other than medical schools, including relevant 

specialty societies and academic groups. 

The NIH shouki Ment#J more @ective ways of encouraging the ihcoqoratibn of 

ConseIUUSjindihgs into continuing educatidn activdk 

The NIH should expiore ways of packaging and disseminating Comenstis i.,.~ings that 

would more readily facilitate their use in continuing education activities, including the 

production of slides, overheads, or other curriculum materials. The NIH should also 

conduct focus-group discussions with key audiences of Consensus findings, including 

medical school department chairs and continuing education directors, to learn their 

perspectives on format issues. The NIH could use its future research efforts as 

opportunities to examine the attitudes and practices of other types of continuing 

education sponsors. 

171eIWHshouMstrengthe?l h#omto un&m@ul and &es basic concerns about the 

Conwruus Devekpnent l+vgram 

Because no dissemination efforts will be successful in reaching and influencing people 

with basic concerns about the effectiveness and usefulness of the Program’s methods, 

NIH should strengthen its efforts to understand and address such concerns. The NIH 

could use focus groups or informal working groups of department chairs and 

continuing education directors, and other key audiences, to learn about and address 

these concerns. As first steps, NIH should identi$ and test better ways of involving a 

wider range of people in the planning stage of upcoming Consensus Conferences to 

maximize acceptance of the outcomes, and should address concerns about panel and 

speaker selection. 

AGENCY COMMENTS 

We solicited and received comments on a draft of this report from the Public Health 

Service (PHS). The PHS concurred with our recommendations and identified several 

activities that NIH currently has underway or has planned that address these issues. 

These include the publication of a revised statement of operating guidelines and 

procedures, efforts to identi~ new ways of exploiting CME as an avenue for 

dissemination, and continuing evaluation of the procedures and impact of the CDP. 

... 

Ill 
The PHS also suggested that criticisms of the Consensus Program by some survey 

respondents may be based on incomplete or out-of-date information. We continue to 

believe that these concerns merit serious attention by NIH, and have modified the text 

to put these issues into clearer perspective. The complete text of the PHS comments 

appears in appendix E. 

iv 

-—— 
TABLE OF CONTENTS


PAGE 

m~SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..i


INTRODUCITON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...1


FINDINGS 

.� Use of NIHConsensus findingsin medical school


continuing education activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...5


e� Limi[s ?0 the w’iderd~’~emfr”+{nn and acceptance ofCDP 

information and iis incorporation into continuing education ~Cth7itit5S: 

� Limited familiarity w”th the Consensus Development Program . . . . . . . . . 8


�	 Unsuitable format of the Consensus Statements for


continuing education activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...10


� Concerns about the Program’s effectiveness andusefilness . . . . . . . . . . . 11


RECOMMENDATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...15


AGENCY CO MMENm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...19


APPENDICES


A Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. A-l


B Continuing Medical Education: Background . . . . . . . . . . . . . . . . . . . . . . B-1


c Summary of Department Chairs’ Survey Responses , . . . . . . . . . . . . . . . . C-1


D Summary of CMEDirectors’ Sumey Responses . . . . . . . . . . . . . . . . . . . . D-1


E PHSComments onthe Draft Repofi . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-1


F Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-1


INTRODUCTION


PURPOSE 

To evaluate the impact of the efforts of the National Institutes of Health to 

disseminate Consensus Development Program information through medical school 

continuing education activities. 

BACKGROUND 

Health-care technology assessment and N~ 

In recent years, health-care providers, policy makers, and the public have expressed 

incl~asin~ ccncern about rising ccsts. v~riaticms in medical practice. and overuse ~f 

certain mechcal procedures. These concerns have spurred greater interest in the 

evaluation of the appropriateness and effectiveness of existing and emerging healthcare technologies, including drugs, devices, and procedures. 

The Congress determined in the late 1970’s that the National Institutes of Health 

(NIH), as the principal biomedical research institution in the nation, should play a role 

in this arena. The NIH director at the time testified that “NIH and the rest of the 

scientific community must assume more responsibility for the effect of research on the 

quality of health care delivered. The need for accelerating the transfer of new 

technology across the ‘interface’ between biomedical research and the health care 

community and systems is a major issue.”1 

The Consensus Development Program 

The Consensus Development Program (CDP) was created as the focus of NIH’s 

technology assessment efforts. The CDP aims “to promote the timely incorporation of 

beneficial medical innovations into clinical practice, encourage the abandonment of 

obsolete technologies in favor of ones that are more efficacious or safe, discourage the 

adoption of technologies that have little value, and form public policy choices that 

encourage or discourage the use of certain medical technologies.”2 

The Consensus Program employs a group-judgment process that brings together nongovernmental panels to address the safety and efficacy of controversial existing or 

emerging medical technologies .3 At the end of each 2-1/2 day Consensus Conference, 

the panel crafts a Consens~s Statement presenting its finding; and conclusions. 

Reaching dktors: Dksemihation of CDP information 

The NIH aims to disseminate each Consensus Statement widely in order to achieve 

the “maximum impact of the Statement on health care practice.”4 The NIH’s efforts 

in this area reflect a recognition that “the term ‘effective dissemination’ includes the 

1 
concept of diffusion of knowledge and information as well as the acceptance, 

inculcation, and utilization of disseminated information;” the “distribution of 

information alone is insufficient to ensure adoption or use.”s 

Several recent evaluations of the Program have raised questions about the NIH’s 

success in disseminating its findings, and have suggested the need for further research 

in this area.6 These reports have noted in particular that NIH has had especially 

limited success in reaching general and family physicians--an important target audience 

because they both use many of the technologies evaluated by the Program and, 

through the referrals they make, serve as gatekeepers who often determine the type of 

care their patients will receive.’ 

Wdical school confining	 medical educalion advities 

asavehiclefor thediw emmation “ of CDPfindings 

In a 1989 study of me ?UH Consensus Program, the Rand Curpurauwi ~m..+ t.ui: 

continuing medical education (CME) could be a “critical dissemination vehicle” for the 

Program, and recommended that it “play a more prominent role in the Program’s 

dissemination strategy.”s Focus-group discussions with physicians recently conducted 

by NIH likewise found that most physicians prefer face-to-face contacts as a method of 

learning, and cited continuing education programs as a key source of information.g 

Continuing medical education activities are sponsored by a wide variety of 

organizations, and sexve as a major means for practicing doctors to keep up to date 

with current knowledge about medical procedures and technologies. According to the 

Accreditation Council for Continuing Medical Education, “medical schools, through 

their faculty members, provide the major resource for CME in this country.”lo (See 

appendix B for more information.) 

Doctors in positions of leadership in medical schools and those who plan and conduct 

continuing education activities seine as “switching points” for new medical information 

and can be particularly influential in spreading CDP findings.11 In addition to their 

role in setting the agenda for undergraduate, graduate, and continuing education at 

their own institutions, department chairs often conduct continuing education activities 

at their respective specialty society conferences, and are invited to speak at community 

hospitals.12 As noted in the Rand report, “information may well follow a “two-step 

flow” from opinion leader to follower. . . . Major communication sources may alert 

specialists and academic physicians to the existence of new medical information 

(eg., Consensus Conference recommendations), and this information may then ‘filter 

down’ to primary care practitioners and other physicians in private practice.”13 

m n?ptm 

The Rand report suggested that a survey of directors of CME regarding actual and 

potential uses of NIH Consensus material would be valuable.14 This report presents 

an examination of the current and potential future utilization of NIH Consensus 

2 
findings by medical school continuing education programs. We provide information on


NIH’s efforts to reach medical school department chairs and CME directors, how


these people view the Program and its products, and the extent to which the


continuing education activities they sponsor serve as vehicles for the wider


dissemination of NIH Consensus information. We do not attempt to assess changes in


medical practice resuhing from either CME activities or the Consensus Program 

itself.15 

METHODO~GY 

Our primary means of data collection was a set of mail surveys of the directors of 

continuing education and of the chairs of the departments or divisions of family 

medicine, neurology, and oncology at all U.S. medical schools. We inquired about 

general familiarity with the Consensus Program, awareness and use of seven recent 

Consensus Statements. and opinions about the Pro~ram. In the findings, we focus on 

the responses of department chairs, wiio are generally key m~di~tii i)pi.ilti,. ,.-..i:-i Uiid 

who are more closely involved than continuing education directors in the subject 

matter of their departments’ continuing education activities (see appendices C and D 

for a summary of survey responses). We supplemented this information with 

telephone interviews with a sample of survey respondents, discussions with NIH staff, 

a review of NIH dissemination materials, a review of the literature, and discussions 

with experts in the field. (See appendix A for detailed methodology.) 

We conducted our review in accordance with the Quality Standards for Inspections 

issued by the President’s Council on Integrity and Efficiency. 

3 

.


FINDINGS


FIFTY-TWO PERCENT OF MEDICAL SCHOOL DEPARTMENT CHAIRS


REPORTED HAVING SPONSORED A CO NTINUING EDUCATION ACITVITY


THAT ADDRESSED THE FINDINGS OF A RECENT NIH CONSENSUS


CONFERENCE.


We asked medical school department chairs if they had sponsored CME activities


during the previous three years on at least one of a sample of seven topic areas, and if


such activities had addressed the findings of the recent NIH Consensus Conference on


each topic.16


Fifty-two percent had sponsored a CME activity that addressed the NIH Consensus


findinw on one of the sample topics. Seventeen percent had sponsored a mlmmnt


continuing education activity, but had not addressed the NIH Conseusus rimmgs on


the topic.17 Sixteen percent had not sponsored a relevant CME activity (see table 1).


Table 1: Use of CDP Fmdinm in CME Activities Percentage of department chairs who 

sponsmed a recent continuing education activity on at least one of the sample topics in 

their field and the percentage of these that addreased the findings of the NIH Consensus 

C2mkxence on that topic 

Conducted a CME Addressed the 

activity on at least N-m Cklsensus 

one of the sample Fmdiqp on that 

topkx in their field topic 

FamiIy Neurology oncology Average 

Medicine Chaim chairs 

chain 

(n=S4) (n=SS) (n=95) (n=267) 

Yes Yes” 30% 47% 77% 52% 

No* 22% 20% 9% 17% 

Don’t know 26% 16% 4% I 15% 

No 23% 17% 9% 16% 

Total


Soumx*
 OIG swvey of medical school department chairs, November 1992


Nom Includes those who responded “definite~ and “prubab~ yea or no.


Differences are statistically significant at the .05 level.


5


I 
We also inquired as to whether department chairs were aware of continuing education


activities sponsored by their departments that had addressed the findings of additional


Consensus Conferences, other than those in our sample. Twelve percent were aware


of such additional CME activities. Of these, however, most had also addressed at least


one of the Statements in the sample; only 6 (2 percent) had not also addressed at


least one of the Statements in the sample.


Chuim of family medkine reprted mom lhnkd use of CDP tiformation in conlbuukg 

educalion activdies than did chaiis of neurobgy or oncology. 

We found significant variation in the reported use of Consensus information in CME


activities among the three specialty groups: 30 percent of family medicine chairs, 47


percent of neurology chairs, and 77 percent of oncology chairs reported such use (see


table 1).


We also found that neurology and oncology chairs were more definitive about their


departments’ use of CDP information in continuing education activities than were


family medicine chairs: 59 percent of oncology chairs reported that they had


sponsored relevant CME activities that had dej_initeZyaddressed CDP findings; 30


percent of neurology chairs and only 12 percent of family medicine chairs reported the


same. This may reflect both less intensive involvement in continuing education on the


part of family medicine chairs (see appendix C, table C-17), and less use of CDP


information in their continuing education activities.


We address some of the reasons for the variation in the use of Consensus information


both within and among specialty groups in our second finding, beginning on page 8.


NH Consensus fhdings have been adlikssed k a varie~ of medical school continuing 

educatim formuts; the mart cornrnon have been grand mwnd.r. 

In-house programs, such as grand rounds, and large-group activities were the most


common forums for CME activities that addressed NIH Consensus findings. Oncology


chairs reported significantly greater use of small-group activities, such as discussion


groups, than did neurology or family medicine chairs (see appendix C, table C-10).


A number of respondents explained how they had made use of Consensus findings in


their continuing education activities. One department chair reported that the CDP


findings had been the central topic of a regularly scheduled department session on


current topics in the field, while several noted that the findings had been addressed


only peripherally, as part of larger programs. Several indicated that Consensus


Statements have been useful in subspecialty conferences, departmental meetings, and


formal lectures. Some respondents noted that the concise nature of the Consensus


Statements makes them more appropriate for shorter, small-group discussions; they


are already “distilled beyond usefulness” for longer activities. A CME director


6

proposed that they are “best used in small groups with an ‘expert’ moderator or in


self-instruction format.”


iUedkal school continuing ea!ucation activitk addkwzn“ g NEl Conwnsus jidkgs have 

been tmgeted to both in-houses@ and facully, and pructicingphysicians frvm outside 

the medkal school setting. 

Of those department chairs who reported addressing CDP information in their CME


activities, 89 percent noted that these activities had been targeted to residents, 81


percent to in-house faculty, and 68 percent to in-house hospital staff.


These activities also served to reach practicing doctors beyond the medical school


setting: 72 percent of department chairs reported that their activities had been


targeted to local or nonlocal practicing physicians--35 percent to practicing physicians


from outside the iucai ar~~ (see appenuix C, tabie C-11).18


7

SEVERAL FACI’ORS HINDER WIDER DISSEMINATION AND ACCEPTANCE


OF CDP INFORMATION AND INCORPORATION OF THAT INFORMATION


INTO CONTINUING EDUCATION ACHWTI13.


Lhnited f~ with the Ctmsensus Development ??vgmm 

Sixteen percent of department chairs knew nothing about the CDP; an additional 17


percent were aware of the Program but knew little about it.19


Family medicine chairs reported much more limited familiarity with the Program than


did oncology and neurology chairs: 27 percent of family medicine chairs, 18 percent of


neurology chairs, and only 3 percent of oncology chairs reported that they had never


heard of the Program. There was similar variation among the groups with regard to


those who were aware of the Program but knew little about it (see table 2).


Table 2 Awareness of the CDP Percentage of medieal school CME directors and 

department chairs who were aware of the NIH Consensus Development program 

. . 

: 

Very familiar


-F EXii?i5 

Somewhat familiar,


but not sure of all


the details


Aware, but didn’t


know much about it


Not at all aware


11% 14% 31% 47% 31%


48% 33% 34% 41% 36%


23% 25% 17% 8% 17%


18$Z0 27% 18% 3% 16%


Sourax OIGsuneyof medicalschoolCMEdwectorsanddepartmentchairs,Nmemher1992


Note Differences amongdepartmentchairsarestadstically significant
 at the.05level.


Awareness of s~ecific Consensus Statements was more limited than ~eneral awareness 

of the CDP.n ‘Again, chairs of family medicine were less likely to b: aware of 

Consensus Statements relevant to their field than were other department chairs; for 

example, 89 percent of oncology chairs but only 57 percent of family medicine chairs 

were aware of the Statement on EarZy Stage Breast Cancer. Even many department 

chairs who had sponsored continuing education activities on topics that had been 

addressed by Consensus Conferences were not aware of the relevant Consensus 

8

Statements on those topics. For example, only 44 percent of the chairs of family


medicine who had sponsored a CME activity on gastrointestinal surgery for severe


obesity were aware of the Consensus Statement on that subject (see table 3).21


Table 3: Awareness of Sued “ c Consensus Statement Percentage of all responden~ and 

of tkxe who sponsored a continuing education activity that mhkssed each topiq who wxxe 

aware of each Statement 

. 

‘B ~ “i;:: ~~ “o;;; 

i, 

‘ ‘=61 “a’~~::~: ‘ ‘=69 “at:~= ‘ ‘=” ~ ‘k’t::~:z: 

Suqety for 

Epilepsy 

Clinical use of 

Botulinum Toxz”n 

Intravenous 

Immunoglobulin 

Adjuvant 

Therapy for 

Colon and 

Rectum Cancer 

Early Stage 

Breast Cancer 

Gastrointestinal 

Sugery for 

Severe Obesity 

Treatmentof 

Panic Disorder 

. 85% 87% - (n=54) 

18% 100% 67% 79% - (n=2) (n=43) 

19% 45% 

(n=m) 

- 79% * 88% 

* 

� 

(tt=dll]


57% 60% - - 89% 92% 

(n=35) (n=79)


31% 44% -


(n=9]


56% 55% - (n=42]


SOH OIG suNey of medical school department chaixy Nwemher 1992


Note Adaah (-) indicatea that the group waa not aaked about that Statement.


In their written comments, a number of respondents demonstrated this limited


familiarity with the Program. One neurolo~ chair commented that he “was not aware


of the dimension of the Program; these are worthwhile goods that are insufficiently


known.” A family medicine chair noted that the CDP findings “may go unnoticed at


times.” One CME director noted that “we have never received anything from NIH,”


while another suggested that “this Program is not well recognized.” As one oncology


9


I 
chair remarked, “I don’t know all of the Consensus Conferences--how could a


practicing doctor?”


Department chairs reported that their most common sources of information about the


CDP during the previous three years had been direct-mail copies of Consensus


Statements and journal publications.” Nonetheless:


�� 25 percent reported that they had not received a Consensus Statement in the 

mail (or could not recall having received one) during the previous 3 years. This 

ranged from 8 percent of oncologists, to 36 percent of family physicians, to 39 

percent of neurologists.~ 

�� 40 percent had not read a Statement in either the Journal of the American 

Medical Association (JA.MM) or a specialty journal in the previous 3 years. This 

ranged from 33 nercent of family physicians, to 38 percent of neurologists. to 48 

percwt ui oncoiog~stb.:4 

Other sources of information, including colleagues, the popular media, and continuing


education activities sponsored by outside organizations, were reported by department


chairs as less important ways of hearing about the Program or specific Conferences


(see appendix C, table C-3).X


Unsuitable format of the Consensus Statements for contbuing education activitia 

Only 33 percent of those department chairs who were familiar with the Consensus


Program rated the format in which Conference findings are presented as very


appropriate for CME. This figure ranged from 12 percent of family medicine chairs


to 40 percent of neurology and oncology chairs. Only 18 percent of continuing


education directors gave this rating.


Respondents criticized the Consensus Statements as bland, lacking in adequate


background information, and lacking in sufficient data to support the findings and


conclusions. One family medicine chair noted that the “format is pretty dry and the


findings are not easily recalled. Might be helpful if CDP topics were contrasted with


current prevailing professional practices.”


Respondents also noted that it would require work on their part to translate a


Consensus Statement into a format appropriate for a continuing education activity.


Many suggested that they would be more likely to make use of Consensus information


if they had access to materials that could readily be incorporated into their ongoing


continuing education activities. Our survey respondents rated the potential usefulness


of 10 types of materials for their CME activities; the materials rated most highly were


CME curriculum materials for short sessions (such as grand rounds), reference


10

bibliographies, and conference proceedings. Video tapes of the Conferences received


the lowest ratings for usefulness in continuing education. X (See appendix C,


table C-16.)


A number of respondents offered the suggestion that slides or overheads outlining the


findings and recommendations of Conferences would be very useful for CME


activities, lectures, talks, and meetings. Several people with whom we spoke urged


that such materials could best be used in regularly scheduled activities, such as


monthly departmental update meetings or grand rounds. They suggested that NIH


might make such materials available, or announce their availability, through targeted


mailings to department chairs and others responsible for continuing education. The


NIH could suggest in a cover letter that such people discuss Conference findings at an


upcoming continuing education activity.27


Ldtwms aDoul iw Frogrurn’s ejfecdventw and usef” 

Most medical school department chairs voiced general respect for the Consensus


Program.n Nevertheless, a number of specific concerns were voiced by respondents


familiar with the Program about limitations to its effectiveness and usefulness. The


most prominent of these concerns were related to the appropriateness of Consensus


panelists and speakers, the practicality and directiveness of Consensus Statements, and


the adequacy of CDP procedures. A few respondents expressed fundamental


objections to the use of consensus methods to address controversies in medicine.


Many of these comments parallel criticisms raised in previous evaluations of the NIH


Consensus Program, including NIH-commissioned studies by Rand, the University of


Michigan, and the Institute of Medicine.


The groups we su~eyed represent some of the best informed and most up-to-date


members of the medical community. Their concerns, whether reflective of actual or


perceived weaknesses in the CDP, represent a serious barrier to wider dissemination


and acceptance of NIH Consensus findings.


Choice of panelists and speakens: In unprompted written comments, 10 percent of 

those department chairs who were familiar with the Consensus Program voiced 

concerns about the credentials and intellectual independence of Conference panelists 

and speakers. 29 As one summed up these concerns, “the CDP is highly dependent 

on the panel selected to make the assessment. The stature, knowledge, and respect of 

a single individual can alter the conclusions in an inappropriate manner.” The 

predominant criticism among these respondents was that the process is biased by the 

selection of NIH “insiders,” both to present evidence and to sit on the panels. As one 

neurologist put it, “attendees appear limited and the panel of ‘experts’ seems handpicked; it doesn’t represent a true broad sampling of opinion to develop ‘consensus,’ 

but just reflects a preconceived opinion of NIH or a small panel.” A CME director, 

him~elf a physician, described pa~icipants as the “same insider crowd--’experts’ who 

often have their own conflicts of interest.” 

11

Family physicians were particularly critical of what they regarded as the subspecialty


focus and research-orientation of the panelists and speakers.w In a typical criticism,


one strongly urged that NIH “balance the committees towards practical users of the


information away from so-called experts.” Another voiced the widely noted comment


that “the panels should include family physicians since we are likely to be the


‘consumers’ of much of the Consensus findings.”


Practicality and directiveness of Statements: Only 40 percent of those department chairs 

who were familiar with the Program strongly agreed that Consensus recommendations 

are practical--ranging from 26 percent of family physicians to 52 percent of 

neurologists. Likewise, only 38 percent of department chairs strongly agreed that 

Consensus recommendations are sufficiently directive--ranging from 26 percent of 

family physicians to 46 percent of neurologists (see appendix C, table C-14). 

While there w~re few strongly negative ratings with regard to these characteristics of


Consemus atawnems, we ~elieve mat the fact that 60 percem u. QJpi.LL.---..Ai-~lairs


do not strongly regard the Statements as sufficiently practical or directive may well


serve as a barrier to greater use of NIH Consensus findings in their CME activities.


Family physicians were particularly critical of the usefulness of Consensus Statements


for their practices. In a typical, unprompted comment, one argued that Consensus


Conferences “are biased by a subspecialty focus. The underlying assumption that the


Conference results are relevant for application in general medical practice is, at best,


partially correct. The Program would benefit from a vast increase in practice-based


research.” Another similarly noted that “the requisite knowledge for frontline practice


has not and will not come from subspeciaky, highly selected patients. Thus Consensus


guidelines are often unuseful and sometimes irrelevant to generalists.” Another


argued that the panels should “make the recommendations & more practical and


applicable. Consensus Statements are written by research experts not knowledgeable


about the patients and settings for which they are intended.”31


Adequa~ of CDP procedures: Respondents identified two areas of concern regarding 

the mechanics of the Consensus process: Conference planning and length. 

In unprompted comments, 14 percent of those department chairs who were familiar


with the Program criticized the process used for planning Consensus Conferences.


They urged that there be broader participation in the planning stages of Conferences,


including topic-, panelist-, and speaker selection. Several noted, in particular, that


there should be greater medical school involvement in the plaming process, and urged


that all medical school department chairs in a relevant specialty should be invited to


attend the actual Conferences. Also in unsolicited comments, 18 percent of those


family medicine chairs who were familiar with the Program urged greater involvement


of primary care specialists in the planning of Conferences.


Only 31 percent of respondents strongly agreed that the 2-1/2 day length of the


Conferences was sufficient to adequately consider the issues?2 Their concerns were


12

illustrated by 2 respondents who had particular familiarity with the Program: one


department chair, who had chaired an NIH Consensus Conference, noted that


“minority opinions get lost in the rush--sometime after midnight--to complete the


report and achieve consensus. I find the format a bit too rushed--even a few extra


hours would help.” A second respondent, who had participated on two Consensus


panels, likewise rated the length of the Conferences has highly inadequate.


Objections to consensus methods: In unprompted comments, 6 percent of those 

respondents who were familiar with the CDP expressed a ‘philosophical problem’ with 

the conce t of consensus development as a means of resolving controversies in 

medicine. f

3 Some of these respondents objected to what they regarded as a 

‘cookbook approach’ to medicine, arguing that this type of activity could be harmful to 

the independent judgment of practicing doctors. Many of these respondents asserted 

that Consensus recommendations are the ‘result of compromise,’ and argued for a 

more rigorous method of technolo~ assessment. One urged that NIH use an “explicit, 

evidence-based, pauem pretercnce-kwussed approach.” 

One neurology chair summed up what several believed is the fundamental paradox of


the consensus process: “The extent of information and studies that need to be


reviewed for the consensus cannot be mastered unless the panelists are already expert.


Yet if they are, they will bring their own biases to the panel.”~


13


RECOMMENDATIONS 

Our findings indicate that medical school continuing education activities can play an 

important part in the dissemination of NIH Consensus findings. Indeed, medical 

school department chairs and others responsible for planning and conducting 

continuing education activities are key opinion leaders in their fields and can serve as 

major conduits for transmitting such information more broadly. 

These findings also suggest, however, that this dissemination channel is not being fully 

utilized, in part because (1) a significant minority of those people who play a role in 

medical school continuing education know little or nothing about the Program; (2) 

many of those who are aware of the CDP feel that its products camot be readily 

incorporated into their activities; and (3) some voice basic concerns about the current 

effectiveness and usefulnws of the Program. 

To maximize the potential for medical school continuing education as a vehicle for 

disseminating Consensus findings, we offer the following recommendations: 

l%e IWHshoukii takestepf tohcreaseawaren.4m of the CDP among tlume rqponsible for 

conthubag medical katim lh w shouki make special ~orts m reach those who 

sponsor Contbluing education for gerund and fmphysiciiwm 

Considering the important role of department chairs and CME directors in planning 

and conducting medical school continuing education activities, it is important that NIH 

reach them and ensure that they are familiar with the CDP and its products. To do 

SO,NIH should: 

.	 Maximize the effectiveness of direct-mailing efforts by tailoring such mailings to 

department chairs and CME directors as educatois. While some currently 

receive direct-mailings in their capacity as members of specialty societies, we 

believe that specialized mailings would be more effective. The NIH could use 

such mailings as an opportunity to encourage recipients to cover Consensus 

findings in their continuing education activities. 

.	 Work with individual department chairs, members of CME adviso~ 

committees, deans, and continuing education directors at key institutions--or 

gToups of such individuals--to plan appropriate dissemination strategies. 

In addition, NIH could: 

.	 Work with organizations other than medical schools--including relevant specialty 

societies, associations of medical school professors in particular specialty areas 

(such as the Society of University Professors of Neurology and the Society of 

Teachers of Family Medicine), the Society of Medical College Directors of 

CME, the Association of American Medical Colleges, and the Accreditation 

15 
Council for Graduate Medical Education--to plan appropriate dissemination 

strategies. 

.	 Explore ways of identifying those people in individual departments who are 

most closely involved in the planning and conduct of continuing education 

activities in particular subject areas. 

In each of these areas, NIH should make special efforts to reach general and family 

physicians. They are an important target audience for the CDP because they both use 

the technologies addressed in Consensus Conferences, and serve as gatekeepers who 

determine the type of specialty care their patients receive. Their role is likely to grow 

in importance as a result of expected reforms in the health care system and an 

increasing emphasis on managed care systems. 

2%e NIH should _ more +ective wavs of encoumging the zkcorpomthn of 

Corawnsw jnamgs inw ~waruung txiucuu.onacntiia. 

The NIH should explore ways of packaging and disseminating Consensus findings that 

would facilitate their use in CME activities. In this effort, NIH should: 

.	 Produce--or work with other organizations to produce--slides, overheads, or 

other curriculum materials. The NIH should build upon its experience with 

prior efforts in this area. 

.	 Conduct focus-group discussions with key audiences of Consensus findings, 

including medical school department chairs and CME directors, to learn further 

ideas on format issues. 

The NIH could also examine the attitudes and practices of other types of continuing 

education sponsors. In its fiture research, NIH could consider the following ideas: 

.	 Survey CME directors at community hospitals, State medical societies, State 

and local medical and specialty societies, and local chapters of national societies 

regarding their current and potential uses of Consensus information. 

.	 Conduct focus group discussions with representatives of national specialty 

societies, including the American Academy of Family Practice and other 

prima~-care groups, regarding their current and potential uses of Consensus 

information. 

nae NIHshouus@en@en ihefomto Wdmklnd and &ld&?ssbasic CO?lCtY7LS (ZbOld the 

~rwmus Devekpnent l%ogmm 

The NIH has demonstrated a firm commitment to ongoing evaluation and 

improvement of the Consensus Development Program, including Conference planning, 

conduct, and information dissemination. Because no dissemination efforts will be 

16

successful in reaching and influencing people who have basic concerns about the 

effectiveness and usefulness of the Program’s methods, NIH should intensify its efforts 

to understand and address such concerns. Some of these concerns may be based on 

limited familiarity with or understanding of the CDP. This possibility suggests the 

need for intensified efforts to clari~ the Program’s procedures, targeting in particular 

key groups of opinion leaders, such as those we surveyed. At the same time, as noted 

in the report, the fact that some of these concerns have been raised repeatedly in 

evaluations of the CDP may suggest a genuine need to reexamine some of the 

Program’s methods. 

The NIH could use focus groups or informal working groups of department chairs and 

CME directors, and other key audiences, to learn about and address these concerns. 

As initial steps, NIH should: 

Identify ana test better ways of involving a wider range of pc o~M ,fi L... 

planning stage of upcoming Consensus Conferences to maximize acceptance of 

the outcomes. 

Address concerns about panel and speaker selection by (1) strengthening its 

efforts to increase primary-care representation, and (2) clearly stating in each 

Consensus Statement how panelists and speakers were selected and reporting 

their credentials. 

17


AGENCY COMMENTS 

We solicited and received comments on a draft of this report from the Public Health 

Service (PHS). The PHS concurred with our recommendations and identified several 

activities that NIH currently has underway or has planned that address these issues. 

In particular, PHS noted that the NIH Office of Medical Applications of Research is 

in the process of publishing a revised statement of operating guidelines and 

procedures for the CDP. In addition, NIH will explore new ways of exploiting CME 

as an avenue for dissemination of Consensus findings. The NIH will also continue its 

ongoing efforts to evaluate the procedures and impact of the CDP. 

The PHS also suggested that the report ‘be revised to make clear that respondents’ 

criticisms of Ihc Ccm~sm~MPro&grammav he based on limited famil;atitv -’~tfi the 

Program or out-of-date iniormatlon; PHS suggested that these opiruons may rcxlect 

perceptions of problems, rather than actual problems. We continue-to believe that the 

concerns reported to us by survey respondents merit serious attention by NIH. First, 

our respondents represent some of the best informed and most up-to-date members of 

the medical community, and are in key positions to seine as conduits for the 

dissemination of Consensus findings; their opinions are important. Second, the 

criticisms raised by this critical minority of respondents have been identified in prior 

studies, some of which were commissioned by NIH itsel~ this suggests either 

continuing problems or a need for better communication. We have modified the text 

to put these issues into clearer perspective. 

The complete text of the PHS comments appears in appendix E. 

19


APPENDIX A


METHODOIX3GY 

Our findings and recommendations are based upon five main sources of information: 

1. Mail surveys: We conducted two sets of mail surveys in November 1992. First, we 

selected three specialty areas in which recent Consensus Conferences had been held 

--family medicine, neurology, and oncology--and surveyed the chairs of those 

departments or divisions at all U.S. medical schools, as listed in the Association of 

American Medical Colleges’ Directorv of American Medical Education (see appendix 

C for a summary of responses). Second, we surveyed the directors of CME at all U.S. 

medical schools; the Accreditation Council for Continuing Medical Education provided 

us with its mailinz list (see appendix D for a summary of responses). 

In our sutveys, we inquired about overall familiarity with the Consensus Development 

Program, awareness of a sample of specific Consensus Statements, use of Consensus 

findings in continuing education activities, and opinions about the Program and its 

products. We asked department chairs to consult appropriate colleagues regarding 

specific subject areas, as necessary. 

Of the 12 Consensus Conferences held in 1990 and 1991, we selected 7 for our 

sample, targeted to a variety of audiences (see table A-l). 

TableA-1: Target audiencesfor the 7 sampleConsensusStatements 

Target audiences includti 

Clmsensus Statement 

Swgq for Epilep~ (March 1990) x 

~ ~ 

Chkicaf use of Botulinum Toxin (November 1990) x x 

Intravenous Immunog[obulin (May 1990) x 

Adjuvant Therapyfor Colon and Rectum Cancer x 

(April 1990) 

Earfy Stage Breast Cancer (June 1990) 

II 

x 

I ( 

x 

Gastrointestinal Suqey for Severe Obesity x 

(March 1991) 

Treaonent of Panic D&order (September 1991) x 

Our sample selection was based on discussions with NIH staff and respondents to our 

survey pretest, and was designed to produce a sample that included more than one 

A-1 
Statement addressed toeachof thethree specialty groups. Twoof the Statements 

were addressed to more than one of the groups; the other five were addressed to only 

one of the groups each. We asked family medicine chairs about 4 of the 7, neurology 

chairs about 2 of the 7, and oncology chairs about 3 of the 7. 

We received a response rate of 77 percent among department chairs and 66 percent 

among CME directors. Our respondents represented medical schools of all sizes, 

types of sponsorship, and geographic location. We do not believe that nonrespondents 

differed in any significant way from respondents (see appendices C and D for 

breakdowns of our response rates). 

In the findings, we have focussed on the responses of department chairs, who are 

generally key medical opinion leaders and who are more closely involved than CME 

directors in the subject matter of their departments’ continuing education activities. 

2. In-person and telephone discussions with medical school dep~rmien~ cnati ~ ad 

continuing education directors: We spoke with a large number of CME directors and 

department chairs both before and after the mail survey. In particular, we telephoned 

all those department chairs who reported in their surveys that their departments’ 

relevant continuing education activities had definitely not addressed Consensus findings, 

in order to learn better why not. We also followed up with a sample of those who 

responded that their departments’ CME activities dej7niteZy had addressed CDP 

findings, in order to learn more of their thoughts about the Program and its 

dissemination activities. 

3. Contacts with NIH: We conducted numerous formal and informal discussions with 

the staff of NIH’s Office of Medical Applications of Research and a review of NIH’s 

dissemination materials. We also attended a July 1992 meeting of the NIH 

Coordination Committee on the Assessment and Transfer of Technology, and both the 

public and executive sessions of the September 1992 NIH Consensus Conference on 

Gallstones and Laparoscopic Cholecystectomy. 

4. Other discussions: We spoke with directors of CME at several medical specialty 

societies, including the American Academy of Neurology, the American Society of 

Clinical Oncology, the American College of Cardiology, and the American Psychiatric 

Association; and representatives of the Alliance for Continuing Medical Education, the 

Accreditation Council for Continuing Medical Education, the Society of Medical 

College Directors of Continuing Medical Education, the Association of American 

Medical Colleges, the American Medical Association, the Rand Corporation, the 

Institute of Medicine, the Public Health Service’s Agency for Health Care Policy and 

Research, and the Office of the Assistant Secretary for Planning and Evaluation. 

5. Review of the literature: We examined the literature on medical technology 

assessment, the diffusion of medical innovations, the theory and practice of medical 

education, and physician information habits. 

A-2

APPENDIX B


CONTINUING MEDICAL EDUCATION: 

BACKGROUND 

The American Medical Association and the Accreditation Council for Continuing


Medical Education (ACCME) define continuing education as consisting of


“educational activities which serve to maintain, develop, or increase the knowledge,


skills, and professional performance and relationships that a physician uses to provide


services for patients, the public, or the profession,” and its content as “that body of


knowledge and skills generally recognized by the profession as within the basic medical


sciences, the discipline of clinical medicine, and the provision of health care to the


public.”35 Such information can be conveyed in a variety of settings and formats,


including periodic conferences and formal lectures: ongoing hospital grand v’lnds,


seminars, and discussion groups; ancl seii-instructional materlais.


To ensure some degree of consistency, the ACCME accredits most sponsors of


continuing education for physicians, including most medical schools, national medical


specialty societies, teaching and community hospitals, State and local specialty


societies, voluntary health organizations, pharmaceutical and device firms, and for-


profit educational companies. State medical societies can be accredited both to grant


credit for their own programs, and to accredit community hospitals as sponsors of in-


house and locally marketed activities. The ACCME also accredits the NIH Office of


Education, which grants credit to physicians who attend the public sessions of NIH


Consensus Conferences. In addition, NIH has begun to explore the possibility of


granting credit for self-study of printed Consensus Statements.


Physicians may be required or encouraged to complete a certain number of hours of


CME by their respective State medical societies; national, State, or local professional


organizations; State licensing boards; malpractice insurers; and hospitals. To further


encourage participation in continuing education, the AMA grants the Physician’s


Recognition Award to doctors upon completion of certain requirements.


The primary continuing education activities of national specialty societies are large


annual meetings that often draw more than 10,000 attendees and involve hundreds of


different educational offerings. A society’s planning committee or education office


usually selects such offerings from among proposals submitted by members. Some


societies also conduct smaller educational offerings throughout the year, either at their


headquarters or regionally.


Medical schools and teaching hospitals offer a wide range of continuing education


programs. Some of these are conducted in-house at the school or hospital, and are


intended for in-house faculty and staff. Others are marketed to a broader audience,


require a fee, and are held at hotels, conference facilities, or other hospitals.


B-1

At many institutions, including most medical schools and hospitals, continuing


education activities are coordinated by an office of continuing education, which


provides educational and administrative support. The structure of these offices varies


greatly from one institution to another: Some CME offices are headed by physicians, 

but many are headed by either educational professionals or administrators. 

The CME directors’ expertise in curriculum development, familiarity with physicians’ 

educational activities, and position in the medical school all suggest that they could 

play a useful role in the dissemination of Consensus findings. Many of those who 

responded to our survey urged that they receive Consensus information directly. One 

assistant dean for medical education noted that “the formal CME activities of medical 

schools comprise a major system for disseminating Consensus Statements.” 

At the same time, these CME directors generally have a limited role in deciding what 

information is presented in the activities for which they grant credit. They generally 

rely on ~Ld~dh~ Ur ii~-h~ ~~i~mitt~~s d redid facuit~ or stali LL)rLL)~i~..: . . .Aun 

and program development. The chairs of individual medical school departments are 

key players in setting the agenda for continuing education at their institutions.% The 

specific content of most courses is determined by the individual faculty or staffi as one 

director noted, “the vast majority of our programs are initiated, planned, and executed 

by individual departments.”37 

B-2

APPENDIX C


SUMMARY OF DEPARTMENT CHAIR SURVEY RESPONSES 

Tables Page C-2 

c-1 OIG survey sample sizes and response rates 

c-2 Familiarity with the CDP 

c-3 Sources of information about the CDP 

c-4 Awareness of each of 7 sample Consensus Statements 

‘C-5 ,~wareness of :lkiitional Consensus Statements 

C-6 Percentage whose departments address each type of topic in CME activities 

c-7� Percentage of department chairs who sponsored CME activities in each of 7 sample 

topic areas 

C-8� Percentage of those department chairs who sponsored CME activities on each of 7 

sample topic areas who were aware of a relevant Consensus Statement 

c-9� Percentage of department chairs reporting that a relevant CME activity had addressed 

the findings of at least one of 7 recent Consensus Conferences 

c-lo Formats of CME activities that addressed CDP information 

C-n Target audiences for medical school CME activities that addressed CDP information 

C-12 Additional use of CDP information in CME 

C-13� Of those 28 percent of department chairs that attended a CME activity sponsored by 

another organization, the percentage reporting that this activity had been sponsored by 

each type of organization 

C-14 Comments on the CDP 

C-15 Degree of respect for the findings of the CDP 

C-16� Percentage of department chairs describing each type of material as potentially useful 

for their CME activities 

C-17� Reported degree of department-chair involvement in the selection of topics for CME 

activities in their department 

Blank SmVeyhstrumen~ Page C-15 

c-1 
I Table C-1: OIG survey sample 

Family 

Medieine 

chairs 

Sample Size 110 

Number of 84 

Respondents 

Response Rate 76% 

1 

sizes and response rates I 

118 

WI 

75% I 

Neurology 

Chairs 

(n=88) 

31% 

34V0 

17% 

18% 

II 

119 347 ii 

I 

95 267 ~ 

80% 77% j] 

Oncology Average 

Chairs 

(n=95) (n=267) 

47% 3170 

41% 36V0 

8% 1770 

3% 16570 

Very familiar 

Somewhat familiar, 

but not sure of all 

the details 

Aware, but didn’t 

know much about it 

Not at all aware 

Family 

Medieine rChairs

(n=&l) 

II 14% 

33% 

25% 

II 27% 

Source OIG smwy of medieal school department chairs, November 1992 

Note Differences are statistically significant at the .05 level. 

c-2 
Table C-3: sources of information about the CDP 

—-C 

Received a CDP Statement by mail 

E ~~ ‘ ~~ ~ ‘~~ 

Read a CDP Statement in JAMA ‘ 62% 3170 32% * 40% 

Read about the CDP in JAMA 61% 22% 2270 * 32$Z0 

Read a CDP Statement in a specialty 

journal —. 13% I 47% 299Z0~ E * 3170 

Read about the LDP in a speclaity I’ 

journal 189Z0 3597(0 30% 28% 

Received a CDP Statement at a 

CME activity, professional society 

meeting, or conference 26% 22% 33% 28% 

Heard about the CDP from 

colleagues within own institution 15% 25% 35% * 26% 

Heard about the CDP from 

colleagues at outside institutions 1570 21% 3370 * 24% 

Read or saw something about the 

CDP in the popular media 1890 4% 23% * 16% 

Source OIG survey of medical school department chaim, November 1992


Note F.xciud~ those 42 department chaim (16%) who described themselves as “Not at all aware” of the CDP; an


asterisk (*) indicates that differences among department chaira are statistically significant at the .05 level.


c-3 
Table CA Awareness of each of 7 sanmle Consensus Statements ~~ 

Statement 

Su&ety for Epilep~ 

Clinical Use of Botulinum Toxin 

Intravenous Immunoglobulin 

Adjuvant Theropy for Colon and 79% 

Rectum Cancer (n=92) 

il - I (n=61) I (11=!/1) ,, 

Gastrointestinal Sutgety for ~ 3170 - I 

Severe Obesity (n=61) :! 

I 

Treatment ofPanic Disorder ~ 56% -1 

(n=61) I 

Source OIG sutvcy of medical school department chairs, November 1992


Note: Excludes those 42 department chairs (16%) who described themselves as “1
 “Not at all


aware” of the CDP; a dash (-) indicates that the group was not asked about a given statement. 

c-4 
Table C-5: Awareness of additional Gmsensus 

Statements, not included in our sample 

Number of Statements 

None 

1 

2 

3 

4 

Percentage of 

respondents 

(n=225) 

72% 

19% 

7% 

2% 

1% 

Source OIG survey of medical school department chairs, November 

1992 

.-— _——. . -————

Table C*. Percentage reporting that their departments address each type of 

topic in their CME activities 

r~ “n%’lr 

c. Specialty society 77% 5570 71% * 68%


practice guidelines/


practice parameters (n=82) (n=87) (n=94) (n=263)


Soureex OIG survey of medical school department chairs, November 1992


Note: An asterisk (*) indicates that the differences among department chairs are statistically significant at


the .05 level.


c-5 
Table C-7: Percentage of department chairs who sponsored CME 

activities on each of 7 sample topic areas 

Topic Family Neurology Oncology 

Medicine chairs chairs 

chairs 

(n=S4) (n=88) 

Surgery for Epilepsy 78% 

Clinical use of Botulinum Toxin 2.4% ‘ 57% 

Intravenous Immunoglobulin 23% 

(n=94) 

Adjuvant Therapy for Colon 7570 

and Rectum Cancer (n=94) 

Early Slagc Breast Cancer ~ 56?ZG .: ,. 

~t]=~~j 

Gastrointestinal Surgery for 12% 

Severe Obesity 

Treatment of Panic Disorder 69% 

Source OIG survey of medical school department chain, November 1992 

Note A dash (-) indicates that the group was not asked about a given topic. 

C-6

Table C-& Percentage of those department chairs who sponsored 

CME activities on each of 7 sample topic areas who were aware of a 

relevant COnsensus Statement 

Topic Family Neurology 0nu310gy 

Medicine chairs chairs 

Chaim 

Surge~ for Epilepsy 87% 

(n=54) 

Clinical use of Botulinum Toxin 100% 79% 

(n=2) (n=43) 

Intravenous Immunoglobulin 45% 

(n=20) 

Adjuvant Therapy for Colon w% 

and l<QUILJiil LtiL;..J~ 

Early Stage Breast Cancer I 60% 929io 

I (n=35) (n=79) 

Gastrointestinal Surgery for 44% 

Severe Obesity (n=9) 

Treatment of Panic Disorder 55% 

(n=42) 

Source: OIG survey of medieal school department chairs, November 1992


Note Excludes those who were not at all aware of the program; a dash (-) indicates that the


group was not asked about a given topic.


c-7 

I 
Table C-9 Percentage of department chairs reporting that a relevant


CME activity had addressed the findings of at least one of seven I


recent Consensus Conferences durim the urevious 3 Years
 e.	 .


I


Family Neurology Oncology Average ! 

Medicine chairs chairs 

Chairs 

Definitely yes


Probably yes


Definitely not


Probably not


———— 

Don’t know


(n=65) (n=73) (n=85) (n=223) ~ 

15% 36970 65% 

23% 2190 20% 21% 1’ 

12% 1490 3% 9% ~ 

15% 11% 7% 11% jl


I


3470 19% 5% 1870 I
 -—— 

Source OIG survey 0[ medical school department chairs, Novemoer 1992


Note Inchsdea only those 223 who did addreas at least one of the 7 sample topics in a CME I


activi~ differences are statistically significant at the .05 level. 1


C-8

Table C-l& Formats of CME activities that addressed CDP information 

Family 

Medicine 

chairs 

(n=25) 

Large group activity 6090


Small group activity (eg. discussion


grouphvorkshop) 24%


In-house activity (eg. grand rounds) 60%


Activity open to outside audiences 409’0


Self-directed study 4%


Neurology OncOlogy Average 

chairs chairs 

(n=41) (n=73) (n=139) 

56% 44% 50% 

34% 53% * 42% 

73% 71% 70% 

44% 34% 38% 

5% 5% 5% 

“ ,LIrcc ~!G sumt’v of medical schooi department c!,.aim.November 1992 

II Note: includes only those 139 (62 percent) wno reported that their @lE achwties on the relm’ant tovics naa u.. ‘hlLUYor ,[ 

probabty addressed- CDP findin&; &terisk (*) indie&s that the differences noted are statistically sign~fieant at the .05 level. 

Table C-II: Target audiences for medical school CME activities that addressed CDP 

information 

=EqE!EElr 

In-house hospital staff 44% I 76% 71% * 68940 

In-house medical school faculty 48% I 98% 84% * 81% 

Residents 80% ~ 95% 89% 89% 

Researchers from other institutions o% 5% 16% 10% 

Practicing physicians: local or nonlocal 

68% 76% 71% 72% 

From the local area 

6s% 66% 7070 6a% 

From outaide the local area 

40% qlqo 30% 35% 

Source OIG sutvey of medieal school department chairs, November 1992


Note Includes only those 139 (62 pereent) who reported that their CME activities on the relevant topies had definitely or


probably addreased CDP findingq an asterisk (*) indieatea that the differences noted are statistically significant at the .05


level.


c-9 
Table C-12 Additional use of CDP information in CME 

Family Neurology Oncology Average 

Me&ine Chahx Chairs 

chairs n=264 

Percentage aware of CME activities that 7% 970 18% * 12% 

have been conducted at their institution that 

addressed information from a Consensus 

Conference other than those in our sample (n=83) (n=87) (n=94) (n=264) 

Percentage that have attended a CME 14% 22% 44% * 2870 

activity sponsored by an organization other 

than their own that addressed information 

from a Consensus Conference (n=83) (n=87) (n=95) (n=263) 

Source: (3JG survev of medical school department chaim. November 1992 

Table C-13: Of those 73 (28 percent) that attended a CME activity sponsored by another 

organizitio~ the percentage reporting that this activity had been sponsored by each type of 

organization 

ME 

a. Another medical school/teaching 

=b. A national specialty society 7 : : : 

c. A state or local specialty society 17% 5% 12% 11% 

d.	 A for-profit firm (eg., a 25% 11% 7% 11% 

pharmaceutical company, etc.) 

Souros OIG sumq’ of medical school department chain, November 1992 

c-lo 
Table C-14 Comments on the CDP 

Very Some A Not 1 N/A 

much what little at all ~ 

a. the topics addressed in NIH ~Average 62 32 4 01 2


Consensus Conferences are


timely , Family Medicine 36 51 11 o~ 2


I ~n::oBB� 77 17 1 0 4 

68 30 1 0 0 

b. the issues addressed by the Average 38 47 12 11 3


CDP are appropriate for


resolution through the group Family Medicine 39 41 15 0 5


judgment method of consensus Neurology 46 45 6 o’ 3


development Oncolo~” 30 52 14 2 1


.——— 

c. the Consensus Statements Average * 48 41 9 1 2 

usefully distill large bodies of 

information� Family Medicine 49 31 16 0 3 

Neurology 52 38 4 1 4 

Oncology 45 49 7 0, 0 

d. the 2-1/2 day format of the Average 31 39 9 4 ‘ 18


NIH Consensus Conferences


allows sufficient time to examine Family Medicine 24 32 14 71 24


and consider the issues in Neurology 35 42 7 01 16


question Oncology 33 40 7 4 16


e. the recommendations of the Average * 40 50 7 1 3 

consensus� panels are practical 

Family Medicine 26 54 13 3 3 

Neurology 52 42 3 0 3 

Oncology 39 52 7 0, 2 

L the Consensus Statements are Average * 38 47 10 1 4 

sufficiently directive 

Family Medicine 26 51 15 3 5 

Neurology 46 46 4 0 4 

Oncology 41 46 10 1 2 

1. the recommendations are Average 55 40 3 0 2


iirected towards clinical, not just


“esearch audiences Family Medicine 40 45 12 0 3


Neurolo~ 58 39 0 0 3 

Oncology 62 37 0 0 1 

1.NIH sponsorship of the CDP Average 63 24 9 2’ 3


mhances the credibility of the


?rogram’sfindings Family Medicine 56 23 11 21 8


Neurology 71 22 3 3 1 

Oncology 61 26 11 1 1 

C-n 
Table C-14: Comments on the CDP 

Very 

, much 

I 

i. the topics addressed by the Average 52


CDP are appropriate for


incorporation into CME Family Medicine 38


activities Neurology 58


Oncology 57 

j.� the format in which the Average 33 

findings of Consensus panels are 

presented are useful for CME Family Medicine 12 

activities Neurology 42 

Oncology 40 —.————- . ——— . 

Some A Not NIA 

what little at all 

38 9 1 1 

48 13 0 2 

35 4 1 1 

34 9 1 0 

45 16 2 4 

60 24 2 3 

39 12 3 4 

40 14 2 3 —. 

!1 

Note Excluding these who were not at all aware of the CDP; n=222 (Family Medicine n=61; Neurology n=6q Oncology ‘ 

n=92); an asterisk (*) indicates lhat the differences are statistically significant at the .05 level; numbers are in percxmt. 

Table C-15: Degree of respect for the findings of the ~ 

Average 49 44 

Family Medicine 38 52 

Neurology 64 32 

Oncology 44 46 

I 

A little No 

respect respect 

( I 

8 0 

10 0 

31 0 

10 01 

1 

Source OIG survey of medical school department chairs, November 1992 

Note: Excluding those who were not at all aware of the CDP; n=220 (Family 

Medicine n=61; Neurology n=68; Oncology n=91); the differences among 

department chairs are statistically significant at the .05 level. 

C-12

Table C-16 Percentage of department chairs describing each type of material as potentially 

useful for their CME activities 

Family Neurology ~ Oncology ~ Average 

Medicine Chairs ; Chairs I 

chairs ~ 

CME curriculum materials for small-group I II 

settings 89% 76910 80% 81~o 

CME curriculum materials for large-group 

settings 78% 69$Z0I 62% 70% 

CME curriculum materials for self-directed 

study 80% 64% ~~ 61% ;,F * 6870 

CME curriculum materials for longer CME ~ II 

activities (1 day conferences/ II 

courses) II 45?70 4570 39% I 4370 

Videotapes of Conference highlights II 42% I 49% 35% II 42% 

Videotapes of Conference press [

I 

conferences 9% 11% 6% 9% 

Conference proceedings I 

in booklet form 62% 82% 81% * 75% 

Reference bibliographies 86% 85% II * 81% 

Complete listing of past CDP topics 83% 83% ~ 74% I 80% 

Source OIG suIvey of medical school department chairs, November 1992


Note N is based on all 267 department chair respondents, regardleas of their level of awareness of the CDP; the response


rate for individual questions ranged from 235 to 250 an asterisk (*) indicates that the differences among department chaim


are statisticaifv simificant at the .05 level.


C-13

Table C-17 Reported degree of department-chair involvement in the selection of topics for 

~=F 

, Very involved 44% 78V0 62% 

Somewhat involved ! 47% 16% 33% 32% 

Not at all involved 9% 6% 2% 6% 

Source OIG sumey of medical school department chain, November 1992 

C-14

SURVEY INSTRUMENT MEDICAL SCHOOL DEPARTMENT CHAIRS


The following survey was sent to chairs of family medicine. Similar surveys were sent 

to chairs of neurology and oncdo~, in those surveys, questions A-3 and B-2 addressed 

Casensus topics relevant to those fields. 

C-15

SECI’ION k AWARENESS OF THE NH CONSENSUS DEVELOPMENT PROGRAM 

1. Prior to receiving this survey, how familiar were you with the National Institutes of Health 

Consensus Development Program? Please check one: 

~El 

Very familiar


Somewhat familiar, but not sure of all the details


Aware, but didn’t know much about it


Not at all aware I


If Not at all aware, then please go to Section B on Page 2. Otherwise, please continue. 

2. From what sources have you heard about the CDP during the past three years? Please check all 

that apply: 

a. Received advance notice of at least one upcoming NIH Consensus Conference 

the mail 

b. Received at least one NIH Consensus Statement in the mail 

IIc. Received at least one NIH Consensus Statement at a CME activity, professional 

IIsociety meeting, or conference 

I d. Heard about the CDP from colleagues within my institution 

II e. Heard about the CDP from colleagues at other institutions 

f. Read about the CDP in .MM4 

g. Read about the CDP in a specialty journal. Please specify: 

h. Read or saw something about the CDP in the popular media 

i. Read at least one NIH Consensus Statement in L0L4 

in 

~1----1 

IL----I 

II II 

L-4 

IHI 

I-----4 

IL--! 

IMI


j. Read at least one NIH Consensus Statement in a specialty journal. Please speci~: H 

k. Other. Please speciijc 

C-16

3. Prior to receiving this survey, were you aware of the fo]]owing specific Consensus Statements? 

IITOPIC 

IIa. Clinical Use of Botuhzwn Toxin (November 1990) IL-U 

b. Earfy Stage Breast Cancer (June 1990) 

c. Gastrointestinal Suqgety for Severe Obesity (March 1991) l--+-+ 

d. Treatment of Panic Disorder (September 1991) Iul 

4. Please list any other specific Consensus Statements of which you are aware: 

SECTION B. USE OF NH-I CONSENSUS CONFERENCE INFORMATION 

1. Does your department address the following types of topics in CME activities? 

~EIEl 

a. The results of medical technology 

assessments 

b. Clinical applications of research 

c. Specialty society practice guidelines/ 

practice parameters .B 

2. Has your department conducted a CME activity in the past three years (either in-house or for 

outside audiences) that addressed, at least in part, any of the following topics: 

l’UPIC 

a. Clinical Use of Botulinum Toxin ~‘a ‘0 

b. Early Stage Breast Cancer 

II 1 1 

c. Gastrointestinal Surgery for Severe Obesity II I I 

d. Treatment of Panic Disorder II I I 

C-17

3. If ~ for at least one of these topics, were the findings of the NIH Consensus Conference on that 

topic addressed in the CME activity (ies)? Please check one: 

d


Definitely yes


Probably, but not sure


Don’t know


Probably not, but not sure


Definitely not


3b. If YES, what was the format(s) of the activity(ies)? Please check all that apply: 

a. Large group activity 

b. Small group activity (eg. discussion group/ 

workshop) 

c. In-house activity (eg. grand rounds) 

d. Activity open to outside audiences 

e. Self-directed study 

f. Other: Please describe: 

II II 

l----+ 

L


3C. To whom was the activity(ies) targeted? Please check all that apply: 

~m 

a. In-house hospital staff 

b. In-house medical school faculty 

c. Residents 

d. Researchers from outside your institution 

e. Practicing physicians from the local area 

f. Practicing physicians from outside the local 

area 

g. Other: Please describe: 

II II 

II II 

II II 

II II 

!_____l 

C-18

4.	 Are you aware of any CME activities sponsored by your institution that have addressed information 

from other NIH Consensus Conferences? 

Yes No 

4b. If YES, please list which Consensus Conference(s): 

5.� Have you ATI’ENDED any CME activity sponsored by an organization other than your own that 

addressed the findings of one of the NIH Consensus Conferences noted above? 

Yes No 

5b. If YES, what sort of organization sponsored this activity (ies)? Please check aii that apply: 

IIa. Another medical school/teaching hospital Iul 

b. A national specialty society 

I-----! 

c. A state or local specialty society 

d. A for-profit firm (eg., a pharmaceutical 

company, etc.) 

e. Other (please specify): 

IL IIH Ii 

C-19

SECITON C COMMEN’IS ON THE CDP 

1. The opinions of CME and clinical profSion& tow~ds the ~p and towards group judgment 

efforts in general may affect the way in which specific r~mmendatiom from the Program are received 

In the following table, we ask you to answer several questions by circling a choice on a four-point scale, 

where 1 = Verv Muck 2 = somewha~ 3 = A Little, and 4 = Not at All. If you are unable to answer 

a queatio~ please circle Not Applicable, LA 

! Please rate the degree to which	 Vay Some- A Notat 1NA 

Much what Little Au 

1 a. the topics addressed in NIH Consensus 1234 NA1 

], Conferences are timelv 

I b. the issues addressed by the CDP are appropriate ‘1234 ~NA 

~ for resolution through the group judgment method of I 

‘ consensus development 

c. the Consensus Statements usefully distill large 1234 NA 

‘~bodies of infoma?ion —.. ———.. 

‘1d. the 2 lfl-day format of the NIH Cons&us 1234 I NA ~: 

I Conferences allows sufficient time to examine and I I 

~1consider the issues in question 

e. the recommendations of the consensus panels are 1234 ~NA II 

i practical 1 

‘ f. the Consensus Statements are sufficiently directive 1234 lNA\ 

I 

IIg. the recommendations are directed towards clinical, I 1 2 3 4 I NA II 

not just research audiences 1 

) II I

h. NIH sponsorship of the CDP enhances the 1234 INA1 

, credibility of the Program’s findings ( I \ , 

i. the topics addressed by the CDP are appropriate 1234 ~NA ‘ ~ 

! for incorporation into CME activities ! 4 

i	 j. the format in which the findings of Consensus 1234 ~NA 

panels are presented are useful for CME activities 

2. Please rate your overall degree of respect for the findings of NIH Consensus Development 

Conferences (circle one): 

Great Some A little No 

Respect Respect Respect Respect 

1 2 3 4 

C-20

SECI’fON D: CME ACTIWIES 

In this sectio~ we ask how NIH might make information from the NIH bnsensus Development 

Program more useful for CME activities. 

1. Would it be useful if NIH made the following materials related to Consensus Development Program 

topics available for your use in CME activities? Please check Yes or No: 

I


i MATERIALS IF=F=


I 

I a. CME curriculum materials for small group settings 

~ b. CME curriculum materials for large group settings 

] c. CME curriculum materials for self-directed study 

d. CME curriculum materials for short sessions


(eg. 1 hour grand rounds) A E


c. Ch4E curruuurn mawri~is luI iol]gcr CJIE a-uv]ties 

II(eg. 1 day conferences/courses) ~,1’ 

I f. Videotapes of conference highlights 

g. Videotapes of conference press conferences 

h. Conference proceedings in booklet form 

i. Reference bibliographies 

j. Complete listing of past CDP topics 

k. Other: Please speci~: I 

2. How else might NIH work with medical school faculty to facilitate the use of Consensus 

Development information in CME activities? 

3. Please add any additional comments: 

C-21

SECI’ION E RESPONDENT INFORMATION


In order to better understand the responses to our survey questions, we ask a few questions about your


professional background This information will be used solely in aggregate, and will be kept


confidential.


Your name: 

Your title: 

1. Please note your area of specialty: 

2. How long have you been head of your department 

or section? 

3.� Please rate the degree to which you are 

{~~~1.~~din ~~:: ~ele::~on U! t.,pi~s (or 

CME activities m your department (circle one): 

Very Somewhat Not at all 

Involved Involved Involved 

Thank you for taking time to complete this surv~ we appreciate your assistance. Please return the 

sumey in the enclosed postage-paid envelope to 

HHS/OIG/OEI 

Tel: 

FAX: 

c-22 
APPENDIX D


SUMMARY OF CME DIRECTORS’ SURVEY RESPONSES 

Tabl~ Page D-2 

D-1 OIG survey sample size and response rate


D-2 Familiarity with the CDP


D-3 Sources of information about the CDP


D-4 Awareness of each of 7 sample Consensus Statements


D-5 )waremxs of additicm:]! C’msensus St’ltements


D-6 Percentage reporting that their institutions address each type of topic in their CME activities 

D-7 Use of CDP information by CME offices 

D-8 Comments on the CDP 

D-9 Degree of respect for the findings of the CDP 

D-10� Percentage of CME directors describing each type of material as potentially useful for their 

CME activities 

Blank SWey Instrument Page D-9 

D-1

Table D-1: OIG survey sample size and 

remonse rate 

Number of Respondents I 79 

Response Rate 1 66.4% 

Table D-Z Awareness of the CDP II 

Very familiar -I 

Somewhat familiar, but not sure of all ‘~ 

the details L----l Ii 48%

Aware, but didn’t know much about it II,41

Not at all aware II 1890 

I 

Source OIG survey of medical school CME directo~ November 1992 II 

D-2

------

Table D-3: Sources of information about the CDP II 

1 

Received a CDP Statement by mail 

Received advanced notice of a conference bv mail 

Read a CDP Statement in JAMA 

Read about the CDP in JAMA 

Read a CDP Statement in a specialty journal 

Read about the CDP in a specialty journal 

Received a CDP Statement at a CME activity, 

proiesswna: society meeting, or conicrence 

Heard about the CDP from colleagues within own 

institution 

Heard about the CDP from colleagues at outside 

institutions 

Read or saw something about the CDP in the 

popular media 

_7iiE 

Directors 

(n=65) 

82% 

59?70 

25% 

31% 

6% 

2% 

?gy~ “ 

17% 

12% 

15%

; 

Source: OIG sum’eyof medical school CME directors; November 1992


Note Excludes those 14 CME directom (18%) who described themselves as “Not at all


aware” of the CDP


D-3

Table D-4 Awareness of each of 7 sample Consensus 

Statements II 

Statement 

Direetors 

(n=62) 

Surgey for Epi[ep~ 32% 

Clinical Use of Botulinum Toxin 29% 

I/Intravenous Immuno@obulin 

Adjuvant Therapy for Colon and Rectum 44% 

Cancer 

Source: OIG sutvev of medical school CME directow November 1992 

Note Excludes [hose 14 CME directom (187.) who described Themselves 

~‘ 

Table D-5: Awareness of additional Consensus 

Statements: Percentage reporting that they were 

aware of Statements not in our sample 

Number of Statements 

none 

1 

2 

3 

4 

6 

Percentage of 

respondents 

(n=65) 

7970 

9% 

5% 

3% 

3% 

1.5% 

Source OIG survey of medieal schwl CME directo~ November 1992 

D-4

Table D-& Percentage reporting that their institutions ~ 

address each tvue of to~ic in their CME activities I 

The results of medical technology assessments 

Clinical applications of research 

Specialty society practice guidelines/ 

practice parameters 

======+ 

Directors 

(n=78) I 

d

94% I 

60% 

Source: OIG survey of medical school CME directors; November 1992 j 

D-5

-

r

— Table D-7 Use of CDP information by CME offices

[ 

a. Percentage reporting that their office plays a role in topic selection 82%


for CME activities (n=78)


[

b. Percentage reporting that they are familiar with the content of the 96%


CME activities for which they grant credit (n=79)


c. Percentage of those familiar with the content of their CME 23%


activities who were aware of any CME activities sponsored by their


institution during the past 3 years that had addressed information


from an NIH Consensus Conference I (n=75)


d.� Percentage of those CME directors who had received a I

Consensus Statement by mail during the past 3 years who had

forwarded it to a member of the CME committee or faculty: 

never 67% 

once 4% 

2-4 times 19% 

5 or more times 170 

not sure 9% 

(n=79) 

e.� Percentage reporting that a faculty or staff member had 

approached them during the previous 3 years with a recommendation 

to offer a CME activity addressing a specific NIH Consensus 

Conference: 

never 84% 

once 6% 

2-4 times 5% 

5 or more times 090 

not sure 5% 

(n=79) 

Source OIG sutvey of medical school CME directo~ November 1992 

D-6

Table D-& Comments on the CDP 

a. the topics addressed in NIH Consensus 

Conferences are timely 

b. the issues addressed by the CDP are appropriate 

for resolution through the group judgment method 

of consensus development 

c. the Consensus Statements usefully distill large 

bodies of information 

d. the 2-1/2 day format of the NIH Consensus 

Conferences allows sufficient time to examine and ., ~:~nsl[~~~t~l:. : es in am.x:icn 

Very some 

much what YLui_lE


m 

7 0 36 ~ 

2 0 18 ~

! 

2 0 18 ~ 

3 0’ 16 ~ 

, 

16 0 21 i 

‘1 

46 

20 

37 

18 

33 

51 

35 

24 

30 

33 

28 

20 

31 

44 

e. the recommendations of the consensus panels are 28 

practical 

f. the Consensus Statements are sufficiently directive 25 

g. the recommendations are directed towards clinical, 38 

not just research audiences 

h. NIH sponsorship of the CDP enhances the 61 

credibility of the Program’s findings 

i. the topics addressed by the CDP are appropriate 50 

for incorporation into CME activities 

j.� the format in which the findings of Consensus 18 

panels are presented are useful for CME activities 

sour~ OIG survey of medical school CME directors, November 1992 

Note Excluding thase who were not at all aware of the CDP; n varies from 61 to 63 for individual questions; numbers are in 

percent 

D-7


I 
Table D-9 Degree of respeet for the findings of the CDP II 

I Great Some 1 A little No I 

~1Respeet ~ respeet respect respeet 

CME directors i 59 I 38 3 0 

Source OIG survey of medical school CME directo~ November 1992


Note Excludes those who reported that they were Not at all aware of the CDP; n=58


Table D-l& Percentage of CME direetors deseriiing each 

type of material as potentially useful for their CME 

activities 

IICME curriculum materials for small-group settings II 89% II 

IICME curriculum materials for large-group settings 88% II 1 

IICME curriculum materials for self-directed study II 80% II 

CME curriculum materials for short sessions 

(eg., 1 hour grand rounds) 

CME curriculum materials for longer CME 

activities (1 day conferenceshxmrses) 

IIVideotapes of Conference highlights IHI 

IIVideotapes of Conference press conferences IL---=-I 

IIConferenceproceedhtgsin booldet form II 80% I 

IIReference bibliographies II 81% II 

Complete listing of past CDP topics 80% 

Source OIG survey of medieal school CME directo~ November 1992


Note N is based on all 79 CME director respondents, regardless of their level of


awareness of the CDP; it ranged from 70 to 75 for individual questions


D-8

SURVEY INSTRI.JIWEIT’C MEDICAL SCHOOL CME DIRE~ORS 

D-9 
Ir

SECTION A AWARENESSOF THE NIH CONSIWSW3I)IWELOPMENTPROGRAM 

1. Prior to receivingthissurvey,howfamiliarwereyouwith the NationalInstitutesof Health Consensus 

DevelopmentProgram(CDP)? Pleasecheckone: 

ii


~ Very familiar


1 Somewhat familiar, but not sure of all the details


I Aware, but didn’t know much about it


il Not at all aware


ml 

II
!Ul 

II II 

II II 

lL__J 

If Not at all aware, then please go to section B on Page 2 Otherwise, please continue. 

2. From what sources have you heard about the CDP (iurmg the past three years’? Please check all that apply: 

SOURCE	 ,: 4
 1


a. Received advance notice of at least one upcoming NIH Consensus Conference in the mail II


b. Received at least one NIH Consensus Statement in the mail i


c. Received at least one NIH Consensus Statement at a CME activity, professional society r

II 

meeting, or conference ‘~ 

d. Heard about the CDP from colleagues within my institution E 

e. Heard about the CDP from colleagues at other institutions !1 

f. Read about the CDP in l-4Wl ‘1--

,--- g. Read about the CDP in a specialty journal. Please specitjc
 I


h. Read or saw something about the CDP in the popular media ,1 

i. Read at least one NIH Consensus Statement in .L4M4 y 

;-


II
 j. Read at least one NIH Consensus Statement in a specialty journal. Please speci@: I


k. Other. Please specify: II


D-10

3. Prior to receiving this survey, were you aware of the following specific Consensus Statements? Please check 

Yes or No: 

IIa. Surgeryfor Epilep~ (March 1990) 

II I 

b. Clinical Use of Botzdinum Toxin (November 1990) 

IL--L 

c. IntravenousImrnunoglobulin (May 1990)

II II I 

i d. Adjuvant Thera~ for Colon and Rectum Cancer (April 1990) 

I II I 

IIe. Early Stage Breast Cancer (June 1990) II I 

f. GastrointestinalSurgey for SevereObesity (March 1991) 

g. Treatment of Panic Dtiorder (September 1991) H 

4. Please list any other specific Consensus Statements of which you are aware: 

SECTION B: USE OF NIX-ICONSENSUS DEVELOPMENT PROGRAM INFORMATION 

Medicai school CME offices differ in the degree to which they are involved in the seleetion of topics and 

development of eurrieula for the CME activities for which they grant credit. In this section, we ask some 

questions about how your program operates, and about the use of CDP information at your institution. 

1. Does your institution address the following types of topics in CME activities? Please check Yes or No: 

a. The results of medical technology 

assessments 

b. Clinical applications of research I 

c. Specialty society practice guidelines/ II I II 

practiceparameters 

2. Does your CME office play a role in the choice of topics for CME activities for which you grant credit? 

Yes No 

D-II

3.� Are you familiar with the content of the CME activities for which YOU grant credit? 

Yes No 

3b. If YES, are you aware of any CME activities sponsored by your institution during the past three 

years that have addressed information from an NIH Consensus Conference? 

Yes No 

3c. If YES, please list which Consensus Conference (s): 

4. How many times in the past three years have you forwarded an NIH consensus Statement to a member of 

your CME committee or a member of your faculty for review? 

None one 2-4 times 5 or more times Not Sure 

Which Statement (s)? 

4. How many times in the past three years has a member of your institution’s faculty or staff approached you 

with a recommendation to offer a CME activity addressing a specific NIH Consensus Conference? 

None One 2-4 times 5 or more times Not Sure 

Which Statement (s)? 

D-12

SECI’ION C COMMENTS ON THE CDP 

1. The opinions of CME and clinical professionals towards the CDP and towards group judgment efforts in 

general may affect the way in which specific recommendations from the Program are received In the following 

tabl~ we ask you to answer scweral questions by circling a choice on a four-point scale, where 1 = Vew MUCIL2 

= Somewha& 3 = A Utt.le, and 4 = Not at All. If you are unable to answer a question, please circle Not 

Applicable &A 

Please rate the degree to which VW ~- A Notat NA 

MudI what little Au 

a. the topicsaddressedht NIH Consensus(lmferences are 1234 NA


timely


b. the issuesaddressedby the CDP are appropriate for 1 234 NA


resolution through the groupjudgmentmethodof


consensusdevelopment


c. the ConsensusStatementsusefullydistill large bodies of 1 234 NA 

int’orm.’llloll - _—

d. the 2 1/2 day format of the NIH Consensus Conferences 1 234 NA	

allows sufficient time to examine and consider the issues	

in question	

e. the recommendations of the consensus panels are 1 234 NA	

practical	

f. the Consensus Statements are sufficiently directive 1234 NA	

g. the recommendations are directed towards clinical, not 1 234 NA	

just research audiences	

h. NIH sponsorship of the CDP enhances the credibility of 1234 NA	

the Program’s findings	

i. the topics addressed by the CDP are appropriate for 1 234 NA	

incorporation into CME activities	

j. the format in which the findings of Consensus panels 1234 NA	

are presented are useful for CME activities	

2.� Please rate your overall degree of respect for the findings of NIH Consensus Development Conferences 

(circle one): 

Great Some A little No 

Respect Respect Respect Respect 

1 2 3 4 

D-13

SECI’ION D CME ACmTITES 

IIIthis seetio~ we ask how NIH might make tio~tion from the NIH COIMXMUSDevelopment Program more 

useful for CME activities. 

1. Would it be useful if NIH made the following matefials related to consensus Development Program topics 

available for your use in CME activities? Please check Yes or No: 

II a. CME curriculum materials for small group settings r I I 

IIc

II b. CME curriculum materials for large group settings IL>II 

CME curriculum materials for self-directed study II I I 

IId. Videotapes of conference highlights 

i I 

IIe.Videotapesof conferencepress conferences ,, I I 

IIg.Referencx bibliographies (1 I 

IIi. II h. Complete listing of past CDP topics r 1 

Other: Please specify: II I 

2. How else might NIH work with medical school CME offices or facul~. to facilitate the use of Consensus 

Development in~ormation in CME activities? 

3. Please add any additional comments: 

D-14

SECTION E2 RESPONDENT INFORMATION 

In order to better understand the responses to our survey questions, we ask a few questions about your 

educational and professional background. This information will be used solely in aggregate, and will be kept 

confidential. 

Your name: 

Your title: 

1. Please note your educational degree (s): 

2. If you are an M.D., please note your area 

of specialty 

2.� How long have vou been in the field of C,ME? . 

3. How long have you been in your current position? 

‘I%ankyou for taking time to complete this sunq we appreciate your assistance. Please return the survey in 

the enclosed postage+aid envelope to 

HHS/OIG/OEI 

Tel: 

FAX: 

D-15


APPENDIX E 

PUBLIC HEALTH SERVICE 

COMMENTS ON THE DW REPORT 

E-1 
,, ...””,, 

/+ ‘$ 

~’ 

: DEPARTMENT OF HEALTH & HUJ}lW SERVICES

;

5 Public Health Sewi< 

‘*, -g 

+%,,” >

Rockville
 MD 20857~


DEC27 1~ 

From: Deputy Assistant Secretag for Health Management 

‘ Operations 

Subject:	 Office of Inspector General (OIG) Draft Report 

NIH Consensus Development Progrm: Dissemination “The 

Findings Through Medical School Continuing Education of 

Activities, “ OEI-01-91-01760 

To : lnspeczor General, OS 

Attached are the Public Health Service comments on the subject 

OIG draft report. We concur with the recommendations and the 

National Institutes of Health has taken or plans to take 

actions to implement them. 

Attachment 

E-2

----

PUBLIC HEALTH SERVICE (PHS~ COMMENTSON THE OFFICE OF


INSPECTOR GENERAL (OIG~ DRAFT REPORT “THE NIH


CONSENSUS DEVELOPMENT PROGRAM: DISSEMINATION


OF FINDINGS THROUGH MEDICAL SCHOOL CONTINUING


EDUCATION ACTIVITIES, ” OEI-01-91-01760


General Comments


The National Institutes of Health (NIH) appreciates the study


performed by OIG of an important avenue for information 

dissemination that could enhance the effectiveness of the 

Consensus Development Program (CDP). Continuing Medical 

Education (WE) was identified in previous evaluation studies 

as an important channel wi.thi.n American medicine that could be 

better utilized by the CDP. The OIG report reaffirms this 

observation and suggests several important steps that could be 

‘-.]<en. ~~-,?:7T::f ~ of:: (-~ of ::2~~:--- .=1 ApplLcat~mM of Research ‘GO 

clarify’’medical treatment issues and inform the community of 

practitioners. 

OIG Recommendation


1.	 The NIH should take steps to increase awareness of the 

CDP among those responsible for C14E. NIH should make 

special efforts to reach those who sponsor continuing 

education for general and family physicians. 

PHS Comment


We concur. To promote awareness of the CDP, the Office of 

Medical Applications of Research is in the process of 

publishing its recently revised and clarified statement of 

operating guidelines and procedures. The updated procedures 

will serve to share the results of NIH~s growing experience 

with key persons in the international community of health 

technology professionals. Special efforts will be made to 

i.dentj.fy leaders in the CM13community and make them aware of 

the cDP and of lllH’s procedures for dissemination of findings. 

OIG Recommendation


2.	 The NIH should identify more effective ways ~of 

encouraging%,the incorporation of consensus findings into 

continuing education activities. 

PHS Comment


We concur. NIH will attempt to icient~fy or create new ways to 

adapt their materials and exploit the CME activities to reach 

more practitioners. In this-process, NIH will solicit ideas


from CME leaders and other professionals, and will consider


E-3

2,


developing slides, overheads, and other curriculum materials


as suggested in the report.


OIG Recommendation 

3.	 The NIH should strengthen its efforts to understand and 

address basic concerns about the CDP. 

PHS Comment ‘


We agree to continue evaluating the procedures used to 

initiate a conference, develop a consensus statement, and


disseminate the results. NIH will also continue its efforts 

to evaluate the impact of conferences on medical practice to 

&scertair& whic1-Lt~~..esc.f.:cmfesenc’e :G.LCS, zcGt2ment .t~rmats, 

and dissemination efforts have the greatest impact. The 

concerns expressed by the critical minority of respondents in 

this OIG study will be considered and assessed seriously.


Technical Comment


Paqes 10 to 12. We suggest that the report indicate that the 

negative views of the CDP that were volunteered by a minority 

of respondents, while sincerely held, may be inaccurate or 

based on out-of-date information. 

E-4

APPENDIX F 

NOTES 

1. National Journal, January 22, 1977, p. 142. 

2.	 “An NH-I Overview of a Report of the Council on Health Care Technology of 

the Institute of Medicine on NIH Consensus Conferences,” NIH Coordinating 

Committee on Assessment and Transfer of Technology, February 10, 1992, p. 3. 

The CDP is administered by the Office of Medical Applications of Research 

(OMAR), within the Office of the NIH Director; each Conference is 

cosponsored by OMAR and one or more of NIH’s institutes, centers, or 

diw”sions. Since its inception, OM~ has conducted almost nine~ Consensus 

Conferences on topics across the spectrum oi medicai science and practice. 

3.	 The founders of the Program stressed that NIH’s role was to facilitate 

‘technical consensus’ on medical technologies, which was defined as 

“scientific/medical agreement on the scientific facts that a given innovation is 

deemed optimal and potentially feasible for introduction into practice.” The 

NIH was not to become directly involved in what was termed ‘interface 

consensus’ on issues such as cost-effectiveness, patient preference, or ethics. 

Indeed, NIH administrators expressed a “critical assumption,” that %roadened 

responsibilities [would] ~ draw NIH into activities inappropriate to its primary 

research mission.” Responsibilities of NIH at the Health Research/Health Care 

lnte~ace, Seymour Perry, MD, Office of the Director, NIH, February 1977, p. 

A-4, A-2. 

In 1989, Congress also created the Agency for Health Care Policy and 

Research (AHCPR), within the Public Health Service, to evaluate medical 

technologies and treatments and to develop practice guidelines. 

4.	 Guidelines for the Selection and Management of Consensus Development 

Conferences, OffIce of Medical Applications of Research, National Institutes of 

Health, September 1988, p. 3. 

The NIH develops specialized dissemination strategies for individual 

Conferences. Dissemination activities include press conferences; press releases; 

distribution of video highlights of Conferences; coverage in the popular press; 

publication of Statements in both general and specialty medical journals; and 

direct mailing of Statements to physicians, hospitals, medical schools, third-party 

payers, policy makers, continuing medical education programs, and others. In 

addition to maintaining a mailing list of standard contacts, NIH also purchases 

targeted mailing lists from relevant specialty societies and other groups for 

individual Conferences. The NIH does not target medical school department 

F-1

chairs asa discrete category (OMAR I)kec,torfCommunications,
 Personal


CommunicationwiththeOIG, October14,1992).


5. Information Dissemination to Health Care Practitioner and Policymakens: 

Annotated� Bibliography, Agency for Health Care Policy and Research 

(AHCPR), Public Health Service, U.S. Department of Health and Human 

Services, April 1992, p. 1. 

6. The most prominent of these have been funded by NIH itself, and include: 

Evaluation of the NIH Consensus Development Process, Paul Wortman, Center 

for Research on Utilization of Scientific Knowledge, Institute for Social 

Research, University of Michigan, 1982. 

Changing Medical Practice through Technolow Assessment: An Evaluation of 

,.le i tii. _Gil>~tCSUS ~evei:jpment Program, Dawd km~ JL3, et w., kind 

Corporation, March 1989. 

Consensus Development at the NIH: Imuroving the Pronam, Committee to 

Improve the NIH Consensus Development Program, Council on Health Care 

Technology, Institute of Medicine (IOM), 1990. 

The OMAR has also conducted several smaller evaluation projects, including 

an examination of the role of medical opinion leaders in the diffusion of 

medical innovations (Expert Panel Meeting Summay The Role of Medical 

Opinion Leaden in Disseminating Consensus Development Conference 

Recommendations, January 31, 1989) and a series of focus groups with 

practicing physicians on their attitudes towards the CDP (Physician Focus 

Groups 10 Identifi Strategies for Dikremination of Consensus Development 

Statements, June 1990). The focus group report noted that “more information 

concerning the specific topic of physician awareness and perception of 

[Consensus] statements is needed.” NIH Physicitirz Focus Groups, p. 3. 

7.	 Rand, p. 244. The report oversight committee further urged that “reaching 

these primary care physicians should have high priority” (p. x). 

The NIH plans to target generalist physicians and internists in an upcoming 

series of surveys regarding practice patterns and awareness of Consensus 

findings. The NIH will suxvey them regarding every Conference, because “they 

are more likely to be first in contact with patients with the disorder.” OiWIR 

Quick-Launch Physician Practice SurvqY, NIH Clearance Package submitted to 

the Office of Management and Budget (OMB), September 1991, p. 32. 

8.	 Rand p. 245, p. 233. The study found that CME courses, conferences, and 

meetings were rated by responding physicians as the most important 

information sources for both finst hearing about and deciding to use a new 

F-2

procedure.Italsonoted thattheetient ofpafiicipation
nCMEwas'tthe


9. 

10. 

11. 

12. 

13. 

14. 

15. 

single most important predictor of whether physicians are aware of specific 

conference recommendations” (p. xiii, p. 97). 

NIH Physician Focus Groups, p. 9. 

Continuing Medical Education: A Primer, Alliance for Continuing Medical 

Education, 1992, p. 96. 

Administrators at NIH noted at the time of the CDP’S founding that “medical 

school teaching hospitals and main disease research centers currently represent 

the most effective transfer points for the movement of research knowledge into 

health practice,” and that “any proposed solutions to the dissemination problem 

will have to utilize these strengths already in place.” Perry, appendix A-1, A-3. 

l%e Rand study also determmed that medical schcmis were one of the wo n:cm 

important sources of CME for doctors who were aware of the Program; local 

hospitals were the other most important source (p. 95). 

David Kanouse, Rand Corporation, personal communication with the OIG, 

October 9, 1992. 

Van Harrison, Ph.D., Director of CME, University of Michigan, personal 

communication with the OIG, October 16, 1992. 

The OMAR has identified “opinion leaders” as people holding “positions of 

leadership,” or having “high status title, rank, or office.” The Role Of Medical 

Opinion Leaden, p. 4, 6. 

Rand also found that medical school facuhy members were more likely than the 

average physician to be aware of the Program and were among those most 

likely to first hear about CDP information (p. 72). 

Rand, p. 101. 

Everett M. Rogers, Diffusion of Innovations, 3rd cd., Free Press, New Yorlq 

1983, p. 331. 

Rand, p. 245. 

The impact of CME on physician behavior is a matter of considerable 

controversy. Many researchers have suggested that reimbursement, discipline, 

quality-assurance, and peer-review activities have greater effect on physician 

behavior. This report does not address such questions. 

Furthermore, this report does not explore the issue of getting doctors into 

CME. As an NIH administrator noted at the time of the founding of the 

-.

16. 

17. 

18. 

19. 

20. 

21. 

Consensus Program, “while the results of consensus building should be useful in


meeting continuing education needs of physicians and other health


professionals, this process will not resolve the problem of individuals who are


unable or unwilling to involve themselves in such activities.” Perry, p. 9.


We asked family physicians about four topics, oncologists about three topics,


and neurologists about two topics. The topics were surgery for epilepsy, clinical


use of botulinum toxin, intravenous immunoglobulin, adjuvant therapy for colon


and rectum cancer, early stage breast cancer, gastrointestinal surgery for severe


obesity, and treatment of panic disorder. Each of these had been the subject of


an NIH Consensus Conference held in 1990 or 1991. Members of each group


were asked only about the Statements that were relevant to their field (see


appendix A for a listing of the Statements and their audiences).


Of those who reported that their departments’ relevant CME activities had


we~irulay iitit addressed the Consensus findings, one informed us that the


activity had been conducted prior to the given Conference; two informed us


that they were not aware of the relevant Consensus Statement at the time of


the activity. Several others were unable to say exactly when their relevant


CME activity had been held, but told us that they would not have incorporated


the Consensus findings into the activity at any time because they were unaware


of it, did not agree with it, or did not consider it appropriate for CME.


We did not ask respondents how many people had been reached in these CME


activities; it would not have been feasible for them to access such information.


These figures are considerably higher than those reported by Rand researchers


for a sumey they conducted of randomly chosen physicians in all settings (Rand,


p. 71). The groups that we sumeyed, however, represent key opinion leaders in 

their respective fields. Considering their importance, we regard their level of 

familiarity with the Program as limited. Furthermore, we expect that the level 

of awareness of the Program among physicians in community settings is likely to 

be considerably lower than that of these academic groups. 

We also asked department chairs if they were familiar with any additional 

Consensus Statements, other than the 7 in our sample; 72 percent of all 

department chairs (including 72 percent of each specialty group) reported that 

they were not. Only 9 percent were aware of 2 or more additional statements. 

These figures may somewhat underestimate total awareness of specific 

Statements and specific recommendations in those Statements, as people 

generally separate the ‘message’ from the ‘messenger’ and forget the latter over 

time. We did not measure awareness of--or conformity to--the specific 

recommendations made by NIH Consensus Panels. 

One family medicine chair with whom we spoke pointed out that NIH may 

have a difficult time gaining name recognition for the CDP because of the 

F-4 
22. 

23. 

24. 

25. 

recentproliferation ofmedicaltechnology assessment efforts; he deseribed
 the


result ofthesedifferent programsasa “cacophony
ofparameters.”


We found wide-ranging opinions regarding the relative value of direct-mailings.


and journals as vehicles for disseminating CDP information; these reflect, in


part, differing information habits among doctors. In our interviews, some


department chairs told us that direct mail was the only way to reach them; they


miss things published in journals. Others assured us, however, that publishing is


the only way to get to them; as one put it, “I receive three feet of mail a week


and throw most of it away unread.”


Of the 82 percent of CME directors who had received at least one Consensus


Statement in the mail during the previous 3 years, 57 percent had never


forwarded a Statement to a member of their CME committee or other faculty


for review.


As would be expected, family physicians were much more likely to have heard


about the Program or read a Statement in JAMA than were o-ther specialists;


the reverse was also true with regard to specialty journals (see appendix C,


table C-3).


Other sources of information about the CDP and specific Conferences


included:


Colleagues: The NIH working group on the role of medical opinion leaders in


information dissemination predicted that opinion leaders are more likely than


the average doctor to hear about Consensus recommendations from colleagues


outside their own institutions. This appears to be more true of oncologists and


neurologists than of family physicians: Only 15 percent of family medicine


chairs, but 21 percent of neurology and 33 percent of oncology chairs had


heard about the CDP from colleagues at outside institutions.


Popular media; The popular media was not a major source of information


about the Program: Only 16 percent of department chairs had heard about the


Program through the written or print popular media--ranging from 4 percent of


neurology chairs, to 18 percent of family medicine chairs, to 23 percent of


oncology chairs.


Continuing Medical Education: Twenty-eight percent of department chairs


reported that they had attended a CME activity sponsored by an organization


other than their own medical school that had addressed the findings of an NIH


Consensus Conference--ranging from 14 percent of family medicine chairs to 44


percent of oncology chairs. Of these, 85 percent had attended such activities


sponsored by national specialty societies, and 30 percent had attended such


activities sponsored by other medical schools or teaching hospitals (see


appendix C, tables C-12 and C-13).


F-5

26.	 The Rand study suggested that video coverage of Consensus Conferences, 

particularly of the press conferences, could be valuable for CME activities 

(p. 245). Our findings do not bear this out for medical school CME programs. 

Video tapes of the press conferences received the lowest rating for usefulness 

in CME, 9 percent. Video tapes of conference highlights received the secondlowest rating, 42 percent. A number of respondents noted that they do not 

have the time or desire to sit and watch a video; they prefer either a quick 

summa~ to read alone, or interactive materials to use in a group. 

27.	 Several respondents also recommended more ambitious--and expensive--efforts 

on the part of NIH. In particular, a number recommended that NIH directly 

sponsor, and finance, local CME activities and/or provide visiting speakers. 

One respondent recommended the preparation of case examples and pre- and 

post-tests to facilitate the development of CME programs. 

~1- -0. ~;i~iety-three percent ofthosedepartment chairs who werefamiliarwiththe


CDP expressed some degreeofrespect foritsfindings--49 percent
voicedgreat


respect, ranging from38 percent offamily medicinechairs,
 to44 percentof


oncologychairs, to65 percentofneurology chairs. Sixty-three
 percentalso


strongly agreedthatNIH sponsorship
 oftheprogramenhancesthecredibility


ofitsfindings
 (seeappendixC, table C-14, C-15). 

29.	 University of Michigan researchers also identified the problem of “selection 

bias” in the choice of Conference topics and panelists as a potential “threat to 

the credibility” of the Program. The comments we received from department 

chairs suggest that their concerns regarding this issue persist. 

Wortman, Paul, et. al., “Do Consensus Conferences Work? A Process 

Evaluation of the NIH Consensus Development Program,” Journal of Health 

Politics, Policy, and Law, vol. 13, no. 3, Fall 1988. 

30.	 Even among some specialists in oncology and neurology, there was concern 

about too great a @specialty focus. One radiation oncolo~-st, for example, 

objected to the ‘overrepresentation’ of medical oncologists vs. surgical and 

radiation oncologists. 

31.	 One family physician argued that “NIH has to get into the trenches with the 

people who are going to use this information, rather than bypassing them. NIH 

has traditionally been antagonistic towards family physicians, and then it 

wonders why it can’t reach them.” 

32.	 Again, we believe that the fact that 69 percent of department chairs do not 

strongly regard the length of Consensus Conferences as adequate may well 

serve as a barrier to greater use NIH Consensus findings in their CME 

activities. 

F-6

33.	 Typical comments were: “’consensus’ is not the same as correct,” and “consensus 

is not necessarily equal to state of the art and not necessarily w“dely 

applicable.” As one respondent described this view, “you have one consensus 

and then three months later you have another consensus;” medicine is always in 

a state of flux. 

34.	 An oncologist also asked, “can individuals not knowledgeable about the area 

under study truly contribute to consensus development?” Another argued, “the 

information is generally available. Most conferences and clinicians review the 

original data and journal reviews and draw their own conclusions; why should 

an academic clinician care what these non-experts say?” 

Some respondents expressed doubts that any panel could objectively arrive at a


consensus of opinion that would be adequately grounded in fact. As one


expressed this sentiment, “the process can be self-serving and represent


primariiy the vmvpoint of a few individuals with vested interests rather lhan a


consensus of the scientific community after it has had an opportunity to review


the data in the scientific, peer-reviewed literature.”


35.	 7%e Physician h Recognition Award: 1991 Information Booklet, American 

Medical Association, 1991. 

36.	 In our survey, 62 percent of department chairs reported that they are vey 

involved in the selection of topics for CME activities; another 32 percent 

reported that they were somewhat involved. In addition, 52 percent of the 

department chairs who were familiar with the CDP rated the topics addressed 

in Consensus Conferences as very appropriate for incorporation into CME 

activities--ranging horn 38 percent of family medicine chairs to about 57 

percent of both neurology and oncology chairs. 

37.	 Although 83 percent of CME directors reported in our survey that their ofllces 

play a role in the selection of topics for CME activities, this role tends to be 

limited. For example, 96 percent of CME directors reported that they were 

familiar with the content of the CME activities for which they grant credit, but 

only 23 percent of these were aware of a CME activity at their institution that 

had addressed findings from a Consensus Conference. This number is much 

lower than that reported by the chairs of individual departments. 

F-7


—.—