Evidence Report/Technology Assessment 

Number 149 

Effectiveness of Continuing Medical Education 

Prepared for: 

Agency for Healthcare Research and Quality 

U.S. Department of Health and Human Services 

540 Gaither Road 

Rockville, MD 20850 

www.ahrq.gov 

Contract No. 290-02-0018 

Prepared by: 

The Johns Hopkins University, Evidence-based Practice Center, Baltimore, MD 

Investigators 

Spyridon S. Marinopoulos M.D., M.B.A. 

Todd Dorman, M.D. 

Neda Ratanawongsa, M.D. 

Lisa M. Wilson, Sc.M. 

Bimal H. Ashar, M.D. 

Jeffrey L. Magaziner, M.D. 

Redonda G. Miller, M.D., M.B.A. 

Patricia A. Thomas, M.D. 

Gregory P. Prokopowicz, M.D. 

Rehan Qayyum, M.D. 

Eric B. Bass, M.D., M.P.H. 

AHRQ Publication No. 07-E006 

January 2007
This report is based on research conducted by the Johns Hopkins University Evidence-based 

Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality 

(AHRQ), Rockville, MD (Contract No. 290-02-0018). The findings and conclusions in this 

document are those of the author(s), who are responsible for its content, and do not 

necessarily represent the views of AHRQ. No statement in this report should be construed as 

an official position of AHRQ or of the U.S. Department of Health and Human Services. 

The information in this report is intended to help clinicians, employers, policymakers, and 

others make informed decisions about the provision of health care services. This report is

intended as a reference and not as a substitute for clinical judgment. 

This report may be used, in whole or in part, as the basis for the development of clinical 

practice guidelines and other quality enhancement tools, or as a basis for reimbursement and 

coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of 

such derivative products may not be stated or implied. 
ii

This document is in the public domain and may be used and reprinted without permission except 

those copyrighted materials noted for which further reproduction is prohibited without the 

specific permission of copyright holders. 

Suggested Citation: 

Marinopoulos SS, Dorman T, Ratanawongsa N, Wilson LM, Ashar BH, Magaziner JL, Miller 

RG, Thomas PA, Prokopowicz GP, Qayyum R, Bass EB. Effectiveness of Continuing Medical 

Education. Evidence Report/Technology Assessment No. 149 (Prepared by the Johns Hopkins 

Evidence-based Practice Center, under Contract No. 290-02-0018.) AHRQ Publication No. 

07-E006. Rockville, MD: Agency for Healthcare Research and Quality. January 2007.

No investigators have any affiliations or financial involvement (e.g., employment, 

consultancies, honoraria, stock options, expert testimony, grants or patents received or 

pending, or royalties) that conflict with material presented in this report. 
iii

Preface 

 The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-Based 

Practice Centers (EPCs), sponsors the development of evidence reports and technology 

assessments to assist public- and private-sector organizations in their efforts to improve the 

quality of health care in the United States. The reports and assessments provide organizations 

with comprehensive, science-based information on common, costly medical conditions and new 

health care technologies. The EPCs systematically review the relevant scientific literature on 

topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to 

developing their reports and assessments. 

 To bring the broadest range of experts into the development of evidence reports and health 

technology assessments, AHRQ encourages the EPCs to form partnerships and enter into 

collaborations with other medical and research organizations. The EPCs work with these partner 

organizations to ensure that the evidence reports and technology assessments they produce will 

become building blocks for health care quality improvement projects throughout the Nation. The 

reports undergo peer review prior to their release. 

 AHRQ expects that the EPC evidence reports and technology assessments will inform 

individual health plans, providers, and purchasers as well as the health care system as a whole by 

providing important information to help improve health care quality. 

 We welcome comments on this evidence report. They may be sent by mail to the Task Order 

Officer named below at: Agency for Healthcare Research and Quality, 540 Gaither Road, 

Rockville, MD 20850, or by e-mail to epc@ahrq.gov. 

Carolyn M. Clancy, M.D. 

Director 

Agency for Healthcare Research and Quality 

Jean Slutsky, P.A., M.S.P.H. 

Director, Center for Outcomes and Evidence 

Agency for Healthcare Research and Quality 

Beth A. Collins-Sharp, Ph.D., R.N. 

Director, EPC Program 

Agency for Healthcare Research and Quality 

Ernestine Murray, B.S.N., R.N., M.A.S. 

EPC Program Task Order Officer 

Agency for Healthcare Research and Quality 
iv

Acknowledgments 

The Evidence-based Practice Center thanks Karen Robinson for her assistance in developing the 

search strategies; Lori Bash, Gabriel Lai, Rachel Millstein, and Chidinma Ibe for their assistance 

with article reviewing and data entry; Renee Wilson, Brenda Zacharko, and Laura Barnes for 

their assistance with final preparations of the report. 
v 

Structured Abstract 

Objective: Despite the broad range of continuing medical education (CME) offerings aimed at 

educating practicing physicians through the provision of up-to-date clinical information, 

physicians commonly overuse, under-use, and misuse therapeutic and diagnostic interventions. It 

has been suggested that the ineffective nature of CME either accounts for the discrepancy 

between evidence and practice or at a minimum contributes to this gap. Understanding what 

CME tools and techniques are most effective in disseminating and retaining medical knowledge 

is critical to improving CME and thus diminishing the gap between evidence and practice. The 

purpose of this review was to comprehensively and systematically synthesize evidence regarding 

the effectiveness of CME and differing instructional designs in terms of knowledge, attitudes, 

skills, practice behavior, and clinical practice outcomes. 

Methods: We formulated specific questions with input from external experts and representatives 

of the Agency for Healthcare Research and Quality (AHRQ) and the American College of Chest 

Physicians (ACCP) which nominated this topic. We systematically searched the literature using 

specific eligibility criteria, hand searching of selected journals, and electronic databases 

including: MEDLINE®, EMBASE®, the Cochrane Database of Systematic Reviews, The 

Cochrane Central Register of Controlled Trials (CENTRAL), the Cochrane Database of 

Abstracts of Reviews of Effects (DARE), PsycINFO, and the Educational Resource Information 

Center (ERIC®). Two independent reviewers conducted title scans, abstract reviews, and then 

full article reviews to identify eligible articles. Each eligible article underwent double review for 

data abstraction and assessment of study quality. 

Results: Of the 68,000 citations identified by literature searching, 136 articles and 9 systematic 

reviews ultimately met our eligibility criteria. The overall quality of the literature was low and 

consequently firm conclusions were not possible. Despite this, the literature overall supported the 

concept that CME was effective, at least to some degree, in achieving and maintaining the 

objectives studied, including knowledge (22 of 28 studies), attitudes (22 of 26), skills (12 of 15), 

practice behavior (61 of 105), and clinical practice outcomes (14 of 33). Common themes 

included that live media was more effective than print, multimedia was more effective than 

single media interventions, and multiple exposures were more effective than a single exposure. 

The number of articles that addressed internal and/or external characteristics of CME activities 

was too small and the studies too heterogeneous to determine if any of these are crucial for CME 

success. Evidence was limited on the reliability and validity of the tools that have been used to 

assess CME effectiveness. Based on previous reviews, the evidence indicates that simulation 

methods in medical education are effective in the dissemination of psychomotor and procedural 

skills. 

Conclusion: Despite the low quality of the evidence, CME appears to be effective at the 

acquisition and retention of knowledge, attitudes, skills, behaviors and clinical outcomes. More 

research is needed to determine with any degree of certainty which types of media, techniques, 

and exposure volumes as well as what internal and external audience characteristics are 

associated with improvements in outcomes. 

vii 

Contents 

 

Executive Summary.................................................................................................................. 1 

Evidence Report ..................................................................................................................... 9

Chapter 1. Introduction ............................................................................................................ 11 

Chapter 2. Methods.................................................................................................................. 13 

Topic Development............................................................................................................. 13 

Search Strategy ................................................................................................................... 13 

Study Selection ................................................................................................................... 14 

Data Extraction ................................................................................................................... 15 

Data abstracted to assess the effectiveness of CME (Key Questions 1 and 2)............. 16 

Data abstracted from systematic reviews on the effectiveness of simulation in 

medical education (Key Question 3).......................................................................... 19 

Data abstracted to assess the influence of audience characteristics and/or external 

factors on the effectiveness of CME (Key Questions 4 and 5).................................. 19 

Data abstracted to assess the validity and/or reliability of tools (Key Question 6) ...... 20 

Quality Assessment............................................................................................................. 20 

Data Synthesis..................................................................................................................... 20 

Data Entry and Quality Control .......................................................................................... 21 

Rating the Body of Evidence .............................................................................................. 21 

Peer Review and Public Commentary.............................................................22 

Chapter 3. Results .................................................................................................................... 23 

Results of Review of Primary Literature ............................................................................ 23 

Results of Review of Systematic Reviews.......................................................................... 23 

Summary of Study Characteristics and Evaluation Methods ............................................. 23 

Summary of Study Participants and Study Setting............................................................. 23 

Summary of CME Activity Characteristics ........................................................................ 26 

Study Quality of Primary Literature ................................................................................... 27 

Quality of the Systematic Reviews..................................................................................... 27 

Reporting of Adult Learning Principles.............................................................................. 28 

Key Question 1: Is there evidence that particular methods of delivering CME are 

more effective in: a) imparting knowledge to physicians, b) changing physician 

attitudes, c) acquiring skills, d) changing physician practice behavior, or e) 

changing clinical practice outcomes? .............................................................................. 29 

Key Question 2: Do changes in knowledge, attitudes, skills, practice behavior, or 

clinical practice outcomes produced by CME persist over time (greater than or 

equal to 30 days)? ............................................................................................................ 29 

Knowledge Outcomes................................................................................................ 29 

Attitude Outcomes ..................................................................................................... 33 

Skills Outcomes ......................................................................................................... 37 

Practice Behavior Outcomes...................................................................................... 40 

Clinical Outcomes...................................................................................................... 44 
viii 

Key Question 3: What is the evidence from systematic reviews about the 

effectiveness of simulation methods in medical education outside of CME? ................. 46 

Characteristics of the Systematic Reviews ................................................................... 46 

Effectiveness of Simulation in Teaching Procedural Skills.......................................... 47 

Effectiveness of Simulation in Teaching Physical Examination .................................. 48 

Effectiveness of Simulation in Teaching Communication Skills ................................. 48 

Effectiveness of Simulation in Knowledge Acquisition............................................... 48 

Features of High-Fidelity Simulators for Effective Learning....................................... 49 

Summary....................................................................................................................... 49 

Key Question 4: Which characteristics of the audience by themselves or in 

combination with other characteristics influence the effectiveness of certain 

educational techniques? ................................................................................................... 49 

Key Question 5: Which external factors by themselves or in combination with other 

factors reinforce the effects of CME in changing behavior? ........................................... 49 

Audience Characteristics .............................................................................................. 50 

External Factors ............................................................................................................ 51 

Key Question 6: What is the reported validity and reliability of the methods that have 

been used for measuring the effects of CME in terms of: a) imparting knowledge, b) 

changing attitudes, c) acquiring skills, d) changing practice behavior, or e) changing 

clinical practice outcomes?.............................................................................................. 52 

Background................................................................................................................... 52

Results........................................................................................................................... 52 

Conclusions................................................................................................................... 54 

Chapter 4. Discussion .............................................................................................................. 57 

Conclusions................................................................................................................... 57 

Limitations .................................................................................................................... 57 

Future Research Implications ....................................................................................... 59

References................................................................................................................................. 61 

List of Acronyms/Abbreviations............................................................................................... 69

Figures 

Figure 1. Summary of literature search and review process for primary literature (number 

of articles) .............................................................................................................................. 24 

Figure 2. Summary of literature search and review process for systematic reviews 

(number of articles)................................................................................................................ 25 

Tables 

Table 1. Definitions of Media Methods................................................................................... 17 

Table 2. Definitions of Techniques/Educational Methods ...................................................... 18 

Table 3. Definitions of Simulation Types................................................................................ 19 

Table 4. Summary of CME Activity Characteristics............................................................... 26 
ix 

Table 5. Quantity and Direction of Evidence for Effectiveness of Simulation ....................... 47 

Table 6. Internal Audience Characteristics and External Factors Examined in the Review ... 50 

Table 7. Number of Methods Evaluated for Each Type of Validity or Reliability, 

Organized by Outcome Type................................................................................................. 56 

Appendixes 

Appendix A: Technical Experts and Peer Reviewers 

Appendix B: Hand Searched Journals 

Appendix C: Detailed Electronic Database Search Strategies 

Appendix D: List of Excluded Studies 

Appendix E: Data Abstraction Forms 

Appendix F: Evidence Tables 

Appendixes and Evidence Tables for this report are provided electronically at 

http://www.ahrq.gov/downloads/pub/evidence/pdf/cme/cme.pdf.

1

Executive Summary 

Introduction 

Continuing medical education (CME) is defined as educational activities that serve to 

maintain, develop, or increase the knowledge, skills, performance, and relationships a physician 

uses to provide services for patients, the public, or the profession. Despite the broad range of 

CME aimed at educating practicing physicians, researchers have found that physicians 

commonly overuse, under use, and misuse therapeutic and diagnostic interventions. It has been 

suggested that CME may not be effective enough to significantly narrow the gap between what is 

done in clinical practice and what should be done based on current evidence. Understanding 

what CME tools and techniques are most effective in disseminating and retaining medical 

knowledge is critical to improving the effectiveness of CME and thus diminishing the gap 

between evidence and practice. 

To date, relatively little has been done to comprehensively and systematically synthesize 

evidence regarding the effectiveness of CME and the comparative effectiveness of differing 

instructional designs for CME in terms of impact on knowledge, attitudes, skills, practice 

behavior, and clinical practice outcomes. Review of evidence elucidating the value of CME (and 

ways the activities could be improved, if appropriate) could yield tremendous value to policy 

makers and professional organizations seeking to make recommendations regarding the optimal 

delivery of medical care. 

The American College of Chest Physicians (ACCP) recognized the potential value of 

identifying and synthesizing the evidence in this area, and nominated this topic to the Evidencebased Practice Center (EPC) Program of the Agency for Healthcare Research and Quality 

(AHRQ). In response to this request by the ACCP, the Johns Hopkins EPC performed a 

systematic review to address the following key questions (KQ) pertaining to the effectiveness of 

CME: 

KQ1 Is there evidence that particular methods of delivering CME are more effective in: a) 

imparting knowledge to physicians, b) changing physician attitudes, c) acquiring 

skills, d) changing physician practice behavior, or e) changing clinical practice 

outcomes? 

KQ2 Do changes in knowledge, attitudes, skills, practice behavior, or clinical practice 

outcomes produced by CME persist over time (greater than or equal to 30 days)? 

KQ3 What is the evidence from systematic reviews about the effectiveness of 

simulation methods in medical education outside of CME? 

KQ4 Which characteristics of the audience by themselves or in combination with other 

characteristics influence the effectiveness of certain educational techniques? 

KQ5 Which external factors by themselves or in combination with other factors reinforce 

the effects of CME in changing behavior? 

KQ6 What is the reported validity and reliability of the methods that have been used for 

measuring the effects of CME in terms of: a) imparting knowledge, b) changing 

attitudes, c) acquiring skills, d) changing practice behavior, or e) changing clinical 

practice outcomes? 
2

Methods 

To answer these questions, we identified primary literature on the effectiveness of CME and 

systematic reviews on the effectiveness of simulation techniques in medical education by 

running searches through February 2006 of the following databases: MEDLINE®, EMBASE®, 

the Cochrane Database of Systematic Reviews, The Cochrane Central Register of Controlled 

Trials (CENTRAL), the Cochrane Database of Abstracts of Reviews of Effects (DARE), 

PsycINFO©, and the Educational Resource Information Center (ERIC®). Additionally, we 

searched by hand the references of included articles and the table of contents of selected journals 

from February 2005 through February 2006. 

 Two independent reviewers conducted title scans in a parallel fashion. If either reviewer felt 

that a title was potentially eligible, it was promoted to abstract review. The abstract review phase 

was designed to identify studies reporting on the effects of CME or simulation on clinical 

practice in terms of knowledge, attitudes, skills, practice behaviors, or clinical outcomes. 

Abstracts were promoted to full article review if both reviewers agreed the abstract met our 

specific inclusion criteria. 

 Each included article underwent double review by our investigators for data abstraction and 

assessment of study quality. For all articles containing original data, reviewers extracted 

information on study characteristics (e.g., study participants, sample size, study setting, 

evaluation methods, and study design), CME characteristics, and outcomes. We assessed which 

of 16 CME methods were employed (see Table 1 in full report), such as lecture, discussion 

group, academic detailing, etc. – most incorporated more than one method. We also rated 

interventions on the extent to which the CME activity incorporated 9 adult learning principles, 

such as enabling learners to be active contributors to their learning, relating the curriculum to 

learner’s current experiences, etc. The study quality form was based on the Jadad criteria. For 

reviews that applied to KQ 3, reviewers abstracted information on: characteristics of the review; 

outcomes evaluated and type of objective; meta-analyses conducted; summary of results; 

conclusions; and quality of the review. 

We graded the quantity, quality, and consistency of the available evidence addressing KQs 1, 

2, and 3 by adapting an evidence grading scheme recommended by the GRADE Working Group. 

We applied evidence grades to bodies of evidence on each type of objective (i.e., knowledge, 

attitudes, skills, practice behaviors, and clinical outcomes). 

Results 

Key Question 1: Is there evidence that particular methods of 

delivering CME are more effective in: a) imparting knowledge to 

physicians, b) changing physician attitudes, c) acquiring skills, d) 

changing physician practice behavior, or e) changing clinical practice 

outcomes? 
3

Key Question 2: Do changes in knowledge, attitudes, skills, practice 

behavior, or clinical practice outcomes produced by CME persist over 

time (greater than or equal to 30 days)? 

Knowledge Outcomes 

• A total of 39 studies addressed 41 knowledge objectives. Only 28 of those studies had a 

control group. 

• Seventy-eight percent of the 28 studies with an adequate control group demonstrated that 

CME activities were effective at improving knowledge with the majority (68 percent) of 

these studies demonstrating long-term improvements in knowledge. 

• The studies were heterogeneous, making it difficult to determine how results differed 

according to media type, educational technique, or number of exposures. 

• The only recognized trends regarding differences by media type were that combine 

multimedia interventions (e.g., use of live and print media) were better than a single 

media intervention and that print interventions were either not beneficial or very weak in 

their ability to improve knowledge. We defined “media” as the method through which 

the CME activity is delivered. 

• When these studies were reviewed according to educational technique, it appeared that 

multiple techniques that most commonly included case-based learning were more likely 

to improve knowledge when compared to a single technique. Case-based learning is an 

educational technique where actual or authored clinical cases are created to highlight 

learning objectives; clinical material is presented and followed with questions usually 

determined by the instructor. 

• The evidence also suggested that multiple exposures produced better knowledge gains 

than a single exposure to content. Exposure was defined as one session versus more than 

one. An additional session could have used print media, computer media, a repeat live 

performance, or audio tape. 

Attitudinal Outcomes 

• A total of 35 studies addressed 45 attitude objectives. Thirty one of the studies had a 

control group. 

• Seventy-one percent of the 31 studies with an adequate control group demonstrated that 

CME activities were effective at improving attitudes, such as attitudes regarding use of 

screening tests or clinical management options. The majority (68 percent) of these studies 

demonstrated long-term improvements in attitudes. 

• The studies were heterogeneous, again making it difficult to determine how results 

differed according to media type, educational technique, or number of exposures. 

• The only recognized trends regarding differences by media type were that multimedia 

interventions were better than a single media intervention and that print interventions 

were either not beneficial or very weak in their ability to improve attitudes. 

• The only recognized trend regarding differences by educational technique was that use of 

multiple techniques that most commonly include case-based learning seemed to be more 

likely to improve attitudes than use of a single technique. 
4

• The evidence suggested a trend toward multiple exposures being of greater benefit for 

attitudinal change then a single exposure, although it must be pointed out that all seven 

studies that evaluated a single exposure indeed demonstrated improvements in attitudes. 

Skill Outcomes 

• Twelve (80 percent) of the 15 studies that reported skill outcomes involved cognitive 

skills (i.e., ability to apply knowledge), with the remaining three involving psychomotor 

skills (i.e., procedures or physical examination techniques). Little can be said about the 

effectiveness of CME for psychomotor skills given the paucity of data in this area. 

• Seven (47 percent) of the 15 studies reporting skill outcomes had an evaluation beyond 

30 days after the CME activity. Six of seven studies addressed the long-term effect of 

CME on cognitive skills and five of the six demonstrated a positive effect. 

• Given the dominance of live methods (seven live, four print, two video, two audio, three 

Internet/computer) among the studies that met their skill objectives, the data suggested 

that live methods had the greatest impact on the effectiveness of CME regarding skillrelated outcomes. Given the paucity of data and the varied results, little can be said about 

the relative effectiveness of other CME media methods on skills. Based on the limited 

data, it is difficult to draw conclusions on particular media methods that have an impact 

on skill long-term. 

• Given the limited number of studies, the wide variety of techniques described, and the 

conflicting results, it is difficult to draw conclusions about the educational techniques that 

have the greatest short- and long-term effects on skills. 

• Most of the studies that met their skill objectives had multiple exposures to the CME 

activity as did most of the studies that evaluated the long-term effect on skills. 

Practice Behavior Outcomes 

• A total of 105 studies evaluated the impact of CME on short- and long-term physician 

practice behavior objectives, and 61 (58 percent) of the studies met practice objectives. 

Fifty studies with evaluation duration greater than 30 days met objectives, suggesting not 

only short-term, but also long-term CME effectiveness. 

• Most studies that evaluated the impact of different types of CME media found that use of 

single live media had both a short- and long-term effect on practice behavior objectives, 

and that single print media is ineffective. Most of the studies suggested that multimediabased CME have both a short- and long-term effect on practice behavior objectives. 

• Of the studies evaluating the short- and long-term impact of different types of CME 

techniques on practice behavior objectives, most reported mixed results for a single 

technique, and overall effectiveness for use of multiple techniques. The use of multiple 

techniques may be advantageous over the use of a single technique. 

• Most studies evaluating the short- and long-term impact of different volumes of exposure 

to CME on practice behavior objectives suggested that both single and multiple 

exposures are effective overall. 

Clinical Outcomes 

• Thirty-seven studies evaluated the impact of CME on clinical outcomes. 
5

• Only one of the 37 studies measured short-term clinical outcomes and it suggested an 

inconclusive effect. Three of the studies did not report the time at which clinical 

outcomes were measured. 

• Of the 33 studies that measured long-term clinical outcomes, 14 (42 percent) were 

successful in demonstrating a beneficial effect of CME. 

• When evaluating the impact of different types of CME media on long-term clinical 

outcomes, the use of multiple media was more effective than use of single media in six of 

the seven studies that made this comparison. 

• Of the five studies that compared clinical outcomes with use of single versus multiple 

educational techniques in CME, three showed that multiple techniques were superior to a 

single technique. 

• In four (57 percent) of the seven studies that evaluated the impact of a single CME 

exposure on clinical outcomes, the CME objective was met. However, insufficient data 

were available on whether multiple CME exposures produce better clinical outcomes 

than single exposure CME. 

Key Question 3: What is the evidence from systematic 

reviews about the effectiveness of simulation methods in 

medical education outside of CME? 

• Eight reviews evaluated the role of simulation in skill acquisition, while two reviews 

evaluated the role of simulators in knowledge acquisition. Simulation methods used in 

these reviews included computer simulations (e.g., learning cardiovascular physiology), 

virtual reality (e.g., learning laparoscopic procedures), standardized patients (e.g., 

learning effective communication), and manikins (e.g., learning physical diagnosis). 

• Overall, the direction of evidence pointed to the effectiveness of simulation training, 

especially in psychomotor skills (i.e., procedures or physical examination techniques) and 

communication skills. 

• The strength of the evidence was considered low, due to the small number of appropriate 

studies, the scarcity of quantitative data, and a number of study limitations. 

Key Question 4: Which characteristics of the audience by themselves 

or in combination with other characteristics influence the 

effectiveness of certain educational techniques? 

• Thirteen studies examined the influence of audience characteristics on the educational 

intervention. These included such characteristics as age, gender, practice setting, and 

years in practice, among others. 

• The small and heterogeneous studies available did not allow us to reach definitive 

conclusions regarding the influence of audience characteristics on the effectiveness of 

CME. 
6

Key Question 5: Which external factors by themselves or in 

combination with other factors reinforce the effects of CME in 

changing behavior? 

• Five studies examined the influence of external factors on the educational intervention. 

The small and heterogeneous studies available did not allow us to reach definitive 

conclusions regarding the influence of external factors on the effectiveness of CME. 

Key Question 6: What is the reported validity and reliability of the 

methods that have been used for measuring the effects of CME in 

terms of: a) imparting knowledge, b) changing attitudes, c) acquiring 

skills, d) changing practice behavior, or e) changing clinical practice 

outcomes? 

• Forty-five of 136 articles (33 percent) reported the validity and/or reliability within the 

study population of at least one evaluation method for assessing the effectiveness of 

CME, for a total of 61 methods. Validity refers to the degree to which a method truly 

measures what it is intended to measure. Reliability refers to the consistency or 

reproducibility of measurements.

• Of the 61 evaluation methods with validity or reliability reported, 29 evaluation methods 

were drawn from previous studies, and 24 were created for the current studies. For eight 

methods, the source was unclear. Authors frequently did not report reliability testing 

within the new study population for methods found to be reliable in other populations.

• Of 61 evaluation methods with validity or reliability reported, 16 (26 percent) included 

descriptions of validity alone, 28 (46 percent) included descriptions of reliability alone, 

and nine (15 percent) had descriptions of both validity and reliability. For six methods 

(10 percent), the methods were described as valid and/or reliable, but the specific type of 

validity or reliability was not reported.

• The most common type of outcome evaluated by valid and/or reliable evaluation methods 

involved practice behaviors, for 31 out of 61 methods (51 percent). Knowledge or 

cognitive skills were evaluated by 15 methods (24 percent). Attitudes were evaluated by 

seven methods (12 percent). Skills were evaluated by 11 methods (18 percent). Clinical

outcomes (with or without practice behaviors) were evaluated by 8 methods (13 percent).

• Among these 61 methods, content validity (i.e., the degree to which an instrument 

accurately represents the skill or characteristic it was designed to measure) was the most 

commonly reported type of validity (27 percent).

• Among these 61 methods, inter-rater reliability (28 percent) and internal consistency (23 

percent) were the most common types of reliability reported. Inter-rater reliability is the 

degree to which measurements are the same when obtained by different persons. Internal 

consistency is a measure of how well items reflecting the same construct yield similar 

results.
7

Discussion 

Conclusions 

Overall, despite the generally low quality of the evidence, most of the studies reviewed 

suggested that CME is effective, at least to some degree, in not only achieving, but also in 

maintaining the objectives studied. Despite the wide variety of CME techniques, media, and 

exposures used, and the heterogeneity of the studies, we found common themes among studies 

which applied across objectives. For example, when assessing the effectiveness of CME across 

domains, print media seemed to be less effective than live media, and multimedia activities 

generally seemed more effective than single media. In addition, interactive techniques seem to 

be more effective than non-interactive ones, and multiple exposures to the CME activity seem to 

be more effective than a single exposure. Thus, the evidence supports consideration of these 

attributes of effective educational interventions when designing a CME course. 

We evaluated the effect of simulation methods in medical education by conducting a review 

of systematic reviews. Although we found that simulation training generally was effective, 

especially in the dissemination of psychomotor skills (e.g., procedures or physical examination 

techniques), studies which examined simulation did not review outcomes along the entire 

continuum of domains (i.e., knowledge through clinical outcomes), and were heterogeneous 

enough that few other conclusions could be drawn. 

 We also studied whether certain internal (audience) and external characteristics or factors 

may affect the effectiveness of CME. We found that the small and heterogeneous studies 

available did not allow us to reach definitive conclusions regarding the influence of audience 

characteristics or external factors on the effectiveness of CME. 

Limitations 

This review has several important limitations. The heterogeneous nature of the studies 

precludes a quantitative summary of the effectiveness of CME. The educational interventions 

studied targeted different types of audiences, using multiple types of objectives across diverse 

content areas. This makes it difficult to generalize results from one field of CME to another. 

 Furthermore, we cannot draw firm conclusions about the effectiveness of CME because of 

the generally low quality of study designs, the variable quality of reporting in studies, and the 

lack of valid and reliable CME evaluation tools. Although we used a comprehensive search 

strategy, we cannot rule out some degree of publication bias. The review does point out a lack of 

standardization of approaches to CME research in general, including the lack of standardization 

for definitions of controls. The CME literature in general lacks standardization of terminology 

related to media type, educational techniques, and exposure volume, which makes it difficult to 

determine the impact of these factors on the effectiveness of CME. 

Future Research Implications 

• Educators should develop strategies for identifying and prioritizing the gaps in our 

knowledge about CME that should be the focus of further research. 
8

• More randomized controlled studies of CME should be performed with clear definition of 

intervention and control groups and measurement of effectiveness at multiple points postintervention. Such studies should focus on high priority areas given the resource 

limitations that educators face in conducting research on CME. 

• To advance such research on CME, leaders in medical education could develop a national 

agenda on what is needed most to improve the effectiveness of CME. 

• In developing a national agenda for research on CME, educational leaders should 

establish a clear definition of what constitutes CME. For example, does quality 

improvement or practice improvement alone constitute CME? 

• Future research on CME should include development of more standardized approaches to 

describe CME interventions, media, techniques, and exposure volumes. 

• Further studies should examine emerging methods of CME such as Internet-based CME 

that could be available to clinicians at the point of care. 

• Future research on CME should be based on a sound conceptual model of what 

influences the effectiveness of CME. 
Evidence Report 

11 

Chapter 1. Introduction 

Continuing medical education (CME) was initiated by the American Academy of General 

Practice, which has required CME for membership since 1947.1

 The American Medical 

Association (AMA) presently defines continuing medical education as “educational activities 

that serve to maintain, develop, or increase the knowledge, skills, and professional performance 

and relationships a physician uses to provide services for patients, the public, or the profession.”2

The AMA defines the content of CME as “that body of knowledge and skills generally 

recognized and accepted by the profession as within the basic medical sciences, the discipline of 

clinical medicine, and the provision of health care to the public.”2

 In 1971 New Mexico became 

the first state to require CME credit for relicensure. Physicians commonly spend an average of 

50 hours per year in CME activities geared toward improving their performance and optimizing 

their care of patients. Participation in CME is not voluntary as, in 41 states, it is a requirement 

for continued physician licensure and is often a requirement for hospital credentials and 

participation in many managed care plans. This has led to the proliferation of CME offerings 

aimed at providing physicians with the CME credits they require. 

Despite the broad range of CME offerings aimed at educating the nation’s practicing 

physicians through the provision of up-to-date clinical information, researchers have found that 

physicians commonly overuse, underuse, and misuse therapeutic and diagnostic interventions.3

Some medical educators have suggested that CME may not be effective enough to significantly 

narrow the gap between what is done in clinical practice and what should be done based on 

current evidence. Understanding what CME tools and techniques are most effective in 

disseminating and retaining medical knowledge is critical to improving the effectiveness of CME 

and thus diminishing the gap between evidence and practice. 

To date, relatively little has been done to comprehensively and systematically synthesize 

evidence regarding the effectiveness of CME and the comparative effectiveness of differing 

instructional designs for CME and their impact on knowledge, attitudes, skills, practice behavior, 

and clinical practice outcomes. Review of evidence elucidating the value of CME (and ways the 

activities could be improved, if appropriate) could yield tremendous value to policy makers and 

professional organizations seeking to make policy recommendations regarding the optimal 

delivery of medical care. Such a review could have particular significance in light of the 

American Board of Medical Specialties (ABMS) recently initiated robust Maintenance of 

Certification (MOC) program that requires evidence of self-directed learning which is commonly 

accomplished through attendance at CME activities.4

 In addition, as the National Institutes of 

Health (NIH) and the Agency for Healthcare Research and Quality (AHRQ) place more 

emphasis on translating scientific knowledge into clinical practice,5

 educators will need to apply 

evidence-based approaches to CME that will support translating knowledge into practice. 

Furthermore, the published literature demonstrates a large gap between recommended processes 

of care and those delivered.6

 Effective CME is required to help fill that gap. 

The American College of Chest Physicians (ACCP) recognized the potential value of 

identifying and synthesizing the evidence in this area, and nominated this topic to the Evidencebased Practice Center Program (EPC) of AHRQ. In response to this request by the ACCP, the 

Johns Hopkins EPC performed a systematic review to address the following key questions 

pertaining to the effectiveness of CME: 
12 

1. Is there evidence that particular methods of delivering CME are more effective in: a) 

imparting knowledge to physicians, b) changing physician attitudes, c) acquiring skills, 

d) changing physician practice behavior, or e) changing clinical practice outcomes? 

2. Do changes in knowledge, attitudes, skills, practice behavior, or clinical practice 

outcomes produced by CME persist over time (greater than or equal to 30 days)? 

3. What is the evidence from systematic reviews about the effectiveness of simulation 

methods in medical education outside of CME? 

4. Which characteristics of the audience by themselves or in combination with other 

characteristics influence the effectiveness of certain educational techniques? 

5. Which external factors by themselves or in combination with other factors reinforce the 

effects of CME in changing behavior? 

6. What is the reported validity and reliability of the methods that have been used for 

measuring the effects of CME in terms of a) imparting knowledge, b) changing attitudes, 

c) acquiring skills, d) changing practice behavior, or e) changing clinical practice 

outcomes?
13 

Chapter 2. Methods 

 The ACCP requested an evidence report to review and synthesize published literature 

regarding the effectiveness of CME, the comparative effectiveness of instructional designs for 

CME, and their impact on imparting knowledge, changing attitudes, acquiring skills, changing 

practice behavior, and changing clinical practice outcomes. Our EPC established a team and a 

work plan to develop the evidence report. The project consisted of recruiting technical experts, 

formulating and refining the specific questions, performing a comprehensive literature search, 

summarizing the state of the literature, constructing evidence tables, synthesizing the evidence, 

and submitting the report for peer review. 

Topic Development 

 The topic for this report was nominated in a public process. At the beginning of the project, 

we recruited a panel of external technical experts to give input on key steps including the 

selection and refinement of the questions to be examined. The panel included external experts 

who have strong expertise in CME (see Appendix Aa).

 We worked with the technical experts and representatives of AHRQ and ACCP to develop 

the Key Questions that are presented in the Scope and Key Questions section of Chapter 1 

(Introduction). Based on the feedback from the technical experts, AHRQ, ACCP, and our team 

members, we expanded the preliminary questions to include knowledge, attitude, skills, practice 

behavior, and clinical outcomes and to address potential synergies between learning methods. 

We refined Key Question 3 to focus on the effectiveness of simulation methods used in medical 

education. Additionally, we added Key Question 6, which assesses the validity and reliability of 

tools used to evaluate the effectiveness of CME. The Key Questions focus on the effectiveness of 

CME in (1) imparting knowledge, (2) changing attitudes, (3) acquiring skills, (4) changing 

practice behaviors, or (5) changing clinical practice outcomes. We considered any test of 

physician or CME participant knowledge as knowledge. Attitudes were limited to physician or 

CME participant attitudes; attitudes could include physician attitudes toward a medical topic, 

physician comfort level, or satisfaction with the course. Skills were divided into cognitive skills 

(ability to apply knowledge) and psychomotor skills (e.g., procedures or physical examination 

techniques). Practice behavior referred to any type of physician behavior. We defined clinical 

outcomes as any change in patient health status, health-related behavior of patients, or attitudes 

of the patients about the physicians toward whom the CME intervention was directed. Thus, in 

addition to direct measures of health status such as blood pressure and fasting blood glucose, we 

also included indirect measures such as patient satisfaction, medication adherence, and smoking 

cessation. 

Search Strategy 

 Searching the literature included the steps of identifying reference sources, formulating a 

search strategy for each source, and executing and documenting each search. Additionally, we 

 

a Appendixes cited in this report are provided electronically at http://www.ahrq.gov/clinic/tp/cmetp.htm.
14 

searched for medical subject heading (MeSH) terms that were relevant to CME. We used a 

systematic approach for searching the literature, with specific eligibility criteria, to minimize the 

risk of bias in selecting articles for inclusion in the review. The systematic approach was 

intended to help identify gaps in the published literature. 

Our comprehensive search plan included electronic and hand searching. Beginning in 

February of 2006 we ran searches of the following databases: MEDLINE®, EMBASE®, the 

Cochrane Database of Systematic Reviews, The Cochrane Central Register of Controlled Trials 

(CENTRAL), the Cochrane Database of Abstracts of Reviews of Effects (DARE), PsycINFO©, 

and the Educational Resource Information Center (ERIC®), to identify primary literature on the 

effectiveness of CME and systematic reviews on the effectiveness of simulation techniques in 

medical education. 

Hand searching for possibly relevant citations took several forms. From our electronic 

search, we identified the 13 journals (see Appendix Ba

) that were most likely to publish articles 

on this topic (i.e., these journals had the highest number of abstracts and articles included in the 

review). We scanned the table of contents of each issue of these journals for relevant citations 

from February 2005 through February 2006. For the second form of hand searching, reviewers 

received eligible articles and flagged references of interest for the team to compare to the 

existing database. 

 Search strategies, specific to each database, were designed to enable the team to focus 

available resources on articles most likely to be relevant to the Key Questions. Initially, we 

developed a core strategy for MEDLINE®, accessed via PubMed®, based on an analysis of the 

MeSH terms and text words of key articles identified a priori. The PubMed® strategy formed the 

basis for the strategies developed for the other electronic databases (see Appendix Ca

). 

 The results of the searches were downloaded and imported into ProCite® version 5 (ISI 

ResearchSoft, Carlsbad, CA). We used the duplication scan feature in ProCite® to delete citations 

already retrieved. From ProCite®, the articles were uploaded to SRS 3.0 (TrialStat! Corporation, 

Ottawa, Ontario, Canada), a Web-based software package developed for systematic review data 

management. Additionally, this database was used to store citations in portable document format 

(PDF) and to track the search results at title review, abstract review, article inclusion/exclusion, 

and data abstraction levels. A list of excluded articles is presented in Appendix Da. 

Study Selection 

 After the electronic databases were searched, citations were downloaded into ProCite®, and 

uploaded to the SRS 3.0 tracking system. The study team scanned all titles. Two independent 

reviewers conducted title scans in a parallel fashion. For a title to be eliminated at this level, both 

reviewers had to indicate that it was ineligible. If the two reviewers did not agree on the 

eligibility of an article, it was promoted to the next level (see Appendix Ea, Title Review Form). 

The title review phase was designed to capture as many studies as possible reporting on the 

effectiveness of CME or as many systematic reviews reporting on the effectiveness of simulation 

in medical education. All titles that were thought to address the above effectiveness issues were 

promoted to the abstract review phase. 

 The abstract review phase was designed to identify studies reporting on the effects of CME 

or simulation in medical education on clinical practice in terms of knowledge, attitudes, skills, 

 

a

 Appendixes cited in this report are provided electronically at http://www.ahrq.gov/clinic/tp/cmetp.htm.
15 

practice behaviors, or clinical outcomes. All articles with abstracts meeting these criteria were 

kept for further review. Abstracts were reviewed independently by two investigators. Abstracts 

concerning the effectiveness of CME were excluded if both investigators agreed that the article 

met one or more of the following exclusion criteria: (1) not written in English; (2) contained no 

human data; (3) no original data and did not apply to Key Question 3; (4) was a meeting abstract, 

editorial, commentary, or letter; (5) did not include at least 15 fully trained physicians or less 

than 50 percent of the CME participants were fully trained physicians and there was not a 

separate analysis for fully trained physicians; (6) did not include training or education; (7) did 

not evaluate an educational activity; (8) published prior to 1981; (9) not conducted in the United 

States or Canada; (10) did not apply to a Key Question; (11) did not include data from a 

concurrent or historical comparison group; or (12) involved quality improvement without an 

educational activity. Since CME accreditation from the Accreditation Council for Continuing 

Medication Education began in 1981, we decided to limit our review to studies published after 

that date. We decided to exclude studies not conducted in the United States or Canada because 

we felt that the medical education systems in other countries could be very different from and not 

relevant to CME in the United States. To qualify for Key Question 6, an abstract must also 

address at least one of the other key questions. 

Abstracts concerning the effectiveness of simulation in medical education were excluded if 

both investigators agreed that the article met one or more of the following exclusion criteria: (1) 

not written in English; (2) contained no human data; (3) was not a systematic review (i.e., 

identified a question, described a search strategy, described eligibility criteria, and synthesized 

results either quantitatively or qualitatively); (4) was a meeting abstract, editorial, commentary, 

or letter; (5) did not include medical students or physicians-in-training; (6) did not include 

medical training or education; (7) did not evaluate an educational activity; (8) did not involve 

simulation, virtual reality, manikins, or standardized patients; (9) published prior to 1990; (10) 

did not apply to Key Question 3; (11) included only fully trained physicians or CME; or (12) did 

not report separately on the effects of simulation. The decision to limit our review to reviews 

published after 1990 was based partly on when simulation began to be used in medical education 

and partly on a desire to focus on reviews that are not too out-of-date. The cut-off date for 

inclusion in this review was February 2006. Differences in opinions regarding abstract inclusion 

or exclusion were resolved through consensus adjudication. At this level of inclusion/exclusion, 

the reviewers were also asked to identify which Key Questions the article might apply to if the 

article was eligible. 

 Because of the broad array of potentially eligible articles obtained at the abstract review 

phase, full articles initially selected for review underwent another independent parallel review by 

investigators to determine if they should be included for full data abstraction. At this phase of the 

review, investigators determined which of the Key Questions each article addressed (see 

Appendix E, Article Inclusion/Exclusion Form). If articles were still deemed to have applicable 

information, they were included in the final article review. Differences in opinions regarding 

article inclusion or exclusion were resolved through consensus adjudication. 

Data Extraction 

 The purpose of the article review was to confirm the relevance of each article to the research 

questions, to determine methodological characteristics pertaining to study quality, and to collect 
16 

evidence that addressed the research questions. Articles eligible for full review could address one 

or more of the Key Questions. We used standardized forms for data extraction to minimize the 

risk of bias in how data were extracted from eligible studies and to maximize consistency in 

identifying all pertinent data available for synthesis. Additionally, we developed definitions and 

created examples, which were reviewed by the study team, to enhance the consistency of data 

extraction. 

 Each article underwent double review by study investigators for full data abstraction and 

assessment of study quality. For all data abstracted from studies, we used a sequential review 

process. In this process, the primary reviewer completed all data abstraction forms. The second 

reviewer confirmed the first reviewer’s data abstraction forms for completeness and accuracy. 

Reviewer pairs were formed to include personnel with both clinical and methodological 

expertise. A third reviewer re-reviewed a random sample of articles by the first two reviewers to 

ensure consistency in the classification of the articles. Reviewers were not masked to the articles’ 

authors, institution, or journal.7

 In most instances, data were directly abstracted from the article. 

If possible, relevant data were also abstracted from figures. Differences in opinion were resolved 

through consensus adjudication. For assessments of study quality and the reporting of adult 

learning principles, each reviewer independently judged study quality and reporting and rated 

items on quality assessment and adult learning principles forms (see Appendix E, Data 

Abstraction Review Forms). The second reviewer provided the assessment of study quality and 

reporting of adult learning principles used in this report. 

 All information from the article review process was entered into the SRS 3.0 database by the 

individual completing the review. Reviewers entered comments into the system whenever 

applicable. The SRS 3.0 database was used to maintain and clean the data, as well as to create 

detailed evidence tables and summary tables (see Appendix F and Summary Tables). 

Data Abstracted to Assess the Effectiveness of CME (Key Questions 1 

and 2) 

 For all articles containing original data, reviewers extracted information on general study 

characteristics, CME activity characteristics, and outcomes. The general study characteristics 

abstracted included study participants, sample size, study setting, evaluation methods, and study 

design. Data abstracted to the CME activity forms included: intervention or control; number 

assigned and number analyzed per group; description of group; timing of the educational 

exposure (i.e., single or multiple exposures); media utilized (i.e., the method through which the 

CME activity is delivered; see Table 1 for definitions); technique or educational method used 

(see Table 2 for definitions); whether the intervention was designed for individuals or practice 

settings; average time spent in the CME activity; average duration of the educational 

intervention; the number of days/week exposed to the CME activity; length of time from the end 

of the CME activity to the last evaluation period; whether the CME activity was accredited; and 

whether the CME activity was a part of a quality improvement project. Reviewers used the 

outcomes form to abstract data regarding: the main outcome measure; the type of objective 

(knowledge, attitudes, skills, practice behaviors, and clinical outcomes); whether the learning 

objectives were met; a qualitative summary of the results; and the author's overall conclusions. 

For whether learning objectives were met, reviewers were instructed to mark "yes" if most of the 

measures showed improvement, "no" if most of the measures showed a lack of improvement, 
17 

"mixed results" if the results were mixed, "no control group" if there was not an appropriate 

control group to answer the question appropriately and "unclear" if the results were unclear. 

 Additionally, each reviewer independently completed a study quality form and a reporting of 

adult learning principles form. The study quality form was based on the Jadad criteria8

 to assess 

randomization, blinding, and withdrawals, and included additional questions regarding power 

calculations. On the reporting of adult learning principles form, reviewers assessed how well the 

article reported the extent to which the CME activity incorporated adult learning principles, such 

as enabling learners to be active contributors to their learning, relating the curriculum to learners' 

current experiences, and tailoring the curriculum to learners' past experiences (Romsai 

Boonyasai, personal communication). The questions on the reporting of adult learning principles 

form was derived from a review of adult education.9

Table 1. Definitions of media methods 

Media method Definition 

Live Any CME activity that is conducted in-person 

Computer-based, offline Any CME activity that is conducted on the computer, but is not conveyed through the 

Internet (e.g., CD-ROM) 

Internet, real-time (e.g., 

streaming) 

Any CME activity that is conducted real-time via the Internet 

Internet, not real-time Any CME activity that is conducted via the Internet, but is not conducted in real-time 

Video Any CME activity that uses a videotape to convey its message 

Audio Any CME activity that uses an audiotape to convey its message 

Handheld Any CME activity that involves handheld materials (e.g., laminated card) 

Print Any CME activity that is conducted via educational printed materials or readings 
18 

Table 2. Definitions of techniques/educational methods 

Technique/Educational 

method Definition 

Academic detailing Detailing provided by an institution or hospital 

Audience response 

systems 

Addresses knowledge objectives. Used in combination with live lectures or discussion 

groups, these are computerized feedback tools that allow the teacher/instructor to pose a 

question to a large group and receive immediate feedback from each learner which is 

collated and presented on a screen. Instructor may choose to alter content based on 

audience response 

Case-based learning Addresses higher order knowledge and skill objectives. Actual or authored clinical cases 

are created to highlight learning objectives; clinical material is presented and followed with 

questions usually determined by the instructor 

Clinical experiences Addresses skill, knowledge and attitudinal objectives. Generally refers to a preceptorship 

or observership with an expert, as in attending a specialty clinic or operating room 

Demonstration Addresses skill and or knowledge (knows how) objectives; can be presented live, or with 

video or audio. Teacher determines amount and pace of content 

Discussion group Addresses knowledge, especially application or higher order knowledge, or affective 

objectives; usually requires preparation with readings, or another experience, such as 

viewing a videotape, or a role play. Can be facilitated by instructor, but group often 

determines content 

Feedback The provision of information about an individual's performance to learners 

Lecture Presentation of knowledge content; live, video, audio or slide presentation available 

online. Teacher/instructor determines amount and pace of content 

Mentor/Preceptor Addresses higher order cognitive, skill and affective objectives. Learner is paired with a 

mentor who may observe, review documentation of performance, advise, coach, and 

facilitate learning 

Point of care Addresses knowledge and higher order cognitive objectives (decision-making). 

Information which is provided at the time of clinical need, integrated into chart or electronic 

medical record 

Problem-based learning 

or team-based learning 

Addresses higher order knowledge objectives, metacognition and some skill (group work) 

objectives. A clinical scenario is presented to a team, who identify the learning objectives, 

assign information-seeking tasks, and return to share information and answer questions 

about the case. Can be facilitated or non-facilitated 

Programmed learning Addresses knowledge objectives. Content is delivered in sequential steps, which are 

tested with the learner, before moving to the next, usually more complicated step. Pace is 

determined by the learner, but objectives are set by the program (teacher). Can be 

delivered in text or online 

Readings Presentation of knowledge content or background for attitudinal objectives. Requires 

learner to complete; can be done at learner’s pace. Teacher/instructor directed or selfdirected (e.g., journals, newsletters, searching online) 

Role play Addresses skill, knowledge and affective objectives. Learners assume role of patients 

and/ or clinicians in practicing focused encounters around training problems, usually when 

standardized patients are unavailable. Encounter may be recorded and reviewed or 

followed with a discussion group. Rarely used as sole method of education 

Simulation (other than 

standardized patient or 

role-play) 

Addresses knowledge and skill objectives; ability to simulate potentially addresses higher 

order integrative objectives, such as responding to an emerging clinical situation, 

understanding the unfolding of a protein structure, working in teams. Technology can be 

used for simulation training of procedures, as in endoscopy virtual reality trainers or 

anesthesia simulators. Includes also models, such as joint injection and suture. Requires 

active participation of learner; can use multiple learners in some scenarios 

Standardized patient Addresses skill and some knowledge and affective objectives. Usually used for 

communication skills training and assessment, the standardized patient or simulated 

patient is trained in a specific patient scenario and presentation of a clinical problem. 

Encounter may be audio or videotaped and timed. Review offers opportunity for reflection 

and “replay” of the scenario 

Writing/Authoring Addresses knowledge and affective objectives. Can include authoring test items and 

participation in test development. Journaling is used frequently for affective objectives, 

and may be followed with discussion groups or review with a mentor 
19 

Data Abstracted from Systematic Reviews on the Effectiveness of 

Simulation in Medical Education (Key Question 3) 

 We abstracted data from systematic reviews that evaluated the effectiveness of simulation in 

medical education (see Appendix E, Data Abstraction Review Forms). Data from systematic 

review articles were abstracted regarding the types of simulation and comparisons included in 

the review; types of healthcare professionals included in the review; exclusion criteria; search 

strategies (types of searches and end date of search); number of articles in the review; outcomes 

evaluated and the type of objective; meta-analyses conducted; summary of results; subgroup 

analyses, sensitivity analyses, and meta-regressions conducted; and overall conclusions. 

Definitions for the types of simulations are presented in Table 3 (Elizabeth Hunt, MD, Director, 

JHH Simulation Center, personal communication). 

 Each reviewer also independently completed a study quality form. The study quality form 

was based off of the QUORUM statement10 and assessed the reporting of the study question, 

search methods, inclusion/exclusion criteria, analysis, quality assessment, and conclusions. 

Additional questions regarding assessment of publication bias were included. 

Table 3. Definitions of simulation types 

Simulation type Definition 

Full simulation Whole room or whole patient simulations 

Partial task simulation The use of products to learn or practice a specific skill, such as intubation heads, central 

venous line chests, intraosseous line legs, or umbilical artery cannulation trainers 

Computer simulation The use of computer programs that allow the student to practice decision making skills, 

specific knowledge sets such as Advanced Cardiac Life Support (ACLS) trainers and 

trauma management trainers 

Virtual reality The use of advanced computerized technology to allow students to learn or practice how 

to perform cardiac catheterizations, colonoscopies, bronchoscopies, ureteroscopies, 

laparoscopic surgery, hysteroscopy, arthroscopy, ocular surgery, intravenous line 

placement, etc. 

Standardized patient The use of individuals trained to play the roles of patients, family members, or others to 

allow students to practice physical exam skills, history taking skills, communication skills, 

etc. 

Role play Participants play roles of patients, family members, or others to allow practice of 

communication skills, etc. 

Data Abstracted to Assess the Influence of Audience Characteristics 

and/or External Factors on the Effectiveness of CME (Key Questions 4 

and 5) 

 For studies addressing the influence of audience characteristics and/or external factors (Key 

Questions 4 and 5), an audience characteristics/external factors form (see Appendix E, Data 

Abstraction Review Forms) was completed. Data abstracted to this form included the audience 

characteristic or external factor that was being analyzed, whether a primary goal of the study was 

to assess the effects of this audience characteristic or external factor, any covariates used in the 

analysis, and a qualitative summary of the results. Additionally, reviewers abstracted data 
20 

regarding general study characteristics, CME activity characteristics, outcomes, study quality, 

and the reporting of adult learning principles. 

Data Abstracted to Assess the Validity and/or Reliability of Tools (Key 

Question 6) 

 Data regarding validity and reliability of methods used to assess the effectiveness of CME 

(Key Question 6) were abstracted to a validity/reliability of tools form (see Appendix E, Data 

Abstraction Review Forms). Articles need not have used the specific terms “validity” or 

“reliability” to be included in Key Question 6. If authors did not label the specific type of 

validity or reliability reported, we classified the type based on the definitions from Reed et al.11

Articles that used a previously validated/reliable method were included if the authors described 

the method as valid/reliable or described a process or statistic used for psychometric testing. 

Reviewers also abstracted data regarding general study characteristics, CME activity 

characteristics, outcomes, study quality, and the reporting of adult learning principles. 

Quality Assessment 

Article quality was assessed differently for clinical trials and systematic reviews. The dual, 

independent review of article quality judged articles on several aspects of each study type’s 

internal validity. Quality assessment of trials was based on the Jadad criteria8

 and included: (1) 

appropriateness of the randomization scheme, (2) appropriateness of the blinding, and (3) 

description of withdrawals and drop-outs. For each trial, we created a score between 5 (high 

quality) and 0 (low quality). Two questions regarding power calculations were added to this 

form, however, the answers to these questions did not factor into the quality score. 

The quality of each systematic review was assessed using criteria based off the QUORUM 

statement10: (1) whether the question being addressed by the review was clearly stated; (2) 

comprehensiveness of search methods used and described in the report; (3) whether 

inclusion/exclusion criteria were clearly defined and appropriate; (4) whether analyses were 

conducted to measure variability in efficacy; (5) whether study quality was assessed and done 

appropriately (using validated instruments); (6) whether differences in how outcomes were 

reported and analyzed across studies were taken into consideration; (7) whether the study 

methodology was reproducible; and (8) whether conclusions were supported by the data 

presented. Additional questions regarding assessment of publication bias were included. 

Data Synthesis 

 For each Key Question, we created a set of detailed evidence tables containing all 

information extracted from eligible studies. The investigators reviewed the tables and eliminated 

items that were rarely reported. For Key Questions 1 and 2, the results were categorized and 

sorted based on the media method used, educational technique used, and the amount of exposure. 

Media methods were categorized into single print media (i.e., the CME activity used only print 

methods), single live media, single Internet media, other single media, multiple media (i.e., the 

CME activity used more than 1 media methods), and single vs. multiple media (i.e., the CME 

activity for one group used only 1 media method and the CME activity for the another group 
21 

used more than 1 media methods). Educational techniques were categorized into single 

educational techniques (i.e., the CME activity used only 1 educational technique), multiple 

educational techniques (i.e., the CME activity used more than 1 educational techniques), single 

vs. multiple, and other/not reported. The amount of exposure was categorized into single 

exposure (i.e., the CME participants were exposed to the activity on only 1 occasion), multiple 

exposures (i.e., the CME participants were exposed to the activity on multiple occasions), single 

vs. multiple exposures, and other/not reported. Investigators used the resulting versions of the 

evidence tables to prepare the text of the report and selected summary tables. 

 For Key Question 6, the data were grouped according to similar evaluation methods to 

facilitate evaluating validity and reliability of these methods. 

Data Entry and Quality Control 

 Initial data were abstracted by investigators and entered directly into Web-based data 

collection forms using SRS® 3.0 (TrialStat! Corporation, Ottawa, Ontario, Canada). After a 

second reviewer reviewed data, adjudicated data were re-entered into the Web-based data 

collection forms by the second reviewer. Second reviewers were generally more experienced 

members of the research team, and one of their main priorities was to check the quality and 

consistency of the first reviewers’ answers. In addition to the second reviewers checking the 

consistency and accuracy of the first reviewers, a lead investigator examined a random sample of 

the reviews to identify problems with the data abstraction. If problems were recognized in a 

reviewer’s data abstraction, the problems were discussed at a meeting with the reviewers. In 

addition, research assistants used a system of random data checks to assure data abstraction 

accuracy. 

Rating the Body of Evidence 

 At the completion of our review, we graded the quantity, quality and consistency of the best 

available evidence addressing Key Questions 1, 2, and 3 by adapting an evidence grading 

scheme recommended by the GRADE Working Group.12 We applied evidence grades to bodies 

of evidence on each type of objective (i.e., knowledge, attitudes, skills, practice behaviors, and 

clinical outcomes). We assessed the strength of the study designs with randomized controlled 

trials considered best, followed by non-randomized controlled trials, and observational studies. 

To assess the quantity of evidence, we focused on the number of studies with the strongest 

design. We also assessed the quality and consistency of the best available evidence, including 

assessment of limitations to individual study quality (using individual quality scores), certainty 

regarding the directness of the observed effects in studies, precision and strength of findings, and 

availability (or lack thereof) of data to answer the Key Question. We classified evidence bodies 

pertaining to Key Questions 1, 2, and 3 into four basic categories: (1) “high” grade (indicating 

confidence that further research is very unlikely to change our confidence in the estimated effect 

in the abstracted literature); (2) “moderate” grade (indicating that further research is likely to 

have an important impact on our confidence in the estimates of effects and may change the 

estimates in the abstracted literature); (3) “low” grade (indicating further research is very likely 

to have an important impact on confidence in the estimates of effects and is likely to change the 

estimates in the abstracted literature); and (4) “very low” grade (indicating any estimate of effect 
22 

is very uncertain). We did not grade the body of evidence for Key Questions 4 and 5 since this is 

a subset of Key Questions 1 and 2. Also, we did not grade the body of evidence for Key 

Question 6 since the grading criteria do not apply to our questions about the validity and 

reliability of educational assessment methods. 

Peer Review and Public Commentary 

A draft of the completed report was sent to the technical experts and peer reviewers, as well 

as to the representatives of AHRQ and the Scientific Resource Center. In response to the 

comments of the technical experts, peer reviewers, and AHRQ, revisions were made to the 

evidence report, and a summary of the comments and their disposition was submitted to AHRQ. 
23 

Chapter 3. Results 

Results of Review of Primary Literature 

A summary of the search results for the primary literature review is presented in Figure 1. 

From the search, we retrieved 59,116 unique citations. After reviewing the titles and abstracts, 

659 seemed eligible for further review and the full articles were retrieved. A total of 136 

primary literature articles are included in this review. 

Results of Review of Systematic Reviews 

A summary of the search results for the review of systematic reviews is presented in Figure 

2. The search yielded 1617 unique citations. After reviewing the titles and abstracts, 43 seemed 

eligible for review and they were retrieved for further review. A total of nine systematic review 

articles are included in this review. 

Summary of Study Characteristics and Evaluation Methods 

Over two-thirds (68 percent) of the studies were randomized controlled trials; the remaining 

studies were predominantly non-randomized controlled trials (see Appendix F∗, Evidence Table 

1). An evaluation was conducted both before and after the intervention in 76 percent of the 

randomized controlled trials and 88 percent of the non-randomized trials. 

Participant questionnaire was the most frequently used evaluation method (59 percent of the 

studies). Of those that used a participant questionnaire, over two-thirds (69 percent) used a 

written questionnaire. A few studies administered the questionnaire via computer (5 studies), 

orally (5), or over the phone (7). About half (47 percent) of the studies used a performance audit 

to evaluate the CME program. Performance audits were usually conducted through chart review 

(39 studies) and health plan databases (18). Patient questionnaires were used in 39 studies. 

Twenty studies included a qualitative evaluation. Seventeen studies evaluated the CME program 

through observer assessment. Most of these studies used a live standardized patient to assess the 

observer (11 studies). 

Summary of Study Participants and Study Setting 

The CME participants in most of the studies were exclusively fully-trained physicians (see 

Appendix F, Evidence Table 2). Other CME participants in the other studies most often included 

nurse practitioners, physician assistants, or nurses. Family medicine (73 studies), internal 

medicine (58 studies), general medicine (32 studies), pediatrics (21 studies), and primary care 

(18 studies) were the most common specialties included in the studies. Most study participants 

worked in private practice (52 studies), for health plans (32 studies), or in hospitals (19 studies). 

Many of the studies did not report on other characteristics of the study participants. Gender, age 

 

∗

 Appendixes cited in this report are provided electronically at http://www.ahrq.gov/clinic/tp/cmetp.htm
24 

and years of training/experience were reported in 50 percent, 28 percent, and 34 percent of 

studies.

 Central = The Cochrane Central Register of Controlled Trials; ERIC = Educational Resource Information Center 

Figure 1. Summary of literature search and review process for primary literature (number of articles)

Electronic databases 

MEDLINE® (38174) 

EMBASE® (44765) 

Cochrane CENTRAL 

(1843) 

ERIC® (2002) 

PsycINFO© (8738) 

Retrieved 

95560 

Title review 

59116 

Abstract review 

3574 

Article review 

659 

Hand searching 

38 

Duplicates 

36444 

Excluded 

55542 

Excluded 

2915 

Included studies 

136 

Excluded 

523 

Reasons for exclusion at the abstract review level* 

Not in English: 0 

Did not include human data: 11 

No original data, but may apply to KQ3: 22 

No original data and did not apply to KQ3: 107 

Meeting abstract: 1 

Did not include at least 15 fully trained physicians: 

417 

Did not include training or education: 165 

No evaluation of an educational activity: 238 

Published prior to 1981: 0 

Not conducted in US or Canada: 335 

Did not apply to a key question: 382 

Evaluation did not include data from a concurrent 

or historical comparison group: 402 

Other reason: 65 

Reasons for exclusion at the article review level* 

Not in English: 0 

Did not include human data: 1 

No original data, but may apply to KQ3: 5 

No original data and did not apply to KQ3: 60 

Meeting abstract: 5 

Did not include at least 15 fully trained physicians: 

95 

Did not include training or education: 11 

No evaluation of an educational activity: 12 

Published prior to 1981: 0 

Not conducted in US or Canada: 94 

Did not apply to a key question: 16 

Evaluation did not include data from a concurrent 

or historical comparison group: 80 

Involves quality improvement: 9 

Other reason: 12 

* Total may exceed number in corresponding box, as articles could be excluded for more than one reason at this level. 
25 

 Central = The Cochrane Central Register of Controlled Trials; ERIC = Educational Resource Information Center 

Figure 2. Summary of literature search and review process for systematic reviews (number of articles)

Electronic databases 

MEDLINE® (466) 

EMBASE® (2359) 

Cochrane CENTRAL 

(3) 

ERIC® (14) 

PsycINFO© (34) 

Retrieved 

2876 

Title review 

1617 

Abstract review 

148 

Article review 

43 

Hand searching 

0 

Duplicates 

1259 

Excluded 

1469 

Excluded 

105 

Included studies 

9 

Excluded 

34 

Reasons for exclusion at the abstract review level* 

Not in English: 0 

Did not include human data: 0 

Not a systematic review: 91 

Meeting abstract: 0 

Did not include medical students or physicians-intraining: 2 

Did not include medical training or education: 1 

No evaluation of an educational activity: 0 

Does not involve simulation, virtual reality, 

manikins, or standardized patients: 1 

Published prior to 1990: 0 

Did not apply to Key Question 3: 9 

Only includes fully trained physicians but could 

apply to other key questions: 0 

Other reason: 1 

* Total may exceed number in corresponding box, as articles could be excluded for more than one reason at this level. 

Reasons for exclusion at the abstract review level* 

Not in English: 0 

Did not include human data: 0 

Not a systematic review: 20 

Meeting abstract: 0 

Did not include medical students or physicians-intraining: 1 

Did not include medical training or education: 0 

No evaluation of an educational activity: 2 

Does not involve simulation, virtual reality, 

manikins, or standardized patients: 2 

Published prior to 1990: 0 

Did not apply to Key Question 3: 1 

Only includes fully trained physicians but could 

apply to other key questions: 0 

Did not report separately on the effects of 

simulation: 2 

Other reason: 0 
26 

One-third of the studies did not report the setting of the CME activity. In 52 studies, the 

CME activity occurred in the practice setting. The CME activity was not linked to a physical 

setting in 27 settings. Government agencies sponsored about 40 percent of the studies. 

Pharmaceutical agencies, professional societies, and insurance/health plan companies sponsored 

about 10 percent of the studies each. In about two-thirds of the studies, some type of physician, 

the majority of which were academic, taught the CME activity. The type of educator was not 

mentioned in about one-quarter of the studies. 

Summary of CME Activity Characteristics 

Table 4 provides an overview of the characteristics of the CME activities evaluated (for 

details of the characteristics of CME activities, see Appendix F∗, Evidence Table 3). Fifty 

studies evaluated a single media method. The most common single media method used in the 

CME activity was live (29 studies). Print media was evaluated in 14 studies and the Internet in 6 

studies. Nearly half of the studies used multiple media methods in the CME activities they were 

evaluating. Eighteen studies compared using a single media method to using multiple media 

methods. The type of media was not reported in one study. 

Table 4. Summary of CME Activity Characteristics 

Type of CME activity Number (%) of studies 

Media method 

Single media used in CME activity 50 (37) 

Live only media 29 (21) 

Print only media 14 (10) 

Internet only media 6 (4) 

Other type of single media 4 (3) 

Multiple media used in CME activity 67 (49) 

Single vs. multiple media used in CME activity 18 (13) 

Type of media not reported 1 (1) 

Educational technique 

Single technique used in CME activity 13 (10) 

Multiple techniques used in CME activity 95 (70) 

Single vs. multiple techniques used in CME activity 25 (18) 

Type of technique not reported 3 (2) 

Amount of exposure 

Exposed to CME activity once 44 (32) 

Exposed to CME activity multiple times 69 (51) 

One vs. multiple exposures to CME activity 12 (9) 

Other amount of exposure 7 (5) 

Amount of exposure not reported 4 (3) 

 

∗

 Appendixes cited in this report are provided electronically at http://www.ahrq.gov/clinic/tp/cmetp.htm
27 

Most (70 percent) of the studies used multiple educational techniques in the CME activity. 

Twenty-five studies compared using a single educational technique to using multiple educational 

techniques. Thirteen studies evaluated a single technique. Six articles evaluated reading only,13-

18 two evaluated only academic detailing,19 20 and one evaluated each of the following: problembased learning,21 conference calls,22 feedback,23 lecture,24 and lecture versus case-based 

learning.25

The participants were exposed to the CME activity only once in about a third of the studies 

and multiple times in about half. Twelve studies compared participants who were exposed once 

to a CME activity to participants who were exposed multiple times. 

The CME activities were designed for individuals in 90 studies and practice settings/teams in 

24 studies. For 17 studies, the CME activities were designed for both individuals and practice 

settings/teams. The CME activity was reported as being accredited in about a quarter (31 studies) 

of the studies. 

In 41 studies, the CME activity was a part of a quality improvement project. CME activities 

that were a part of quality improvement projects used multiple media methods significantly more 

often than CME activities with no quality improvement project (63% versus 43%; p=0.03). 

CME activities with quality improvement projects were also significantly less likely to use single 

live media (10% versus 26%; p=0.03). CME activities with and without quality improvement 

projects were similar to each other in terms of educational techniques and amount of exposure 

(data not shown). 

Study Quality of Primary Literature 

More than two-thirds of the trials were randomized (see Appendix F, Evidence Table 4). Just 

over half of these trials described their randomization techniques adequately, while five included 

descriptions of suboptimal randomization schemes. Because participants in a study of an 

educational intervention cannot be blinded to the intervention, we evaluated the trials for 

evidence that the outcomes evaluation was blinded. Approximately one-fifth of the trials 

described a blinded evaluation, with the majority of these including adequate descriptions of the 

blinding technique. Only one-quarter of the trials described losses to follow-up or reasons for 

withdrawal. Nearly two-thirds of the trials described power calculations to detect outcomes, but 

one-quarter of these were not adequately powered. 

Based on our quality scoring system described in the Methods chapter, over three-quarters of 

the trials were rated at two points or less: 

• 8 studies achieved a score of 5 

• 11 studies achieved a score of 4 

• 12 studies achieved a score of 3 

• 38 studies achieved a score of 2 

• 29 studies achieved a score of 1 

• 36 studies achieved a score of 0. 

Quality of the Systematic Reviews 

All reviews, except one,26 described search methods in sufficient detail to allow replication 

of the reviews (see Appendix F, Evidence Table 5). All reviews used one or more electronic 
28 

databases and hand-searching to identify relevant articles. However, only three reviews26-28

supplemented their search by soliciting expert opinion to identify relevant articles. Two 

systematic reviews29 30 did not report study inclusion (and/or exclusion) criteria and one review28

did not report inclusion criteria in enough detail to allow replication.

Only two reviews performed quality assessment of the studies included in the review using a 

validated quality scale.27 30 In addition, one review31 provided a descriptive assessment of study 

quality. Evaluation for publication bias was not reported by any review. All reviews, except one, 

synthesized evidence qualitatively. One review,26 that combined individual study results 

quantitatively to generate a summary effect size, used both fixed-effects and random-effects 

models. 

Four reviews26 27 30 31 discussed the variation in the results of the original studies in a 

qualitative manner, one additional study29 discussed it only partially, and four studies did not 

discuss the variations in the results of the original studies. Qualitative evaluation of 

heterogeneity, either by subgroup analyses or meta-regression, was not performed by any of the 

reviews included in our study. 

Reporting of Adult Learning Principles 

Only 13 percent of trials were rated as “good” in enabling learners to be active contributors 

to their learning, while about half of trials were rated as fair (see Appendix F, Evidence Table 6). 

More than three-quarters of the trials were rated as “good” in relating to learners’ current work or 

life experiences. Only 12 percent of the trials were rated as “good” in tailoring their curricula to 

learners’ current or past experiences, and almost half were rated as “poor.” Only 6 percent of the 

trials were rated as “good” in allowing learners to identify their own learning goals and direct 

their education, and over 80 percent were rated as “poor.” About 10 percent of trials were rated 

as “good” in allowing learners to practice what they learned in simulated activities, providing 

support to self-directed learners, and receiving feedback from teachers or peers during active 

learning. Two-thirds were rated as “poor” in these areas. Less than 10 percent of trials were rated 

as “good” in allowing learners to reflect on their learning, while three-quarters of trials were 

rated as “poor.” Only four trials were rated as “good” in allowing learners to observe the faculty 

role-model behaviors, while nearly 90 percent of trials were rated as “poor.” 

A notable example of a trial that received high ratings for all adult learning principles was 

Gerrity et al.32
29 

Key Question 1: Is there evidence that particular methods of 

delivering CME are more effective in: a) imparting knowledge 

to physicians, b) changing physician attitudes, c) acquiring 

skills, d) changing physician practice behavior, or e) 

changing clinical practice outcomes? 

Key Question 2: Do changes in knowledge, attitudes, skills, 

practice behavior, or clinical practice outcomes produced by 

CME persist over time (greater than or equal to 30 days)? 

Knowledge Outcomes 

Short-term and long-term effects of CME on knowledge (See Appendix F, Evidence 

Table 7). A total of 39 studies addressed 41 knowledge objectives.17 19 25 32-67 Eleven studies that 

addressed 12 objectives had a comparison group, but no control group and thus could not be 

utilized to determine whether or not the intervention was effective at knowledge transfer.17 25 58-61 

63-67 This left 28 studies addressing 29 objectives. These studies were classified by the outcome 

of the objective into those in which the objective was met, objective not met, or those with mixed 

results. In addition, the 28 studies were classified into three groups by the duration of knowledge 

acquisition/retention into those studies in which the duration was not clearly reported, studies 

that addressed short-term knowledge changes and long-term knowledge changes. 

A total of 22 studies addressing 23 objectives demonstrated improvements in knowledge.19 33-

53 This represents 79% of the 28 studies with an adequate control group. Of these, 6 studies33-38

addressing 6 objectives did not clearly report the duration of evaluation, 1 study39 addressing 1 

objective demonstrated short-term improvement in knowledge, and 15 studies addressing 16 

objectives demonstrated long-term improvements in knowledge. The studies were too 

heterogeneous too identify any global similarities, but any specific information related to the 

type of media, educational technique or exposure volume will be addressed in those subsequent 

sections. 

Four studies addressing four objectives failed to show improvements in knowledge.32 54-56 

No study demonstrated a regression in knowledge. Of these four studies, one study32 did not 

clearly report on the duration of evaluation while the three remaining studies evaluated long-term 

knowledge changes. No study was identified that evaluated short-term knowledge and failed to 

show a change in knowledge. Of the three studies that considered long-term outcomes, the study 

by Elliott et al demonstrated trends toward improved knowledge but these slight improvements 

lacked statistical significance.55 In the remaining two studies, one failed to show improvements 

in knowledge regarding bioterrorist attacks through the voluntary participation in a web-based 

educational program and the other failed to show an improvement in knowledge regarding blood 

pressure control after a mailed CME program. 

Two studies addressing two objectives demonstrated mixed results.57 62 One study did not 

clearly report on the duration of evaluation and one reported on long-term effects. The study that 

did not clearly report duration evaluated the impact of an educational intervention aimed at 

medical care evaluation committees and demonstrated a statistical improvement in only one of 
30 

the three committees.57 The study by Chodosh et al considered an evaluation nine months after 

the intervention. In this study, more intervention group physicians answered knowledge-related 

questions on capacity determination for patients with possible dementia, but found no differences 

between intervention group and control group on questions regarding dementia evaluation, 

patient safety, or depression treatment. 

In summary, 78% of the studies with an adequate control group demonstrated that CME 

activities were effective at improving knowledge with the majority (68%) of these studies 

demonstrating long-term improvements in knowledge. 

Short-term and long-term effects of CME media methods on knowledge. Studies were 

classified by objectives met and evaluation duration as described above and by media utilized for 

the educational intervention. The media classifications included multi-media, single media and 

single versus multi-media. 

As stated in the above section, 22 studies addressing 23 objectives demonstrated 

improvements in knowledge. Seven studies addressing eight objectives evaluated a single media 

intervention.39 40 43 45 48 50 53 56 Three of the studies utilized the internet and the remaining four 

utilized live media. The three studies that utilized the internet demonstrated improved short-term 

knowledge in one study39 and long-term knowledge in two studies.43 50 All four of the studies 

that utilized live media demonstrated long-term improvements in knowledge. Twelve studies 

addressing 12 objectives utilized multimedia interventions to demonstrate knowledge benefits.19 

33-37 41 42 46 49 51 52 Five studies did not clearly report on the duration and seven studies 

demonstrated long-term improvements in knowledge. Three studies compared a single mediabased intervention to a multimedia intervention.38 44 47 All three studies included at least one 

concurrent control group that only utilized print material and in all three cases multi-media 

outperformed the print-based single media intervention group. 

The four studies that did not demonstrate improvements in knowledge utilized single media 

(print) in one study56 that evaluated long-term knowledge, multimedia (live, audio and print) in 

one study32 in which evaluation duration was not clearly defined, multimedia (live and print) in 

one study55 that considered long-term knowledge changes, and one study54 that compared single 

to multi-media for long-term benefits. In this study the comparison was between a live 

intervention and a live internet with non-real time reading material. 

Of the two studies that demonstrated mixed results, one study57 utilized a single media 

intervention (live) and the other study62 utilized multimedia intervention (live internet with not 

real time print). 

When grouped solely by media classification, 9 studies addressing 10 objectives used a 

single media with 7 demonstrating benefits, 15 studies addressing 15 objectives considered 

multi-media interventions with 12 demonstrating benefits, and 4 studies addressing 4 objectives 

compared single to multi-media with 3 of these studies demonstrating that multi-media 

interventions had greater benefit. 

Given the heterogeneity of the studies the only recognized trends were that multimedia seems 

better than a single media intervention and that print interventions are either not beneficial or 

very weak in their ability to lead to improved knowledge. 

Short-term and long-term effects of CME educational techniques on knowledge. 

Studies were classified by objectives met and evaluation duration as described above and by 

technique utilized for the educational intervention. The technique classifications included 
31 

multiple techniques, single technique and single versus multiple techniques. Two of the studies 

that have not been previously included because of a lack of a control group can be utilized in this 

section because the concurrent comparison group did indeed utilize a different technique 

permitting some head to head comparisons.25 66 This means that for this section there are 30 

studies addressing 31 objectives. 

Only one study that met its objectives utilized a single technique, academic detailing, and this 

study demonstrated improvements in long-term knowledge.19 Eighteen studies addressing 18 

objectives utilized multiple techniques to improve knowledge.33-37 39 41-46 48-53 Five studies33-37

that demonstrated improved knowledge did not clearly report the duration, 1 study39

demonstrated short-term knowledge improvement and 12 studies demonstrated long-term 

knowledge improvements. The majority of the studies that demonstrated improvement but did 

not clearly report the duration included case-based learning as a technique in combination with 

techniques ranging from discussion groups to independent reading. Of the 12 studies that 

demonstrated long-term improvements in knowledge the majority integrated multiple techniques 

and most commonly combined case-based learning with discussion groups with independent 

reading, several utilized standard lectures with readings, and some technique combinations 

ranged from lecture with a standardized patient to lecture with team-based training. Three studies 

addressing four objectives compared a single technique versus multiple techniques demonstrated 

improvements in knowledge.38 40 47 One study did not clearly report duration and two studies 

addressing three objectives demonstrated long-term knowledge improvements. The study that 

did not clearly report duration demonstrated a greater benefit to the combination of case-based 

learning with readings when compared to readings alone.38 One study that addressed two 

objectives compared problem-based learning to the combination of lecture and discussion groups 

and demonstrated that that problem-based learning group was more effective regarding 

knowledge of diagnosis and management of headache. The other study compared discussion 

groups with readings to readings alone and the combination was more effective at increasing 

knowledge regarding compliance adherence. 

Four studies of four objectives did not demonstrate improved knowledge.32 54-56 One study 

that utilized multiple techniques that included lecture, discussion groups, role playing and 

feedback failed to demonstrate improved knowledge and did not clearly report the duration.32

Two studies failed to demonstrate improvements in knowledge despite utilizing multiple 

techniques.55 56 One combined lecture with case-based learning with discussion groups and 

readings and failed to show improvements in pain knowledge, while the other combined readings 

with chart cue materials and failed to show improvements in knowledge of hypertension. One 

study compared a single technique, lecture to a combination of techniques that included lecture 

with case-based learning and readings and failed to show an improvement in knowledge of 

bioterrorism. 

The two studies that were added to this section deserve a greater explanation. Greenberg et al 

compared lecture to case-based learning and the case-based learners demonstrated improvements 

in post-test questions regarding common pediatric problems as compared to 29% improvement in 

the lecture group.25 Unfortunately, these improvements seen in the case-based group were not 

significant at six and nine months after the intervention. Heale et al compared three groups, one 

receiving lectures, one case-based learning with discussion groups, and the final group received 

problem-based learning.66 This study also failed to demonstrate benefits of any technique either 

in short or long-term knowledge. Thus these two studies, when added to the four above studies 
32 

that did not demonstrate improvements means a total of six studies did not show improved 

knowledge. 

One of the two studies that demonstrated mixed results did not clearly report duration while 

the other considered long-term knowledge improvements.57 62 The study57 that did not report 

duration compared a combination of case-based learning with discussion groups to control while 

the other compared lecture with discussion group to control. 

When grouped solely by technique classification then two studies addressing two objectives 

utilized a single technique with one study showing improvements through the use of academic 

detailing and one showing no knowledge improvement through the use of readings alone. 

Twenty-three studies addressing 23 objectives utilized multiple techniques with 18 

demonstrating benefits, 3 no benefits, and 2 mixed results. Five studies addressing six objectives 

compared single versus multiple techniques. Two studies addressing two objectives 

demonstrated benefits of multiple techniques as compared to single, one study addressing two 

objectives demonstrated benefits to single technique(problem-based learning) compared to 

multiple and two studies addressing two objectives did not demonstrate any benefit of single as 

compared to multiple techniques. 

The outcomes from this section are also heterogeneous but it does appear that multiple 

techniques that most commonly include case-based learning seem to be more likely to be 

associated with improvements in knowledge. 

Short-term and long-term effects of the amount of exposure on knowledge. Studies were 

classified by objectives met and evaluation duration as described above and by exposure volume 

for the educational intervention. The exposure volume classifications included multiple 

exposures, single exposure and single versus multiple exposures. The two studies added in the 

previous section do not apply to this section given the lack of control groups. In addition, one 

additional study38 did not adequately describe the exposure and thus is excluded from analysis in 

this section leaving 27 studies addressing 28 objectives. 

There were 21 studies addressing 22 objectives which demonstrated an improvement in 

knowledge.19 33-37 39-53 Five studies addressing five objectives evaluated a single exposure volume 

with one study36 not clearly reporting the duration and four studies44 45 48 53 demonstrating longterm improvements. Three of these considered knowledge at 6 months after intervention and one 

at 24 months after intervention. Twelve studies addressing 12 objectives utilized multiple 

exposure volumes. Three studies did not clearly report duration,33 35 37 one study demonstrated 

short-term knowledge gains,39 and the final eight studies all demonstrated long-term knowledge 

gains.19 41 43 46 49-52 The shortest time interval to evaluation in these long-term beneficial studies 

was 3 months and the longest was 15 months. Four studies addressing five objectives compared a 

single exposure to multiple exposures and demonstrated knowledge gains. One study that did not 

clearly report duration demonstrated that multiple exposures were better than a single exposure 

at improving knowledge of office-based dermatologic procedures.34 The other three studies 

addressing four objectives all demonstrated that multiple exposures were better than a single 

exposure at improving knowledge.40 42 47 

Four studies addressing four objectives did not demonstrate knowledge improvements. No 

study was identified that evaluated a single exposure only. Three studies evaluated multiple 

exposures with one32 not clearly reporting the duration and the other two55 56 failing to 

demonstrate long-term knowledge improvements. One study that compared single versus 
33 

multiple exposures did not show any difference from baseline knowledge at one and six months 

after intervention in either group.54

No studies were identified that utilized a single exposure only or compared a single versus 

multiple that demonstrated mixed results. Two studies demonstrated mixed results and both 

utilized multiple exposures.57 62 One study57 did not clearly report on duration and the other study 

demonstrated mixed results at the nine month evaluation. 

When grouped solely by exposure volume then all five studies that evaluated a single 

exposure demonstrated improved knowledge. Twelve of the 17 studies that utilized multiple 

exposures demonstrated knowledge improvements with an additional 2 demonstrating mixed 

results. The majority (67%) of these were able to demonstrate long-term knowledge benefits. Of 

the five studies that compared a single exposure to multiple exposures, four (80%) demonstrated 

a greater benefit to multiple exposures as compared to a single exposure. 

In summary, despite the heterogeneity of these studies it appears that despite the fact that all 

five studies that utilized a single exposure demonstrated benefit the head-to-head comparison 

studies imply that when possible multiple exposures produces better knowledge gains. 

Summary of the effects of CME on knowledge. The heterogeneity of the studies precludes 

firm conclusions, but the trends demonstrated that CME is effective at producing both short-term 

and long-term knowledge gains and that when possible, multimedia, multiple techniques, and 

multiple exposures should be used. 

Quality of the evidence for the short-term and long-term effects of CME on knowledge 

(see Appendix F∗

, Evidence Table 8). Taking into consideration the quantity and quality, and 

consistency of evidence on the effectiveness of CME on knowledge, we graded the strength of 

evidence as very low. 

Attitude Outcomes 

Short-term and long-term effects of CME on physician attitudes (See Appendix F, 

Evidence Table 9). A total of 35 studies addressed 45 attitude objectives.13 34-37 39 40 42 47 50 52 53 55 

58 59 62-64 66 68-83 Eight studies that addressed 11 objectives had a comparison group, but no control 

group and thus could not be utilized to determine whether or not the intervention was effective at 

attitudinal change.58 59 63 64 66 79 80 83 One additional study that addressed two objectives could not 

be utilized as results were not reported by groups and thus its results were unclear and it was also 

not included.82 This left 26 studies addressing 32 objectives. These studies were classified by the 

outcome of the objective into those in which the objective was met, objective not met, or those 

with mixed results. In addition, the 26 studies were classified into three groups by the duration 

of attitudinal change into those studies in which the duration was not clearly reported, studies 

that addressed short-term attitudinal changes and long-term attitudinal changes. 

A total of 22 studies addressing 26 objectives demonstrated improvements in attitude.13 34-36 

39 40 42 47 50 52 53 68-78 This represents 71% of the 31 studies with an adequate control group. Of 

these, six studies addressing seven objectives did not clearly report the duration of evaluation,13 

34-36 68 69 one study addressing two objectives demonstrated short-term improvement in 

attitudes,39 and 15 studies addressing 17 objectives demonstrated long-term improvements in 

 

∗

 Appendixes cited in this report are provided electronically at http://www.ahrq.gov/clinic/tp/cmetp.htm
34 

attitude.40 42 47 50 52 53 70-78 The studies were too heterogeneous too identify any global similarities, 

but any specific information related to the type of media, educational technique or exposure 

volume will be addressed in those subsequent sections. 

Four studies addressing four objectives failed to show improvements in attitude.37 52 55 62 No 

study demonstrated a regression in attitude. Of these four studies, one study37 did not clearly 

report on the duration of evaluation while the three remaining studies evaluated long-term 

attitudinal changes. No study was identified that evaluated short-term attitudes and failed to 

show a change in attitudes. Of the three studies that considered long-term outcomes, the study by 

Elliott et al demonstrated trends toward improved attitudes but these slight improvements lacked 

statistical significance.55 In the remaining two studies, one failed to show improvements in 

attitudes regarding providers’ perceptions of quality of care nine months after intervention.62 The 

other failed to show an improvement in attitude regarding efficacy of cholesterol lowering 

practices.52 

Two studies addressing two objectives demonstrated mixed results.62 81 Both studies reported 

on long-term effects. The study by Chodosh et al considered an evaluation nine months after the 

intervention. In this study, more intervention group physicians endorsed the statement, “Older 

patients with dementia are difficult to manage in primary care,” but no other differences in 

attitudes regarding dementia were identified.62 The study by Norris et al utilized an evaluation 

six months after intervention and noted that intervention group providers had significant 

improvements in three of ten tested attitudes, including they felt by self report that they were 

more likely to counsel patients regarding physical activity than control providers.81

In summary, 85 percent of the studies with an adequate control group demonstrated that 

CME activities were effective at improving attitudes with the majority (68%) of these studies 

demonstrating long-term improvements in attitudes. 

Short-term and long-term effects of CME media methods on physician attitudes. 

Studies were classified by objectives met and evaluation duration as described above and by 

media utilized for the educational intervention. The media classifications included multi-media, 

single media and single versus multi-media. 

As stated in the above section, 22 studies addressing 26 objectives demonstrated 

improvements in attitudes. Seven studies addressing eight objectives evaluated a single media 

intervention.13 39 40 50 53 71 72 Two of the studies39 50 utilized the internet, one13 utilized a computerbased program and the remaining four utilized print media.40 53 71 72 The two studies addressing 

three objectives that utilized the internet demonstrated improved short-term attitudes in one study 

and long-term attitudes in the other. All four of the studies that utilized print media demonstrated 

long-term improvements in attitudes. The one study that utilized computer-based education for 

its intervention did not clearly report the duration. Twelve studies addressing 15 objectives 

utilized multi-media interventions to demonstrate attitudinal benefits.34-36 42 52 68 69 73 75-78 Five 

studies addressing six objectives did not clearly report on the duration and seven studies 

addressing nine objectives demonstrated long-term improvements in attitudes. Three studies 

compared a single media-based intervention to a multi-media intervention.47 70 74 All three studies 

included at least one concurrent control group that only utilized print material and in all three 

cases multi-media outperformed the print-based single media intervention group. All three 

studies showed long-term improvements in attitudes. 

The four studies that did not demonstrate improvements in attitude all utilized multimedia.37 

52 55 62 One study combined live with print and did not clearly report on duration.37 The other 3 
35 

studies considered long-term attitudinal changes at 9 months in one study and 15 months in the 

other 2 studies. The study that considered outcomes at nine months utilized a combination of live 

internet with print material while the other two studies combined live with print in one and live 

with video and print in the other. 

Both of the two studies that demonstrated mixed results utilized multimedia approaches.62 79 

One study utilized live internet with print material and the other study utilized live in conjunction 

with print and a followup phone call. 

When grouped solely by media classification, then 7 studies addressing 8 objectives utilized 

a single media with all 7 demonstrating attitudinal benefits, 18 studies addressing 26 objectives 

considered multimedia interventions with 12 demonstrating benefits, and 3 studies addressing 3 

objectives compared single to multi-media with all 3 of these studies demonstrating that 

multimedia interventions had greater benefit. 

Given the heterogeneity of the studies the only recognized trends were that multimedia 

appears better than a single media intervention and that print interventions are either not 

beneficial or very weak in their ability to lead to improve attitudes. 

Short-term and long-term effects of CME educational techniques on physician 

attitudes. Studies were classified by objectives met and evaluation duration as described above 

and by technique utilized for the educational intervention. The technique classifications included 

multiple techniques, single technique and single versus multiple techniques. One study that had 

not been previously included because of a lack of a control group can be utilized in this section 

because the concurrent comparison group did indeed utilize a different technique permitting 

some head to head comparisons.66 This means that for this section there are 23 studies addressing 

27 objectives. 

Only one study that met its objectives utilized a single technique, reading, and this study did 

not clearly report duration.13 Seventeen studies addressing 21 objectives utilized multiple 

techniques to improve attitudes. Five studies34-36 68 69 addressing 6 objectives that demonstrated 

improved attitudes did not clearly report the duration, 1 study39 addressing 2 objectives 

demonstrated short-term attitudinal improvement and 11 studies42 50 52 53 71-73 75-78 addressing 13 

objectives demonstrated long-term attitudinal improvements. 

The majority of the studies that demonstrated improvement but did not clearly report the 

duration included case-based learning as a technique in combination with techniques ranging 

from discussion groups to independent reading. Of the 11 studies that demonstrated long-term 

improvements in attitude the majority integrated multiple techniques and most commonly 

combined case-based learning with discussion groups with independent reading, several utilized 

standard lectures with readings, and some technique combinations ranged from lecture with a 

standardized patient to lecture with team-based training. Four studies addressing four objectives 

compared a single technique versus multiple techniques demonstrated improvements in 

attitudes.40 47 70 74 All four studies demonstrated long-term knowledge improvements. Three of 

these four studies utilized readings as the sole technique and one utilized a discussion group. All 

four studies demonstrated that multiple techniques were better than single techniques. 

Four studies of four objectives did not demonstrate improved attitudes.37 52 55 62 One study 

that utilized multiple techniques that included live and print failed to demonstrate improved 

attitudes and did not clearly report the duration.37 Two studies failed to demonstrate 

improvements in attitudes despite utilizing multiple techniques.52 62 One combined lecture with 

discussion group and failed to show improvements in provider perceptions about quality of care 
36 

for patients with dementia, while the other combined case-based learning with discussion with 

readings, with standardized patients and failed to show improvements in attitudes regarding 

cholesterol lowering practices. Only one study compared a single technique versus multiple and 

failed to show an improvement in provider attitudes regarding pain.55

The one study that was added to this section deserves a greater explanation. Heale et al66

compared three groups, one receiving lectures, one case-based learning with discussion groups, 

and the final group received problem-based learning. This study demonstrated improvements in 

attitudes that were greatest in the problem-based learning participants. 

Both studies that demonstrated mixed results assessed long-term attitudinal change62 81 and 

both utilized multiple techniques. These included lecture with a point-of-care opinion leader in 

one study and lecture with discussion groups in the other. 

When grouped solely by technique classification then one study addressing one objective 

utilized a single technique and it demonstrated attitudinal improvement. Twenty-two studies 

addressing 26 objectives utilized multiple techniques with 17 demonstrating benefits, 3 no 

benefits, and 2 mixed results. Five studies addressing five objectives compared single versus 

multiple techniques. Four studies demonstrated greater attitudinal change with the utilization of 

multiple techniques as compared to a single technique and one study showed no improvement. 

The outcomes from this section are also heterogeneous but it does appear that multiple 

techniques that most commonly include case-based learning seem to be more likely to be 

associated with improvements in attitudes. 

Short-term and long-term effects of the amount of exposure on physician attitudes. 

Studies were classified by objectives met and evaluation duration as described above and by 

exposure volume for the educational intervention. The exposure volume classifications included 

multiple exposures, single exposure and single versus multiple exposures. The one study added 

in the previous section does not apply to this section given the lack of control group. In addition, 

one additional study58 did not adequately describe the exposure and thus is excluded from 

analysis in this section leaving 24 studies addressing 25 objectives. 

There were 22 studies addressing 25 objectives which demonstrated an improvement in 

attitude.13 34-36 39 40 42 47 50 52 53 68-78 Seven studies addressing eight objectives evaluated a single 

exposure volume with two studies36 68 not clearly reporting the duration and five studies53 70 71 74 

77 addressing six objectives demonstrating long-term improvements. Eleven studies addressing 

12 objectives utilized multiple exposure volumes. Three studies13 35 69 did not clearly report 

duration, one study39 demonstrated short-term knowledge gains, and the final seven studies50 52 72 

73 75 76 78 addressing eight objectives all demonstrated long-term knowledge gains. Four studies 

addressing five objectives compared a single exposure to multiple exposures and demonstrated 

attitudinal improvements.34 40 42 47 One study34 addressing two objectives did not clearly report 

duration, while the remaining three studies all demonstrated long-term improvements in 

attitudes. 

Four studies addressing four objectives did not demonstrate attitude improvements. No study 

was identified that evaluated a single exposure only or that performed a comparison between 

single and multiple exposures. Three studies evaluated multiple exposures with one37 not clearly 

reporting the duration and the other two55 62 failing to demonstrate long-term knowledge 

improvements. 
37 

No studies were identified that utilized a single exposure only or compared a single versus 

multiple that demonstrated mixed results. Two studies demonstrated mixed results and both 

utilized multiple exposures.62 81 Both studies assessed long-term attitudes. 

When grouped solely by exposure volume then all seven studies that evaluated a single 

exposure demonstrated improved attitudes. Eleven of the 17 studies that utilized multiple 

exposures demonstrated attitudinal improvements with an additional two demonstrating mixed 

results. The majority (64%) of these were able to demonstrate long-term attitudinal benefits. Of 

the four studies that compared a single exposure to multiple exposures all demonstrated a greater 

benefit to multiple exposures as compared to a single exposure. 

In summary, despite the heterogeneity of these studies it appears that there is a trend toward 

multiple exposures being of greater benefit for attitudinal change then a single exposure, 

although it must be pointed out that all seven studies that evaluated a single exposure indeed 

demonstrated improvements in attitudes. 

Summary of the effects of CME on physician attitudes. The heterogeneity of the studies 

precludes firm conclusions, but the trends demonstrated that CME is effective at producing both 

short-term and long-term attitudinal gains and that when possible, use of multimedia, multiple 

techniques, and multiple exposures should produce better attitudinal outcomes. 

Quality of the evidence for the short-term and long-term effects of CME attitudes (see 

Appendix F, Evidence Table 8). Taking into consideration the quantity and quality, and 

consistency of evidence on the effectiveness of CME on attitudes, we graded the strength of 

evidence as very low. 

Skills Outcomes 

Short-term effects of CME on skills (See Appendix F, Evidence Table 10). Skill-related 

outcomes were categorized as either cognitive or psychomotor. The skills reported were varied 

including cognitive skills such as diagnostic accuracy for psychiatric conditions,36 or 

communication skills with patients.84 Psychomotor skills included examples such as performing 

a physical exam,60 or doing a joint injection.80 A total of 15 studies reported skill outcomes, with 

a total of 18 skill outcomes.13 32 33 36 40 49 60 77 80 84-89 Twelve of the 15 studies that reported skill 

outcomes were cognitive skills,13 32 33 36 40 49 77 84-88 with the remaining three being psychomotor 

skills.60 72 80 

Of those 15 studies, 10 had skill outcomes that met the objectives of the study.32 33 36 40 49 72 84-

87 One study had three skill outcomes that all met the objectives,84 and another study had two 

reported skill outcomes that met the objectives.33 The remaining studies each had one skill 

outcome. Two studies had skill outcomes that did not achieve the study objectives,13 77 and one 

study had mixed results that did not clearly meet the study objectives.88 Two studies compared 

different methods and techniques of CME without a separate control group that did not receive 

CME;60 80 therefore, the overall effectiveness of CME cannot be discerned from these studies. 

Given that 10 out of 13 studies that included control groups and reported on skills (13 out of 16 

outcomes) met the objectives and given the varied nature of the studies, the literature does 

indicate that CME is effective in this area, particularly at developing cognitive skills. Little can 

be said about the effectiveness of CME for psychomotor skills given the paucity of data in this 

area. 
38 

Long-term effects of CME on skills (See Appendix F, Evidence Table 10). Seven of the 

15 studies reporting skill outcomes had an evaluation beyond 30 days after the CME activity.40 49 

72 77 84 86 90 Six of the studies addressing long-term skill outcomes addressed cognitive skills,40 49 

77 84 86 90 while one addressed psychomotor skills.72 Of those seven studies, six met the objectives 

regarding long-term skill outcomes,40 49 72 84 86 90 while one did not meet its objectives.77 The one 

study that did not meet its objectives regarding long-term skill outcomes addressed cognitive 

skills.77 Overall, the data supports the positive effect of CME on long-term skill outcomes. 

Short-term effects of CME media methods on skills. The media methods of CME that 

were included in the studies that met the study objectives regarding skills included seven that 

used live media,32 33 36 40 49 72 84 four that included print materials,32 33 49 85 two that included video 

methods,32 36 two that included audio methods,32 49 two that used the Internet (not real time),86 87

and one that used computers (off-line).33 However, in several cases the same media methods 

were used in all experimental arms and therefore no conclusions can be drawn from the study 

outcomes applying to specific CME methods. This situation applied to live methods in one 

study,40 audio methods in one study,49 and print materials in two studies.49 85 In addition, one 

study was not clear about which groups received particular methods including live, print, and 

computer (off-line) media.33 The studies that did not clearly meet skills objectives included 

live,77 88 video,77 88 and computer (off-line) methods.13 Two studies directly compared different 

methods of CME but did not include a control group without CME.60 80 One of these studies 

showed no difference between print, computer (off-line), and live methods.80 However, another 

study showed that live methods were superior to video and print combined.60 Given this result 

and the dominance of live methods among the studies that met their skills objectives, the data 

suggests that live methods have the greatest impact on the effectiveness of CME regarding skills 

outcomes. Given the paucity of data and the varied results, little can be said about the relative 

effectiveness of other CME media methods on affecting skills. 

Long-term effects of CME media methods on skills. The six studies that addressed longterm skill outcomes beyond 30 days and met their skills objectives used a variety of media 

methods including four using live methods,40 49 72 91 two using print materials,49 85 one using 

audio methods,49 and one using the Internet (not real time).86 However, the one study that used 

audio methods had audio in all experimental arms,49 the two that used print media did so in all 

groups,49 85 and one of the four that used live methods did so in all groups.40 Therefore, the 

experimental effect on long-term outcomes in studies that met their skills objectives was seen 

only for three studies using live media,49 72 84 and one study that used the Internet (not real 

time).86 The one study that did not meet its objectives regarding long-term skill outcomes used 

live and video as the methods.77 The studies that directly compared different methods of CME 

did not address long-term skill outcomes.60 80 Based on the limited data, it is difficult to draw 

conclusions on particular media methods of CME that have a greater or lesser impact on longterm skill outcomes. 

Short-term effects of CME educational techniques on skills. Varied educational 

techniques were used in the studies that met their objectives regarding short-term skill outcomes. 

These included lectures in six studies,32 36 40 49 72 84 discussion groups in five studies,32 49 72 84 85

readings in five studies,32 33 49 85 86 case-based learning in four studies,36 49 86 87 feedback in three 
39 

studies,32 49 86 role play in three studies,32 72 84 and clinical experiences in two studies.32 72

Listserv,85 programmed learning,86 problem-based learning,40 audio-taped encounters,72 and 

standardized patients,84 were seen in one study each in the studies that met their skills objectives. 

Of note, readings were used as a technique in both experimental and control groups in one 

study.85 In the studies where the skills objectives were not clearly met by the outcomes, the 

following techniques were used: demonstration and lecture in two studies,77 88 and readings,13

discussion groups,88 feedback,88 programmed learning,88 and role play88 in one study each. Two 

studies that did not include a control group compared the techniques of readings versus readings 

with demonstration versus mentor/preceptor with simulation.80 No difference was seen with 

these different techniques. In another study without a control group, the techniques of 

demonstration and simulation were compared with demonstration and simulation plus 

feedback.60 The group that included feedback was significantly better when skills acquisition 

was assessed. Finally, two of the studies that met their objectives regarding skills and included 

control groups did a comparison of individual techniques of delivering CME. One study 

compared problem-based learning with lecture, and problem-based learning was significantly 

better.40 Another study compared discussion groups and readings to discussion groups and 

readings with feedback.85 In this study, feedback had no additional effect. Given the limited 

number of studies, the wide variety of techniques described, and the conflicting results, it is 

difficult to draw conclusions about the educational techniques that have the greatest and least 

effect on skills. 

Long-term effects of CME educational techniques on skills. The educational techniques 

used in the studies that met the study objectives regarding long-term skill outcomes included 

discussion groups in four studies,49 72 84 85 lectures in four studies,40 49 72 84 readings in three 

studies,49 85 86 role play in two studies,72 84 case-based learning in two studies,49 86 and feedback in 

two studies.49 86 The techniques of clinical experiences,72 listserv,85 programmed learning,86

problem-based learning,40 audio-taped encounters,72 and standardized patients,84 were seen in 

one study each that met skills retention objectives. The one study that did not meet its objectives 

regarding long-term skill outcomes used the techniques of demonstration, lecture, and 

simulation.77 The two studies that compared different techniques in CME without a control 

group did not address long-term skill outcomes.60 80 However, in two studies that did have a 

control group, one showed that problem based learning is superior to lectures in long-term skill 

outcomes,40 but the other showed no advantage of feedback over demonstration and readings 

alone.85 Given the limited number of studies and the varied techniques, it is difficult to draw 

conclusions about the educational techniques that that have a greater or lesser effect on long-term 

skill outcomes. 

Short-term effects of the amount of exposure on skills. The majority of the studies that 

met their skills objectives had multiple exposures to the CME activity. Seven of these studies 

used multiple exposures,32 40 49 72 84-86 while one used a single exposure.36 It was unclear how 

many exposures there were in two of the studies whose skills outcomes met the study 

objectives.33 87 The studies that did not meet the skills objectives included two that used a single 

exposure,77 88 and one that used multiple exposures.13 These results suggest that multiple 

exposures to CME for skills objectives is superior to a single exposure. 
40 

Long-term effects of the amount of exposure on skills. All six of the studies that 

addressed long-term skill outcomes and met the study objectives used multiple exposures to 

CME.40 49 72 84-86 The one study that did not meet its study objectives regarding long-term skill 

outcomes used a single exposure.77 These results support multiple exposures as having a greater 

impact on long-term skill outcomes. 

Quality of the evidence for the short-term and long-term effects of CME on skills (see 

Appendix F∗

, Evidence Table 8). Taking into consideration the quantity and quality, and 

consistency of evidence on the effectiveness of CME on skills, we graded the strength of 

evidence as low. 

Practice Behavior Outcomes 

Short-term and long-term effects of CME on practice behavior (See Appendix F, 

Evidence Table 11). A total of 105 studies evaluated the short- and long-term impact of CME 

activity on 135 practice behavior objectives. Of the 105 studies, 61 studies met 70 practice 

behavior objectives. Ten of these, reporting on 11 objectives, met objectives but did not report 

evaluation duration, leaving us unable to determine whether the effect of CME was short-term or 

long-term.15 32 35 36 68 69 79 87 92 93 Four of these studies reported improvement with regard to 

medication prescribing, and three reported improvement with regard to screening test referrals. 

One study with evaluation duration of 30 days or less met objectives in three out of six smoking 

cessation and documentation behaviors.94

Fifty studies with evaluation duration greater than 30 days met 58 objectives, suggesting 

long-term retention of CME effectiveness.19 44-49 52 53 62 70 72 75-78 81 82 82 95-125 Among these, 

evaluation duration ranged from 6 months or less after the educational intervention (17 studies) 

to 1 year or greater (30 studies). 

A wide mix of objectives was studied. For example, 12 objectives were related to medication 

prescribing, eight were related to screening standards, 14 were related to physician counseling 

behaviors (mostly smoking cessation, but also dietary counseling, sexual practices counseling, 

etc.), 11 were related to guideline adherence, and the remainder were related to physician 

behaviors pertaining to other topics. 

Twenty nine total studies, reporting on 38 objectives, did not meet objectives. Of these, twenty 

four studies reporting on 33 objectives were evaluated at greater than 30 days.14 16 43 56 70 73 74 96 99 

101 119 122 126-137 Two studies with evaluation duration of 30 days or less did not meet objectives.18 

138 Three studies did not report evaluation duration and did not meet objectives.13 57 139 

Nine studies, evaluating 9 objectives, showed mixed results in terms of their objectives 

being met.20 42 71 73 88 137 140-142 One study was unclear as to whether it met objectives.101 Fourteen 

studies, evaluating 17 objectives lacked a control group and therefore did not allow us to assess 

effectiveness.23-25 59 64-67 83 143-147 

Overall, CME interventions were effective in the short- and long-term achievement of 

practice behavior objectives. 

Short-term and long-term effects of CME media methods on practice behavior. The 

different types of media evaluated included single-media live presentations (20 studies), single-

 

∗

 Appendixes cited in this report are provided electronically at http://www.ahrq.gov/clinic/tp/cmetp.htm
41 

media print materials (9 studies), internet (1 study), other single media (2 studies), multimedia 

(57 studies), and also single versus multimedia comparisons (15 studies). 

Of 20 studies using single live media, 10 studies met 11 objectives, three studies did not meet 

objectives, three studies showed mixed results and four studies did not have a control group. Of 

these 20 studies, nine studies with evaluation duration of greater than 30 days met 10 

objectives,45 48 53 72 82 102 107 121 125 suggesting that the use of single live media had a favorable 

long-term effect on practice behavior objectives. One study had an evaluation duration of 30 

days or less and met objectives using single live media.94 Three studies using single live media 

reported evaluation duration greater than 30 days but showed mixed results.20 71 140 Four studies 

lacked a control group, and no meaningful conclusions could be drawn with regard to the 

comparative effectiveness of single live media.24 25 64 66 Three studies did not meet practice 

behavior objectives using single live media. Of these, one did not report evaluation duration,57

one reported evaluation duration of 30 days or less,138 and one reported evaluation duration 

greater than 30 days.133

Out of nine studies with ten objectives that examined the impact of single print media, only 

one met objectives,15 but it did not report evaluation duration. One study did not meet objectives 

using single print media and did not report evaluation duration.139 One study with evaluation 

duration of 30 days or less did not meet objectives using single print media.18 Four studies (five 

objectives) with evaluation duration greater than 30 days did not meet objectives using single 

print media.14 56 130 134 Two studies using single print media reported evaluation duration greater 

than 30 days, but lacked a control group and no meaningful conclusions could be drawn with 

regard to the comparative effectiveness of single print media.23 143 The evidence suggests that 

single print media is not effective in the short- or long-term achievement of practice behavior 

objectives. 

The only study using single internet media reported an evaluation duration greater than 30 

days and it was unclear whether it met objectives.43 One study did not meet objectives using 

other single media and did not report evaluation duration.13 One study using other single media 

did not report evaluation duration and lacked a control group.147 One study with evaluation 

duration greater than 30 days did not report the media used and lacked a control group.145

Out of 57 studies (78 total objectives) using multimedia, 40 studies met 47 objectives, 14 

studies did not meet 19 objectives, 4 studies showed mixed results, and 4 studies lacked a control 

group. Of the 40 studies which met objectives, 31 studies with 37 objectives were evaluated at 

greater than 30 days suggesting that multimedia-based CME has a favorable long-term effect on 

practice behaviors.19 46 49 52 62 75-78 81 95 98 99 101 103 104 106 108-110 112 113 116-120 122-124 148 Nine studies met 

10 objectives using multimedia but did not report evaluation duration.32 35 36 68 69 79 87 92 93 One 

study using multimedia did not report evaluation duration; it also lacked a control group.146 Four 

studies using multimedia reported an evaluation duration greater than 30 days, and showed 

mixed results.42 73 88 142 Three studies (six objectives) with evaluation duration greater than 30 

days lacked a control group and precluded meaningful conclusions.65 83 144 Fourteen studies with 

19 objectives with evaluation duration greater than 30 days did not meet objectives using 

multimedia.16 73 99 101 119 122 126-129 131 132 135 136 The evidence suggests that multimedia may have a 

positive short- and long-term effect on practice behavior objectives. 

Out of 15 studies comparing single media and multimedia, 10 studies met 11 objectives and 

all were evaluated at an interval of greater than 30 days after the educational intervention,44 47 70 

96 97 100 105 111 114 115 suggesting that both single media and multimedia have a positive short and 

long-term effect on practice behavior objectives, and that multimedia have an advantageous 
42 

effect. One study comparing single media and multimedia did not report evaluation duration and 

lacked a control group.59 Two studies comparing single media and multimedia reported 

evaluation duration greater than 30 days and showed mixed results.137 141 One study reported 

evaluation duration greater than 30 days, but lacked a control group.67 Four studies with 

evaluation duration greater than 30 days did not meet seven objectives.70 74 96 137

One study did not report the type of media used, reported evaluation duration greater than 30 

days, lacked a control group, and was inconclusive about meeting objectives.145

The evidence suggests that both single media and multimedia may have positive short- and 

long-term effects on practice behavior, with use of multimedia being advantageous. 

Short-term and long-term effects of CME educational techniques on practice behavior. 

A total of 11 studies reporting on 12 objectives evaluated the impact of a single technique on 

practice behavior objectives. One study met objectives using a single technique but did not 

report evaluation duration.15 One study with evaluation duration of 30 days or less met objectives 

using a single technique.94 One study with evaluation duration greater than 30 days met 

objectives using a single technique.19 One study did not report evaluation duration and did not 

meet objectives using a single technique.13 One study with evaluation duration of 30 days or less 

did not meet objectives using a single technique.18 Two studies with evaluation duration of 

greater than 30 days did not meet objectives using a single technique.14 16 One study was judged 

inconclusive because of mixed results.20 Three studies using a single technique reported 

evaluation duration of greater than 30 days, but lacked a control group.23-25 Two of these three 

studies did not reach statistical significance. This suggests that a using a single technique may 

not have a short- or long-term positive effect on practice behavior objectives. 

A total of 76 studies with 98 objectives evaluated the short- and long-term impact of multiple 

techniques on practice behavior objectives. Eight studies met 11 objectives using multiple 

techniques but did not report evaluation duration32 35 36 68 69 79 92 93 and thus did not allow us to 

distinguish between short- and long-term effects of CME. Thirty-nine studies with evaluation 

duration greater than 30 days met 45 objectives using multiple techniques.44-46 48 49 52 53 62 72 76-78 81 

82 82 95 98 99 101-113 116-120 122-124 

Two studies did not report evaluation duration and did not meet objectives using multiple 

techniques.57 139 One study with evaluation duration of 30 days or less did not meet objectives 

using multiple techniques.138 Sixteen studies with evaluation duration of greater than 30 days 

reporting on 22 objectives did not meet objectives using multiple techniques.43 56 73 101 119 122 126-

129 131-136

One study using multiple techniques did not report evaluation duration and lacked a control 

group.146 One study using multiple techniques did not report evaluation duration and showed 

mixed results.149 Seven studies reporting on 10 objectives using multiple techniques reported an 

evaluation duration of greater than 30 days but lacked a control group.64 65 67 83 143-145 Six studies 

showed mixed results.42 71 73 88 140 142 One of the six showed a positive effect which was lost 6 

months after the intervention.73

The evidence suggests that the use of multiple techniques in CME may have an overall 

positive short- and/or long-term effect on practice behavior objectives. 

A total of 18 studies reporting on 24 objectives compared the use of single and multiple 

educational techniques in CME.47 59 66 70 74 87 96 97 100 114 115 121 125 130 137 141 147 148 Ten studies with 

evaluation duration greater than 30 days met 12 objectives comparing single and multiple 

techniques,47 70 96 97 100 114 115 121 125 148 and indicated that multiple techniques may have an 
43 

advantageous short- and long-term effect on practice behavior objectives. Two studies using 

single versus multiple techniques did not report evaluation duration and neither had a control 

group, precluding further meaningful conclusions.59 147 Two studies using single versus multiple 

techniques reported an evaluation duration of greater than 30 days and showed mixed results,137 

141 One study with evaluation duration greater than 30 days lacked a control group. Five studies 

with evaluation duration of greater than 30 days reporting on eight objectives did not meet 

objectives and thus were unable to identify a difference when using single versus multiple 

techniques.70 74 96 130 137

Short-term and long-term effects of the amount of exposure on practice behavior. A 

total of 37 studies reporting on 41 objectives evaluated the impact of single exposure to the CME 

activity. Two studies met objectives using single exposure, but did not report evaluation 

duration.36 68 One study with evaluation duration of 30 days or less met objectives using a single 

exposure to the CME activity.94 Sixteen studies with evaluation duration greater than 30 days 

met 18 objectives using single exposure to the CME activity.44 45 48 53 70 77 82 98 100 102-104 106 107 111 

114 This suggests that single CME exposure may have a positive short- and long-term effect on 

practice behavior objectives. Two studies with evaluation duration of 30 days or less did not 

meet objectives using single exposure to the CME activity.18 138 Only six studies with evaluation 

duration greater than 30 days did not meet objectives using single exposure to the CME 

activity.14 16 70 74 128 129 Five studies with evaluation duration greater than 30 days lacked a control 

group,24 25 64-66 although one of the five did not meet objectives for either intervention group.24 

Six studies with evaluation duration greater than 30 days showed mixed results.20 71 88 140-142

A total of 55 studies (72 objectives) evaluated the impact of multiple exposures to the CME 

activity. Five studies met objectives using multiple exposures to the CME activity, but did not 

report evaluation duration.15 32 35 69 93 Thirty studies with evaluation duration greater than 30 

days met 36 objectives using multiple exposures to the CME activity.19 46 49 52 62 72 75 76 78 81 95 97 99 

101 105 108-110 112 113 115-121 123 125 148 One study using multiple exposures to the CME activity did not 

report evaluation duration and lacked a control group.147 Five studies of evaluation duration 

greater than 30 days lacked a control group24 25 64-66 whereas two studies showed mixed results.73 

137 No meaningful conclusions could be drawn from these studies. Two studies did not meet 

objectives using multiple exposures to the CME activity, and did not report evaluation 

duration.13 57 Sixteen studies with evaluation duration greater than 30 days did not meet 21 

objectives using multiple exposures to the CME activity.43 56 73 99 101 119 126 127 130 132-137 150 Overall, 

we conclude that multiple exposure volume to the CME activity may have positive short- and 

long-term effects on practice behaviors. 

A total of eight studies reporting on 17 objectives did a head-to-head comparison between 

single and multiple exposures.42 47 59 79 83 96 122 144 Whereas four of these studies indicated that 

multiple exposures may be better than single exposure, six showed mixed or negative results, 

thus not allowing us to draw any strong conclusions. One study met objectives using single 

versus multiple exposures to the CME activity, but did not report evaluation duration.79 Three 

studies with evaluation duration greater than 30 days met objectives using single versus multiple 

exposures to the CME activity.47 96 122 

One study using single versus multiple exposures to CME activity did not report evaluation 

duration and had no control group.59 Two studies using single versus multiple exposures to 

CME activity reported evaluation duration greater than 30 days but did not have a control 

group.83 144 No meaningful conclusions could be drawn from these studies. One study using 
44 

single versus multiple exposures to CME activity reported evaluation duration greater than 30 

days and showed mixed results.42 

Two studies with evaluation duration greater than 30 days did not meet objectives using 

single versus multiple exposures to the CME activity.96 122 

One study did not meet objectives and did not report exposure to CME activity or evaluation 

duration.139

Two studies met objectives using other exposures to the CME activity, but did not report 

evaluation duration.87 92 One study with evaluation duration greater than 30 days met objectives 

using other exposures to the CME activity.124 One study using other exposure to CME activity 

did not report evaluation duration and lacked a control group.146 

Quality of the evidence for the short-term and long-term effects of CME on practice 

behavior (see Appendix F, Evidence Table 8). Taking into consideration the quantity and 

quality, and consistency of evidence on the effectiveness of CME on practice behaviors, we 

graded the strength of evidence as very low. 

Clinical Outcomes 

Short-term effects of CME on clinical outcomes (See Appendix F, Evidence Table 12). 

Only one study measured the short-term effects of CME on clinical outcomes, i.e., less than 30 

days after the educational intervention.18 This study reported on the effect of printed CME on 

adherence to beta-blocker use, and was successful in achieving the stated objective of the 

intervention. However, the effects of CME were evaluated in non-equivalent patient groups, 

rendering the results of the study inconclusive. Three additional studies did not clearly report the 

time at which clinical outcomes were measured.32 33 151 Of these, two reported that the desired 

objective of the intervention was met,33 151 and a third showed mixed results.32 The first of these 

studies reported on a direct measure of health status: depression. In that study, depression was 

improved to a greater degree among patients of physicians who had received a quality 

improvement intervention that included CME in the form of academic detailing, discussion 

groups, printed materials, and feedback, than among a control group.151 However, the effect of 

CME was difficult to separate from the effect of the quality improvement component. The 

second study addressed antimicrobial drug use with a complex statewide intervention that 

targeted education directly at patients as well as their physicians; this made the actual effect of 

the CME component difficult to estimate.33 The third study evaluated a multi-modality 

depression education program and found significantly improved patient satisfaction among 

women, but not among men.32 

Long-term effects of CME on clinical outcomes. Thirty-three studies, reporting on 42 

clinical outcomes, measured the long-term effect of CME, i.e., more than 30 days following the 

educational intervention.22 43 47 55 56 65 72-75 77 78 81 84 95 99 109 111-115 117 120 131 132 135 137 152-156 Fourteen 

of these studies were successful in achieving the desired effect of the CME intervention on 

clinical outcomes.22 43 47 74 78 84 95 109 111 115 152-155 One study showed mixed results, impacting 

frequency of office visits but not emergency room visits or hospitalizations.120 In the remaining 

23 studies, either no effect of CME was observed, or the effect was uncertain due to ambiguous 

results or problems in study design. Of the 14 studies that did show a long-term effect, six 

reported on direct measures of health status of the target patient population. These outcomes 
45 

were arthritis pain and disability,95 depression,22 152 general health and function,109 emotional 

distress,84 and lost work due to back pain.153 Eight studies reported on health-related behaviors or 

attitudes. These outcomes were: percent of patients taking medication,152 patient adherence with 

antibiotics,47 patient satisfaction with care,74 frequency of physician visits,111 hospitalizations,78

hospital length of stay,155 and smoking cessation rates.115 154 One study reported a mixed 

outcome, “quality of practice,” which combined direct measures of patient health status, such as 

whether the blood pressure was below 130/80 mm Hg, with behavioral or physician-related 

outcomes, such as whether the physician had recorded a family history of diabetes.43 

Short-term effects of CME media methods on clinical outcomes. Only one study was 

available to assess the relative effectiveness of different types of media (live, print, internet, or 

multiple) on short-term clinical outcomes (less than 30 days). In this study, a print intervention 

improved adherence with beta-blocker use.18 Five studies did not report on duration of clinical 

outcome.32 33 146 151 157 Thus, no conclusions could be drawn about the differential effectiveness 

of CME media in the short term. 

Long-term effects of CME media methods on clinical outcomes. Of the studies that had 

information about the effectiveness of different single media forms of CME on long-term clinical 

outcomes, five used a live CME intervention;72 84 152 154 155 four of these five achieved the stated 

goal of the study. Two studies used print media and neither achieved its objective.56 157 Another 

used Internet-based CME and did achieve its objective.43 Most of the studies, however, used 

multiple CME media. Twenty-two studies used multiple media CME in comparison to a 

control.55 65 73 73 75 75 77 78 81 95 99 109 112 113 117 120 120 131 131 132 135 156 In four of these, the study 

achieved its stated aim. Seven studies compared multiple media CME to single media CME.47 74 

111 114 115 137 153 Six of these achieved the stated aim; each found multiple media CME to be more 

effective than single media in improving clinical outcomes. 

Short-term effects of CME educational techniques on clinical outcomes. A total of 15 

different educational techniques were identified in the studies that reported clinical outcomes: 

readings, conference calls, academic detailing, discussion groups, lectures, point of care CME, 

feedback, physician visits, case-based learning, role-play, standardized patients, demonstrations, 

clinical experiences, simulation, and problem-based learning. Only one CME technique was 

evaluated singly in comparison to control for short-term clinical outcomes: provision of 

educational readings was associated with increased use of beta-blockers.18 

Long-term effects of CME educational techniques on clinical outcomes. Only one study 

evaluated the effect of a single CME technique in comparison to control for long-term clinical 

outcomes. In this study, the use of conference calls was associated with improvements in 

depression.22 With only one such study, no conclusions could be drawn regarding the 

comparative effectiveness of single CME techniques on long-term clinical outcomes. Most of the 

studies evaluated used multiple CME techniques. Thirty-eight studies reported on the use of 

multiple simultaneous CME techniques in comparison to control. Twelve of these reported that 

the desired clinical outcome of the CME intervention was achieved.43 47 74 78 84 95 109 111 115 152 153 155

Two studies yielded mixed results, i.e., some of the outcomes showed improvement while others 

did not.120 151 No individual CME techniques were common to the studies that did or did not 

achieve their stated objective; thus, one cannot draw any conclusions regarding the differential 
46 

effectiveness of specific educational techniques. Five studies compared single to multiple CME 

interventions.47 74 115 137 152 In three of the five studies, the use of multiple simultaneous CME 

techniques was superior to the use of a single CME technique (readings). 

Short-term effects of the amount of exposure on clinical outcomes. Only one study 

assessed the short-term effect of CME on clinical outcomes (less than 30 days). In this study, a 

one-time print intervention improved adherence with beta-blocker use.18 Five studies did not 

report on duration of clinical outcome,32 33 146 151 157 and all other studies reported long-term 

outcomes. Thus, no conclusions could be drawn about the differential effectiveness of amount of 

CME exposure (one-time vs. multiple exposures) on short term outcomes. 

Long-term effects of the amount of exposure on clinical outcomes. Seven studies 

evaluated the long-term effect of a single CME exposure on clinical outcomes.65 74 77 111 114 152 154

Four of these studies reported that the CME objective had been met.74 111 152 154 In one study, the 

objective was not met.114 Of the remaining two studies, one lacked a well-defined control group65

and one yielded unclear results.77 Most studies employed multiple CME exposures. In 24 studies, 

the multiple CME exposures were compared to a control with no CME.22 43 55 56 72 73 75 75 78 81 84 95 

99 109 112 113 115 117 120 131 132 135 137 156 In seven of the studies, the objectives were met.22 43 78 84 95 109 

115 In 16 studies, the objectives were not met or it was unclear if they were met.55 56 72 73 75 75 81 99 

112 113 117 131 132 135 137 156 One study produced mixed results, as described above.120 In one study, 

a single printed CME intervention and a combination of printed material plus tutorial were 

compared with control.47 While the CME intervention was deemed successful in changing 

clinical outcomes, the use of multiple CME exposures (tutorial plus reading) outperformed a 

single CME exposure (reading alone) in only 2 of 5 outcomes studied. In summary, both onetime and multiple exposure CME interventions have produced changes in clinical outcomes in 

about half of the studies, but it is unclear whether multiple exposure CME produces better results 

than one-time CME. 

Quality of the evidence for the short-term and long-term effects of CME on clinical 

outcomes (see Appendix F, Evidence Table 8). Taking into consideration the quantity and 

quality, and consistency of evidence on the effectiveness of CME on skills, we graded the 

strength of evidence as low. 

Key Question 3: What is the evidence from systematic 

reviews about the effectiveness of simulation methods in 

medical education outside of CME? 

Characteristics of the Systematic Reviews 

The nine systematic reviews that met the inclusion criteria were a heterogeneous group (see 

Appendix F∗

, Evidence Table 13). All included systematic reviews were published between 1990 

and 2006. Eight of the reviews evaluated the role of simulation in skill acquisition, while two 

reviews28 158 evaluated the role of simulators in knowledge acquisition. Out of the eight reviews 

 

∗

 Appendixes cited in this report are provided electronically at http://www.ahrq.gov/clinic/tp/cmetp.htm
47 

that evaluated the role of simulation in skill acquisition, five reviews26 29-31 159 addressed the 

effectiveness of simulation in training procedural or surgical skills, two reviews158 160 addressed 

the effectiveness of simulation in training on communication skills, and two reviews158 159

examined the effectiveness of simulation in training on physical examination skills. One study27

addressed the general features and uses of high-fidelity simulation in effective learning. Five 

reviews26 27 31 159 160 clearly stated the study question to allow replication of the results, while 

four other reviews28-30 159 only partially stated the study question. 

Types of simulation studied. A wide variety of simulation-based methods were identified 

in these reviews. Two reviews26 30 included studies that had evaluated virtual reality, two 

reviews31 159 included studies with full simulation, three reviews28 158 160 included studies with 

standardized patients or role play, five reviews29-31 158 159 included studies with partial task 

simulation, and six reviews28-31 158 159 included studies with computer simulation. All reviews 

had studies that compared simulation-based training with another type of simulation-based 

training, other educational intervention, standard training, no education, or no training. 

Study populations and study designs. Three reviews26 158 160 restricted inclusion criteria to 

studies that had enrolled only medical students or physicians-in-training. Other reviews also 

included studies that enrolled, in addition to medical students and physicians-in-training, fullytrained physicians, nurses, allied health professionals, and non-medical personnel. One review31

included only randomized controlled trials, while other reviews included non-randomized 

controlled trials and prospective trials in addition to randomized controlled trials. 

Effectiveness of Simulation in Teaching Procedural Skills 

Since the reviews often included a variety of studies, depending on the focus of the review, 

each systematic review was further broken down by the number of studies which addressed the 

effectiveness of simulation as an educational method (see Table 5 and Appendix F, Evidence 

Table 14). 

Table 5. Quantity and direction of evidence for effectiveness of simulation 

Learning objectives 

Number of reviews 

which addressed 

learning objective 

Number of studies 

which addressed 

effectiveness of 

simulation as an 

educational method Direction of evidence 

Psychomotor Skills 626 29-31 158 159 63 Favors simulation 

Communication Skills 2158 160 14 Favors simulation 

Cognitive Skills 228 159 37 Mixed results 

Three reviews evaluated the effectiveness of virtual reality in teaching surgical skills. In 

virtual reality, the surgical field is represented in three dimensions which may help in learning 

more accurate surgical planning and procedures. One meta-analysis26 found that training in a 

virtual reality environment significantly decreases the total amount of time required for task 

completion. There was also a trend toward a decreased error rate which did not reach a 
48 

statistically significant level. Another systematic review29 found four studies that evaluated the 

role of virtual reality in training surgical techniques. Two studies in this review found 

improvement in surgical skills after training with a virtual reality simulator while two other 

studies found no significant improvement after training with virtual reality simulators. A third 

systematic review31 found that trainees trained on computer simulation perform better than those 

who received no training, however, studies found an inconsistent benefit of computer simulation 

if simulation-trained students were compared with those who received standard training. This 

systematic review found only one study in which computer simulation was found to be superior 

to a physical training model. On the other hand, this review found that physical or model 

simulation may be superior to no training and standard training as instructions from mentors or 

manuals. 

Video simulation was studied in one review.31 Video simulation was not superior to standard 

training or no training, and there was insufficient evidence to support the superiority of computer 

simulation to video simulation. 

One review30 evaluated the effectiveness of simulators for training in gastrointestinal 

endoscopy and concluded that flexible sigmoidoscopy can be applied for clinical training of 

residents and fellows for better patient comfort only. However, this review did not find enough 

evidence to support the use of simulators for clinical training in gastrointestinal endoscopy to 

improve clinical outcomes. 

Another review159 found one study with a cross-over design that reported a better post-test 

score (on a 22-object written test) of the anesthesia residents who were trained on a simulator as 

compared to those who were not trained. 

Effectiveness of Simulation in Teaching Physical Examination 

Two systematic reviews evaluated the effectiveness of simulation in teaching physical 

examination. One review158 found that use of standardized patients to teach breast examination to 

medical students was associated with better performance in a clinical skills examination. This 

review further found that use of standardized patients or breast examination models was 

associated with improved ability of the students to detect breast lumps. The second review159

found that use of a patient simulator was associated with improved practical skills as measured 

on a post-test examination. 

Effectiveness of Simulation in Teaching Communication Skills 

Two reviews evaluated the effectiveness of simulation in teaching communication skills. One 

review158 found that when students were taught communication skills by patients with cancer, 

including training for giving bad news, students were more likely to respond empathetically to 

patients and better able to communicate bad news than students who were not taught 

communication skills by patients with cancer. This review further found that role-playing may be 

important in teaching oncology-specific communication skills as well as in communicating bad 

news. The second review160 found that use of standardized patients and role-play was effective in 

teaching medical students smoking cessation counseling skills. This review further found that the 

use of standardized patients in teaching tobacco cessation skills to medical students was 

associated with increased confidence of students in their smoking cessation counseling skills. 
49 

Effectiveness of Simulation in Knowledge Acquisition 

We found two reviews28 159 that addressed the effectiveness of simulation in knowledge 

acquisition. Hmelo28 found that computer assisted models are effective in teaching 

pathophysiologic principles to medical students. The pooled effect size, which measures the 

combined magnitude of the effect of intervention across 33 studies, was 0.63 for use of computer 

assisted learning (in favor of simulation). One study included in Ravert159 looked at computerbased trauma simulation to teach trauma management; individual-study effect sizes ranged from 

-0.04 to 0.35 (did not favor simulation or were neutral). This review159 did not report on the 

pooled effect size. 

Features of High-Fidelity Simulators for Effective Learning 

One review27 systematically evaluated the features of high-fidelity simulators essential for 

effective learning. This review found that the following features of a high-fidelity simulator were 

important for effective learning: should provide feedback during learning experience, allow 

repetitive practice, can integrate in overall curriculum, has increasing levels of difficulty, is 

adaptable to multiple learning strategies, can allow clinical variation in a simulated environment, 

provides controlled environment to make and detect mistakes without consequences, provides 

individualized and standardized learning, defines outcomes clearly, and must have face validityrealism of simulator. 

Summary 

Overall the direction of evidence points to the effectiveness of simulation training, especially 

in psychomotor skills (e.g., procedures or physical examination techniques) and communication 

skills, but the strength of the evidence was considered low, due to the small number of 

appropriate studies, the scarcity of quantitative data, and other limitations. Several factors may 

be responsible for the inadequate quality of evidence in support of this method. In our view the 

most important factor is the lack of widely-accepted and standardized methods to quantify the 

competency in procedural or communication skills. In addition, the high cost of simulation 

methods and difficulty in introducing clinical realism in a simulated environment are other 

factors that may be responsible for inadequate quality of evidence in this field. 

Key Question 4: Which characteristics of the audience by 

themselves or in combination with other characteristics 

influence the effectiveness of certain educational 

techniques? 

Key Question 5: Which external factors by themselves or in 

combination with other factors reinforce the effects of CME 

in changing behavior? 
50 

We sought to evaluate the impact of both internal audience characteristics and external 

factors on the effectiveness of CME. Certain audiences may respond differently to an 

educational intervention, and it would be important to determine the most effective means to 

educate these subgroups. Specifically, we were interested in whether certain audience 

characteristics, such as gender and years in practice, might influence the effectiveness of certain 

educational techniques. Likewise, we wished to determine whether the presence of specific 

external factors or incentives might affect the short-term and long-term outcomes of CME 

activities. Both types of factors could impact future CME course design. Table 6 contains a list 

of the relevant internal and external factors that we considered. 

Table 6. Internal audience characteristics and external factors examined in the review 

Internal Audience Characteristics External Factors 

 Age 

 Gender 

 Practice setting 

 Years in practice 

 Specialty 

 Foreign vs. U.S. medical graduate 

 Country of practice 

 Personal motivation 

 Non-monetary rewards/motivations 

 Learning satisfaction 

 Knowledge enhancement 

 Regulation 

 State licensing board 

 Professional boards 

 Hospital credentialing 

 External audits 

 Monetary/financial rewards 

 Academic advancement 

 Provision of tools 

 Public demand/expectations 

 CME credit 

Audience Characteristics 

Thirteen studies examined the influence of audience characteristics on the educational 

intervention that was evaluated.33 39 53 61 69 70 79 80 92 101 105 140 161 In only one of these studies was 

evaluation of these characteristics considered the primary goal of the paper.105 Evidence Table 

15 contains a description of the studies reviewed, grouped by the audience characteristic 

examined. We excluded characteristics that were unique to individual outcome measures (e.g., 

Medicaid practice size, nursing home practice size) and instead focused on features of the 

audience that were evaluated more frequently (see Table 6). We also specifically searched for 

articles that described personal motivation factors (e.g., knowledge enhancement and nonmonetary rewards) in their analysis but could not find any. 

Six studies examined the effect of years in practice on the educational intervention.33 53 79 80 92 

140 Beaulieu, et al. suggested that physicians with less than 11 years of experience ordered fewer 

unnecessary screening tests than those with more experience, however, these results were not 

stratified by the educational technique actually employed by individual physicians.140 Two other 

studies suggested that physicians with greater experience who underwent educational 

interventions had improvements in attitudes33 and self-reported practice behavior.53 However, 

none of the studies examined revealed a relationship between years in practice and acquiring 

knowledge, acquiring skills, or changing practice outcomes. Similarly, age had no influence on 

the outcomes of educational interventions in the 6 studies that reported on the effects of age.39 79 

80 101 105 161

Of the five studies39 53 79 80 161 that analyzed the effect of gender, only one showed a 

significant association. Leopold, et al. showed that women improved more than men in 
51 

confidence with an objective performance of knee joint injections after an educational 

intervention consisting of printed material, hands-on instruction, or video instruction.80 Only one 

paper described the influence of race on the effectiveness of the educational intervention.79 

Grady, et al. suggested that non-whites who underwent a presentation on mammography 

screening followed by cue enhancement (i.e., chart stickers and clinic posters) improved their 

screening rates more than whites.79 This study also suggested that the intervention had a greater 

effect on solo practitioners compared to those in other practice settings. Three studies that 

examined the influence of board certification on educational outcomes primarily focused on 

internists and family practitioners and failed to show an association between certification and the 

desired outcome.53 79 105 

Conclusions and Limitations. We cannot reach definitive conclusions regarding the 

influence of audience characteristics on the effectiveness of specific educational techniques due 

to the heterogeneity of the educational interventions and characteristics examined. Furthermore, 

there are very limited data regarding any specific characteristic and the overall quality of the 

existing data on these questions are suboptimal. 

External Factors 

The literature is limited about the role of external factors, by themselves or in combination 

with other factors, in reinforcing the effects of CME on changing behavior. Very few studies 

explicitly stated that such factors were examined independently or collected data regarding these 

factors. Only one study rigorously examined external factors as a primary outcome.79 Grady et 

al. studied whether token monetary rewards in addition to an educational intervention and chart 

cues increased the rate of mammography referral. While chart cues were effective in increasing 

mammogram referral and completion rates, the addition of a token monetary incentive provided 

no added benefit. 

The offering of CME credit would not intuitively appear to be an external motivating factor 

for behavior change. Yet, two studies specifically examined the role of offering CME credit for 

this purpose, so it was included in our analysis. Both of these studies looked at results in 

association with earning CME credit for the educational activity.106 156 Chassin and colleagues 

examined in a subgroup analysis the effect of offering CME credit for attendance at educational 

programs designed to decrease inappropriate x-ray pelvimetry rates in 64 study hospitals.162 

They found no significant difference among intervention participants with respect to the offering 

of CME credit; both groups had a comparable decrease in pelvimetry use. Messina et al. found a 

potential association of offering CME credit for a physician educational program with an 

increase in the use of screening mammography in women who had never undergone 

mammography, but it did not reach statistical significance. The trend did not hold true for 

previous mammogram users.156 These findings may be because CME credit may be an 

inducement to attend a CME activity but may not be sufficient to engage the participant in active 

learning. 

Some CME courses utilize the signature of a “commitment-to-change” statement as an 

external motivating factor to improve clinical outcomes. Two studies examined the effectiveness 

of such a practice.161 163 Mazmanian and colleagues randomized 110 physicians to signature 

versus non-signature groups. While they found that those expressing an intent to change were 

more likely to change practice behavior (as documented by self-report on a follow-up survey), 

the act of signing such a commitment-to-change statement had no effect.161 In a much smaller 
52 

study of 16 physicians attending a geriatrics course, Pereles et al. found that the physicians who 

were asked to make written commitments for practice changes (n=7) made more changes than 

controls at both one and three months followup. The results are of unclear statistical significance 

given the small study numbers.163 

Conclusions and Limitations. There are several barriers to collecting data on external 

factors. First, it is methodologically difficult to offer incentives (such as CME credit or financial 

reward) in a controlled fashion. Second, most evaluation of external factors is based on selfreport. Finally, small study sizes often preclude a valid analysis of external factors in subgroup 

analyses due to lack of adequate power. Consequently, it is difficult to draw conclusions 

regarding the effectiveness of external factors in enhancing CME effects on behavior. 

Key Question 6: What is the reported validity and reliability of 

the methods that have been used for measuring the effects 

of CME in terms of a) imparting knowledge, b) changing 

attitudes, c) acquiring skills, d) changing practice behavior, 

or e) changing clinical practice outcomes? 

Background 

Valid and reliable evaluation tools are necessary to demonstrate the effectiveness of CME 

interventions. The validity of the evaluation method is “the degree to which the method truly 

measures what it is intended to measure.”11 A valid evaluation method accurately measures 

achievement of the stated objective of the educational intervention, whether it involves 

knowledge, attitudes, skills, practice behaviors, or clinical outcomes. The reliability of the 

evaluation method is “the consistency or reproducibility of measurements.”11 A reliable 

evaluation method allows educators to have confidence in their assessments of learning across 

multiple measurements. As one measurement expert emphasizes, “small amounts of unreliability 

may cause misclassification errors and large score differences on retesting.”164

An evaluation method may be statistically reliable without being valid for the objective 

intended by the investigators. However, a method cannot be valid without being reasonably 

reliable. Thus, Downing argues that “reliability is a necessary but not sufficient condition for 

validity, and reliability is a major source of validity for all assessments.”164

An educational study may employ a newly created evaluation method or one that previously 

has been shown to be valid and/or reliable in another study population. The creation of a new 

evaluation method consumes time and resources for pilot testing, cognitive testing, and 

psychometric analyses to determine the validity and reliable of the new method. However, a 

previously used method may not be a valid measure for a new educational intervention if it does 

not map appropriately to the stated objective. Also, the reliability of a method changes as it is 

applied in different populations and ideally should be re-measured each time. 

Results 

We found reports of the validity and/or reliability of at least one evaluation method in 46 

out of 136 total articles (33.8 percent). Among these 46 articles, 11 reported on the validity 
53 

and/or reliability of more than one method: eight studies described two methods;42 43 48 52 60 72 137 

144 two studies described three methods;55 84 and one study described four methods.32 Thus, 61 

evaluation methods were accompanied by validity or reliability data. For the results below, 

percentages are based on the total number of methods – rather than articles – since some articles 

reported on multiple methods. 

Among these 61 evaluation methods, 30 (49.2 percent) were drawn from previous studies 

and 28 (45.9 percent) were created for the current studies. For 3 methods (4.9 percent), it was not 

clearly reported whether the method was newly created or previously used. For 22 of the 30 

previously used methods, the authors reported that reliability had been assessed: 13 within the 

current study population, 8 within previous study populations, and 1 within current and previous 

study populations. However, only 14 methods were presented with specific statistical data to 

support this reliability. For 12 of the 30 newly created methods, the authors reported that pilot 

and/or cognitive testing was performed. 

Appendix F∗, Evidence Table 16 presents the 61 evaluation methods, organized by type of 

outcome. 

Knowledge or cognitive skills were evaluated by 15 methods (24.5 percent). 

Attitudes were evaluated by seven methods (11.5 percent). Two methods focused exclusively 

on attitudes, while five methods evaluated a combination of attitudes and knowledge / cognitive 

skills. 

Skills (communication, psychomotor, or procedural) were evaluated by 11 methods (18.0 

percent). One method evaluated physical exam skills in an educational setting. A combination of 

skills (communication, psychomotor, or procedural) and practice behaviors were measured by 10 

methods, using standardized patients to visit physicians at their practice setting or analyzing 

interactions with real patients. 

Practice behaviors (without clinical outcomes) were evaluated by 20 methods (32.8 percent). 

Seven methods used self-report by physicians of their practice behaviors. Three methods used 

patients’ report of their physicians’ behaviors in their medical care. Ten used chart review of 

medical records and/or claims data. 

Clinical outcomes (with or without practice behaviors) were evaluated by 8 methods (13.1 

percent). Two studies used chart review. One used reports by patients or families of their 

attitudes. One study used patient satisfaction. Two studied patient reports of their own behavior 

– including medication adherence and participation in preventive screening – as the outcome. 

One study used patients’ reports of preventative services provided by their physicians. Three 

used measures of the patient’s health. 

The following articles provide notable examples for reporting of the validity or reliability of 

educational outcome measures: 

• Knowledge or cognitive skills: Fordis et al42 describes the development of a knowledge test 

for cholesterol management, including: content validation by experts; description of the test 

and response options; piloting and item number reduction; and high internal consistency 

reliability (Cronbach alpha = 0.79). 

• Attitudes: Mann et al52 describes the development of a knowledge and attitudes test about 

cholesterol management, including: content validation by experts; need for >90 percent 

agreement on question inclusion and consistency; description of the test and response 

 

∗

 Appendixes cited in this report are provided electronically at http://www.ahrq.gov/clinic/tp/cmetp.htm
54 

options; pilot testing with internal consistency testing in the pilot population (KR20 = 0.60); 

and test-retest reliability analysis using control group test scores. 

• Skills (communication, psychomotor, or procedural): Roter et al84 describes a coding system 

to rate physicians’ proficiency in managing standardized patients’ emotional distress, 

including: blinding of coders; dichotomous coding system; and internal consistency 

reliability (Cronbach alpha = 0.20-0.62 for shorter scales, 0.76-0.81 for longer scales, and 

0.62 for overall score). 

• Practice behaviors: Sibley et al136 describes chart abstraction for quality of care, including: 

content validity of pre-determined criteria by experienced clinicians; rating system; training 

of blinded nurse-abstractors; and high interrater and intrarater reliability (kappa >0.8). 

• Clinical outcomes: Roter et al84 describes use of the General Health Questionnaire-28 to 

detect emotional distress in patients, including: citation of original source for this previously 

validated questionnaire; repeated reliability testing with the study population, yielding high 

internal consistency (Cronbach alpha = 0.90-0.92). 

Table 7 presents the number of evaluation methods for each type of validity or reliability, as 

well as tallies for each type of outcome within each validity or reliability. Of note, methods may 

capture more than one type of outcome and may be repeated in more than one column for each 

row. Of 61 evaluation methods with validity or reliability reported, 16 (26 percent) included 

descriptions of validity alone, 29 (48 percent) included descriptions of reliability alone, and ten 

(16 percent) had descriptions of both validity and reliability. For six methods (10 percent), the 

methods were described as valid and/or reliable, but the specific type of validity or reliability 

was not reported.

Validity was reported for 31 of 61 evaluation methods (50.8 percent). Content validity was 

reported for 16 methods. The specific “experts” who reviewed the assessment were reported for 

11 of these 15 methods. Concurrent criterion validity was reported for 8 methods. Predictive 

criterion validity was reported for only 1 method, which involved a comparison between 

physicians’ reports of asthma management behaviors and patients’ reports of physician 

behaviors. Construct validity was reported for 5 methods, usually through “known-group 

validity” (establishing construct validity by demonstrating better scores among those with higher 

levels of training or clinical experience). High statistical validity was only demonstrated for two 

methods.72 137 Five methods were described as valid without specific details. Thus, the vast 

majority of CME studies offered no or limited psychometric data for the validity of their 

evaluation methods. 

Reliability was reported for 43 of 61 evaluation methods (70.5 percent). Internal consistency 

reliability was reported for 19 methods, including 13 learner instruments, 1 observer instrument 

for audio-taped interactions, 1 standardized patient instrument, and 3 clinical patient instruments. 

Inter-rater reliability was reported for 16 methods, including 9 medical data abstractions and 6 

skills assessments. Intra-rater reliability was assessed for 2 medical data abstraction studies. 

Equivalence reliability was reported for 4 methods, and test-retest reliability for 5 methods. Four 

methods were described as reliable without specific details. When reported, statistical tests 

yielded primarily modest evidence of reliability based on Cronbach-alpha, Kappa, or correlation 

statistics. 
55 

Conclusions 

• Forty-six of 136 articles (34 percent) reported the validity and/or reliability of at least one 

evaluation method for assessing the effectiveness of CME. 

• Thirty methods were drawn from previous studies, and 28 were created for the current 

studies. For 3 methods, the source was unclear. Authors did not commonly report reliability 

testing within the new study population for methods found to be reliable in other populations.

• The most common type of outcome evaluated by valid and/or reliable evaluation methods 

involved practice behaviors, for 20 out of 61 methods (34 percent).

• Of 61 evaluation methods with validity or reliability reported, 16 (26 percent) included 

descriptions of validity alone, 29 (48 percent) included descriptions of reliability alone, and 

ten (16 percent) had descriptions of both validity and reliability. For six methods (10 

percent), the methods were described as valid and/or reliable, but the specific type of validity 

or reliability was not reported.

• Among these 61 methods, content validity was the most commonly reported type of validity 

(26 percent).

• Among these 61 methods, internal consistency (31 percent) and inter-rater (28 percent) were 

the most common types of reliability reported.

Although many studies of the effectiveness of CME have considered the validity or 

reliability of their evaluation methods, relatively few studies have used methods that have strong 

evidence of both construct and criterion validity. In addition, relatively few studies have used 

evaluation methods that have strong evidence of each of the specific types of reliability (internal 

consistency, inter-rater, intra-rater, equivalence, and test-retest). We therefore conclude that the 

overall strength of evidence on the effectiveness of CME is limited by weaknesses in the 

evaluation methods that have been used. To strengthen the evidence base on the effectiveness of 

CME, it will be necessary to commit additional resources to the development of valid and 

reliable evaluation methods. This may be quite challenging because of the limited resources that 

generally are available to clinician-educators. Where appropriate, educators may save time and 

resources by using previously validated and reliable methods, but they must demonstrate the 

validity of these methods for their specific educational outcomes and the reliability of these 

methods for their particular study populations. 
56 

Table 7. Number of methods evaluated for each type of validity or reliability, organized by outcome type 

Type of Validity 

Definition* # of 

Methods

Knowledge or cognitive skills†

Attitudes† Skills 

(communication or psychomotor)†

PractBeha

Content Degree to which an instrument accurately 

represents the skill or characteristic it is designed to measure, based on people’s experience and available knowledge 

16 1521 32 36 38 40 

42 46 48 51-54 57 85 142

351-53 0 253 13

Concurrent criterion 

Degree to which an instrument produces the same results as another accepted or proven instrument that measures the same variable 

8 1142 182 125 616 25 

144

Predictive criterion 

Degree to which a measure accurately predicts expected outcomes 

1 0 0 0 178 

Construct Degree to which an instrument measures the 

theoretical construct it intends to measure 

5 336 43 85 0 0 253 10

Type of Reliability 

Definition # of 

Methods

Knowledge or cognitive skills †

Attitudes† Skills 

(communication or psychomotor)†

PractBeha

Internal consistency 

How well items reflecting the same construct yield similar results 

19 1236 40 42 45 48 

49 52 53 55 63 64 85

845 49 52 53 55 

62-64

232 84 632 45 

84

Inter-rater Degree to which measurements are the same 

when obtained by different persons 

16 185 0 632 60 84 88 138 140 1332 4

108 118 136 138 

Intra-rater Degree to which measurements are the same 

when repeated by the same person 

2 0 0 0 248 13

Equivalence Degree to which alternate forms of the same 

measurement instrument produce the same results 

4 317 36 54 0 0 153 

Test-retest Degree to which the same test produces the 

same results when repeated under the same conditions 

5 352‡53 252 53 0 322 53 

Validity: not specifically reported 

 4 0 172 332 43 72 332 72 

Reliability: not specifically reported 

 5 160 0 132 232 12

*Definitions were obtained from Reed, et al.11† Methods referenced may target more than one type of outcome and are listed under each applicable column.

‡ Two assessment methods within the same article. 
57 

Chapter 4. Discussion 

Conclusions 

We conducted a systematic review of the medical literature to evaluate the effectiveness of 

CME in improving knowledge, attitudes, skills, physician behavior and clinical outcomes. 

Overall, despite the generally low quality of the evidence, most of the studies reviewed suggest 

that CME is effective, at least to some degree, in not only achieving, but also in maintaining the 

objectives studied. Despite the wide variety of CME techniques, media, exposures used, and 

despite the heterogeneity of the studies reviewed, we found common themes among studies 

which applied across objectives. For example, when assessing the effectiveness of CME across 

domains, print media seem to be less effective than live media, and multimedia generally seem to 

be more effective than single media. In addition, interactive techniques seem to be more 

effective than non-interactive ones, and multiple exposures to the CME activity seem to be more 

effective than single exposure. Thus, the evidence supports consideration of these attributes of 

effective educational interventions when designing a CME course. 

To ascertain whether broader lessons could be drawn from the non-CME medical education 

realm, we evaluated the effect of simulation methods in medical education by conducting a 

review of systematic reviews. Although we found that simulation training generally was 

effective, especially in the dissemination of psychomotor skills (e.g., procedures or physical 

examination techniques), studies which examined simulation did not review outcomes along the 

entire continuum of domains (i.e., knowledge through clinical outcomes), and were 

heterogeneous enough that few other conclusions could be drawn. 

 We also studied whether certain internal (audience) and external characteristics or factors, 

special to the environment, the participants or the course, may affect the effectiveness of CME. 

We found that the small and heterogeneous studies available did not allow us to reach definitive 

conclusions regarding the influence of audience characteristics or external factors on the 

effectiveness of CME. This is an area where further study might yield useful results in asking 

whether it might be important to marry the CME activity offered with those particular 

characteristics which might enhance its effectiveness and value. 

Limitations 

This evidence report has a number of limitations. First, the heterogeneous nature of the 

studies inhibits a quantitative summary of the effectiveness of CME. There is a lack of 

standardization of the definition of CME or associated performance improvement. The 

educational interventions studied targeted different types of audiences, using multiple types of 

objectives, across diverse content areas. Thus, comparing the effectiveness of educational 

methods and techniques across studies is challenging. Even if multiple studies shared 

comparable objectives, we found that authors did not use standardized reporting of results – such 

as effect sizes – prohibiting a quantitative meta-analysis of the results. Given these limitations, 

we had to pursue a qualitative synthesis of the available data. 

Second, the generally low quality of study designs limits our ability to draw firm conclusions 

about the effectiveness of CME. Although we limited the review to studies with comparison 
58 

groups, including a large number of randomized control trials, many studies lacked adequate 

descriptions of randomization methods or techniques for adjusting for baseline group differences. 

Moreover, too many of the articles we studied were published with comparison groups but did 

not have a control group, which did not allow us to evaluate effectiveness. In addition, only onefifth of the studies described blinding of those evaluating the outcomes, leaving open the 

potential for biased assessment. 

Third, the quality of reporting was variable. Authors rarely described the study design and 

the interventions in enough detail to allow reproducibility. In particular, studies rarely described 

specific learning objectives, prohibiting assessment of whether objectives matched appropriately 

to the evaluation methods/outcomes. 

Fourth, the lack of valid and reliable CME evaluation tools leaves open the possibility of 

overestimation or underestimation of the effectiveness of CME. Most studies lacked 

psychometric data regarding their evaluation methods. Thus, the evaluation methods may not 

have truly assessed the outcomes targeted by the educational interventions. In addition, 

evaluation methods with poor reliability may fail to detect actual improvements in outcomes. 

Fifth, our search strategy may be subject to some publication bias. Our search was limited to 

published English-language articles about educational studies within the United States and 

Canada. Our review does not include studies from other countries where high quality CME 

studies have been conducted. As indicated in the Methods chapter, this methodological choice 

was made because the medical education systems in other countries are very different from the 

system in the United States, thereby limiting the applicability of such studies to CME in the 

United States. Also, educational studies with negative findings are less likely to be published, 

potentially leading to overestimation of the effectiveness of CME. 

Sixth, there is a lack of standardization of approaches to CME research in general, i.e., no 

Phase 1, Phase 2, Phase 3-like process that crafts an organized approach to how aims are set up 

and how comparative groups are organized. This includes the lack of standardization for 

definitions of controls. 

Seventh, there is general lack of standardization of terminology related to media, techniques, 

exposure volume, etc, which makes studying the impact of different methods, techniques, 

exposures etc. on the effectiveness of CME difficult. 

Finally, several limitations were specific to particular key questions: 

For Key Questions 1 and 2, this report does not systematically review the effectiveness of 

quality improvement interventions. Although we included quality improvement studies if they 

included a physician education component, our search strategy did not systematically target all 

quality improvement studies. Thus, we cannot draw definitive conclusions comparing the 

effectiveness of physician education in quality improvement interventions versus quality 

improvement interventions more generally. 

 Moreover, our conclusions are limited secondary to the heterogeneity of studies included 

along multiple domains. In addition, many of the studies lacked a clear control group, which did 

not allow effectiveness of CME to be determined, but only different effectiveness across 

different interventions. 

For Key Question 3, conclusions are limited due to the weaknesses of the systematic reviews 

available, the poor quality of many of the included studies, the heterogeneity of included studies, 

and the rapidly evolving nature of computerized simulation. 

For Key Questions 4 and 5, there was lack of standard definitions of internal and/or external 

factors that might impact CME. In addition, conclusions are limited due to small sample sizes 
59 

prohibiting analysis based on within-group characteristics and infrequent collection of data on 

external and internal motivating factors. In addition, lack of standardization of tools to assess the 

efficacy of CME inhibited our ability to draw firm conclusions. 

For Key Question 6, conclusions are limited due to inconsistent reporting of validity and 

reliability for evaluation methods drawn from previous studies; we may have missed some valid 

or reliable evaluation methods that were not described as “valid” or “reliable” and for which 

psychometric data was not reported. 

Additional limitations for Key Question 3 include: 

a) With the exception of Hmelo,28 these reviews were quite recent (2002-2006), and point to 

evolving educational methods. Virtual reality in 2001 may be difficult to compare to 

virtual reality in 2006. The computer assisted instruction described in the review by 

Hmelo28 is already dated. 

b) No review included tests for publication bias, which would be highly anticipated with any 

new technology. 

c) Simulation can include a variety of tasks and procedures, with varying lengths and 

complexities. Some studies included partial task simulators with complex surgical and 

endoscopic procedures; pooling such disparate skills may be inappropriate. 

d) One of the major advantages of simulation over standard medical education training for 

procedures should be the opportunity to practice and receive feedback in a shorter period 

of time. No study explored this important contributor to effectiveness, i.e., the frequency 

and intensity of the simulation method and whether there is a “dose-response” effect with 

the use of simulation and the outcome of clinical skills competence. This aspect was 

assessed by a systematic review published after our literature search which found an 

association between hours of practice on high-fidelity simulators and standardized 

learning outcomes.165 

e) There is no consensus on the appropriate outcome measures for effectiveness of 

simulation. Haque,26 Sutherland,31 Gerson,30 Aucar,29 and Issenberg27 all included 

validity studies within their reviews, but heterogeneity of tasks and simulators again 

makes it difficult to pool results. 

f) Although nearly every review included a careful description of search methodology, most 

fell short in nearly every quality measure of a systematic review. The exception, 

Issenberg27 is an example of a high quality systematic review of an educational topic; the 

authors unfortunately did not address the outcome of interest to Key Question 3. 

g) For Key Question 3, systematic reviews that addressed the use of simulation in CME 

educational activities were excluded as this aspect was covered by our other Key 

Questions. It is possible that systematic reviews of the efficacy of simulation in CME 

activities may reach a different conclusion. 

Future Research Implications 

We believe that assessing those factors that make CME more or less effective will be 

important for the planning of effective CME activities in the future. Although the overall quality 

of the studies was low, there were a few important trends. CME appears to be generally effective 

not only in the acquisition or achievement of knowledge, attitudes, skills, behaviors, and clinical 

practice outcomes, but also in their retention, and there are certain techniques, methods or 

exposures which seemed to be better than others. Unfortunately, most studies did not describe 
60 

multiple evaluation points after the intervention, which did not allow us to determine at what 

point the CME effect, when persistent, became extinguishable and might have needed 

reinforcement. To enable future systematic reviews of CME, study researchers should refer to an 

excellent review by Reed et al, which summarizes guidelines for standardization in the conduct 

and reporting of educational interventions.11 

Simulation, as a teaching tool, has the potential to affect patient safety and clinical outcomes, 

but no study included in this review used a patient-based clinical outcome as a measure of 

effectiveness. Future research should seek to determine the impact of simulation in improving 

clinical outcomes. 

We believe that educators should develop strategies for identifying and prioritizing the gaps 

in our knowledge about CME that should be the focus of additional research. Future research 

should include high quality randomized controlled studies of CME with clear intervention and 

control groups and measurement of effectiveness at multiple points post-intervention. Such 

studies should focus on high priority areas given the resource constraints that educators typically 

face in conducting research on CME. Educators will need to use a variety of study designs, 

including qualitative research methods, because it will not be feasible to perform randomized 

controlled trials on many of the issues. Indeed, it will be difficult to rely too heavily on 

randomized controlled trials given the difficulty of creating and maintaining effective control 

groups. 

To advance research on CME, leaders in medical education could develop a national agenda 

on what is needed to improve the effectiveness of CME. Such an agenda should include a clear 

definition of what constitutes CME. For example, whether quality improvement or practice 

improvement alone should have been included in our evaluation presented a dilemma. We 

decided that there needed to be a well-defined educational intervention for us to include quality 

improvement or practice improvement studies in our review of the effectiveness of CME. The 

agenda for future research should include development of more standardized approaches to the 

description of CME interventions, media, techniques, and exposure volumes. Ideally, the agenda 

would be based on a sound conceptual model of what influences the effectiveness of CME, 

including participating physician perspectives. Given the large amount of time, effort and money 

invested in CME, it seems reasonable to invest in a national consensus conference that could 

help to lay the foundation for a comprehensive research agenda for CME. In addition, greater 

resources should be devoted to funding educational researchers to design higher quality CME 

studies as well as the tools to evaluate CME outcomes. 
61 

References 

 1. American Academy of Family Practice. CME 

Requirements for Members. Accessed 22 

November 2006 at 

http://www.aafp.org/online/en/home/cme/cmea/cme

requirements.html. 

 2. American Medical Association. House of Delegates 

Policy #300.988. Restoring Integrity to Continuing 

Medical Education. Accessed 12 September 2006 at 

http://www.amaassn.org/apps/pf_new/pf_online?f_n=resultLink&d

oc=policyfiles/HnE/H300.988.HTM&s_t=300.988&catg=AMA/HnE&cat

g=AMA/BnGnC&catg=AMA/DIR&&nth=1&&st_

p=0&nth=1&. 

 3. Chassin MR, Galvin RW. The urgent need to improve 

health care quality. Institute of Medicine National 

Roundtable on Health Care Quality. JAMA 

1998;280(11):1000-5. 

 4. Steinbrook R. Renewing board certification. N Engl J 

Med 2005;353(19):1994-7. 

 5. U.S. Department of Health and Human Services. 

Moving Medical Innovations Forward - New 

Initiatives from HHS, U.S. Department of Health 

and Human Services. January 2005 Accessed 

November 2006 at 

http://www.hhs.gov/reference/medicalinnovations.s

html#_Toc93371787. 

 6. Cabana MD, Rand CS, Powe NR, et al. Why don't 

physicians follow clinical practice guidelines? A 

framework for improvement. JAMA 

1999;282(15):1458-65. 

 7. Berlin JA. Does blinding of readers affect the results 

of meta-analyses? University of Pennsylvania 

Meta-analysis Blinding Study Group. Lancet 

1997;350(9072):185-6. 

 8. Jadad AR, Moore RA, Carroll D, et al. Assessing the 

quality of reports of randomized clinical trials: is 

blinding necessary? Control Clin Trials 

1996;17(1):1-12. 

 9. Kaufman DM. Applying educational theory in 

practice. Br Med J 2003;326(7382):213-6. 

10. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, 

Stroup DF. Improving the quality of reports of 

meta-analyses of randomised controlled trials: the 

QUOROM statement. Quality of Reporting of 

Meta-analyses. Lancet 1999;354(9193):1896-900. 

11. Reed D, Price EG, Windish DM, et al. Challenges in 

systematic reviews of educational intervention 

studies. Ann Intern Med 2005;142(12 Pt 2):1080-9. 

12. Atkins D, Best D, Briss PA, et al. Grading quality of 

evidence and strength of recommendations. Br Med 

J 2004;328(7454):1490. 

13. Mukohara K, Schwartz MD. Electronic delivery of 

research summaries for academic generalist 

doctors: a randomised trial of an educational 

intervention. Med Educ 2005;39(4):402-9. 

14. Bjornson DC, Rector TS, Daniels CE, Wertheimer AI, 

Snowdon DA, Litman TJ. Impact of a drug-use 

review program intervention on prescribing after 

publication of a randomized clinical trial. Am J 

Hosp Pharm 1990;47(7):1541-6. 

15. Dormuth CR, Maclure M, Bassett K, Jauca C, 

Whiteside C, Wright JM. Effect of periodic letters 

on evidence-based drug therapy on prescribing 

behaviour: a randomized trial. CMAJ 

2004;171(9):1057-61. 

16. Schectman JM, Schroth WS, Elinsky EG, Ott JE. The 

effect of education and drug samples on 

antihistamine prescribing costs in an HMO. HMO 

Pract 1996;10(3):119-22. 

17. Slotnick HB. Educating physicians through 

advertising: using the brief summary to teach about 

pharmaceutical use. J Contin Educ Health Prof 

1993;13(4):299-314. 

18. Zuckerman IH, Weiss SR, McNally D, Layne B, 

Mullins CD, Wang J. Impact of an educational 

intervention for secondary prevention of myocardial 

infarction on Medicaid drug use and cost. Am J 

Manag Care 2004;10(7 Pt 2):493-500. 

19. Cohn BA, Wingard DL, Patterson RC, McPhee SJ, 

Gerbert B. The National DES Education Program: 

effectiveness of the California Health Provider 

Intervention. J Cancer Educ 2002;17(1):40-5. 

20. Ray WA, Schaffner W, Federspiel CF. Persistence of 

improvement in antibiotic prescribing in office 

practice. JAMA 1985;253(12):1774-6. 

21. Chan DH, Leclair K, Kaczorowski J. Problem-based 

small-group learning via the Internet among 

community family physicians: a randomized 

controlled trial. MD Comput 1999;16(3):54-8. 
62 

22. Rost K, Nutting P, Smith J, Werner J, Duan N. 

Improving depression outcomes in community 

primary care practice: a randomized trial of the 

quEST intervention. Quality Enhancement by 

Strategic Teaming. J Gen Intern Med 

2001;16(3):143-9. 

23. Winickoff RN, Coltin KL, Morgan MM, Buxbaum 

RC, Barnett GO. Improving physician performance 

through peer comparison feedback. Med Care 

1984;22(6):527-34. 

24. Mazmanian PE, Daffron SR, Johnson RE, Davis DA, 

Kantrowitz MP. Information about barriers to 

planned change: a randomized controlled trial 

involving continuing medical education lectures 

and commitment to change. Acad Med 

1998;73(8):882-6. 

25. Greenberg LW, Jewett LS. The impact of two teaching 

techniques on physicians' knowledge and 

performance. J Med Educ 1985;60(5):390-6. 

26. Haque S, Srinivasan S. A meta-analysis of the training 

effectiveness of virtual reality surgical simulators. 

IEEE Trans Inf Technol Biomed 2006;10(1):51-8. 

27. Issenberg SB, McGaghie WC, Petrusa ER, Lee 

Gordon D, Scalese RJ. Features and uses of highfidelity medical simulations that lead to effective 

learning: a BEME systematic review. Med Teach 

2005;27(1):10-28. 

28. Hmelo CE. Computer-assisted instruction in health 

professions education: a review of the published 

literature. J Educ Technol. 1990;18(2):83-101. 

29. Aucar JA, Groch NR, Troxel SA, Eubanks SW. A 

review of surgical simulation with attention to 

validation methodology. Surg Laparosc Endosc 

Percutan Tech 2005;15(2):82-9. 

30. Gerson LB, Van Dam J. Technology review: the use of 

simulators for training in GI endoscopy. 

Gastrointest Endosc 2004;60(6):992-1001. 

31. Sutherland LM, Middleton PF, Anthony A, et al. 

Surgical simulation: a systematic review. Ann. 

Surg. 2006;243(3):291-300. 

32. Gerrity MS, Cole SA, Dietrich AJ, Barrett JE. 

Improving the recognition and management of 

depression: is there a role for physician education? J 

Fam Pract 1999;48(12):949-57. 

33. Kiang KM, Kieke BA, Como-Sabetti K, Lynfield R, 

Besser RE, Belongia EA. Clinician knowledge and 

beliefs after statewide program to promote 

appropriate antimicrobial drug use. Emerg Infect 

Dis 2005;11(6):904-11. 

34. Curran VR, Hoekman T, Gulliver W, Landells I, 

Hatcher L. Web-based continuing medical 

education. (II): Evaluation study of computermediated continuing medical education. J Contin 

Educ Health Prof 2000;20(2):106-19. 

35. Costanza ME, Zapka JG, Harris DR, et al. Impact of a 

physician intervention program to increase breast 

cancer screening. Cancer Epidemiol Biomarkers 

Prev 1992;1(7):581-9. 

36. Andersen SM, Harthorn BH. Changing the psychiatric 

knowledge of primary care physicians. The effects 

of a brief intervention on clinical diagnosis and 

treatment. Gen Hosp Psychiatry 1990;12(3):177-90. 

37. Block L, Banspach SW, Gans K, et al. Impact of 

public education and continuing medical education 

on physician attitudes and behavior concerning 

cholesterol. Am J Prev Med 1988;4(5):255-60. 

38. Premi J SS. Randomized controlled trial of a combined 

video-workbook educational program for CME. 

Acad Med. 1993;68(10 Suppl):S13-5. 

39. Harris JM Jr, Kutob RM, Surprenant ZJ, Maiuro RD, 

Delate TA. Can Internet-based education improve 

physician confidence in dealing with domestic 

violence? Fam Med 2002;34(4):287-92. 

40. Doucet MD, Purdy RA, Kaufman DM, Langille DB. 

Comparison of problem-based learning and lecture 

format in continuing medical education on 

headache diagnosis and management. Med Educ 

1998;32(6):590-6. 

41. Premi J, Shannon S, Hartwick K, Lamb S, Wakefield 

J, Williams J. Practice-based small-group CME. 

Acad Med 1994;69(10):800-2. 

42. Fordis M, King JE, Ballantyne CM, et al. Comparison 

of the instructional efficacy of Internet-based CME 

with live interactive CME workshops: a randomized 

controlled trial. JAMA 2005;294(9):1043-51. 

43. Stewart M, Marshall JN, Ostbye T, et al. Effectiveness 

of case-based on-line learning of evidence-based 

practice guidelines. Fam Med 2005;37(2):131-8. 

44. Beaulieu M, Choquette D, Rahme E, Bessette L, 

Carrier R. CURATA: A patient health management 

program for the treatment of osteoarthritis in 

Quebec: an integrated approach to improving the 

appropriate utilization of antiinflammatory/analgesic medications. Am J Manag 

Care 2004;10(8):569-75. 
63 

45. Lane DS, Messina CR, Grimson R. An educational 

approach to improving physician breast cancer 

screening practices and counseling skills. Patient 

Educ Couns 2001;43(3):287-99. 

46. Gifford DR, Mittman BS, Fink A, Lanto AB, Lee ML, 

Vickrey BG. Can a specialty society educate its 

members to think differently about clinical 

decisions? Results of a randomized trial. J Gen 

Intern Med 1996;11(11):664-72. 

47. Maiman LA, Becker MH, Liptak GS, Nazarian LF, 

Rounds KA. Improving pediatricians' complianceenhancing practices. A randomized trial. Am J Dis 

Child 1988;142(7):773-9. 

48. White CW, Albanese MA, Brown DD, Caplan RM. 

The effectiveness of continuing medical education 

in changing the behavior of physicians caring for 

patients with acute myocardial infarction. A 

controlled randomized trial. Ann Intern Med 

1985;102(5):686-92. 

49. Terry PB, Wang VL, Flynn BS, et al. A continuing 

medical education program in chronic obstructive 

pulmonary diseases: design and outcome. Am Rev 

Respir Dis 1981;123(1):42-6. 

50. Short LM, Surprenant ZJ, Harris JM Jr. A communitybased trial of an online intimate partner violence 

CME program. Am J Prev Med 2006;30(2):181-5. 

51. Meredith LS, Jackson-Triche M, Duan N, Rubenstein 

LV, Camp P, Wells KB. Quality improvement for 

depression enhances long-term treatment 

knowledge for primary care clinicians. J Gen Intern 

Med 2000;15(12):868-77. 

52. Mann KV, Lindsay EA, Putnam RW, Davis DA. 

Increasing physician involvement in cholesterollowering practices: The role of knowledge, attitudes 

and perceptions. Adv Health Sci Educ Theory Pract 

1997;2(3):237-53. 

53. Gerstein HC, Reddy SS, Dawson KG, Yale JF, 

Shannon S, Norman G. A controlled evaluation of a 

national continuing medical education programme 

designed to improve family physicians' 

implementation of diabetes-specific clinical practice 

guidelines. Diabet Med 1999;16(11):964-9. 

54. Chung S, Mandl KD, Shannon M, Fleisher GR. 

Efficacy of an educational Web site for educating 

physicians about bioterrorism. Acad Emerg Med 

2004;11(2):143-8. 

55. Elliott TE, Murray DM, Oken MM, et al. Improving 

cancer pain management in communities: main 

results from a randomized controlled trial. J Pain 

Symptom Manage 1997;13(4):191-203. 

56. Evans CE, Haynes RB, Birkett NJ, et al. Does a 

mailed continuing education program improve 

physician performance? Results of a randomized 

trial in antihypertensive care. JAMA. 

1986;255(4):501-4. 

57. Maxwell JA, Sandlow LJ, Bashook PG. Effect of a 

medical care evaluation program on physician 

knowledge and performance. J Med Educ 

1984;59(1):33-8. 

58. Kemper KJ, Gardiner P, Gobble J, Mitra A, Woods C. 

Randomized controlled trial comparing four 

strategies for delivering e-curriculum to health care 

professionals. BMC Med Educ 2006;6(1):2. 

59. Rosenthal MS, Lannon CM, Stuart JM, Brown L, 

Miller WC, Margolis PA. A randomized trial of 

practice-based education to improve delivery 

systems for anticipatory guidance. Arch Pediatr 

Adolesc Med 2005;159(5):456-63. 

60. Hergenroeder AC, Chorley JN, Laufman L, Fetterhoff 

A. Two educational interventions to improve 

pediatricians' knowledge and skills in performing 

ankle and knee physical examinations. Arch Pediatr 

Adolesc Med 2002;156(3):225-9. 

61. Des Marchais JE, Jean P, Castonguay LG. Training 

psychiatrists and family doctors in evaluating 

interpersonal skills. Med Educ 1990;24(4):376-81. 

62. Chodosh J, Berry E, Lee M, et al. Effect of a dementia 

care management intervention on primary care 

provider knowledge, attitudes, and perceptions of 

quality of care. J Am Geriatr Soc 2006;54(2):311-7. 

63. White M, Michaud G, Pachev G, Lirenman D, Kolenc 

A, FitzGerald JM. Randomized trial of problembased versus didactic seminars for disseminating 

evidence-based guidelines on asthma management 

to primary care physicians. J Contin Educ Health 

Prof 2004;24(4):237-43. 

64. Lockyer JM, Fidler H, Hogan DB, Pereles L, Lebeuf 

C, Wright B. Dual-track CME: accuracy and 

outcome. Acad Med 2002;77(10 Suppl):S61-3. 

65. Kutcher SP, Lauria-Horner BA, MacLaren CM, BujasBobanovic M. Evaluating the impact of an 

educational program on practice patterns of 

canadian family physicians interested in depression 

treatment. Prim Care Companion J Clin Psychiatry 

2002;4(6):224-31. 

66. Heale J, Davis D, Norman G, Woodward C, Neufeld 

V, Dodd P. A randomized controlled trial assessing 

the impact of problem-based versus didactic 

teaching methods in CME. Res Med Educ 

1988;27:72-7. 
64 

67. Labelle M, Beaulieu M, Renzi P, Rahme E, Thivierge 

RL. Integrating clinical practice guidelines into 

daily practice: impact of an interactive workshop on 

drafting of a written action plan for asthma patients. 

J Contin Educ Health Prof 2004;24(1):39-49. 

68. Bloomfield HE, Nelson DB, van Ryn M, et al. A trial 

of education, prompts, and opinion leaders to 

improve prescription of lipid modifying therapy by 

primary care physicians for patients with ischemic 

heart disease. Qual Saf Health Care 

2005;14(4):258-63. 

69. Lane DS, Polednak AP, Burg MA. Effect of 

continuing medical education and cost reduction on 

physician compliance with mammography 

screening guidelines. J Fam Pract 1991;33(4):359-

68. 

70. Lewis CE, Bursch B, Klau M, Konitsney D, Conrow 

S. Continuing medical education for AIDS: an 

organizational response. AIDS Educ Prev 

1993;5(3):263-71. 

71. Kronick J, Blake C, Munoz E, Heilbrunn L, 

Dunikowski L, Milne WK. Improving on-line skills 

and knowledge. A randomized trial of teaching 

rural physicians to use on-line medical information. 

Can Fam Physician 2003;49:312-7. 

72. Brown JB, Boles M, Mullooly JP, Levinson W. Effect 

of clinician communication skills training on patient 

satisfaction. A randomized, controlled trial. Ann 

Intern Med 1999;131(11):822-9. 

73. Lin EH, Katon WJ, Simon GE, et al. Achieving 

guidelines for the treatment of depression in 

primary care: is physician education enough? Med 

Care 1997;35(8):831-42. 

74. Pinto BM, Goldstein MG, DePue JD, Milan FB. 

Acceptability and feasibility of physician-based 

activity counseling. The PAL project. Am J Prev 

Med 1998;15(2):95-102. 

75. Harris SB, Leiter LA, Webster-Bogaert S, Van DM, 

O'Neill C. Teleconferenced educational detailing: 

diabetes education for primary care physicians. J 

Contin Educ Health Prof 2005;25(2):87-97. 

76. Jennett PA, Laxdal OE, Hayton RC, et al. The effects 

of continuing medical education on family doctor 

performance in office practice: a randomized 

control study. Med Educ 1988;22(2):139-45. 

77. Rodney WM, Albers G. Flexible sigmoidoscopy: 

primary care outcomes after two types of 

continuing medical education. Am J Gastroenterol 

1986;81(2):133-7. 

78. Clark NM, Gong M, Schork MA, et al. Long-term 

effects of asthma education for physicians on 

patient satisfaction and use of health services. Eur 

Respir J 2000;16(1):15-21. 

79. Grady KE, Lemkau JP, Lee NR, Caddell C. Enhancing 

mammography referral in primary care. Prev Med 

1997;26(6):791-800. 

80. Leopold SS, Morgan HD, Kadel NJ, Gardner GC, 

Schaad DC, Wolf FM. Impact of educational 

intervention on confidence and competence in the 

performance of a simple surgical task. J Bone Joint 

Surg Am 2005;87(5):1031-7. 

81. Norris SL, Grothaus LC, Buchner DM, Pratt M. 

Effectiveness of physician-based assessment and 

counseling for exercise in a staff model HMO. Prev 

Med 2000;30(6):513-23. 

82. Schroy PC, Heeren T, Bliss CM, Pincus J, Wilson S, 

Prout M. Implementation of on-site screening 

sigmoidoscopy positively influences utilization by 

primary care providers. Gastroenterology 

1999;117(2):304-11. 

83. Cherkin D, Deyo RA, Berg AO, Bergman JJ, Lishner 

DM. Evaluation of a physician education 

intervention to improve primary care for low-back 

pain. I. Impact on physicians. Spine 

1991;16(10):1168-72. 

84. Roter DL, Hall JA, Kern DE, Barker LR, Cole KA, 

Roca RP. Improving physicians' interviewing skills 

and reducing patients' emotional distress. A 

randomized clinical trial. Arch Intern Med 

1995;155(17):1877-84. 

85. Macrae HM, Regehr G, McKenzie M, et al. Teaching 

practicing surgeons critical appraisal skills with an 

Internet-based journal club: a randomized, 

controlled trial. Surgery 2004;136(3):641-6. 

86. Gerbert B, Bronstone A, Maurer T, Berger T, McPhee 

SJ, Caspers N. The effectiveness of an Internetbased tutorial in improving primary care physicians' 

skin cancer triage skills. J Cancer Educ 

2002;17(1):7-11. 

87. Frush K, Hohenhaus S, Luo X, Gerardi M, Wiebe RA. 

Evaluation of a Web-based education program on 

reducing medication dosing error: a multicenter, 

randomized controlled trial. Pediatr Emerg Care 

2006;22(1):62-70. 

88. Carney PA, Dietrich AJ, Freeman DH Jr, Mott LA. A 

standardized-patient assessment of a continuing 

medical education program to improve physicians' 

cancer-control clinical skills. Acad Med 

1995;70(1):52-8. 
65 

89. Farrin A, Russell I, Torgerson D, Underwood M. 

Differential recruitment in a cluster randomized 

trial in primary care: the experience of the UK back 

pain, exercise, active management and 

manipulation (UK BEAM) feasibility study. Clin 

Trials 2005;2(2):119-24. 

90. Frumovitz M, Ramirez PT, Greer M, et al. 

Laparoscopic training and practice in gynecologic 

oncology among Society of Gynecologic 

Oncologists members and fellows-in-training. 

Gynecol Oncol 2004;94(3):746-53. 

91. Olson AL, Dietrich AJ, Prazar G, et al. Two 

approaches to maternal depression screening during 

well child visits. J Dev Behav Pediatr 

2005;26(3):169-76. 

92. Juzych NS, Banerjee M, Essenmacher L, Lerner SA. 

Improvements in antimicrobial prescribing for 

treatment of upper respiratory tract infections 

through provider education. J Gen Intern Med 

2005;20(10):901-5. 

93. Goldberg HI, Deyo RA, Taylor VM, et al. Can 

evidence change the rate of back surgery? A 

randomized trial of community-based education. 

Eff Clin Pract 2001;4(3):95-104. 

94. Sharif I, Oruwariye T, Cohen R, Ozuah PO. 

Effectiveness of clinician training in smoking 

cessation counseling. Arch Pediatr Adolesc Med. 

2002;156(9):944-5. 

95. Stein CM, Griffin MR, Taylor JA, Pichert JW, Brandt 

KD, Ray WA. Educational program for nursing 

home physicians and staff to reduce use of nonsteroidal anti-inflammatory drugs among nursing 

home residents: a randomized controlled trial. Med 

Care 2001;39(5):436-45. 

96. Maclure M, Dormuth C, Naumann T, et al. Influences 

of educational interventions and adverse news about 

calcium-channel blockers on first-line prescribing 

of antihypertensive drugs to elderly people in 

British Columbia. Lancet 1998;352(9132):943-8. 

97. Rabin DL. Adapting an effective primary care 

provider STD/HIV prevention training programme. 

AIDS Care 1998;10 Suppl 1:S75-82. 

98. Schwartzberg JG, Guttman R. Effect of training on 

physician attitudes and practices in home and 

community care of the elderly. Arch Fam Med 

1997;6(5):439-44. 

99. Mehler PS, Krantz MJ, Lundgren RA, et al. Bridging 

the quality gap in diabetic hyperlipidemia: a 

practice-based intervention. Am J Med 

2005;118(12):1414. 

100. Rahme E, Choquette D, Beaulieu M, et al. Impact 

of a general practitioner educational intervention on 

osteoarthritis treatment in an elderly population. 

Am J Med 2005;118(11):1262-70. 

101. Davis RS, Bukstein DA, Luskin AT, Kailin JA, 

Goodenow G. Changing physician prescribing 

patterns through problem-based learning: an 

interactive, teleconference case-based education 

program and review of problem-based learning. 

Ann Allergy Asthma Immunol 2004;93(3):237-42. 

102. Herbert CP, Wright JM, Maclure M, et al. Better 

Prescribing Project: a randomized controlled trial of 

the impact of case-based educational modules and 

personal prescribing feedback on prescribing for 

hypertension in primary care. Fam Pract 

2004;21(5):575-81. 

103. Anderson JF, McEwan KL, Hrudey WP. 

Effectiveness of notification and group education in 

modifying prescribing of regulated analgesics. Can 

Med Assoc J. 1996;154(1):31-9. 

104. Ozer EM, Adams SH, Lustig JL , et al. Increasing 

the screening and counseling of adolescents for 

risky health behaviors: a primary care intervention. 

Pediatrics 2005;115(4):960-8. 

105. Soumerai SB, Avorn J. Predictors of physician 

prescribing change in an educational experiment to 

improve medication use. Med Care 1987;25(3):210-

21. 

106. Chassin MR, McCue SM. A randomized trial of 

medical quality assurance. Improving physicians' 

use of pelvimetry. JAMA 1986;256(8):1012-6. 

107. Perera DR, LoGerfo JP, Shulenberger E, Ylvisaker 

JT, Kirz HL. Teaching sigmoidoscopy to primary 

care physicians: a controlled study of continuing 

medical education. J Fam Pract 1983;16(4):785-8. 

108. Schectman JM, Schroth WS, Verme D, Voss JD. 

Randomized controlled trial of education and 

feedback for implementation of guidelines for acute 

low back pain. J Gen Intern Med 2003;18(10):773-

80. 

109. Ray WA, Stein CM, Byrd V, et al. Educational 

program for physicians to reduce use of nonsteroidal anti-inflammatory drugs among 

community-dwelling elderly persons: a randomized 

controlled trial. Med Care 2001;39(5):425-35. 

110. Dietrich AJ, Olson AL, Sox CH, Winchell CW, 

Grant-Petersson J, Collison DW. Sun protection 

counseling for children: primary care practice 

patterns and effect of an intervention on clinicians. 

Arch Fam Med 2000;9(2):155-9. 
66 

111. Gonzales R, Steiner JF, Lum A, Barrett PH Jr. 

Decreasing antibiotic use in ambulatory practice: 

impact of a multidimensional intervention on the 

treatment of uncomplicated acute bronchitis in 

adults. JAMA 1999;281(16):1512-9. 

112. Cummings SR, Richard RJ, Duncan CL, et al. 

Training physicians about smoking cessation: a 

controlled trial in private practice. J Gen Intern Med 

1989;4(6):482-9. 

113. Cummings SR, Coates TJ, Richard RJ, et al. 

Training physicians in counseling about smoking 

cessation. A randomized trial of the "Quit for Life" 

program. Ann Intern Med 1989;110(8):640-7. 

114. Kottke TE, Brekke ML, Solberg LI, Hughes JR. A 

randomized trial to increase smoking intervention 

by physicians. Doctors Helping Smokers, Round I. 

JAMA 1989;261(14):2101-6. 

115. Lindsay-McIntyre E, Wilson D, Best JA, et al. The 

impact of a continuing education package for 

smoking cessation on physicians' clinical behavior 

and patient smoking. Res Med Educ 1987;26:14-9. 

116. Stross JK, Bole GG. Evaluation of an educational 

program for primary care practitioners, on the 

management of osteoarthritis. Arthritis Rheum 

1985;28(1):108-11. 

117. Cummings SR, Richard RJ, Duncan CL, et al. 

Training physicians about smoking cessation: A 

controlled trial in private practices. J Gen Intern 

Med. 1989;4(6):482-9. 

118. Margolis PA, Lannon CM, Stuart JM, Fried BJ, 

Keyes-Elstein L, Moore DE Jr. Practice based 

education to improve delivery systems for 

prevention in primary care: randomised trial. Br 

Med J 2004;328(7436):388. 

119. McClellan WM, Millman L, Presley R, Couzins J, 

Flanders WD. Improved diabetes care by primary 

care physicians: results of a group-randomized 

evaluation of the Medicare Health Care Quality 

Improvement Program (HCQIP). J Clin Epidemiol 

2003;56(12):1210-7. 

120. Clark NM, Gong M, Schork MA, et al. Impact of 

education for physicians on patient outcomes. 

Pediatrics 1998;101(5):831-6. 

121. Moran JA, Kirk P, Kopelow M. Measuring the 

effectiveness of a pilot continuing medical 

education program. Can Fam Physician 

1996;42:272-6. 

122. Browner WS, Baron RB, Solkowitz S, Adler LJ, 

Gullion DS. Physician management of 

hypercholesterolemia. A randomized trial of 

continuing medical education. West J Med 

1994;161(6):572-8. 

123. Bunting PS, Van Walraven C. Effect of a controlled 

feedback intervention on laboratory test ordering by 

community physicians. Clin Chem 2004;50(2):321-

6. 

124. Ockene IS, Hebert JR, Ockene JK, Merriam PA, 

Hurley TG, Saperia GM. Effect of training and a 

structured office practice on physician-delivered 

nutrition counseling: the Worcester-Area Trial for 

Counseling in Hyperlipidemia (WATCH). Am J 

Prev Med 1996;12(4):252-8. 

125. Adams A, Ockene JK, Wheller EV, Hurley TG. 

Alcohol counseling: physicians will do it. J Gen 

Intern Med 1998;13(10):692-8. 

126. Hagen BF, Armstrong-Esther C, Quail P, et al. 

Neuroleptic and benzodiazepine use in long-term 

care in urban and rural Alberta: characteristics and 

results of an education intervention to ensure 

appropriate use. Int Psychogeriatr 2005;17(4):631-

52. 

127. Solomon DH, Katz JN, La Tourette AM, Coblyn 

JS. Multifaceted intervention to improve 

rheumatologists' management of glucocorticoidinduced osteoporosis: a randomized controlled trial. 

Arthritis Rheum 2004;51(3):383-7. 

128. Pazirandeh M. Does patient partnership in 

continuing medical education (CME) improve the 

outcome in osteoporosis management? J Contin 

Educ Health Prof 2002;22(3):142-51. 

129. Thom DH. Training physicians to increase patient 

trust. J Eval Clin Pract 2000;6(3):245-53. 

130. Schectman JM, Kanwal NK, Schroth WS, Elinsky 

EG. The effect of an education and feedback 

intervention on group-model and network-model 

health maintenance organization physician 

prescribing behavior. Med Care 1995;33(2):139-44. 

131. Casebeer LL, Klapow JC, Centor RM, et al. An 

intervention to increase physicians' use of 

adherence-enhancing strategies in managing 

hypercholesterolemic patients. Acad Med 

1999;74(12):1334-9. 

132. Gullion DS, And Others. Management of 

hypertension in private practice: a randomized 

controlled trial in continuing medical education. J 

Contin Educ Health Prof 1988;8(4):239-55. 
67 

133. Lin EH, Simon GE, Katzelnick DJ, Pearson SD. 

Does physician education on depression 

management improve treatment in primary care? J 

Gen Intern Med 2001;16(9):614-9. 

134. Socolar RR, Raines B, Chen-Mok M, Runyan DK, 

Green C, Paterno S. Intervention to improve 

physician documentation and knowledge of child 

sexual abuse: a randomized, controlled trial. 

Pediatrics 1998;101(5):817-24. 

135. Brown R, Bratton SL, Cabana MD, Kaciroti N, 

Clark NM. Physician asthma education program 

improves outcomes for children of low-income 

families. Chest 2004;126(2):369-74. 

136. Sibley JC, Sackett DL, Neufeld V, Gerrard B, 

Rudnick KV, Fraser W. A randomized trial of 

continuing medical education. N Engl J Med 

1982;306(9):511-5. 

137. Kim CS, Kristopaitis RJ, Stone E, Pelter M, Sandhu 

M, Weingarten SR. Physician education and report 

cards: do they make the grade? Results from a 

randomized controlled trial. Am J Med 

1999;107(6):556-60. 

138. Levinson W, Roter D. The effects of two continuing 

medical education programs on communication 

skills of practicing primary care physicians. J Gen 

Intern Med 1993;8(6):318-24. 

139. Schectman JM, Elinsky EG, Pawlson LG. Effect of 

education and feedback on thyroid function testing 

strategies of primary care clinicians. Arch Intern 

Med 1991;151(11):2163-6. 

140. Beaulieu MD, Rivard M, Hudon E, Beaudoin C, 

Saucier D, Remondin M. Comparative trial of a 

short workshop designed to enhance appropriate use 

of screening tests by family physicians. Can Med 

Assoc J. 2002;167(11):1241-6. 

141. Tziraki C, Graubard BI, Manley M, Kosary C, 

Moler JE, Edwards BK. Effect of training on 

adoption of cancer prevention nutrition-related 

activities by primary care practices: results of a 

randomized, controlled study. J Gen Intern Med 

2000;15(3):155-62. 

142. Gifford DR, Holloway RG, Frankel MR, et al. 

Improving adherence to dementia guidelines 

through education and opinion leaders. A 

randomized, controlled trial. Ann Intern Med 

1999;131(4):237-46. 

143. Pimlott NJ, Hux JE, Wilson LM, Kahan M, Li C, 

Rosser WW. Educating physicians to reduce 

benzodiazepine use by elderly patients: a 

randomized controlled trial. Can Med Assoc J. 

2003;168(7):835-9. 

144. McBride P, Underbakke G, Plane MB, et al. 

Improving prevention systems in primary care 

practices: the Health Education and Research Trial 

(HEART). J Fam Pract 2000;49(2):115-25. 

145. Howe HL, Lehnherr M, Katterhagen JG. Effects of 

physician outreach programs on rural-urban 

differences in breast cancer management. J Rural 

Health 1997;13(2):109-17. 

146. Goldstein MK, Lavori P, Coleman R, Advani A, 

Hoffman BB. Improving adherence to guidelines 

for hypertension drug prescribing: clusterrandomized controlled trial of general versus 

patient-specific recommendations. Am J Manag 

Care 2005;11(11):677-85. 

147. Allison JJ, Kiefe CI, Wall T , et al. Multicomponent 

Internet continuing medical education to promote 

chlamydia screening. Am J Prev Med 

2005;28(3):285-90. 

148. Myers RE, Turner B, Weinberg D, et al. Impact of a 

physician-oriented intervention on follow-up in 

colorectal cancer screening. Prev Med 

2004;38(4):375-81. 

149. Fick DM, Maclean JR, Rodriguez NA, et al. A 

randomized study to decrease the use of potentially 

inappropriate medications among communitydwelling older adults in a southeastern managed 

care organization. Am J Manag Care 2004;10(11 Pt 

1):761-8. 

150. Peterson L, Tremblay G, Ewigman B, Popkey C. 

The parental daily diary. A sensitive measure of the 

process of change in a child maltreatment 

prevention program. Behav Modif 2002;26(5):627-

47. 

151. Wells KB, Sherbourne C, Schoenbaum M, et al. 

Impact of disseminating quality improvement 

programs for depression in managed primary care: a 

randomized controlled trial. JAMA 

2000;283(2):212-20. 

152. Worrall G, Angel J, Chaulk P, Clarke C, Robbins 

M. Effectiveness of an educational strategy to 

improve family physicians' detection and 

management of depression: a randomized 

controlled trial. Can Med Assoc J. 1999;161(1):37-

40. 

153. Derebery VJ, Giang GM, Saracino G, Fogarty WT. 

Evaluation of the impact of a low back pain 

educational intervention on physicians' practice 

patterns and patients' outcomes. J Occup Environ 

Med 2002;44(10):977-84. 
68 

154. Wilson DM, Taylor DW, Gilbert JR, et al. A 

randomized trial of a family physician intervention 

for smoking cessation. JAMA 1988;260(11):1570-

4. 

155. McMahon SM, McKenna P, Hodgins JL. An 

application of continuing medical education to 

decrease excessive lengths of stay. J Med Educ 

1988;63(5):364-71. 

156. Messina CR, Lane DS, Grimson R. Effectiveness of 

women's telephone counseling and physician 

education to improve mammography screening 

among women who underuse mammography. Ann 

Behav Med 2002;24(4):279-89. 

157. Goldwater SH, Milkovich G, Morrison AJ Jr, 

Lindgren B. Comparison of therapeutic interchange 

with standard educational tools for influencing 

fluoroquinolone prescribing. Am J Health Syst 

Pharm 2001;58(18):1740-5. 

158. Gaffan J, Dacre J, Jones A. Educating 

undergraduate medical students about oncology: a 

literature review. J Clin Oncol 2006;24(12):1932-9. 

159. Ravert P. An integrative review of computer-based 

simulation in the education process (Structured 

abstract). Comput Inform Nurs. 2002;20(5):203-8. 

160. Spangler JG, George G, Foley KL, Crandall SJ. 

Tobacco intervention training: current efforts and 

gaps in US medical schools. JAMA 

2002;288(9):1102-9. 

161. Mazmanian PE, Johnson RE, Zhang A, Boothby J, 

Yeatts EJ. Effects of a signature on rates of change: 

a randomized controlled trial involving continuing 

education and the commitment-to-change model. 

Acad Med 2001;76(6):642-6. 

162. Schaafsma F, Hulshof C, van Dijk F, Verbeek J. 

Information demands of occupational health 

physicians and their attitude towards evidencebased medicine. Scand J Work Environ Health 

2004;30(4):327-30. 

163. Pereles L, Lockyer J, Hogan D, Gondocz T, 

Parboosingh J. Effectiveness of commitment 

contracts in continuing medical education. Acad 

Med 1996;71(4):394. 

164. Downing SM. Reliability: on the reproducibility of 

assessment data. Med Educ 2004;38(9):1006-12. 

165. McGaghie WC, Issenberg SB, Petrusa ER, Scalese 

RJ. Effect of practice on standardised learning 

outcomes in simulation-based medical education. 

Med Educ 2006;40(8):792-7. 
69 

List of Acronyms/Abbreviations 

ABMS American Board of Medical Specialties 

ACCP American College of Chest Physicians 

AHRQ Agency for Healthcare Research and Quality 

AMA American Medical Association 

CI Confidence interval 

CME Continuing medical education 

EPC Evidence-based Practice Center 

KQ Key questions 

KR20 Kuder-Richardson 20 

MOC Maintenance of Certification 

NIH National Institutes of Health 

SD Standard deviation 

US United States 
Appendix A: Technical Experts and Peer Reviewers 

A 1 

Technical Experts and Peer Reviewers 

Alejandro Aparicio, MD, FACP 

Director, Division of Continuing Physician 

Professional Development,

American Medical Association 

Vice President for Medical Affairs, 

Ballard Health Care 

Chicago, IL 

Michael H. Baumann, MD, MS 

Professor of Medicine 

Division of Pulmonary and Critical Care 

Medicine 

University of Mississippi Medical Center 

Jackson, MS 

Frank C. Berry 

Continuing Medical Education Director 

MedChi, The Maryland State Medical 

Society 

Baltimore, Maryland 

Nancy L. Davis, PhD 

Director, Division of Continuing Medical 

Education 

American Academy of Family Physicians 

Leawood, KS 

Robert Galbraith, MD, MBA, FACP 

Executive Director of the Center for 

Innovation, 

National Board of Medical Examiners 

Philadelphia, PA 

James C. Hebert, MD, FACS 

Chair, Committee on Continuous 

Professional Development 

Associate Dean for Graduate Medical 

Education 

Vice Chair for Education, Department of 

Surgery 

American College of Surgeons 

University of Vermont, College of Medicine 

Burlington, Vermont 

S. Barry Issenberg, MD, FACP 

Associate Professor of Medicine 

Assistant Dean, Research in Medical 

Education 

Director, Division of Research and 

Technology 

Assistant Director, Center for Research in 

Medical Education 

University of Miami Miller School of 

Medicine 

Miami, Florida 

Jocelyn Lockyer, PhD 

Director, Continuing Medical Education and 

Professional Development 

Associate Professor, Department of 

Community Health Services 

University of Calgary 

Calgary, Alberta Canada 

Mary Martin Lowe, MA 

Director, Education and Improvement 

Accreditation Council for Continuing 

Medical Education 

Chicago, Illinois 
Appendix A: Technical Experts and Peer Reviewers 

A 2 

Don Moore, Jr., PhD 

Professor of Medical Education and 

Administration 

Director, Division of Continuing Medical 

Education 

Vanderbilt University School of Medicine 

Nashville, TN 

LTC Lisa K. Moores, MC, USA 

Former Chair, Council of NetWorks 

Vice Chair, Continuing Education 

Committee 

Member, Task Force on Performance 

Measurement 

Walter Reed Army Medical Center 

Gaithersburg, MD 

Charles Willis, MBA 

Former Director, Department of AMA PRA 

Standards & Policy Liaison Activities 

American Medical Association 

Administrative Director, Division of 

Continuing Physician Professional 

Development 

Chicago, IL 
 Appendix B: Hand Searched Journals 

B 1 

All Journals Hand Searched 

February 2005-February 2006 

Academic Medicine 

The American Journal of Managed Care 

American Journal of Preventive Medicine 

Annals of Internal Medicine 

Chest 

Canadian Medical Association Journal 

The Journal of Continuing Education in the Health Professions 

Journal of General Internal Medicine 

Journal of Medical Education 

The Journal of the American Medical Association 

Medical Care 

Medical Education 
Appendix C: Detailed Electronic Database Search Strategies

C 1 

Detailed Electronic Database Search Strategies for Primary Literature 

on Effectiveness of Continuing Medical Education 

MEDLINE Strategy 

Terms Returns 

(((("Education, Continuing"[MeSH] OR "Education, Medical"[MeSH]) NOT ("Education, 

Dental, Continuing"[MeSH] OR "Education, Nursing, Continuing"[MeSH] OR "Education, 

Pharmacy, Continuing"[MeSH] OR "Education, Medical, Undergraduate"[MeSH] OR 

"Internship and Residency"[MeSH]) OR ("continuing medical education"[tiab] OR CME[tiab]) 

OR ((educat*[tiab] OR train*[tiab] OR curriculum[tiab]) AND (physician* OR Family 

practi*[tiab] OR Family medicine[tiab] OR General practi*[tiab] OR internist*[tiab] OR 

Surgeon*[tiab] OR Primary care[tiab] OR Allergist*[tiab] OR Immunologist*[tiab] OR 

Anesthesiology*[tiab] OR Dematolog*[tiab] OR Emergency medicine[tiab] OR Forensic 

medicine[tiab] OR Hospitalist*[tiab] OR Internal medicine[tiab] OR Cardiolog*[tiab] OR 

Endocrinolog*[tiab] OR Gastroenterolog*[tiab] OR Hematolog*[tiab] OR Oncolog*[tiab] OR 

Nephrolog*[tiab] OR Pulmonolog*[tiab] OR Rhematolog*[tiab] OR Neurolog*[tiab] OR 

Patholog*[tiab] OR Pediatric*[tiab] OR Psychiatr*[tiab] OR Radiolog*[tiab] OR 

Obstetrician*[tiab] OR Gynecolog*[tiab]))) AND (behav*[tiab] OR practice*[tiab] OR 

evaluat*[tiab] OR assess*[tiab] OR learn*[tiab] OR skill*[tiab] OR outcome*[tiab] OR 

effective*[tiab] OR analy*[tiab] OR intervention*[tiab] OR examin*[tiab])) NOT (dental*[tiab] 

OR dentist*[tiab] OR student*[tiab] OR undergraduate*[tiab] OR athlet*[tiab])) AND 

English[lang] NOT (animal[mh] NOT human [mh]) AND (1981:2006[dp]) NOT (review[pt] OR 

meta-analysis[pt] OR editorial[pt] OR comment[pt] OR letter[pt]) 

38174 

EMBASE Strategy 

(((('medical education':de) NOT ('clinical supervision':de OR 'dental education':de OR 

'medical school':de OR 'physician assistant education':de OR 'residency education':de)) OR 

('continuing medical education':ti,ab) OR cme:ti,ab OR ((educat*:ti,ab OR train*:ti,ab OR 

curriculum:ti,ab) AND (physician*:ti,ab OR (family:ti,ab AND practi*:ti,ab) OR (family:ti,ab 

AND medicine:ti,ab) OR (general:ti,ab AND practi*:ti,ab) OR internist*:ti,ab OR 

surgeon*:ti,ab OR (primary:ti,ab AND care:ti,ab) OR allergist*:ti,ab OR immunologist*:ti,ab 

OR anesthesiolog*:ti,ab OR dermatolog*:ti,ab OR (emergency:ti,ab AND medicine:ti,ab) OR 

(forensic:ti,ab AND medicine:ti,ab) OR hospitalist*:ti,ab OR (internal:ti,ab AND 

medicine:ti,ab) OR cardiolog*:ti,ab OR endocrinolog*:ti,ab OR gastroenterolog*:ti,ab OR 

hematolog*:ti,ab OR oncolog*:ti,ab OR nephrolog*:ti,ab OR pulmonolog*:ti,ab OR 

rhemaolog*:ti,ab OR neurolog*:ti,ab OR patholog*:ti,ab OR pediatric*:ti,ab OR 

psychiatr*:ti,ab OR radiolog*:ti,ab OR obstetric*:ti,ab OR gynecolog*:ti,ab))) AND 

(behav*:ti,ab OR practice*:ti,ab OR evaluat*:ti,ab OR assess*:ti,ab OR learn*:ti,ab OR 

skill*:ti,ab OR outcome*:ti,ab OR effective*:ti,ab OR analy*:ti,ab OR intervention*:ti,ab OR 

examin*:ti,ab)) NOT (dental*:ti,ab OR dentist*:ti,ab OR student*:ti,ab OR undergraduate*:ti,ab 

OR athlet*:ti,ab) AND [English]/lim NOT ([animals]/lim NOT [humans]/lim) AND [1981-

2006]/py NOT ([conference paper]/lim OR [editorial]/lim OR [erratum]/lim OR [letter]/lim OR 

[note]/lim OR [review]/lim) 

44765 
Appendix C: Detailed Electronic Database Search Strategies

C 2 

The Cochrane Central Register of Controlled Trials (CENTRAL) 

(((((Continuing medical education):ti,ab,kw OR (CME):ti,ab,kw) OR ((educat* 

OR train* OR curriculum):ti,ab,kw NEAR (physician* OR Family practi* OR 

Family medicine OR General practice OR internist* OR Surgeon* OR Primary 

care OR Allergist OR Immunologist OR Anesthesiolog* OR Dematolog* OR 

Emergency medicine OR Forensic medicine OR Hospitalist* OR Internal 

medicine OR Cardiolog* OR Endocrinolog* OR Gastroenterolog* OR 

Hematolog* OR Oncolog* OR Nephrolog* OR Pulmon* OR Rhematolog* OR 

Neurolog* OR Patholog* OR Pediatric* OR Psychiatr* OR Radiolog* OR 

Obstetrician* OR Gynecolog*):ti,ab,kw)) AND (behav* OR evaluat* OR 

assess* OR learn* OR skill* OR outcome* OR effective* OR analy* OR 

examin* OR intervention*):ti,ab,kw) NOT (dental* OR dentist* OR student* 

OR undergraduate* OR athlet*):ti,ab,kw), LIMIT DATE RANGE from 1981 to 

2006

1843 

PsycINFO 

(((((MM "Continuing Education" OR MM "Inservice Training" OR MM 

"Medical Education") NOT (MM "Medical Internship" OR MM "Medical 

Residency")) OR (TI “continuing medical education” OR TI "CME" OR AB 

“continuing medical education” OR AB "CME") OR ((AB educat* OR AB 

train* OR AB curriculum) AND (AB physician* OR AB Family practi* OR AB 

Family medicine OR AB General practi* OR AB internist* OR AB Surgeon* 

OR AB Primary care OR AB Allergist OR AB Immunologist OR AB 

Anesthesiology* OR AB Dematolog* OR AB Emergency medicine OR AB 

Forensic medicine OR AB Hospitalist* OR AB Internal medicine OR AB 

Cardiolog* OR AB Endocrinolog* OR AB Gastroenterolog* OR AB 

Hematolog* OR AB Oncolog* OR AB Nephrolog* OR AB Pulmonolog* OR 

AB Rhematolog* OR AB Neurology* OR AB Patholog* OR AB Pediatric* OR 

AB Psychiatr* OR AB Radiolog* OR AB Obstetrician* OR AB Gynecolog*))) 

AND (AB behav* OR AB evaluat* OR AB assess* OR AB learn* OR AB skill* 

OR AB outcome* OR AB effective* OR AB analy* OR AB examin*)) NOT 

(AB dental* OR AB dentist* OR AB student* OR AB undergraduate* OR AB 

athlet*)) AND (LA English NOT (PO animal NOT PO human) AND DT 

198101-200602 NOT (PZ abstract collection OR PZ bibliography OR PZ 

column/opinion OR PZ comment/reply OR PZ editorial OR PZ 

erratum/correction OR PZ letter OR PZ obituary OR PZ all chapters OR PZ 

original chapter OR PZ reprinted chapter OR PZ reprinted journal article OR PZ 

publication information OR PZ review)) [FURTHER LIMITED TO ALL 

JOURNALS]

8738 
Appendix C: Detailed Electronic Database Search Strategies

C 3 

ERIC 

(((((MM "Continuing Education" OR MM "Inservice Training" OR MM 

"Medical Education") NOT (MM "Medical Internship" OR MM "Medical 

Residency")) OR (TI “continuing medical education” OR TI "CME" OR AB 

“continuing medical education” OR AB "CME") OR ((AB educat* OR AB 

train* OR AB curriculum) AND (AB physician* OR AB Family practi* OR AB 

Family medicine OR AB General practi* OR AB internist* OR AB Surgeon* 

OR AB Primary care OR AB Allergist OR AB Immunologist OR AB 

Anesthesiology* OR AB Dematolog* OR AB Emergency medicine OR AB 

Forensic medicine OR AB Hospitalist* OR AB Internal medicine OR AB 

Cardiolog* OR AB Endocrinolog* OR AB Gastroenterolog* OR AB 

Hematolog* OR AB Oncolog* OR AB Nephrolog* OR AB Pulmonolog* OR 

AB Rhematolog* OR AB Neurology* OR AB Patholog* OR AB Pediatric* OR 

AB Psychiatr* OR AB Radiolog* OR AB Obstetrician* OR AB Gynecolog*))) 

AND (AB behav* OR AB evaluat* OR AB assess* OR AB learn* OR AB skill* 

OR AB outcome* OR AB effective* OR AB analy* OR AB examin*)) NOT 

(EL "Early Childhood Education" OR EL "Preschool Education" OR EL 

"Elementary Secondary Education" OR EL "Elementary Education" OR EL 

"Primary Education" OR EL "Adult Basic Education" OR EL "Intermediate 

Grades" OR EL "Secondary Education" OR EL "Middle Schools" OR EL 

"Junior High Schools" OR EL "High Schools" OR EL "High School 

Equivalency Programs" OR EL "Postsecondary Education" OR EL "Two Year 

Colleges")) AND (LA English AND DT 198101-200602 AND PT Journal 

Article NOT (PO animal NOT PO human))

2002 
Appendix C: Detailed Electronic Database Search Strategies

C 4 

Detailed Electronic Database Search Strategies for Systematic 

Reviews on Effectiveness of Simulation Techniques in Medical 

Education 

MEDLINE Strategy 

Terms Returns 

(((“Education, Medical”[MeSH] OR ((educat*[tiab] OR train*[tiab] OR 

curriculum[tiab]) AND (medical*[tiab] OR resident*[tiab] OR residenc*[tiab] 

OR physician*[tiab] OR surgery[tiab] OR surgeon*[tiab] OR surgical*[tiab])))) 

AND (“Patient Simulation”[MeSH] OR Computer Simulation[MeSH] OR 

Manikins[MeSH] OR simulation*[tiab] OR simulat*[tiab] OR mannikin[tiab] 

OR manikin[tiab] OR mannequin*[tiab] OR virtual[tiab] OR computerbased[tiab] OR “standardized patient”[tiab] OR “standardized patients”[tiab])) 

AND ((review[tiab] or review[pt] or meta-analys*[tiab] or meta-analysis[pt]) 

AND English[lang] NOT (letter[pt] or comment[pt] or editorial[pt])) NOT 

(animal[mh] NOT human [mh]) AND ("1990/01/01"[pdat] : "2006/02/28"[pdat])

466 

EMBASE Strategy 

(((('medical education'/exp) NOT ('clinical supervision'/exp OR 'dental 

education'/exp OR 'physician assistant education'/exp)) OR 

((educat*:ti,ab OR train*:ti,ab OR curriculum:ti,ab) AND (medical*:ti,ab 

OR resident*:ti,ab OR residenc*:ti,ab OR surgery:ti,ab OR surgical*:ti,ab OR 

physician*:ti,ab OR (family:ti,ab AND practi*:ti,ab) OR (family:ti,ab AND 

medicine:ti,ab) OR (general:ti,ab AND practi*:ti,ab) OR internist*:ti,ab 

OR surgeon*:ti,ab OR (primary:ti,ab AND care:ti,ab) OR allergist*:ti,ab 

OR immunologist*:ti,ab OR anesthe*:ti,ab OR anaesthe*:ti,ab OR 

dermatolog*:ti,ab OR (emergency:ti,ab AND medicine:ti,ab) OR 

(forensic:ti,ab AND medicine:ti,ab) OR hospitalist*:ti,ab OR 

(internal:ti,ab AND medicine:ti,ab) OR cardiolog*:ti,ab OR 

endocrinolog*:ti,ab OR gastroenterolog*:ti,ab OR hematolog*:ti,ab OR 

oncolog*:ti,ab OR nephrolog*:ti,ab OR pulmonolog*:ti,ab OR 

rhemaolog*:ti,ab OR neurolog*:ti,ab OR patholog*:ti,ab OR 

pediatric*:ti,ab OR psychiatr*:ti,ab OR radiolog*:ti,ab OR 

obstetric*:ti,ab OR gynecolog*:ti,ab))) AND ('skill'/exp OR 'simulator'/exp 

OR 'simulation'/exp OR 'virtual reality'/exp OR simulation*:ti,ab OR 

simulat*:ti,ab OR mannikin:ti,ab OR manikin:ti,ab OR mannequin*:ti,ab OR 

virtual:ti,ab OR computer-based:ti,ab OR (standardized:ti,ab AND 

patient*:ti,ab))) AND (review:ti,ab,it OR 'meta analysis':ti,ab,it OR 

metaanalysis:ti,ab,it) NOT (letter:it OR comment:it OR editorial:it OR 

'conference paper':it OR erratum:it OR note:it) AND [English]/lim NOT 

([animals]/lim NOT [humans]/lim) AND [1981-2006]/py

2359 
Appendix C: Detailed Electronic Database Search Strategies

C 5 

The Cochrane Database of Systematic Reviews and the Cochrane 

Database of Abstracts of Reviews of Effects (DARE) 

(Simulation or simulator or manikin or mannikin or mannequin or virtual or 

computer-based or "standardized patient" or "standardized patients") AND 

education in title, abstract or keywords restricted to reviews

3 

PsycINFO 

((((MM "Medical Education") OR ((AB educat* OR AB train* OR AB 

curriculum) AND (AB physician* OR AB Family practi* OR AB "Family 

medicine" OR AB General practi* OR AB internist* OR AB Surgeon* OR AB 

"Primary care" OR AB Allergist OR AB Immunologist OR AB Anesthesiolog* 

OR AB Dematolog* OR AB "Emergency medicine" OR AB "Forensic 

medicine" OR AB Hospitalist* OR AB "Internal medicine" OR AB Cardiolog* 

OR AB Endocrinolog* OR AB Gastroenterolog* OR AB Hematolog* OR AB 

Oncolog* OR AB Nephrolog* OR AB Pulmonolog* OR AB Rhematolog* OR 

AB Neurolog* OR AB Patholog* OR AB Pediatric* OR AB Psychiatr* OR AB 

Radiolog* OR AB Obstetrician* OR AB Gynecolog* OR AB Medical OR AB 

resident* OR AB residenc* OR AB surgery OR AB surgical*))) AND (MM 

"Simulation" or MM "Virtual Reality" or MM "Human Machine Systems" or AB 

"simulation" or AB simulat* or AB manikin or AB mannikin or AB mannequin 

or AB virtual or AB "standardized patient" or AB "standardized patients")) AND 

(LA English NOT (PO animal NOT PO human) AND DT 199001-200602 NOT 

(PZ abstract collection OR PZ bibliography OR PZ column/opinion OR PZ 

comment/reply OR PZ editorial OR PZ erratum/correction OR PZ letter OR PZ 

obituary OR PZ all chapters OR PZ original chapter OR PZ reprinted chapter 

OR PZ reprinted journal article OR PZ publication information))) AND (PZ 

review OR AB review OR AB meta-analys*)

34 
Appendix C: Detailed Electronic Database Search Strategies

C 6 

ERIC 

((((DE "Medical Education") OR ((AB educat* OR AB train* OR AB 

curriculum) AND (AB physician* OR AB "Family practice" OR AB "Family 

practitioner" OR AB "Family medicine" OR AB "General practice" OR AB 

"General practitioner" OR AB internist* OR AB Surgeon* OR AB "Primary 

care" OR AB Allergist OR AB Immunologist OR AB Anesthesiolog* OR AB 

Dematolog* OR AB "Emergency medicine" OR AB "Forensic medicine" OR 

AB Hospitalist* OR AB "Internal medicine" OR AB Cardiolog* OR AB 

Endocrinolog* OR AB Gastroenterolog* OR AB Hematolog* OR AB Oncolog* 

OR AB Nephrolog* OR AB Pulmonolog* OR AB Rhematolog* OR AB 

Neurolog* OR AB Patholog* OR AB Pediatric* OR AB Psychiatr* OR AB 

Radiolog* OR AB Obstetrician* OR AB Gynecolog* OR AB Medical OR AB 

resident* OR AB residenc* OR AB surgery OR AB surgical*))) AND (DE 

"Simulation" or DE "Virtual Reality" or AB "simulation" or AB simulat* or AB 

manikin or AB mannikin or AB mannequin or AB virtual or AB "standardized 

patient" or AB "standardized patients")) AND (LA English AND DT 198101-

200602 AND PT Journal Article NOT (PO animal NOT PO human))) AND (AB 

review* OR AB meta-analys* OR TI review* OR TI meta-analys*)

14 
Appendix D: List of Excluded Articles

D-1 

Abidi F, Lee-Gorman M. Learning from experience: 

developments in forensic investigation from case 

histories. Sci Justice 2005;45(1):45-51 

Not US or Canada, Does not apply 

Adelson R, Hepburn K, Reed R et al. Effective 

dissemination of the AHCPR guideline: 

prevention and early management of pressure 

ulcers. Abstract Book/Association for Health 

Services Research 97;14167-8 

Abstract 

Ahluwalia N S, Das A, Verity R. Radical curriculum 

design: an experiment in learner empowerment. 

Med Educ 2005;39(5):509-10 

Not US or Canada 

AIDS education in general practice. N Z Med J 

88;101(859):835-6 

No evaluation, Does not apply 

A-Latif A. Continuing medical education: merits of a 

surgical journal club. Med Teach 90;12(2):219-

21 

Not US or Canada 

Alexander B, Nasrallah H A, Perry P J. The impact of 

psychopharmacology education on prescribing 

practices. HOSP. COMMUNITY 

PSYCHIATRY 83;34(12):1150-1153 

<15 trained physicians, No comparison 

Aloisio G, Barone L, Bergamasco M et al. Computerbased simulator for catheter insertion training. 

Stud Health Technol Inform 2004;984-6 

No original data, No training, Not US or 

Canada 

Alroy G, Ber R, Kramer D. An evaluation of the 

short-term effects of an interpersonal skills 

course. MED. EDUC. 84;18(2):85-89 

Not US or Canada 

Amatayakul M. Self-assessment for continuing 

education. Council on Certification of the 

American Health Information Management 

Association. J AHIMA 92;63(2):83-4, 109-11 

No original data, No evaluation 

Amiel G E, Ungar L, Alperin M et al. Ability of 

primary care physician's to break bad news: a 

performance based assessment of an educational 

intervention. Patient Educ Couns 2006;60(1):10-

5 

Not US or Canada 

Amin Z, Hoon Eng K, Chay Hoon T. A novel 

approach to faculty development programme 

evaluation. Med Educ 2004;38(11):1187-8 

<15 trained physicians, Not US or Canada 

Amos E, White M J. Problem-based learning. Nurse 

Educ 98;23(2):11-4 

<15 trained physicians 

Anders K T. Click and learn. Continuing education 

on the Web. Contemp Longterm Care 

2000;23(11):suppl 12-4 

No original data, No evaluation 

Andrews J O, Tingen M S, Waller J L et al. Provider 

feedback improves adherence with AHCPR 

Smoking Cessation Guideline. Prev Med 

2001;33(5):415-21 

<15 trained physicians 

Athanasiadis L, Papaharitou S, Salpiggidis G et al. 

Educating physicians to treat erectile dysfunction 

patients: development and evaluation of a course 

on communication and management strategies. J 

Sex Med 2006;3(1):47-55 

Not US or Canada, No comparison 

Avorn J, Soumerai S B, Everitt D E et al. A 

randomized trial of a program to reduce the use 

of psychoactive drugs in nursing homes. N Engl 

J Med 92;327(3):168-73 

<15 trained physicians 

Awad S S, Fagan S P, Bellows C et al. Bridging the 

communication gap in the operating room with 

medical team training. Am J Surg 

2005;190(5):770-4 

Does not apply, No comparison 

Ayoub M M, Clark J A. Reduction of fresh frozen 

plasma use with a simple education program. 

Am Surg 89;55(9):563-5 

No comparison 

Backhaus J, Junghanns K, Mueller-Popkes K et al. 

Short-term training increases diagnostic and 

treatment rate for insomnia in general practice. 

Eur Arch Psychiatry Clin Neurosci 

2002;252(3):99-104 

Not US or Canada 

Baldwin C D, Gephart D, Maulitz R. Collaborative 

planning of a Web-based learning resource for 

primary care education. Acad Med 

2001;76(5):549-50 

<15 trained physicians 
Appendix D: List of Excluded Articles

D-2 

Barsuk D, Ziv A, Lin G et al. Using advanced 

simulation for recognition and correction of gaps 

in airway and breathing management skills in 

prehospital trauma care. Anesth Analg 

2005;100(3):803-9, table of contents 

<15 trained physicians, Not US or Canada 

Barta P J, Phillips W D, Davidson R. Physician 

education on five complications of diabetes 

mellitus. N J Med 94;91(4):269-70 

No comparison 

Bashook P, Meyer T C, Richards R K et al. 

Symposium: Self-directed learning and 

physicians'' practice changes: concepts, research 

and implications for CME. Proc Annu Conf Res 

Med Educ 86;25329-38 

No original data 

Belda T E, Gajic O, Rabatin J T et al. Practice 

variability in management of acute respiratory 

distress syndrome: bringing evidence and 

clinician education to the bedside using a webbased teaching tool. Respir Care 

2004;49(9):1015-21 

No comparison 

Ben-Arye E, Frenkel M, Hermoni D. An approach to 

teaching primary care physicians how to 

integrate complementary medicine into their 

daily practices: a pilot study. J Altern 

Complement Med 2006;12(1):79-83 

<15 trained physicians, Not US or Canada, No 

comparison 

Ben-Arye E, Frenkel M. An approach to teaching 

physicians about complementary medicine in the 

treatment of cancer. Integr Cancer Ther 

2004;3(3):208-13 

Not US or Canada 

Benjamin E M, Schneider M S, Hinchey K T. 

Implementing practice guidelines for diabetes 

care using problem-based learning. A 

prospective controlled trial using firm systems. 

Diabetes Care 99;22(10):1672-8 

<15 trained physicians, PI or QI, Other 

Bennett P. The process of postgraduate education in 

general practice. Practitioner 

87;231(1436):1298-302 

No original data, No evaluation 

Beno L, Hinchman J, Kibbe D et al. Design and 

implementation of training to improve 

management of pediatric overweight. J Contin 

Educ Health Prof 2005;25(4):248-58 

No comparison 

Berkenstadt H, Ziv A, Barsuk D et al. The use of 

advanced simulation in the training of 

anesthesiologists to treat chemical warfare 

casualties. Anesth Analg 2003;96(6):1739-42, 

table of contents 

Not US or Canada 

Berkowitz K J, Anderson L A, Panayioto R M et al. 

Mini-residency on diabetes care for healthcare 

providers: enhanced knowledge and attitudes 

with unexpected challenges to assessing behavior 

change.. Diabetes Educ 98;24(2):143-144, 149 

<15 trained physicians, No comparison 

Berntson A, Goldner E, Leverette J et al. Psychiatric 

training in rural and remote areas: increasing 

skills and building partnerships. Can J Psychiatry 

2005;50(9):1-8 

No original data, No original data 

Birnbaum M L, Robinson N E, Kuska B M et al. 

Effect of advanced cardiac life-support training 

in rural, community hospitals. Crit Care Med 

94;22(5):741-9 

No comparison 

Bjornson DC, Rector TS, Daniels CE et al. Impact of 

a drug-use review program intervention on 

prescribing after publication of a randomized 

clinical trial [see comment]. American journal of 

hospital pharmacy. 90;47(7):1541 

Other 

Black F. "You show me yours and I'll show you 

mine". A new CME option--learning from your 

own consultations. Aust Fam Physician 

94;23(9):1788-90 

No original data, Not US or Canada 

Blackstien-Hirsch P, Anderson G, Cicutto L et al. 

Implementing continuing education strategies for 

family physicians to enhance asthma patients' 

quality of life. J Asthma 2000;37(3):247-57 

No comparison 

Blum R H, Raemer D B, Carroll J S et al. Crisis 

resource management training for an anaesthesia 

faculty: a new approach to continuing education. 

Med Educ 2004;38(1):45-55 

No comparison 
Appendix D: List of Excluded Articles

D-3 

Boland M R, Hornblow A R, Gibbs J M et al. The 

learning process in medical education. NEW 

ZEALAND MED. J. 85;98(788):861-863 

No original data 

Booth M B. Teaching and learning in a neonatal 

intensive care unit. Arch Dis Child 98;78(3):275-

7 

No original data, Not US or Canada 

Bordley W C, Travers D, Scanlon P et al. Office 

preparedness for pediatric emergencies: a 

randomized, controlled trial of an office-based 

training program. Pediatrics 2003;112(2):291-5 

PI or QI 

Borgiel A E, Williams J I, Davis D A et al. 

Evaluating the effectiveness of 2 educational 

interventions in family practice. CMAJ 

99;161(8):965-70 

PI or QI 

Botelho R J, McDaniel S H, Jones J E. Using a 

family systems approach in a balint-style group: 

an innovative course for continuing medical 

education. Fam Med 90;22(4):293-5 

<15 trained physicians, No comparison 

Boudreau D, Tamblyn R, Dufresne L. Evaluation of 

consultative skills in respiratory medicine using 

a structured medical consultation. Am J Respir 

Crit Care Med 94;150(5 Pt 1):1298-304 

<15 trained physicians, Does not apply 

Bratton S L, Cabana M D, Brown R W et al. Asthma 

educational seminar targeting Medicaid 

providers. Respir Care 2006;51(1):49-55 

No comparison 

Bravata D M, Huot S J, Abernathy H S et al. The 

development and implementation of a curriculum 

to improve clinicians' self-directed learning 

skills: a pilot project. BMC Med Educ 2003;37 

<15 trained physicians 

Brezis M, Cohen R. Interactive learning with voting 

technology. Med Educ 2004;38(5):574-5 

No original data, <15 trained physicians 

Brown R L. Evaluation of a continuing medical 

education program for primary care physicians 

on the management of alcoholism. J Med Educ 

88;63(6):482-4 

No comparison 

Brown R M. What do phonemes have to do with 

learning medicine?. Acad Med 2005;80(5):455 

No original data 

Brown R M. What do phonemes have to do with 

learning medicine?. Acad Med 2005;80(5):455 

Does not apply 

Browning P, Foss G. Evaluation of rehabilitation 

continuing education. Annu Rev Rehabil 83;364-

92 

No original data 

Burlingame G M, Earnshaw D, Hoag M et al. A 

systematic program to enhance clinician group 

skills in an inpatient psychiatric hospital. Int. J. 

Group Psychother. 2002;52(4):555-587 

No original data, <15 trained physicians 

Burr R, Johanson R. Continuing medical education: 

an opportunity for bringing about change in 

clinical practice. Br J Obstet Gynaecol 

98;105(9):940-5 

No original data 

Byrne A J, Sellen A J, Jones J G et al. Effect of 

videotape feedback on anaesthetists' performance 

while managing simulated anaesthetic crises: a 

multicentre study. Anaesthesia 2002;57(2):176-9 

Not US or Canada 

Byrnes J A, Kulick T A, Schwartz D G. Informationseeking behavior changes in community-based 

teaching practices. J Med Libr Assoc 

2004;92(3):334-40 

No comparison 

Cabana MD, Slish KK, Evans D et al. Randomized 

control trial of physician education to improve 

pediatric outcomes for asthma [Abstract]. 

American Thoracic Society 2005 International 

Conference; May 20-25; San Diego, California 

2005;[D93] [Poster: 626] 

Abstract 

Cahan J I. Continuing medical education. An 

effective tool or a faulty fantasy?. Md Med 

2001;2(3):14-6 

No original data, No evaluation 

Cameron C, Naylor C D. No impact from active 

dissemination of the Ottawa Ankle Rules: further 

evidence of the need for local implementation of 

practice guidelines. CMAJ 99;160(8):1165-8 

<15 trained physicians, Not US or Canada 
Appendix D: List of Excluded Articles

D-4 

Campbell C, Parboosingh J, Gondocz T et al. A study 

of the factors that influence physicians'' 

commitments to change their practices using 

learning diaries. Acad Med 99;74(10 

Suppl):S34-6 

No training, No comparison 

Campbell D D. Implementing and evaluating 

continuing professional education: elements of a 

strategy. Physiother Can 83;35(5):253-6 

No original data 

Campbell H S, Fletcher S W, Pilgrim C A et al. 

Improving physicians' and nurses' clinical breast 

examination: a randomized controlled trial. Am J 

Prev Med 91;7(1):1-8 

<15 trained physicians 

Campbell N R, McAlister F A, Brant R et al. 

Temporal trends in antihypertensive drug 

prescriptions in Canada before and after 

introduction of the Canadian Hypertension 

Education Program. J Hypertens 

2003;21(8):1591-7 

No comparison 

Carbonell J L, Chez R A, Hassler R S. Florida 

physician and nurse education and practice 

related to domestic violence. Womens Health 

Issues 95;5(4):203-7 

No evaluation, No comparison 

Cardenas V M, Roces M C, Wattanasri S et al. 

Improving global public health leadership 

through training in epidemiology and public 

health: the experience of TEPHINET. Training 

Programs in Epidemiology and Public Health 

Interventions Network. Am J Public Health 

2002;92(2):196-7 

No original data, Does not apply 

Carlson B. A day spent learning how to satisfy 

patients. Manag Care 97;6(9):57-60 

No original data, Does not apply 

Carpenter J L, Battles J B, McIntire D et al. 

Assessing the usefulness of using standardized 

patients in a clinical medicine course.. Acad Med 

92;67(4):286 

<15 trained physicians 

Casebeer L L, Strasser S M, Spettell C M et al. 

Designing tailored Web-based instruction to 

improve practicing physicians' preventive 

practices. J Med Internet Res 2003;5(3):e20 

No original data, Does not apply, Other 

Casebeer L, Kristofco R E, Strasser S et al. 

Standardizing evaluation of on-line continuing 

medical education: physician knowledge, 

attitudes, and reflection on practice. J Contin 

Educ Health Prof 2004;24(2):68-75 

No original data, No comparison 

Cauffman J G, Rasgon I M, Mayne J C. Relationship 

between quality of CME instruction and changes 

in physicians' patient-management plans. J. 

MED. EDUC. 85;60(6):486-488 

No comparison 

Centor R, Casebeer L, Klapow J. Using a combined 

CME course to improve physicians'' skills in 

eliciting patient adherence. Acad Med 

98;73(5):609-10 

No comparison 

Cerne F. Learning to survive. Continuing education is 

helping health care managers thrive in today''s 

managed care-driven environment. Hosp Health 

Netw 95;69(17):47-8, 50 

No original data, Does not apply 

Chen F M, Burstin H, Huntington J. The importance 

of clinical outcomes in medical education 

research. Med Educ 2005;39(4):350-1 

No original data 

Cherkin D, Deyo R A, Berg A O. Evaluation of a 

physician education intervention to improve 

primary care for low-back pain. II. Impact on 

patients. Spine 91;16(10):1173-8 

No comparison 

Chopra V, Gesink B J, de Jong J et al. Does training 

on an anaesthesia simulator lead to improvement 

in performance?. Br J Anaesth 94;73(3):293-7 

Not US or Canada 

Clark K M. Notes on continuing education: SelfDirected Learning Network. J Contin Educ Nurs 

86;17(3):101-3 

No original data, No evaluation 

CME evaluation. Semin. Oncol. 2005;32(SUPPL. 

2):S23 

Does not apply 

CME evaluation. Semin. Oncol. 2005;32(SUPPL. 

2):S23 

No original data 
Appendix D: List of Excluded Articles

D-5 

CME examination for volume 41Include entire 

volume with the exception of the CME articles. J 

Am Acad Dermatol 2000;42(6):33-6 

No original data, No evaluation 

CME examination. Am J Obstet Gynecol 98;179(6 Pt 

2):114-8 

No original data 

CME examination. J. Am. Acad. Dermatol. 

2005;53(2):210-212 

No original data, No evaluation, Does not 

apply 

CME examination: July 2005. Pediatr. Emerg. Care 

2005;21(7):457 

No original data, No evaluation, Does not 

apply a 

CME quiz & evaluation. Ca Cancer J. Clin. 

2005;55(4):260-264 

No original data, No evaluation, Does not 

apply 

Cohen S J, Stookey G K, Katz B P et al. Encouraging 

primary care physicians to help smokers quit. A 

randomized, controlled trial. Ann Intern Med 

89;110(8):648-52 

<15 trained physicians 

Cole T B, Glass R M. Learning associated with 

participation in journal-based continuing medical 

education. J Contin Educ Health Prof 

2004;24(4):205-12 

No comparison 

Coleman E A, Stewart C B, Wilson S et al. An 

evaluation of standardized patients in improving 

clinical breast examinations for military women. 

Cancer Nurs 2004;27(6):474-82 

<15 trained physicians, PI or QI 

Collins R, Hammond M. Self-directed learning to 

educate medical educators, Part 2: Why do we 

use self-directed learning?. Med Teach 

87;9(4):425-32 

No original data., Not US or Canada 

Connolly Nancy K, Williams Mark E. Facilitating 

Change through CME in Geriatrics.. Journal of 

Continuing Education in the Health Professions 

92;12(4):215-224 

No comparison 

Continuing education: keeping pace with the 

changing scene. Council on Practice Continuing 

Education Committee. J Am Diet Assoc 

88;88(10):1224-5 

No original data, Does not apply 

Continuing Medical Education examination: EBV the 

prototypical human tumor virus - Just how bad is 

it?. J. Allergy Clin. Immunol. 2005;116(2):262 

No original data, Does not apply 

Continuing Medical Education examination: Innate 

immune responses to infection. J. Allergy Clin. 

Immunol. 2005;116(2):250 

No original data, Does not apply 

Cook D A. Internet-based continuing medical 

education. JAMA 2006;295(7):758; author reply 

758-9 

No original data 

Craig M, Nichols A, Price D. Education for general 

practitioners proposing to administer anaesthesia 

in rural general practice. Anaesth Intensive Care 

93;21(4):432-41 

Not US or Canada 

Craychee G A. The psychosocial dimension of 

professional continuing education: behavioral 

intentions. Radiol Technol 87;58(6):529-35 

<15 trained physicians, Does not apply 

Crow W C. Using "pearls" to ensure that learners 

learn key information. Fam Med 

2004;36(9):619-21 

No original data, <15 trained physicians, No 

training, No evaluation 

Cunningham A M, Edwards A, Jones K V et al. 

Evaluation of a service development to increase 

detection of urinary tract infections in children. J 

Eval Clin Pract 2005;11(1):73-6 

Not US or Canada 

Curtis J R, Rubenfeld G D, Hudson L D. Training 

pulmonary and critical care physicians in 

outcomes research: should we take the 

challenge?. Am J Respir Crit Care Med 98;157(4 

Pt 1):1012-5 

No original data 
Appendix D: List of Excluded Articles

D-6 

Curtis J R, Rubenfeld G D, Hudson L D. Training 

pulmonary and critical care physicians in 

outcomes research: should we take the 

challenge?. Am J Respir Crit Care Med 98;157(4 

Pt 1):1012-5 

Does not apply 

Curtis P, Carey T S, Evans P et al. Training primary 

care physicians to give limited manual therapy 

for low back pain: patient outcomes. Spine 

2000;25(22):2954-60; discussion 2960-1 

No comparison 

Da Costa M. Practice-based continuing medical 

education. Educ. Gen. Pract. 99;10(1):70-72 

<15 trained physicians, No comparison 

D'Alessandro M P, Galvin J R, Erkonen W E et al. 

The instructional effectiveness of a radiology 

multimedia textbook (HyperLung) versus a 

standard lecture. Invest Radiol 93;28(7):643-8 

<15 trained physicians 

Dankelman J, Wentink M, Grimbergen C A et al. 

Does virtual reality training make sense in 

interventional radiology? Training skill-, ruleand knowledge-based behavior. Cardiovasc. 

Intervent. Radiol. 2004;27(5):417-421 

No original data, Not US or Canada 

Dark G G. Learning on the Internet. Br J Hosp Med 

97;58(11):572-4 

No original data 

Dauphinee W D. Role of examinations of the 

Medical Council of Canada in improving 

medical standards. CAN. MED. ASSOC. J. 

81;124(11):1425-1427 

No original data 

David T J, Dolmans D H, Patel L et al. Problembased learning as an alternative to lecture-based 

continuing medical education. J R Soc Med 

98;91(12):626-30 

No original data, Does not apply 

Davis D, O''Brien M A T, Freemantle N et al. 

Review: Interactive, but not didactic, continuing 

medical education is effective in changing 

physician performance. Evid.-Based Med. 

2000;5(2):64 

No original data 

Davis D. Evaluating continuing medical education: 

common sense and science. CMAJ 

86;134(5):485-6 

No original data 

Davis M H, Karunathilake I. The place of the oral 

examination in today''s assessment systems. Med 

Teach 2005;27(4):294-7 

No original data 

Dayal R, Faries P L, Lin S C et al. Computer 

simulation as a component of catheter-based 

training. J Vasc Surg 2004;40(6):1112-7 

<15 trained physicians 

de Burgh S, Mant A, Mattick R P et al. A controlled 

trial of educational visiting to improve 

benzodiazepine prescribing in general practice. 

Aust J Public Health 95;19(2):142-8 

Not US or Canada 

De Muylder R, Tonglet R, Nackers F et al. 

Randomised evaluation of a specific training of 

general practitioners in cardiovascular 

prevention. Acta Cardiol 2005;60(2):199-205 

Not US or Canada 

Deeb L C, Pettijohn F P, Shirah J K et al. 

Interventions among primary-care practitioners 

to improve care for preventable complications of 

diabetes. Diabetes Care 88;11(3):275-80 

<15 trained physicians, PI or QI 

Delmont G. Using electronic portfolios to document 

learning. Athl. Ther. Today 2003;8(4):22-23 

No evaluation, Does not apply 

Delvaux N, Razavi D. Psychological training for 

health-care professionals in oncology. A way to 

improve communication skills. Ann N Y Acad 

Sci 97;809336-49 

No original data 

Devitt P, Worthley S, Palmer E et al. Evaluation of a 

computer based package on electrocardiography. 

Aust N Z J Med 98;28(4):432-5 

Not US or Canada 

Diarrhoeal Diseases Control Programme. Supervisory 

skills training course. Wkly Epidemiol Rec 

90;65(42):326-7 

No original data, <15 trained physicians, No 

evaluation, Not US or Canada, Does not apply 
Appendix D: List of Excluded Articles

D-7 

Dibbern D A, Wold E. Workshop-based learning: a 

model for teaching ethics. JAMA 95;274(9):770-

1 

No original data, Does not apply 

DiMatteo M R. Evidence-based strategies to foster 

adherence and improve patient outcomes. 

JAAPA 2004;17(11):18-21 

No original data 

Din-Dzietham R, Porterfield D S, Cohen S J et al. 

Quality care improvement program in a 

community-based participatory research project: 

example of Project DIRECT. J Natl Med Assoc 

2004;96(10):1310-21 

No comparison 

DM program cuts back pain costs by educating, 

supporting primary care physicians. Healthc 

Demand Dis Manag 99;5(10):149-53 

Abstract, No evaluation, PI or QI 

Dolan N C, Ng J S, Martin G J et al. Effectiveness of 

a skin cancer control educational intervention for 

internal medicine housestaff and attending 

physicians. J Gen Intern Med 97;12(9):531-6 

<15 trained physicians 

Dornan T, David T. Adult learning and continuing 

education. Diabet Med 2000;17(1):78-80 

No original data, Does not apply 

Dornbusch D, Allegra C, Willey J et al. How Do U.S. 

Medical Oncologists Learn and Apply New 

Clinical Trials Information from Press Releases 

in Nonmedical Media? A Case Study Based on 

ECOG 4599. Oncologist 2006;11(1):31-8 

No training, No comparison 

D''Orsi C J, Karellas A, Costanza M E et al. 

Preliminary report of an intervention to improve 

mammography skills of radiologists. Prog Clin 

Biol Res 89;293151-7 

No comparison 

Dougherty M. Going the distance. Balancing work 

and education in a distance learning program. J 

AHIMA 2005;76(6):38-42 

No human data, <15 trained physicians, Does 

not apply 

Dowell J, Pagliari C, McAleer S. Development and 

evaluation of a concordance training course for 

medical practitioners. Med Teach 

2004;26(4):384-6 

<15 trained physicians, Not US or Canada, No 

comparison 

Du Pen A R, Du Pen S, Hansberry J et al. An 

educational implementation of a cancer pain 

algorithm for ambulatory care. Pain Manag Nurs 

2000;1(4):116-28 

<15 trained physicians 

Duggan C M. Designing effective training. J AHIMA 

2005;76(6):28-32; quiz 33-4 

Does not apply 

Dumay A C, Jense G J. Endoscopic surgery 

simulation in a virtual environment. Comput 

Biol Med 95;25(2):139-48 

No original data, Not US or Canada 

Duncan A K, Multari A, Li J. Curriculum in 

physician-patient communication skills for new 

faculty. Acad Med 2002;77(5):462 

No original data, Does not apply 

Dutta P K. Continuing medical education of general 

practitioners on MCH care through distance 

learning. Indian J Public Health 96;40(3):68-70 

Not US or Canada 

Dykes P C, Acevedo K, Boldrighini J et al. Clinical 

practice guideline adherence before and after 

implementation of the HEARTFELT (HEART 

Failure Effectiveness & Leadership Team) 

intervention. J Cardiovasc Nurs 2005;20(5):306-

14 

<15 trained physicians 

Eisenberg J. Changing provider behavior: is it 

possible?. Clin Perform Qual Health Care 

96;4(4):204-5 

No original data, Does not apply 

El Ansari W, Pearson D, Davis T. Satisfaction with 

interprofessional education? Influences of 

learners'' demographic and academic 

characteristics. J. Interprof. Care 

2002;16(2):174-175 

<15 trained physicians, Not US or Canada 
Appendix D: List of Excluded Articles

D-8 

Elkin P L, Gorman P N. Continuing medical 

education and patient safety: an agenda for 

lifelong learning. J Am Med Inform Assoc 

2002;9(6 Suppl):S128-32 

No original data, No evaluation 

Emlet C A, da Silva P. Medications, behavior and the 

elderly: a continuing education program for 

board and care facility operators. Gerontol 

Geriatr Educ 86;6(4):41-51 

<15 trained physicians, No comparison 

Evans A W, McKenna C, Oliver M. Self-assessment 

in medical practice. J. R. Soc. Med. 

2002;95(10):511-513 

No original data 

Evans P R. CME in general practice. Postgrad Med J 

96;72 Suppl 1S27-9 

No training, Not US or Canada 

Ewart C K, Li V C, Coates T J. Increasing physicians' 

antismoking influence by applying an 

inexpensive feedback technique. J Med Educ 

83;58(6):468-73 

<15 trained physicians, No comparison 

Fallowfield L, Jenkins V, Farewell V et al. Enduring 

impact of communication skills training: results 

of a 12-month follow-up. Br J Cancer 

2003;89(8):1445-9 

Not US or Canada 

Farris K B, Kirking D M, Shimp L A et al. Design 

and results of a group counter-detailing DUR 

educational program. Pharm Res 

96;13(10):1445-52 

<15 trained physicians 

Fasel J, Sieber R, Rohr H P. Laservision-disk and 

computer-assisted medical learning. J Audiov 

Media Med 86;9(1):15-6 

No evaluation, Does not apply 

Featherstone H J, LoGerfo J P, Barnes R H. 

Performance-based continuing medical 

education. Prophylactic cephalosporin use in 

hysterectomy patients. QUAL. REV. BULL. 

83;9(6):169-174 

<15 trained physicians 

Feighny K M, Monaco M, Arnold L. Empathy 

training to improve physician-patient 

communication skills. Acad Med 95;70(5):435-6 

<15 trained physicians 

Fiallo V M, O'Connor F X, Reed W P. Preceptored 

introduction of laparoscopic techniques for 

cholecystectomy into a large university-affiliated 

medical center. Surg Endosc 94;8(9):1063-6 

<15 trained physicians, No comparisonv 

Fincher R M, Abdulla A M, Sridharan M R et al. A 

prospective educational trial comparing efficacy 

of computer-assisted learning and weekly 

seminars in teaching EKG interpretation.. Proc 

Annu Conf Res Med Educ 86;25(-):3-7 

<15 trained physicians 

Finestone A J, Lanzilotti S S, Marks A D et al. An 

assessment of practice-based CME. Pa Med 

87;90(6):45-6, 48 

No comparison 

Fishbein R H. Professionalism and ''the master 

clinician''--an early learning experience. J Eval 

Clin Pract 2000;6(3):241-3 

No original data, Does not apply 

Folberg R, Dickinson L K, Christiansen R A et al. 

Interactive videodisc and compact discinteractive for ophthalmic basic science and 

continuing medical education. Ophthalmology 

93;100(6):842-50 

No original data, No evaluation 

Fox R D, Bennett N L. Continuing medical 

education. Learning and change: Implications for 

continuing medical education. Br. Med. J. 

98;316(7129):466-468 

No original data 

Fox S, Tsou C V, Klos D S. Increasing 

mammography screening: An application of 

general principles of CME methodology. J. 

PSYCHOSOM. OBSTET. GYNECOL. 

85;4(2):95-104 

<15 trained physicians 

Frenkel M, Ben-Arye E, Hermoni D. An approach to 

educating family practice residents and family 

physicians about complementary and alternative 

medicine. Complement Ther Med 2004;12(2-

3):118-25 

Not US or Canada, No comparison 

Friedenberg R M. Medical education and practice in 

a new environment. Radiology 97;202(3):33A36A 

No original data, Does not apply 
Appendix D: List of Excluded Articles

D-9 

Friedmann P, Selbovitz L G. Continuous quality 

improvement and physician training. Qual 

Manag Health Care 92;1(1):13-9 

No original data, No evaluation, Does not 

apply 

Friss L, Lass S. An evaluation approach for 

continuing education programs in the health 

professions. Mobius 82;2(1):5-13 

No original data, No comparison 

Fustukian S, Macdonald J. Health action is: 

continuing education for PHC. HealthAction 

94;(8):2 

Other 

Gainford M C, McCready D, Cohen Z et al. The 

latest is the greatest? Results of a structured 

lecture about aromatase inhibitor use for breast 

cancer. Breast Cancer Res Treat 2005;1-4 

No comparison 

Gallagher A G, McClure N, McGuigan J et al. 

Virtual reality training in laparoscopic surgery: a 

preliminary assessment of minimally invasive 

surgical trainer virtual reality (MIST VR). 

Endoscopy 99;31(4):310-3 

Not US or Canada 

Gallagher R E, Smith D U. Formulation of 

teaching/learning objectives useful for the 

development and assessment of lessons, courses, 

and programs. J Cancer Educ 89;4(4):231-4 

No original data, No evaluation 

Gask L, Dowrick C, Dixon C et al. A pragmatic 

cluster randomized controlled trial of an 

educational intervention for GPs in the 

assessment and management of depression. 

Psychol Med 2004;34(1):63-72 

Not US or Canada, Not US or Canada 

Gautam V, Heyworth J. A method to measure the 

value of formal training in trauma management: 

comparison between ATLS and induction 

courses. Injury 95;26(4):253-5 

Not US or Canada 

Gemson D H, Ashford A R, Dickey L L et al. Putting 

prevention into practice. Impact of a multifaceted 

physician education program on preventive 

services in the inner city. Arch Intern Med 

95;155(20):2210-6 

<15 trained physicians 

Gilbert J H. Interprofessional education for 

collaborative, patient-centred practice.. Can J 

Nurs Leadersh 2005;18(2):32-36, 38 

No original data, Does not apply 

Giovino G A, Cummings K M, Koenigsberg M R et 

al. An evaluation of a physician training program 

on patient smoking cessation. Prog Clin Biol Res 

90;33927-48 

<15 trained physicians 

Girgis A, Sanson-Fisher R W, Howe C et al. A skin 

cancer training programme: evaluation of a 

postgraduate training for family doctors. Med 

Educ 95;29(5):364-71 

Not US or Canada 

Glassman P A, Luck J, O'Gara E M et al. Using 

standardized patients to measure quality: 

evidence from the literature and a prospective 

study. Jt Comm J Qual Improv 2000;26(11):644-

53 

No evaluation 

Glazier R H, Badley E M, Lineker S C et al. Getting 

a Grip on Arthritis: an educational intervention 

for the diagnosis and treatment of arthritis in 

primary care. J Rheumatol 2005;32(1):137-42 

<15 trained physicians 

Godfrey J, Dennick R, Welsh C. Training the 

trainers: do teaching courses develop teaching 

skills?. Med Educ 2004;38(8):844-7 

Not US or Canada 

Goettner P. Effective e-learning for healthcare.. 

Health Manag Technol 2000;21(12):64, 63 

No original data 

Goldberg H I, Wagner E H, Fihn S D et al. A 

randomized controlled trial of CQI teams and 

academic detailing: can they alter compliance 

with guidelines?. Jt Comm J Qual Improv 

98;24(3):130-42 

PI or QI 

Gomel M K, Wutzke S E, Hardcastle D M et al. 

Cost-effectiveness of strategies to market and 

train primary health care physicians in brief 

intervention techniques for hazardous alcohol 

use. Soc Sci Med 98;47(2):203-11 

Not US or Canada 
Appendix D: List of Excluded Articles

D-10 

Gonzalez-Willis A, Rafi I, Boekeloo B et al. Using 

simulated patients to train physicians in sexual 

risk assessment and risk reduction. Acad Med 

90;65(9 Suppl):S7-8 

No comparison 

Gopalakrishnan G, Devarajan V. StapSim: a virtual 

reality-based stapling simulator for laparoscopic 

hemiorrhaphy. Stud Health Technol Inform 

2004;98111-3 

No training, Does not apply 

Gordon M S, Ewy G A, Felner J M et al. A 

cardiology patient simulator for continuing 

education of family physicians. J Fam Pract 

81;13(3):353-6 

No comparison 

Gordon S M, Troncale J. Evaluation of the 

effectiveness of an addiction treatment training 

program for physicians. Am Clin Lab 

2002;21(5):22-4 

<15 trained physicians 

Gore M J. Outcomes assessment: Latest way to gauge 

the value of CLS education. CLIN. LAB. SCI. 

91;4(2):70-73 

No original data 

Gormley G J, Steele W K, Stevenson M et al. A 

randomised study of two training programmes 

for general practitioners in the techniques of 

shoulder injection. Ann Rheum Dis 

2003;62(10):1006-9 

Not US or Canada 

Goulet F, Jacques A, Gagnon R. An innovative 

approach to remedial continuing medical 

education, 1992-2002. Acad Med 

2005;80(6):533-40 

Does not apply, No comparison 

Greco M, Buckley J, Francis W. Triads: An effective 

method for learning the art of listening. EDUC. 

GEN. PRACT. 97;8(4):329-337 

Not US or Canada 

Greenberg L W, Jewett L S. The impact of two 

educational techniques on physician knowledge, 

performance and patient care. Mobius 

84;4(4):51-4 

Other 

Greene R A, Beckman H, Chamberlain J et al. 

Increasing adherence to a community-based 

guideline for acute sinusitis through education, 

physician profiling, and financial incentives. Am 

J Manag Care 2004;10(10):670-8 

No comparison 

Griffin G A, Barry S M. Muscle and joint pain: 

design and evaluation of courses for general 

practitioners. J R Coll Gen Pract 

81;31(232):661-8 

Not US or Canada 

Grogan E L, Stiles R A, France D J et al. The impact 

of aviation-based teamwork training on the 

attitudes of health-care professionals. J Am Coll 

Surg 2004;199(6):843-8 

No comparison 

Groveman H D, Ganiats T G, Klauber M R et al. 

Computer-assisted assessment of family 

physicians'' knowledge about cancer screening 

guidelines. West J Med 85;143(4):541-4 

No comparison 

Groveman H D, Sanowski R A, Klauber M R. 

Training primary care physicians in flexible 

sigmoidoscopy--performance evaluation of 

17,167 procedures. West J Med 88;148(2):221-4 

No comparison 

Grunwald T, Clark D, Fisher S S et al. Using 

cognitive task analysis to facilitate collaboration 

in development of simulator to accelerate 

surgical training. Stud Health Technol Inform 

2004;98114-20 

No training, No evaluation 

Guenther S M, Laube D W, Matthes S. Effectiveness 

of the gynecology teaching associate in teaching 

pelvic examination skills. J. MED. EDUC. 

83;58(1):67-69 

<15 trained physicians 

Gullion D S, Adamson T E, Watts M S. The effect of 

an individualized practice-based CME program 

on physician performance and patient outcomes. 

West J Med 83;138(4):582-8 

No comparison 

Gullion D S, Adamson T E. A practice-based CME 

program in hypertension using a medication and 

behavioral treatment approach. Proc Annu Conf 

Res Med Educ 85;24207-12 

No comparison 
Appendix D: List of Excluded Articles

D-11 

Gurwitz JH, Noonan JP, Soumerai SB. Reducing the 

use of H2-receptor antagonists in the long-termcare setting [see comment]. Journal of the 

American Geriatrics Society. 92;40(4):359 

No comparison 

Hall P, Hupe D, Scott J. Palliative care education for 

community-based family physicians: the 

development of a program, the evaluation, and 

its consequences. J Palliat Care 98;14(3):69-74 

No comparison 

Halm E A, Horowitz C, Silver A et al. Limited 

impact of a multicenter intervention to improve 

the quality and efficiency of pneumonia care. 

Chest 2004;126(1):100-7 

<15 trained physicians, No comparison, PI or 

QI

Hamstra S J, Dubrowski A. Effective training and 

assessment of surgical skills, and the correlates 

of performance. Surg Innov 2005;12(1):71-7 

No original data, <15 trained physicians 

Handler S. Does continuing medical education affect 

medical care. A study of improved transfusion 

practices. Minn Med 83;66(3):167-80 

Does not apply, Other 

Handmaker N S, Hester R K, Delaney H D. 

Videotaped training in alcohol counseling for 

obstetric care practitioners: A randomized 

controlled trial. Obstet. Gynecol. 99;93(2):213-

218 

<15 trained physicians 

Hardern R D. Teaching and learning evidence based 

medicine skills in accident and emergency 

medicine. J. Accid. Emerg. Med. 99;16(2):126-

129 

No original data 

Harewood G C, Yusuf T E, Clain J E et al. 

Assessment of the impact of an educational 

course on knowledge of appropriate EUS 

indications. Gastrointest Endosc 2005;61(4):554-

9 

No comparison 

Harper G, Norris D, Woo B. Physician education in a 

diverse society: listening to and learning from 

new voices. Pharos Alpha Omega Alpha Honor 

Med Soc 95;58(2):39-42 

No original data, No training 

Harris G D. Professionalism: part II -- teaching and 

assessing the learner''s professionalism. Fam 

Med 2004;36(6):390-2 

No original data, No training, Does not apply 

Harris I B, Wempner J. Continuing medical 

education reconceived: evaluation of a sabbatical 

program for physicians. Acad Med 96;71(10 

Suppl):S46-8 

No comparison 

Harrison R V, Gallay L S, McKay N E et al. The 

association between community physician's 

attendance at a medical center's CME courses 

and their patient referrals to the medical center. J 

Contin Educ Health Prof 90;10(4):315-20 

Does not apply 

Haynes R B, Johnston M E, McKibbon K A et al. A 

program to enhance clinical use of MEDLINE. A 

randomized controlled trial. Online J Curr Clin 

Trials 93;Doc No 56[4005 words; 39 paragraphs] 

<15 trained physicians 

Heins H C, Miller J M, Childs D et al. Perinatal 

educational programs and perinatal outcomes. J 

S C Med Assoc 84;80(9):431-5 

<15 trained physicians 

Henderson R W. Learners in your practice.. Can Fam 

Physician 95;41(-):32-33 

No original data 

Herdson P B. Pathology, pathologists and problembased learning.. Pathology 98;30(3):326-327 

No original data, Not US or Canada 

Hinchman J, Beno L, Dennison D et al. Evaluation of 

a training to improve management of pediatric 

overweight. J Contin Educ Health Prof 

2005;25(4):259-67 

<15 trained physicians 

Hirsch I B, Goldberg H I, Ellsworth A et al. A 

multifaceted intervention in support of diabetes 

treatment guidelines: a cont trial. Diabetes Res 

Clin Pract 2002;58(1):27-36 

<15 trained physicians 
Appendix D: List of Excluded Articles

D-12 

Hochberger J, Euler K, Naegel A et al. The compact 

Erlangen Active Simulator for Interventional 

Endoscopy: a prospective comparison in 

structured team-training courses on "endoscopic 

hemostasis" for doctors and nurses to the "EndoTrainer" model. Scand J Gastroenterol 

2004;39(9):895-902 

Not US or Canada 

Holloway R G, Gifford D R, Frankel M R et al. A 

randomized trial to implement practice 

recommendations: design and methods of the 

Dementia Care Study. Control Clin Trials 

99;20(4):369-85 

No training, Other 

Holmes D R, Fox R D, Tommaso C et al. Renal and 

iliac artery stenting by interventional 

cardiologists and vascular surgeons: the 

Foundation to Advance Medical Education 

(FAME) initiative. Am Heart J 2005;149(5):883-

7 

No comparison 

Holmes G R, Smith M E, Donald A G. Behavioral 

science in medical education: a 1985 updated 

bibliography. Psychol Rep 85;57(3 Pt 1):895-9 

No original data 

Holsgrove G. Getting the most out of distance 

learning. Practitioner 91;235(1500):196-200 

No evaluation, Does not apply 

Holsgrove G. Techniques for distance learning. 

Practitioner 91;235(1501):296-300 

No evaluation, Does not apply 

Hood A F. CME examination. J. Am. Acad. 

Dermatol. 2005;53(3):389-392 

No original data, Does not apply 

Horowitz C R, Goldberg H I, Martin D P et al. 

Conducting a randomized controlled trial of CQI 

and academic detailing to implement clinical 

guidelines. Jt Comm J Qual Improv 

96;22(11):734-50 

PI or QI 

Houge D R. Participant/observer evaluation in 

continuing medical education. J Med Educ 

81;56(6):527 

Does not apply, No comparison 

HSJ awards 2005. Skills development. Winner: new 

standards for doctors'' training that take in 

general management skills. Health Serv J 

2005;115(5982):suppl 51-2 

No original data, Does not apply 

Huas D, Wallace P. Is participation in research as an 

investigator an effective form of continuing 

medical education?. Br J Gen Pract 

2000;50(461):982-3 

Not US or Canada 

Hull A L, Wasman J, Goodnough L T. Effects of a 

CME program on physicians' transfusion 

practices. Acad Med 89;64(11):681-5 

<15 trained physicians, PI or QI 

Hull S A. Rheumatology education for general 

practice. Ann Rheum Dis 91;50 Suppl 3449-52 

No original data 

Hummel L J. An investigation of physician selfdirected learning activities. Proc Annu Conf Res 

Med Educ 85;24213-8 

No evaluation, Does not apply 

Hunt D M. Appraisal and assessment. Ann R Coll 

Surg Engl 98;80(6 Suppl):281-3 

No original data, No training 

Hunter T B. Electronic aids to education and practice 

in radiology: the personal computer in the 

radiologist''s office. J Digit Imaging 89;2(2):71-4 

No original data 

Hutchinson L. Evaluating and researching the 

effectiveness of educational interventions. BMJ 

99;318(7193):1267-9 

No original data 

Hux J E, Melady M P, DeBoer D. Confidential 

prescriber feedback and education to improve 

antibiotic use in primary care: a controlled trial. 

CMAJ 99;161(4):388-92 

PI or QI 

Hyler S E. APA Online CME Practice Guideline for 

the Treatment of Patients with Major Depressive 

Disorder. J Psychiatr Pract 2002;8(5):315-9 

No original data, <15 trained physicians, No

evaluation, Does not apply 

Illes J, Glover G H, Wexler L et al. A model for 

faculty mentoring in academic radiology. Acad. 

Radiol. 2000;7(9):717-724 

No comparison 
Appendix D: List of Excluded Articles

D-13 

Inui T S, Yourtee E L, Williamson J W. Improved 

outcomes in hypertension after physician 

tutorials. A controlled trial. Ann Intern Med 

76;84(6): 646-51 

Pre 1981 

Irvine Doran D M, Baker G R, Murray M et al. 

Achieving clinical improvement: an 

interdisciplinary intervention. Health Care 

Manage Rev 2002;27(4):42-56 

<15 trained physicians, Other 

Jacobs B. Hardly child''s play: implementing a 

pediatric-specific, integrated CPOE system. 

Midwest pediatric hospital tackles all the 

hurdles--needs assessment, clinician buy-in, 

training and measuring results--as it strengthens 

its patient safety efforts with wireless CPOE. 

Health Manag Technol 2004;25(8):30-2 

Does not apply, Other 

Jacobs K, Aja D, Hermenau D. Adult learning 

through case simulation. Am J Occup Ther 

94;48(11):1089-92 

No original data 

Jansen J J, Scherpbier A J, Metz J C et al. 

Performance-based assessment in continuing 

medical education for general practitioners: 

construct validity. Med Educ 96;30(5):339-44 

Not US or Canada, No comparison 

Jennett P A, Laxdal O E, Hayton R C et al. Designing 

education interventions to improve physician 

performance in office practice. Mobius 

84;4(4):55-61 

No evaluation 

Johannes B, Salnitski V P, Goeters K M et al. 

Learning with simulation only--artificial skills. J 

Gravit Physiol 2004;11(2):P27-8 

No original data, <15 trained physicians 

Johnston M. Supporting learners. Nurs Times 

91;87(12):47-8 

No evaluation, Does not apply 

Jones D L. Viability of the commitment-for-change 

evaluation strategy in continuing medical 

education. Acad Med 90;65(9 Suppl):S37-8 

No comparison 

Jones J M, James J, Rodin G et al. The evaluation of 

a two-day interdisciplinary continuing education 

event in psychosocial oncology. J Cancer Educ 

2004;19(3):161-4 

<15 trained physicians 

Jones R, Spencer J. Conference report: teaching old 

docs new tricks: research dissemination and 

professional behavioural change. Fam Pract 

93;10(2):229-30 

No original data, No training 

Joos S K, Hickam D H, Gordon G H et al. Effects of 

a physician communication intervention on 

patient care outcomes. J Gen Intern Med 

96;11(3):147-55 

<15 trained physicians 

Jordan J A, Gallagher A G, McGuigan J et al. Virtual 

reality training leads to faster adaptation to the 

novel psychomotor restrictions encountered by 

laparoscopic surgeons. Surg Endosc 

2001;15(10):1080-4 

Not US or Canada 

Kaegi L. Using guidelines to change clinical 

behavior: dissemination through Area Health 

Education Centers and Geriatric Education 

Centers. QRB Qual Rev Bull 93;19(5):165-9 

No original data, Does not apply 

Kanter M H, van Maanen D, Anders K H et al. A 

study of an educational intervention to decrease 

inappropriate preoperative autologous blood 

donation: its effectiveness and the effect on 

subsequent transfusion rates in elective 

hysterectomy. Transfusion 99;39(8):801-7 

<15 trained physicians 

Karlsten R, Strom K, Gunningberg L. Improving 

assessment of postoperative pain in surgical 

wards by education and training. Qual Saf Health 

Care 2005;14(5):332-5 

<15 trained physicians, Not US or Canada 

Karp J G, Hyler I, Wald M et al. The use of an 

audiotaped analysis in a continuous case 

seminar. Psychoanal Q 93;62(2):263-9 

<15 trained physicians 

Katz H P, Goldfinger S E, Fletcher S W. Academiaindustry collaboration in continuing medical 

education: description of two approaches. J 

Contin Educ Health Prof 2002;22(1):43-54 

No evaluation, Does not apply 
Appendix D: List of Excluded Articles

D-14 

Katz S, Feigenbaum A, Pasternak S et al. An 

interactive course to enhance self-efficacy of 

family practitioners to treat obesity. BMC Med 

Educ 2005;5(1):4 

Not US or Canada, No comparison 

Kaufman J, Forman W B. Hospice and palliative 

care: an educational intervention for healthcare 

professionals in a rural community. Am J Hosp 

Palliat Care 2005;22(6):415-8 

<15 trained physicians, No comparison 

Kauth M R, Sullivan G, Henderson K L. Supporting 

clinicians in the development of best practice 

innovations in education. Psychiatr Serv 

2005;56(7):786-8 

No original data, No evaluation, Does not 

apply 

Kemper K J, Amata-Kynvi A, Sanghavi D et al. 

Randomized trial of an internet curriculum on 

herbs and other dietary supplements for health 

care professionals. Acad Med 2002;77(9):882-9 

<15 trained physicians 

Kennedy A, Gask L, Rogers A. Training 

professionals to engage with and promote selfmanagement. Health Educ Res 2005;20(5):567-

78 

Not US or Canada 

Keyserling T C, Ammerman A S, Davis C E et al. A 

randomized controlled trial of a physiciandirected treatment program for low-income 

patients with high blood cholesterol: the 

Southeast Cholesterol Project. Arch Fam Med 

97;6(2):135-45 

<15 trained physicians, PI or QI, Other 

Khan M S, Bann S D, Darzi A et al. Assessing 

surgical skill. Plast Reconstr Surg 

2003;112(7):1886-9 

No training, No evaluation 

Khan M S, Bann S D, Darzi A et al. Assessing 

surgical skill. Plast Reconstr Surg 

2003;112(7):1886-9 

No original data 

Kiessling A, Henriksson P, O''Malley P G et al. Case 

method learning for general practitioners reduces 

cholesterol concentrations in coronary artery 

disease. Evid.-Based Med. 2003;8(3):95 

No original data, Not US or Canada 

Kim M, Kim Y, Park J H. Development and 

utilization of computer-assisted learning (CAL) 

program for medical education.. Medinfo 95;8 Pt 

2(-):1173-1176 

Not US or Canada 

Kinsinger L S, Harris R, Qaqish B et al. Using an 

office system intervention to increase breast 

cancer screening. J Gen Intern Med 

98;13(8):507-14 

PI or QI 

Kirby R L. Inspection-palpation-percussionauscultation and an outcome-oriented alternative 

approach to the musculoskeletal examination. 

MED. EDUC. 81;15(2):106-109 

<15 trained physicians 

Kjeldmand D, Holmstrom I, Rosenqvist U. Balint 

training makes GPs thrive better in their job. 

Patient Educ Couns 2004;55(2):230-5 

Not US or Canada 

Klein L E, Charache P, Johannes R S. Effect of 

physician tutorials on prescribing patterns of 

graduate physicians. J Med Educ 81;56(6):504-

11 

<15 trained physicians 

Knapp J F, Dowd M D, Kennedy C S et al. 

Evaluation of a curriculum for intimate partner 

violence screening in a pediatric emergency 

department. Pediatrics 2006;117(1):110-6 

No comparison 

Knox J D E. Training of teachers in general medical 

practice. MED. EDUC. 84;18(5):379-380 

Not US or Canada, Does not apply 

Kort W J, Marquet R L, Smith A R et al. Evaluation 

of five training courses in microsurgery. Neth J 

Surg 88;40(3):90-2 

Not US or Canada, No comparison 

Kramer J. Teaching and learning in rural general 

practice. Aust Fam Physician 2004;33(9):737-9 

No original data, No training, No evaluation, Not 

US or Canada, Does not apply

Kristofco R E, Hall S A, Chick E. Needs assessment 

survey. CME preferences, practices of West 

Virginia physicians. W V Med J 87;83(5):223-5 

No training, Does not apply 
Appendix D: List of Excluded Articles

D-15 

Krombach G, Ganser A, Fricke C et al. Virtual 

placement of frontal ventricular catheters using 

frameless neuronavigation: an "unbloody 

training" for young neurosurgeons. Minim 

Invasive Neurosurg 2000;43(4):171-5 

Not US or Canada 

Kuban K C, O'Shea M, Allred E et al. Video and CDROM as a training tool for performing 

neurologic examinations of 1-year-old children 

in a multicenter epidemiologic study. J Child 

Neurol 2005;20(10):829-31 

No comparison 

Labelle M, Beaulieu M, Paquette D et al. An 

integrated approach to improving appropriate use 

of anti-inflammatory medication in the treatment 

of osteoarthritis in Quebec (Canada): the 

CURATA model. Med Teach 2004;26(5):463-70 

No comparison 

LaDuca A, Leone-Perkins M, De Champlain A. 

Evaluating continuing competence of physicians 

through multiple assessment modalities: the 

Physicians'' Continued Competence Assessment 

Program (PCCAP). Acad Med 97;72(5):457-8 

No evaluation, No comparison 

Ladyshewsky R. Simulated patients and assessment. 

Med. Teach. 99;21(3):266-269 

No original data, Not US or Canada 

Lalonde J. How physicians learn. Better methods of 

delivering CME using online practice-based 

learning. Cost Qual 2000;6(3):29-31 

No original data, No evaluation 

Lang L. CME series focuses on Patient-Centered GI 

Practice. Gastroenterology 2004;127(1):5 

No original data 

Lau M A, Dubord G M, Parikh S V. Design and 

feasibility of a new cognitive-behavioural 

therapy course using a longitudinal interactive 

format. Can J Psychiatry 2004;49(10):696-700 

No comparison 

Laube D W, Kretzschmar R M, Guenther S M et al. 

A clinical skills instruction program: the acute 

abdomen. J Med Educ 82;57(9):726-8 

<15 trained physicians 

Lawson K A, Wilcox R E, Littlefield J H et al. 

Educating treatment professionals about 

addiction science research: demographics of 

knowledge and belief changes. Subst Use Misuse 

2004;39(8):1235-58 

<15 trained physicians, No comparison 

Lee E, McNally D L, Zuckerman I H. Evaluation of a 

physician-focused educational intervention on 

medicaid children with asthma. Ann 

Pharmacother 2004;38(6):961-6 

No comparison 

Lehmann H P, Lehmann C U, Freedman J A. The use 

of simulations in computer-aided learning over 

the World Wide Web. JAMA 97;278(21):1788 

No original data 

Lennard M. Vocational training for general practice--

achievements of the last 15 years. Bristol Med 

Chir J 85;100(375):72-3 

No original data, No evaluation 

Lenow J L, Bales R, Smullens S N. The JeffCare 

preceptor model for asthma: a primary care 

physician tutorial training model. Dis Manag 

2003;6(1):35-42 

No comparison 

Leviton L C, Goldenberg R L, Baker C S et al. 

Methods to encourage the use of antenatal 

corticosteroid therapy for fetal maturation: a 

randomized controlled trial. JAMA 

99;281(1):46-52 

<15 trained physicians 

Levy K, Aghababian R V, Hirsch E F et al. An 

Internet-based exercise as a component of an 

overall training program addressing medical 

aspects of radiation emergency management. 

Prehospital Disaster Med 2000;15(2):18-25 

Not US or Canada 

Lewis C C, Pantell R H, Sharp L. Increasing patient 

knowledge, satisfaction, and involvement: 

randomized trial of a communication 

intervention. Pediatrics 91;88(2):351-8 

<15 trained physicians 

Linder J F, Blais J, Enders S R et al. Palliative 

education: a didactic and experiential approach 

to teaching end-of-life care. J Cancer Educ 

99;14(3):154-60 

No comparison 
Appendix D: List of Excluded Articles

D-16 

Lloyd-Jones G, Margetson D, Bligh J G. Problembased learning: a coat of many colours. Med 

Educ 98;32(5):492-4 

No original data 

Lorenz R A. Training health professionals to improve 

the effectiveness of patient education programs. 

Diabetes Educ 86;(12 Suppl):204-9 

No original data 

Love R R. Changing the health promotion behaviors 

of primary care physicians: lessons from two 

projects. Jt Comm J Qual Improv 95;21(7):339-

43 

No original data, No comparison 

Lowy L. Continuing education in the later years: 

learning in the third age. Gerontol Geriatr Educ 

83;4(2):89-106 

No original data 

Lozano P, Finkelstein J A, Carey V J et al. A 

multisite randomized trial of the effects of 

physician education and organizational change in 

chronic-asthma care: health outcomes of the 

Pediatric Asthma Care Patient Outcomes 

Research Team II Study. Arch Pediatr Adolesc 

Med 2004;158(9):875-83 

<15 trained physicians, Other 

Lyden P, Raman R, Liu L et al. NIHSS training and 

certification using a new digital video disk is 

reliable. Stroke 2005;36(11):2446-9 

No comparison 

Lytle J S, Lytle B V, Youmans K G. Learning at a 

distance: people, technology, and learning 

experiments. J AHIMA 95;66(7):64-7 

No original data 

Mackay D M, Hardy S. Learning and teaching 

resource discovery in the Health and Life 

Sciences--partnership and interoperability. 

Health Info Libr J 2005;22 Suppl 270-4 

Not US or Canada 

Madridejos-Mora R, Amado-Guirado E, PerezRodriguez M T. Effectiveness of the 

combination of feedback and educational 

recommendations for improving drug 

prescription in general practice. Med Care 

2004;42(7):643-8 

Not US or Canada 

Mainous A G, 3rd Hueston W J, Love M M et al. An 

evaluation of statewide strategies to reduce 

antibiotic overuse. Fam Med 2000;32(1):22-9 

PI or QI 

Makam R, Rajan C S, Brendon T et al. Training and 

assessment of psychomotor skills for performing 

laparoscopic surgery using BEST-IRIS virtual 

reality training simulator. Stud Health Technol 

Inform 2004;98228-30 

<15 trained physicians, Not US or Canada 

Manfredi C, Czaja R, Freels S et al. Prescribe for 

health. Improving cancer screening in physician 

practices serving low-income and minority 

populations. Arch Fam Med 98;7(4):329-37 

<15 trained physicians 

Manley M, Epps R P, Husten C et al. Clinical 

interventions in tobacco control. A National 

Cancer Institute training program for physicians. 

JAMA 91;266(22):3172-3 

No original data, No evaluation 

Margalit A P, Glick S M, Benbassat J et al. Effect of 

a biopsychosocial approach on patient 

satisfaction and patterns of care. J Gen Intern 

Med 2004;19(5 Pt 2):485-91 

Not US or Canada 

Margolis K L, Money B E, Kopietz L A et al. 

Evaluation of two interventions to improve 

physician diagnostic accuracy in recognizing 

glaucomatous changes of the optic disk.. Proc 

Annu Conf Res Med Educ 87;26(-):28-33 

<15 trained physicians 

Marguet C G, Young M D, L'Esperance J O et al. 

Hand assisted laparoscopic training for 

postgraduate urologists: the role of mentoring. J 

Urol 2004;172(1):286-9 

No training, No comparison 

Markert R J, Barton J C, Rodin A E. Measuring 

participant learning and change in practice 

behavior for continuing medical education 

programs. Annu Conf Res Med Educ 81;2034-9 

No original data, No comparison 

Markert R J, O'Neill S C, Bhatia S C. Using a quasiexperimental research design to assess 

knowledge in continuing medical education 

programs. J Contin Educ Health Prof 

2003;23(3):157-61 

Other 
Appendix D: List of Excluded Articles

D-17 

Massey R U. Problem-based learning: a better way?. 

Conn Med 94;58(12):753 

No original data, Does not apply 

Mast L. Application of the problem-based learning 

model for continuing professional education: a 

continuing medical education program on 

managed care issues--Part II. Am J Manag Care 

97;3(1):77-82 

<15 trained physicians, No comparison 

Mathers N J, Challis M C, Howe A C et al. Portfolios 

in continuing medical education--effective and 

efficient?. Med Educ 99;33(7):521-30 

Not US or Canada 

Mayooran Z, Rombauts L, Brown T I et al. 

Reliability and validity of an objective 

assessment instrument of laparoscopic skill. 

Fertil Steril 2004;82(4):976-8 

Not US or Canada 

Mazze R, Deeb L, Palumbo P J. Altering physicians' 

practice patterns--a nationwide educational 

experiment: evaluation of the Clinical Education 

Program of the American Diabetes Association. 

Diabetes Care 86;9(4):420-5 

No comparison 

McBride P, Underbakke G, Plane MB et al. 

Improving prevention systems in primary care 

practices: the Health Education and Research 

Trial (HEART) [see comment]. The Journal of 

family practice. 2000;49(2):115 

Other 

McKenzie A, Ngobeni O, Bonongo F. Is health 

education effective?. Nurs RSA 92;7(7):26-7 

No original data, No training, No evaluation 

McNabb W L, Cook S, Fischer B et al. 

Dissemination of a continuing education 

program in diabetes to health care professionals. 

Diabetes Educ 94;20(1):35-40 

<15 trained physicians 

Medicare learning online. Optometry 

2004;75(10):658-9 

Does not apply 

Mendoza N, Park CL, Thammasitboon S et al. 

Physician education using case-based learning 

seminar in peditric asthma management. 

Pediatric research. 2001;49(4):138A 

Abstract, <15 trained physicians 

Merckaert I, Libert Y, Delvaux N et al. Factors that 

influence physicians' detection of distress in 

patients with cancer: can a communication skills 

training program improve physicians' detection?. 

Cancer 2005;104(2):411-21 

Not US or Canada 

Modell H I. Use of computer simulations to promote 

active learning in multiple teaching settings.. 

Physiologist 85;28(5):445-446 

No evaluation, No comparison 

Modell H I. Use of computer simulations to promote 

active learning in multiple teaching settings.. 

Physiologist 85;28(5):445-446 

<15 trained physicians 

Monette J, Tamblyn R M, McLeod P J et al. Do 

medical education and practice characteristics 

predict inappropriate prescribing of sedativehypnotics for the elderly?. Acad Med 94;69(10 

Suppl):S10-2 

No training, No evaluation, Does not apply 

Moody L, Waterworth A. A flexible virtual reality 

tutorial for the training and assessment of 

arthroscopic skills. Stud Health Technol Inform 

2004;98244-6 

No original data, Not US or Canada 

Moonie S A, Strunk R C, Crocker S et al. 

Community Asthma Program improves 

appropriate prescribing in moderate to severe 

asthma. J Asthma 2005;42(4):281-9 

<15 trained physicians 

Moore M, Van Schaik J, Montgomery C L. Physician 

retraining, lifelong learning, and the library. Bull 

Med Libr Assoc 92;80(4):374-6 

No original data, No evaluation 

Moore R. Continuing education in practice. 

Practitioner 91;235(1506):663-5 

No evaluation, Does not apply 

Moorthy K, Jiwanji M, Shah J et al. Validation of a 

web-based training tool for lumbar puncture. 

Stud Health Technol Inform 2003;94219-25 

<15 trained physicians, Not US or Canada 

Moral R R, Alamo M M, Jurado M A et al. 

Effectiveness of a learner-centred training 

programme for primary care physicians in using 

a patient-centred consultation style. Fam Pract 

2001;18(1):60-3 

Not US or Canada 
Appendix D: List of Excluded Articles

D-18 

Moran T. A new approach. Physicians learn to better 

care for the dying. Tex Med 99;95(4):42-7 

No original data 

Morris P D, Pracy R. Training for ENT problems in 

general practice. Practitioner 83;227(1380):995-

9 

No original data, Does not apply 

Morrison J C, Sumrall D D, Chevalier S P et al. The 

effect of provider education on blood utilization 

practices. Am J Obstet Gynecol 93;169(5):1240-

5 

No comparison 

Mozes B, Lubin D, Modan B et al. Evaluation of an 

intervention aimed at reducing inappropriate use 

of preoperative blood coagulation tests. Arch 

Intern Med 89;149(8):1836-8 

Not US or Canada 

Mozes B, Lubin D, Modan B et al. Evaluation of an 

intervention aimed at reducing inappropriate use 

of preoperative blood coagulation tests. Arch 

Intern Med 89;149(8):1836-8 

No comparison 

Murphy A W, Bury G, Dowling J et al. The teaching 

of immediate cardiac and trauma care to general 

practitioners in a skills-based outreach format: an 

assessment in terms of information gain. Med 

Educ 99;33(10):774-6 

Not US or Canada 

Murphy P S. Effect of nutrition education on 

nutrition counseling practices of family 

physicians. Acad Med 89;64(2):98-102 

<15 trained physicians, Does not apply 

Mutch P B, Wenberg B G. Continuing learning needs 

assessment for Michigan practitioners. J Am 

Diet Assoc 86;86(2):247-9 

<15 trained physicians, Does not apply 

Myers P, Mahmood K. Teaching exchange. GP 

educational needs assessment: The identification 

of ''key facts''. EDUC. GEN. PRACT. 

97;8(3):238-241 

Not US or Canada, Does not apply 

Naimark D M J, Bott M T, Tobe S W et al. 

Promotion of urine microalbuminuria screening 

among primary care physicians: a randomized, 

controlled, educational intervention trial 

[abstract]. Journal of the American Society of 

Nephrology : JASN. 2001;12(Program and 

Abstracts):231A 

Abstract 

Naimark D M J, Bott M T, Tobe S W. Facilitating the 

adoption of microalbuminuria (mau) screening 

among type II diabetic patients in primary care: 

preliminary results of a randomized educational 

intervention trial [abstract]. Journal of the 

American Society of Nephrology : JASN. 

2002;13(September, Program and 

AbstractsAlbuminuria; di [Diagnosis]; Kidney; 

Patient Education; Diabetes Mellitus,NonInsulin-Dependent): 

Abstract 

Nakada S Y, Hedican S P, Bishoff J T et al. Expert 

videotape analysis and critiquing benefit 

laparoscopic skills training of urologists. JSLS 

2004;8(2):183-6 

No comparison 

Nasmith Louise, Steinert Yvonne. The evaluation of 

a workshop to promote interactive lecturing.. 

Teaching and Learning in Medicine 

2001;13(1):43-48 

Other

Neff J A, Gaskill S P, Prihoda T J et al. Continuing 

medical education versus clinic-based STD and 

HIV education interventions for primary care 

service providers: replication and extension. 

AIDS Educ Prev 98;10(5):417-32 

<15 trained physicians 

Neff James, Alan Amodei, Nancy Martinez et al. 

HIV/AIDS mental health training for health care 

providers: An evaluation of three models.. 

American Journal of Orthopsychiatry 

99;69(2):240-246 

No original data, <15 trained physicians 

Neff James, Alan Amodei, Nancy Martinez et al. 

HIV/AIDS mental health training for health care 

providers: An evaluation of three models.. 

American Journal of Orthopsychiatry 

99;69(2):240-246 

No comparison 
Appendix D: List of Excluded Articles

D-19 

Neff James, Alan Gaskill, Sharon P et al. Preliminary 

evaluation of continuing medical educationbased versus clinic-based sexually transmitted 

disease education interventions for primary care 

practitioners.. Teaching and Learning in 

Medicine 98;10(2):74-82 

<15 trained physicians 

Neff-Smith M, Scott G, Spencer E M et al. Ethics 

program evaluation: The Virginia hospital ethics 

fellows example. HEC FORUM 97;9(4):375-388 

No original data, Does not apply, No 

comparison 

Nestel D, Kneebone R, Barnet A. Teaching 

communication skills for handover: perioperative 

specialist practitioners. Med Educ 

2005;39(11):1157 

<15 trained physicians, No comparison 

Neumayer L, Wako E, Fergestaad J et al. Impact of 

journal articles and grand rounds on practice: CT 

scanning in appendicitis. J Gastrointest Surg 

2002;6(3):338-41 

No comparison 

Newman M, Van den, Bossche P et al. Responses to 

the pilot systematic review of problem-based 

learning. Med Educ 2004;38(9):921-3 

No original data, Does not apply 

Newman P, Peile E. Valuing learners'' experience and 

supporting further growth: Educational models 

to help experienced adult learners in medicine. 

Br. Med. J. 2002;325(7357):200-202 

No original data 

Noguchi M, Minami Y, Iijima T et al. 

Reproducibility of the diagnosis of small 

adenocarcinoma of the lung and usefulness of an 

educational program for the diagnostic criteria. 

Pathol Int 2005;55(1):8-13 

Not US or Canada 

Nohr C, Bygholm A. A problem-oriented, project 

organized, distance learning program in health 

informatics. Medinfo 95;8 Pt 21274-7 

Not US or Canada 

Norton P G, Ginsburg L S, Dunn E et al. Educational 

interventions to improve practice of nonspecialty 

physicians who are identified in need by peer 

review. J Contin Educ Health Prof 

2004;24(4):244-52 

No original data, Does not apply 

Nyquist J G, Naylor A J, Woodward-Lopez G et al. 

Use of performance-based assessment to 

evaluate the impact of a skill-oriented continuing 

education program. Acad Med 94;69(10 

Suppl):S51-3 

<15 trained physicians 

O'Connor P J, Desai J, Solberg L I et al. Randomized 

trial of quality improvement intervention to 

improve diabetes care in primary care settings. 

Diabetes Care 2005;28(8):1890-7 

PI or QI 

O''Dowd T C, Sprackling P D. Continuing medical 

education in general practice. BMJ 

89;298(6686):1472 

No original data, Not US or Canada 

Olson L, Anctil C, Fullerton L et al. Increasing 

emergency physician recognition of domestic 

violence. Ann Emerg Med 96;27(6):741-6 

No comparison 

Omori J, Jacobs J. Multimedia solutions in a 

problem-based learning curriculum. Hawaii Med 

J 2004;63(12):369-70 

No original data, Does not apply 

Osteen A M, Gannon M I. Continuing medical 

education. JAMA 88;260(8):1105-9 

No original data 

Oswald N T, Alderson T S. A core curriculum in 

general practice. Med Educ 97;31(5):352-3 

Not US or Canada 

Ota D, Loftin B, Saito T et al. Virtual reality in 

surgical education. Comput Biol Med 

95;25(2):127-37 

No original data 

Pangaro L. Assessment and outcomes in medical 

education. Mil Med 2003;168(9 Suppl):21-6 

No original data, No training, No evaluation 

Panikkar J, Draycott T, Cook J. The evaluation of 

computer-aided learning in medicine. Postgrad. 

Med. J. 98;74(878):706-708 

No original data 

Parslow G R. Computer-based learning. Biochem. 

Educ. 99;27(3):159 

No original data 
Appendix D: List of Excluded Articles

D-20 

Parzakonis J. Kidney Learning System: 21st century 

education. Nephrol News Issues 2004;18(8):49, 

51 

No original data, Does not apply 

Pasternak Nancy, Stern Rau, Lou Ann L et al. From 

Chaos to Consistency: The Implementation of 

Wound Treatment TIERS in home care.. Home 

Health Care Management & Practice 

2003;15(4):314-324 

<15 trained physicians, No evaluation, Does 

not apply 

Patil N G, Saing H, Wong J. Role of OSCE in 

evaluation of practical skills. Med. Teach. 

2003;25(3):271-272 

<15 trained physicians, Not US or Canada 

Peters J H, Fried G M, Swanstrom L L et al. 

Development and validation of a comprehensive 

program of education and assessment of the 

basic fundamentals of laparoscopic surgery. 

Surgery 2004;135(1):21-7 

<15 trained physicians, No evaluation, Does 

not apply 

Physician training spreads people skills. Profiles 

Healthc Mark 92;(45):27 

No original data, No evaluation 

Pichert J W. Outcomes of a diabetes professional 

education seminar. Diabetes Educ 84;9(4):37-9 

No comparison 

Pinsky L, Robins L, benIzzy J. The story in 

medicine: skills for MDs. Med Educ 

2004;38(11):1200-1 

<15 trained physicians, No comparison 

Portfolio-based learning in general practice. Report 

of a working group on higher professional 

education. Occas Pap R Coll Gen Pract 

93;(63):1-22 

Not US or Canada, Does not apply 

Poses R M, Cebul R D, Wigton R S. You can lead a 

horse to water--improving physicians' knowledge 

of probabilities may not affect their decisions. 

Med Decis Making 95;15(1):65-75 

<15 trained physicians 

Preparing for the certification and recertification 

examination. J Am Board Fam Pract 

2000;13(2):155-8 

No original data, No evaluation 

Price D W, Xu S, McClure D. Effect of CME on 

primary care and OB/GYN treatment of breast 

masses. J Contin Educ Health Prof 

2005;25(4):240-7 

No comparison 

Purkis I E. Commitment for changes: an instrument 

for evaluating CME courses. J Med Educ 

82;57(1):61-3 

No original data, No comparison 

Puterman M, Gorodischer R, Leiberman A. 

Tracheobronchial foreign bodies: the impact of a 

postgraduate educational program on diagnosis, 

morbidity, and treatment. Pediatrics 82;70(1):96-

8 

Not US or Canada 

QI project improves patient outcomes. Hosp Peer 

Rev 2002;27(6):77-8 

Abstract, No comparison, PI or QI 

Raasch B A, Hays R, Buettner P G. An educational 

intervention to improve diagnosis and 

management of suspicious skin lesions. J Contin 

Educ Health Prof 2000;20(1):39-51 

Not US or Canada 

Ramamurthi B. Postgraduate course in general 

practice to improve primary level health care. J 

Indian Med Assoc 94;92(8):275-6 

No original data, Not US or Canada, Does not 

apply 

Randell D. E-learning for continuing education: 

exploring a new frontier. MLO Med Lab Obs 

2001;33(8):24-8 

No original data, No evaluation 

Rappolt S, Pearce K, McEwen S et al. Exploring 

organizational characteristics associated with 

practice changes following a mentored online 

educational module. J Contin Educ Health Prof 

2005;25(2):116-24 

Does not apply, No comparison 

Raskova J, Trelstad R L. Replacing lectures with 

reading, small-group discussion, and computerassisted learning. Acad Med 96;71(5):537-8 

<15 trained physicians 

Raub A C, Bowler F L, Escovitz G H. A physician 

retraining program. Assessment update. JAMA 

82;248(22):2994-8 

No comparison 
Appendix D: List of Excluded Articles

D-21 

Razavi D, Merckaert I, Marchal S et al. How to 

optimize physicians' communication skills in 

cancer care: results of a randomized study 

assessing the usefulness of posttraining 

consolidation workshops. J Clin Oncol 

2003;21(16):3141-9 

Not US or Canada 

Razavi Darius, Delvaux Nicole, Farvacques Christine 

et al. Brief psychological training for health care 

professionals dealing with cancer patients: A 

one-year assessment.. General Hospital 

Psychiatry 91;13(4):253-260 

<15 trained physicians, Not US or Canada 

Razavi Darius, Delvaux Nicole, Farvacques Christine 

et al. Immediate effectiveness of brief 

psychological training for health professionals 

dealing with terminally ill cancer patients: A 

controlled study.. Social Science & Medicine 

88;27(4):369-375 

<15 trained physicians, Not US or Canada 

Reeves S. A joint learning venture between new 

nurses and junior doctors.. Nurs Times 

2000;96(38):39-40 

<15 trained physicians, Not US or Canada 

Regan C, Regan B. Teaching dermatology in general 

practice: The potential of digital cameras and 

information technology. Educ. Gen. Pract. 

2000;11(2):193-197 

No original data, No evaluation 

Reid D S, Weaver L E, Sargeant J M et al. 

Telemedicine in Nova Scotia: report of a pilot 

study. Telemed J 98;4(3):249-58 

No comparison 

Reintgen D, Cruse C W, Wells K et al. The 

effectiveness of skin cancer screening and 

continuing medical education programs toward 

increasing the survival of patients with malignant 

melanoma. Surg Oncol 92;1(6):379-84 

No training 

Requirements for infection control education and 

practice compliance established for physicians 

and other healthcare workers in Minnesota. 

Infect Control Hosp Epidemiol 93;14(8):503 

No original data, No training 

Ricci M A, Caputo M P, Callas P W et al. The use of 

telemedicine for delivering continuing medical 

education in rural communities. Telemed J E 

Health 2005;11(2):124-9 

No comparison 

Richmond R, Mendelsohn C, Kehoe L. Family 

physicians' utilization of a brief smoking 

cessation program following reinforcement 

contact after training: a randomized trial. Prev 

Med 98;27(1):77-83 

Not US or Canada 

Ricketts C, Price J, Chamberlain S. Interactive online 

assessment training for busy practitioners. Med 

Educ 2005;39(5):525-6 

Not US or Canada 

Riker R R, White B W. The effect of physician 

education on the rates of donation request and 

tissue donation. Transplantation 95;59(6):880-4 

<15 trained physicians 

Robbins J A. Training the primary care internist to 

provide care in skilled nursing facilities. J Med 

Educ 83;58(10):811-3 

<15 trained physicians 

Robinson G E, Stewart D E. A curriculum on 

physician-patient sexual misconduct and teacherlearner mistreatment. Part 1: Content.. Can Med 

Assoc J 96;154(5):643-649 

No original data, Does not apply 

Robinson K, Sutton S, von Gunten C F et al. 

Assessment of the Education for Physicians on 

End-of-Life Care (EPEC) Project. J Palliat Med 

2004;7(5):637-45 

No comparison 

Robinson S, Lawson S. Evaluating the impact of 

Information Skills Training within primary care. 

Health Info Libr J 2005;22(1):63-5 

Not US or Canada 

Rodney W M, Ruggiero C. Outcomes following 

continuing medical education on flexible 

sigmoidoscopy. Fam Pract 87;4(4):306-10 

No comparison 

Roe M H. Short course updates physicians'' 

microbiology skills. MLO Med Lab Obs 

91;23(5):37-8, 40, 42-6 

No evaluation, No comparison 
Appendix D: List of Excluded Articles

D-22 

Rogers J, Swee D, Vallario R. Role of case studies in 

evaluating medical problem solving. J. FAM. 

PRACT. 84;18(5):775-778 

No original data 

Rogers J, Swee D, Vallario R. Role of case studies in 

evaluating medical problem solving. J. FAM. 

PRACT. 84;18(5):775-778 

Does not apply 

Rolfsson G, Nordgren A, Bindzau S et al. Training 

and assessment of laparoscopic skills using a 

haptic simulator. Stud Health Technol Inform 

2002;85409-11 

No original data, No training, Not US or 

Canada 

Rosch J, Hajek M, Svarz R et al. Interactive internet 

broadcasting of a complex educational 

interventional radiology symposium. J Vasc 

Interv Radiol 2003;14(7):833-6 

Not US or Canada, No comparison 

Rosen G M, Harris I, Mahowald M W. Objective 

structured clinical examinations (OSCE) for 

sleep. Sleep Med. 2005;6(1):75-80 

<15 trained physicians 

Rucker L, Morrison E. A longitudinal 

communication skills initiative for an academic 

health system. Med. Educ. 2001;35(11):1087-

1088 

<15 trained physicians, Not US or Canada, No 

comparison, Other 

Rucker L, Morrison E. The "EBM Rx": an initial 

experience with an evidence-based learning 

prescription. Acad Med 2000;75(5):527-8 

<15 trained physicians, Does not apply 

Rutala P J, Fulginiti J V, McGeagh A M et al. 

Predictive validity of a required 

multidisciplinary standardized-patient 

examination. Acad Med 92;67(10 Suppl):S60-2 

<15 trained physicians 

Rutherford R B. Physicians in the vascular diagnostic 

laboratory: educational background, prerequisite 

skills, credentialing, and continuing medical 

education. Semin Vasc Surg 94;7(4):217-22 

No original data, Does not apply 

Ryan C A, Finer N N. Changing attitudes and 

practices regarding local analgesia for newborn 

circumcision. Pediatrics 94;94(2 Pt 1):230-3 

No comparison 

Sanci L A, Day N A, Coffey C M M et al. 

Simulations in evaluation of training: A medical 

example using standardised patients.. Evaluation 

and Program Planning 2002;25(1):35-46 

Not US or Canada 

Sanders M R, Tully L A, Turner K M et al. Training 

GPs in parent consultation skills. An evaluation 

of training for the Triple P-Positive Parenting 

Program. Aust Fam Physician 2003;32(9):763-8 

Not US or Canada 

Sandlow L J, Bashook P G, Maxwell J A. Medical 

care evaluation: an experience in continuing 

medical education. J Med Educ 81;56(7):580-6 

Does not apply 

Sandrick K. Effective online CME: it''s only a matter 

of time. Bull Am Coll Surg 2000;85(4):33-6 

No original data 

Sandrick K. Effective online CME: it''s only a matter 

of time. Bull Am Coll Surg 2000;85(4):33-6 

No evaluation 

Sargeant J M, Purdy R A, Allen M J et al. Evaluation 

of a CME problem-based learning internet 

discussion. Acad Med 2000;75(10 Suppl):S50-2 

No comparison 

Savatsky P D, Haitz M C, Sterns N S. Patterns of 

continuing medical education: a generation unit 

analysis of physicians in three community 

hospitals. Soc Sci Med [A] 81;15(5):665-72 

No training, Does not apply 

Schatzki S C. The art of learning medicine. AJR Am 

J Roentgenol 92;158(3):518 

No original data, No evaluation 

Schectman J M, Schorling J B, Nadkarni M M et al. 

Can prescription refill feedback to physicians 

improve patient adherence?. Am J Med Sci 

2004;327(1):19-24 

<15 trained physicians 

Scherer L A, Chang M C, Meredith J W et al. 

Videotape review leads to rapid and sustained 

learning. Am J Surg 2003;185(6):516-20 

<15 trained physicians 

Schoenknecht H. How to use humor to help educate 

physicians in necessary PPS skills. Hosp Top 

85;63(6):30-1, 38-9 

No original data, Does not apply 
Appendix D: List of Excluded Articles

D-23 

Schramm M, Gollnick H. Continuous education for 

acne--basics, pathogenesis, differential 

diagnosis, clinic and therapy. Development of an 

interactive hypermedia application on CD-ROM 

for computer-based learning and instruction. 

Dermatology 98;196(1):100-1 

No evaluation 

Schroeder S A, Myers L P, McPhee S J et al. The 

failure of physician education as a cost 

containment strategy. Report of a prospective 

controlled trial at a university hospital. JAMA 

84;252(2):225-30 

<15 trained physicians 

Schroeder S A. Strategies for reducing medical costs 

by changing physicians'' behavior: efficacy and 

impact on quality of care. Int J Technol Assess 

Health Care 87;3(1):39-50 

No original data 

Schwid HA, Rooke GA, Ross BK et al. Use of a 

computerized advanced cardiac life support 

simulator improves retention of advanced cardiac 

life support guidelines better than a textbook 

review.. Critical care medicine. 99;27(4):821-4 

<15 trained physicians 

Schwiebert L, Peter Davis, Alan. Impact of Feedback 

on Teaching by Volunteer Faculty in a ThirdYear Family Medicine Clerkship.. 93;(4Clinical 

Experience/Family Practice 

(Medicine)/Feedback/Medical 

Education/Supervision/Volunteer 

Training/Higher Education/Instructional 

Improvement/Physicians/Professional 

Education/Volunteers): 

<15 trained physicians 

Seale J P, Shellenberger S, Boltri J M et al. Effects of 

screening and brief intervention training on 

resident and faculty alcohol intervention 

behaviours: a pre- post-intervention assessment. 

BMC Fam Pract 2005;646 

<15 trained physicians, No comparison 

Searle J, Grover S, Santin A et al. Randomised trial 

of an integrated educational strategy to reduce 

investigation rates in young women with 

dysfunctional uterine bleeding. Aust N Z J 

Obstet Gynaecol 2002;42(4):395-400 

Not US or Canada 

Sedlack R E, Petersen B T, Kolars J C. The impact of 

a hands-on ERCP workshop on clinical practice. 

Gastrointest Endosc 2005;61(1):67-71 

No comparison 

See W A, Cooper C S, Fisher R J. Predictors of 

laparoscopic complications after formal training 

in laparoscopic surgery. JAMA 

93;270(22):2689-92 

No evaluation 

Shade S D, Barber G M. When and where you want 

it: continuing education from a distance. 

Gerontol Geriatr Educ 2004;24(4):95-114 

No comparison 

Shain R N, Crouch S H, Weinberg P C. Evaluation of 

the gynecology teaching associate versus pelvic 

model approach to teaching pelvic examination. 

J. MED. EDUC. 82;57(8):646-647 

<15 trained physicians 

Shakespeare T P, Mukherjee R K, Lu J J et al. 

Evaluation of an audit with feedback continuing 

medical education program for radiation 

oncologists. J Cancer Educ 2005;20(4):216-21 

<15 trained physicians, Not US or Canada 

Shannon S. Facilitating learning groups in CME. 

Lancet 2004;363(9411):826 

No original data, No training 

Shannon S. Needs assessment for CME. Lancet 

2003;361(9361):974 

No original data, No evaluation 

Shannon S. Practice-based CME. Lancet 

2003;361(9357):618 

No original data, No evaluation 

Shapiro M J, Morey J C, Small S D et al. Simulation 

based teamwork training for emergency 

department staff: does it improve clinical team 

performance when added to an existing didactic 

teamwork curriculum?. Qual Saf Health Care 

2004;13(6):417-21 

<15 trained physicians 

Shapiro S L, Schwartz G E. Mindfulness in medical 

education: Fostering the health of physicians and 

medical practice. Integr. Med. 98;1(3):93-94 

No original data, <15 trained physicians 
Appendix D: List of Excluded Articles

D-24 

Shifman L, Scott C S, Fawcett N et al. Utilizing a 

game for both needs assessment and learning in 

adolescent sexuality education. Soc Work 

Groups 86;9(2):41-56 

<15 trained physicians, Does not apply 

Shunk R. A cardiac physical examination 

curriculum.. Acad Med 2000;75(5):552-553 

<15 trained physicians 

Simpson D E, Ullian J A. Curriculum design and 

evaluation in faculty development. Fam Med 

97;29(4):251 

No original data, Does not apply 

Singh N P, Gupta S. Objective structured clinical 

examination.. J Assoc Physicians India 

98;46(7):645-652 

No original data, Does not apply 

Skeff K M, Campbell M, Stratos G. Assessment by 

attending physicians of a seminar method to 

improve clinical teaching. J. MED. EDUC. 

84;59(12):944-950 

Other

Skeff K M, Stratos G, Campbell M et al. Evaluation 

of the seminar method to improve clinical 

teaching. J Gen Intern Med 86;1(5):315-22 

Does not apply, Other 

Sliwa J A, Makoul G, Betts H. Rehabilitationspecific communication skills training: 

improving the physician-patient relationship. Am 

J Phys Med Rehabil 2002;81(2):126-32 

<15 trained physicians, No comparison 

Sloan R E, McMillan J. Developing mentoring skills 

for general practitioners using a simulated 

doctor. Med Educ 2003;37(11):1044-5 

Not US or Canada 

Sloan R E, McMillan J. Developing mentoring skills 

for general practitioners using a simulated 

doctor. Med Educ 2003;37(11):1044-5 

<15 trained physicians 

Smedley R R. Analyzing the questions of physicians 

participating in CME programs. Acad Med 

97;72(5):456 

No evaluation, Does not apply 

Smith W D, Berguer R. A simple virtual instrument 

to monitor surgeons' workload while they 

perform minimally invasive surgery tasks. Stud 

Health Technol Inform 2004;98363-9 

No training, Does not apply 

Smits P B A, Verbeek J H A M, De Buisonje C D. 

Problem based learning in continuing medical 

education: A review of controlled evaluation 

studies. Br. Med. J. 2002;324(7330):153-156 

No original data, Not US or Canada, Not US 

or Canada 

Solomon D, Townley P, Dunn K. A distance-learning 

program for telemedicine presenters. Acad Med 

98;73(5):612-3 

No evaluation, Does not apply 

Sommers L S, Marton K I. The curriculum template: 

creating continuing medical education curricula 

for physicians in practice in managed care 

settings. West J Med 2000;173(5):337-40 

No evaluation, Does not apply 

Soumerai S B, McLaughlin T J, Gurwitz J H et al. 

Effect of local medical opinion leaders on quality 

of care for acute myocardial infarction: a 

randomized controlled trial. JAMA 

98;279(17):1358-63 

<15 trained physicians, No comparison, PI or 

QI 

Stammer L. Take-out training. E-learning on the 

menu can improve the bottom line. Healthc 

Inform 2001;18(8):19-22, 24 

No original data, No evaluation 

Steginga S K, Pinnock C, Baade P D et al. An 

educational workshop on the early detection of 

prostate cancer--a before-after evaluation. Aust 

Fam Physician 2005;34(10):889-91 

Not US or Canada, No comparison 

Stein M R, Parish S J, Arnsten J H. The OSCE as a 

formative evaluation tool for substance abuse 

teaching.. Med Educ 2005;39(5):529-530 

<15 trained physicians 

Steinert Y, Nasmith L, Daigle N. Executive skills for 

medical faculty: a workshop description and 

evaluation. Med Teach 2003;25(6):666-8 

No comparison 

Stevenson G W, Cockshott W P. Continuing medical 

education and assessment of radiologists. Clin 

Radiol 88;39(6):575-7 

No original data, No evaluation 
Appendix D: List of Excluded Articles

D-25 

Straus S E, Ball C, Balcombe N et al. Teaching 

evidence-based medicine skills can change 

practice in a community hospital. J Gen Intern 

Med 2005;20(4):340-3 

Not US or Canada, No comparison 

Stross J K, Hiss R G, Watts C M et al. Continuing 

education in pulmonary disease for primary-care 

physicians. Am Rev Respir Dis 83;127(6):739-

46 

No comparison 

Sullivan S D, Lee T A, Blough D K et al. A multisite 

randomized trial of the effects of physician 

education and organizational change in chronic 

asthma care: cost-effectiveness analysis of the 

Pediatric Asthma Care Patient Outcomes 

Research Team II (PAC-PORT II). Arch Pediatr 

Adolesc Med 2005;159(5):428-34 

No evaluation, Does not apply 

Sullivan S, Lee T, Blough D et al. Cost-effectiveness 

of physician peer leader education and practicebased redesign in managed care: the pediatric 

asthma care PORT-II trial [Abstract]. The 

Journal of allergy and clinical immunology. 

2004;113(2 Suppl):S339 

Abstract 

Swiggart W, Spickard A, Dodd D T. Lessons learned 

from a CME course in the proper prescribing of 

controlled drugs. Tenn Med 2002;95(5):192-3 

No evaluation, Does not apply 

Symposium: Improving clinical performance 

assessment: a multi-institutional trial of the 

OSCE (objective structured clinical 

examination).. Proc Annu Conf Res Med Educ 

87;26(-):267-274 

No original data, <15 trained physicians 

Szonyi G, Millard R J. Controlled trial evaluation of 

a General Practitioner education package on 

incontinence: use of a mailed questionnaire. Br J 

Urol 94;73(6):615-20 

Not US or Canada 

Taffinder N, Sutton C, Fishwick R J et al. Validation 

of virtual reality to teach and assess psychomotor 

skills in laparoscopic surgery: results from 

randomised controlled studies using the MIST 

VR laparoscopic simulator. Stud Health Technol 

Inform 98;50124-30 

<15 trained physicians, Not US or Canada 

Teshima D Y, Sekiguchi L. Collaborating with a 

problem-based learning physician curriculum. 

CLIN. LAB. SCI. 93;6(5):274-275 

<15 trained physicians 

The endocrinologist CME examination: July/August 

2005. Endocrinologist 2005;15(4):249-251 

No original data, No evaluation, Does not 

apply 

Thomas Craig W, Guy Sybille M, Ogilvie Larry P. 

An evaluation of a practitioner training program 

designed to assist families of people with severe 

psychiatric disorders.. Psychiatric Rehabilitation 

Journal 99;23(1):34-41 

<15 trained physicians, Not US or Canada, No 

comparison 

Thompson R S, Michnich M E, Friedlander L et al. 

Effectiveness of smoking cessation interventions 

integrated into primary care practice. Med Care 

88;26(1):62-76 

Does not apply, Other 

Tinterow M M. Mandatory CME: an assessment by 

the Kansas medical profession. J Kans Med Soc 

82;83(6):295-8 

No evaluation, Does not apply 

Townsend R N, Clark R, Ramenofsky M L et al. 

ATLS-based videotape trauma resuscitation 

review: education and outcome. J Trauma 

93;34(1):133-8 

<15 trained physicians 

Training and assessment of competence. Surg Endosc 

94;8(6):721-2 

No original data, Not US or Canada, Does not 

apply 

Trowbridge Randall, Dugan William, Jay Stephen J 

et al. Determining the effectiveness of a clinicalpractice intervention in improving the control of 

pain in outpatients with cancer.. Academic 

Medicine 97;72(9):798-800 

<15 trained physicians 

Ultrasound Quarterly CME Examination December 

2005: Postmark Deadline: November 30, 2006. 

Ultrasound Q 2005;21(4):309-314 

No human data, No original data, Does not 

apply, No comparison 

Ultrasound quarterly CME examination September 

2005. Ultrasound Q. 2005;21(3):207-208 

No original data, Does not apply 
Appendix D: List of Excluded Articles

D-26 

Umble K E, Cervero R M, Yang B et al. Effects of 

traditional classroom and distance continuing 

education: a theory-driven evaluation of a 

vaccine-preventable diseases course. Am J 

Public Health 2000;90(8):1218-24 

<15 trained physicians 

Use physician education to cut prescription drug 

costs. Capitation Manag Rep 99;6(7):100-3, 97 

No comparison, Other 

Vacek J L. Practice-based continuing education 

combined with process improvement methods 

improves delivery of preventive services to 

children. Evid.-Based Healthc. 2004;8(4):177-

179 

Other 

Valdiserri R O, Koziol S M, Korth W W et al. A 

workshop to improve the teaching skills of 

physician faculty members. J. MED. EDUC. 

86;61(6):469-471 

No comparison 

van Dulmen A M, Holl R A. Effects of continuing 

paediatric education in interpersonal 

communication skills. Eur J Pediatr 

2000;159(7):489-95 

Not US or Canada 

van Olden G D, Meeuwis J D, Bolhuis H W et al. 

Clinical impact of advanced trauma life support. 

Am J Emerg Med 2004;22(7):522-5 

Not US or Canada 

van Os T W, van den, Brink R H et al. Are effects of 

depression management training for General 

Practitioners on patient outcomes mediated by 

improvements in the process of care?. J Affect 

Disord 2004;80(2-3):173-9 

Not US or Canada 

Vanchieri C. Virtual reality: will practice make 

perfect?. J Natl Cancer Inst 99;91(3):207-9 

No original data 

Vaughan W P, Morlock L L, Lenhard R E et al. 

Demonstration of the effectiveness of the 

professional education component of a 

comprehensive cancer control project using 

serial "patterns of care" (POC) studies. Prog Clin 

Biol Res 83;130425-32 

No original data, PI or QI 

Vegni E, Moja E A. Effects of a course on 

ophthalmologist communication skills: a pilot 

study. Educ Health (Abingdon) 2004;17(2):163-

71 

Not US or Canada 

Verstappen W H, van der, Weijden T et al. 

Improving test ordering in primary care: the 

added value of a small-group quality 

improvement strategy compared with classic 

feedback only. Ann Fam Med 2004;2(6):569-75 

Not US or Canada 

Walker D E, Balvert L. A practical program to 

maintain neonatal resuscitation skills. CMAJ 

94;151(3):299-304 

No original data, <15 trained physicians, No 

evaluation 

Wallace R K, Ford A B, Wallace R W. Geriatric 

medical education: a project in faculty 

development. J Am Geriatr Soc 83;31(2):106-8 

No evaluation, No comparison 

Walsh P L. An assessment of an operational 

educational accountability system for continuing 

education in the health professions. Mobius 

82;2(4):28-38 

No original data, No evaluation 

Ward R, Fidler H, Lockyer J et al. Physician 

outcomes and implications for planning an 

intensive educational experience on attentiondeficit hyperactivity disorder. Acad Med 

99;74(10 Suppl):S31-3 

No comparison 

Warren D K, Zack J E, Mayfield J L et al. The effect 

of an education program on the incidence of 

central venous catheter-associated bloodstream 

infection in a medical ICU. Chest 

2004;126(5):1612-8 

No comparison, Other 

Watts M S. CEHP (continuing education in the health 

professions)--a new discipline rooted in the 

practice?. J Contin Educ Health Prof 89;9(1):61-

5 

No original data, No evaluation 

Waxman H S, Kimball H R. Assessing continuing 

medical education. Am J Med 99;107(1):1-4 

No original data 
Appendix D: List of Excluded Articles

D-27 

Weir E, Stieb D M, Abelsohn A et al. Design, 

delivery and evaluation of an email-based 

continuing professional development course on 

outdoor air pollution and health. Med Teach 

2004;26(2):166-73 

No comparison 

Weiss K B, Lozano P, Finkelstein J A et al. A 

randomized controlled clinical trial to improve 

asthma care for children through provider 

education and health systems change: A 

description of the pediatric asthma care patient 

outcome research team (PAC-PORT II) study 

design. Health Serv. Outcomes Res. Methodol. 

2003;4(4):265-282 

No original data, Other 

Weller J, Dowell A, Kljakovic M et al. Simulation 

training for medical emergencies in general 

practice. Med Educ 2005;39(11):1154 

<15 trained physicians, Not US or Canada, No 

comparison, PI or QI 

Wells K, Sherbourne C, Duan N et al. Quality 

improvement for depression in primary care: do 

patients with subthreshold depression benefit in 

the long run?. Am J Psychiatry 

2005;162(6):1149-57 

<15 trained physicians, No comparison 

Wenzel R P. The development of academic programs 

for quality assessment. Arch Intern Med 

91;151(4):653-4 

No original data 

West R F. A construct validity study of Kolb''s 

learning style types in medical education. J Med 

Educ 82;57(10 Pt 1):794-6 

<15 trained physicians, Does not apply 

Westerhoff K, McCarthy W H, Menzies S W. 

Increase in the sensitivity for melanoma 

diagnosis by primary care physicians using skin 

surface microscopy. Br J Dermatol 

2000;143(5):1016-20 

Not US or Canada 

Wexler R. Open learning. Pointing the way. Nurs 

Times 91;87(13):32-3 

<15 trained physicians, Does not apply 

White B D, Zaner R M. Clinical ethics training for 

staff physicians: designing and evaluating a 

model program. J Clin Ethics 93;4(3):229-35 

No comparison 

White M I, Grzybowski S, Broudo M. Commitment 

to change instrument enhances program 

planning, implementation, and evaluation. J 

Contin Educ Health Prof 2004;24(3):153-62 

Does not apply, No comparison 

White M K, Malik T. Teaching clinician-patient 

communication in the treatment of breast 

diseases. J Womens Health 99;8(1):39-44 

<15 trained physicians 

Whitman N, Magill M K. Is attending a teaching 

skills workshop worth your time?. Fam Med 

98;30(4):255-6 

No evaluation, No comparison 

Whittle S R, Murdoch-Eaton D G. Development of 

lifelong learning and self-evaluation skills 

through special study modules. Med Educ 

2001;35(11):1073-4 

<15 trained physicians, Not US or Canada 

Wilkes M, Bligh J. Evaluating educational 

interventions. BMJ 99;318(7193):1269-72 

No original data 

Williams B C, Woolliscroft J O, Heindel J E. A 

managed care curriculum implemented across 

four academic departments using mandated 

evaluation instruments.. Acad Med 

99;74(5):604-605 

<15 trained physicians, No evaluation 

Williams M Q, Olson S. Teaching critical thinking 

skills through visual formats. CLIN. LAB. SCI. 

92;5(6):360-365 

No original data, Does not apply 

Winston I, Szarek J L. Problem-based learning using 

a human patient simulator.. Med Educ 

2005;39(5):526-527 

<15 trained physicians 

Wirtschafter D D, Sumners J, Jackson J R et al. 

Continuing medical education using clinical 

algorithms. A controlled-trial assessment of 

effect on neonatal care. Am J Dis Child 

86;140(8):791-7 

<15 trained physicians 

Wolters R, Wensing M, Klomp M et al. Effects of 

distance learning on clinical management of 

LUTS in primary care: a randomised trial. 

Patient Educ Couns 2005;59(2):212-8 

Not US or Canada 
Appendix D: List of Excluded Articles

D-28 

Worley P, March R, Worley E. Scanning the horizon 

of training for general practice. Med. Teach. 

2000;22(5):452-455 

No original data 

Wright F C, Law C H, Last L D et al. A blended 

knowledge translation initiative to improve 

colorectal cancer staging. BMC Health Serv Res 

2006;6(1):4 

No evaluation, Other 

Yanoff J M, Leedy R F, Seitchik M et al. A model for 

training family practice preceptors. J Med Educ 

82;57(10 Pt 1):809 

No evaluation, Does not apply 

Yens D P, Peters J T, Stimmel B et al. Evaluation of 

an innovative self-instructional program in the 

surgical subspecialties.. Proc Annu Conf Res 

Med Educ 84;23(-):105-110 

<15 trained physicians 

Yentis S M. The use of patients for learning and 

maintaining practical skills. J R Soc Med 

2005;98(7):299-302 

No training, No evaluation, Does not apply 

Young J M, Ward J. Can distance learning improve 

smoking cessation advice in family practice? A 

randomized trial. J Contin Educ Health Prof 

2002;22(2):84-93 

Not US or Canada 

Youngblood P, Stringer J, Moreno E. Development 

and formative evaluation of a longitudinal webbased nutrition curriculum. Acad Med 

2000;75(5):540-1 

<15 trained physicians, Does not apply 

Zak R. Continuing medical education. Interactive 

technology enhances instruction for both teacher 

and learner. Healthc Inform 2004;21(7):40 

No original data, No training, No evaluation, 

Does not apply 

Zaza C, Sellick S. Assessing the impact of evidencebased continuing education on 

nonpharmacologic management of cancer pain. J 

Cancer Educ 99;14(3):164-7 

No comparison, Other 

Zebrack J R, Anderson R C, Torre D. Enhancing 

EBM skills using goal setting and peer teaching.. 

Med Educ 2005;39(5):513-514 

<15 trained physicians 

Zhang H, Payandeh S, Dill J et al. Acquiring 

laparoscopic manipulative skills: a virtual tissue 

dissection training module. Stud Health Technol 

Inform 2004;98419-21 

No original data, No original data 

Zwar N A, Gordon J J, Sanson-Fisher R W. 

Evaluation of an educational program in rational 

prescribing for GP trainees. Aust Fam Physician 

95;24(5):833-8 

<15 trained physicians, Not US or Canada 

Zwar N, Henderson J, Britt H et al. Influencing 

antibiotic prescribing by prescriber feedback and 

management guidelines: a 5-year follow-up. Fam 

Pract 2002;19(1):12-7 

Not US or Canada 
Previewing Only: You cannot submit data from this form

Previewing at Level 2 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy 

Decomposition Analysis Method, Chemistry, 2006 

State: Excluded, Level: 1

Save to finish later Submit Data

For articles with refid<100000, please answer Q1 or Q2. Otherwise, skip to 

Q3.

1. QUESTION 1. 

Do not review article because article (check 1 or more):

Not in English

Does not include human data

No original data and does not apply to KQ3

No original data, but may apply to KQ3

Meeting abstract (no full article for review)

Does not include at least 15 fully trained physicians (includes only medical students or physiciansin-training, other non-physician providers, or patients)

Does not include training or education

No evaluation of an educational activity

Published prior to 1981

Not conducted in US or Canada (and no other exclusion criteria apply)

Does not apply to a key question

Evaluation does not include data from a concurrent or historical comparison group

Other (please specify):

Unclear: retrieve article to decide

2. QUESTION 2. 

Article relates to Key Question (check all that apply):

Effectiveness of a method of CME (KQ1)

Retention of effectiveness of CME over time (KQ2)

Characteristics of the audience (KQ4)

External factors (KQ5)

For articles with refid>100000, please answer Q3 or Q4. Note: Q3 and Q4 

applies only to KQ3.

3. QUESTION 3. 

Do not review article because article (check 1 or more):

Not in English

Does not include human data

Not a systematic review (i.e., identifies a question, describes a search strategy, describes eligibility 

criteria, AND synthesizes results either quantitatively or qualitatively)

Meeting abstract (no full article for review)

SRS Form Page 1 of 2

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Form took 0.75 seconds to render

Does not include medical students or physicians-in-training (the review could also include other 

health professionals so long as medical students or physicians-in-training are included as well)

Does not include medical training or education

No evaluation of an educational activity

Does not involve simulation, virtual reality, manikins, or standardized patients

Published prior to 1990

Does not apply to KQ3 (effectiveness of simulation methods in medical education)

Only includes fully trained physicians or CME but could apply to the other key questions

Other (please specify:)

Unclear: retrieve article to decide

4. QUESTION 4. 

Article relates to Key Question (check all that apply):

Effectiveness of simulation methods in medical education (KQ3)

May not apply to KQ3, but could apply to the other key questions

Save to finish later Submit Data

SRS Form Page 2 of 2

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Previewing Only: You cannot submit data from this form

Previewing at Level 9 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy 

Decomposition Analysis Method, Chemistry, 2006 

State: Excluded, Level: 1

Save to finish later Submit Data

Reporting of Adult Learning Principles (formerly known as Quality of CME 

Activity Form)

Please complete this form for ALL included articles. Please complete this 

form independently.

 

1. To what extent does the curriculum enable learners to be active contributors to their learning? 

Good (Two or more of the following: leaners identify/choose a question OR actively contribute to finding the answer OR teach the 

results of their learning to others)

Fair (Only one of the above OR none of the above but the curriculum employed partially active learning methods such as interactive 

lectures or group discussions)

Poor (None of the above are described)

Clear Selection

2. To what extent does the curriculum relate to learners' current work or life experiences? 

Good (Learners would recognize the curriculum as having practical or immediate value to their work or lives)

Fair (Learners would recognize the curriculum as having theoretical or future value to their work or lives)

Poor (Learning addresses an issue that the learners do not recognize as having value to their work or lives OR the curriculum's

relevance to the learners is not clear)

Clear Selection

3. To what extent is the curriculum tailored to learners' current or past experiences? 

Good (Authors describe a needs assessment AND describe how the curriculum is tailored towards the needs of the learners)

Fair (Authors describe only one of the above)

Poor (Authors NEITHER describe a needs assessment NOR do they describe how the curriculum is tailored towards the needs of 

learners)

Clear Selection

4. To what extent does the curriculum allow learners to identify their own learning goals and direct their education? 

Good (Learners received complete freedom to pursue independent studies or projects during part or all of the curriculum)

Fair (Learners may choose from a range of learning methods or projects, but the range is limited)

Poor (Learners are limited to a single curriculum plan OR not described)

Clear Selection

5. To what extent does the curriculum allow learners to practice what they learn? 

Good (Learners engage in applied or simulated activities during at least 50% of curriculum time)

Fair (Learners engage in applied or simulated activities during <50% of curriculum time)

Poor (The curriculum does not provide opportunities for practicing knowledge or skills OR not described)

Clear Selection

6. To what extent does the curriculum provide support to self-directed learners? 

Good (Curriculum specifically allots faculty time/resources for supporting learners during independent learning of projects)

Fair (Curriculum provides only self-learning materials (e.g., online library or bulletin boards) OR faculty are available for but not 

dedicated to supporting self-directed learning)

SRS Form Page 1 of 2

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Form took 0.359375 seconds to render

Poor (Curriculum provides none of the above OR not described)

Clear Selection

7. To what extent do learners receive feedback from teachers and/or peers during active learning? 

Good (Curriculum includes mechanisms for providing formative AND summative feedback to learners

Fair (Curriculum only includes mechanisms for providing one of the above)

Poor (Feedback to learners is not provided OR is not described)

Clear Selection

For the purpose of our review: 

Formative feedback is defined as feedback that is intended to help learners adjust their learning or activities prior to completion of the 

curriculum. 

Summative feedback is defined as feedback intended to inform learners of their progress upon completion of the curriculum. 

For example, feedback on a QI project that is provided to learners at the end of a curricular program will be considered summative 

feedback, even if the learners use this feedback to modify their project on their own after the curriculum. 

 

8. To what extent does the curriculum allow learners to reflect on their learning? 

Good (Curriculum describes mechanisms that are specifically intended to help learners reflect on their learning. These may include 

structured reflection time and debriefing meetings or presentations that are intended for self-reflection)

Fair (Curriculum describes learning sessions, such as debriefing meetings or summary presentations, which are not specifically 

intended for reflection on learning but are likely to involve some reflection by learners)

Poor (Opportunities for self-reflection are not included in the curriculum OR are not described)

Clear Selection

9. To what extent can learners observe the faculty role-model behaviors? 

Good (Learners observe role models actually practicing goal behavior in clinical settings)

Fair (Learners observe role models in simulated settings)

Poor (Role modeling is not used OR is not described)

Clear Selection

10. Comments: 

Enlarge Shrink 

11. Reviewed by: 

Enlarge Shrink 

Save to finish later Submit Data

SRS Form Page 2 of 2

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Previewing Only: You cannot submit data from this form

Previewing at Level 3 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy 

Decomposition Analysis Method, Chemistry, 2006 

State: Excluded, Level: 1

Save to finish later Submit Data

For articles with refid<100000, please answer Q1 or Q2. Otherwise, skip to 

Q3.

1. QUESTION 1. 

Do not review article because article (check 1 or more):

Not in English

Does not include human data

No original data and does not apply to KQ3

No original data, but may apply to KQ3

Meeting abstract (no full article for review)

Does not include at least 15 fully trained physicians (includes only medical students or physiciansin-training, other non-physician providers, or patients) OR less than 50% of the CME participants were 

fully trained physicians or there was not a separate analysis for fully trained physicians

Does not include training or education

No evaluation of an educational activity

Published prior to 1981

Not conducted in US or Canada (and no other exclusion criteria apply)

Does not apply to a key question

Evaluation does not include data from a concurrent or historical comparison group

Involves practice improvement or quality improvement

Other (please specify):

Unclear: retrieve article to decide

2. QUESTION 2. 

Article relates to Key Question (check all that apply):

Effectiveness of a method of CME (KQ1)

Retention of effectiveness of CME over time (KQ2)

Characteristics of the audience (KQ4)

External factors (KQ5)

For articles with refid>100000, please answer Q3 or Q4. Note: Q3 and Q4 

applies only to KQ3.

3. QUESTION 3. 

Do not review article because article (check 1 or more):

Not in English

Does not include human data

Not a systematic review (i.e., identifies a question, describes a search strategy, describes eligibility 

criteria, AND synthesizes results either quantitatively or qualitatively)

SRS Form Page 1 of 2

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Form took 0.34375 seconds to render

Meeting abstract (no full article for review)

Does not include medical students or physicians-in-training (the review could also include other 

health professionals so long as medical students or physicians-in-training are included as well)

Does not include medical training or education

No evaluation of an educational activity

Does not involve simulation, virtual reality, manikins, or standardized patients

Published prior to 1990

Does not apply to KQ3 (effectiveness of simulation methods in medical education)

Only includes fully trained physicians or CME but could apply to the other key questions

Does not report separately on the effects of simulation

Other (please specify:)

Unclear: retrieve article to decide

4. QUESTION 4. 

Article relates to Key Question (check all that apply):

Effectiveness of simulation methods in medical education (KQ3)

May not apply to KQ3, but could apply to the other key questions

Save to finish later Submit Data

SRS Form Page 2 of 2

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Previewing Only: You cannot submit data from this form

Previewing at Level 10 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy 

Decomposition Analysis Method, Chemistry, 2006 

State: Excluded, Level: 1

KQ4&5 Form

Save to finish later Submit Data

KQ 4 & 5 Outcomes Form

Audience Characteristics & External Factors Analysis

Please complete this form for ALL included articles.

 

1. Was an analysis performed to examine which audience characteristics (e.g., age, gender, specialty) or external factors (e.g., 

regulatory factors, incentives, audits) by themselves or in combination influence the effectiveness of certain educational 

techniques? (Check all that apply.)

Audience characteristics (Continue)

External factors (Continue)

Neither (Please do NOT fill out remainder of form. Hit Submit.)

2. Was a primary goal of the study or was a subgroup analysis performed to examine which audience characteristics by 

themselves or in combination influence the effectiveness of certain educational techniques?

Yes

No

Clear Selection

3. Was a primary goal of the study or was a subgroup analysis performed to examine which external factors by themselves 

or in combination reinforce the effects of CME in changing behavior?

Yes

No

Clear Selection

4. Was the data adjusted for: (Check all that apply)

Age

Gender

Practice setting

Years in practice

Personal motivation

Specialty

Non-monetary rewards/motivations

Learning satisfaction

Knowledge enhancement

Years from medical school graduation

Foreign medical graduate vs. US graduate

Country of practice

SRS Form Page 1 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Regulation

State licensing board

Professional boards

Hospital credentialing

External audits

Monetary/financial rewards

Academic advancement

Provision of tools

Public demand/patient expectations

Other (specify:)

Other (specify:)

Other (specify:)

Other (specify:)

Other (specify:)

Other (specify:)

Other (specify:)

Other (specify:)

Other (specify:)

5. What outcome is being reported?

Enlarge Shrink 

What were the individual audience characteristics or external factors analyzed by the study?

Audience Characteristic or External Factor Qualitatively summarize the results. 

6. (Check all that apply)

Age

Gender

Practice setting

Years in practice

Personal motivation

Specialty

Non-monetary rewards/motivations

Learning satisfaction

Knowledge enhancement

Years from medical school graduation

Foreign medical graduate vs. US graduate

Country of practice

Regulation

7. 

Enlarge Shrink 

SRS Form Page 2 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Form took 0.390625 seconds to render

State licensing board

Professional boards

Hospital credentialing

External audits

Monetary/financial rewards

Academic advancement

Provision of tools

Public demand/patient expectations

Other (specify:)

Other (specify:)

Other (specify:)

Other (specify:)

8. Comments: 

Enlarge Shrink 

Thanks!

9. Reviewed by: 

Enlarge Shrink 

Save to finish later Submit Data

SRS Form Page 3 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Previewing Only: You cannot submit data from this form

Previewing at Level 4 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy Decomposition Analysis Method, 

Chemistry, 2006 

State: Excluded, Level: 1

General Form Part 1

Save to finish later Submit Data

General Form Part 1

Please complete this form for ALL included articles.

 

1. Type of educator teaching CME activity (check all that apply)

Academic (i.e., medical school faculty)

Non-faculty physicians

Industry (for-profit commercial entity) employee

Employee of not-for-profit organization

Health plan employee

Government employee

Other (specify): 

Not reported

Of all those who were in the study, how many and what percent were physicians who have completed 

medical training? (If reported by group, then record the range for N and percent.)

N (If reported by total) % (If reported by total) N Range (If reported by group) % Range (If reported by group)

2. Physicians who have 

completed medical training

3. Which of the following types of health care professionals also attended the CME activity? (Check all that apply)

Physicians-in-training (residents or fellows)

Physician assistants

Nurse practitioners

Nurses

Medical students

Pharmacists

Other (specify)

Other (specify)

Other (specify)

Other (specify)

Not reported

None

Gender of health professional participating in the study. (If reported by group, then record the range for N and percent.)

N (If reported by total) % (If reported by total) N Range (If reported by group) % Range (If reported by group)

4. Male

5. Female

6. Not reported

Age of learners attending CME activity (If reported by group, then record the range.)

(If reported by total) Range (If reported by group)

7. Mean age

SRS Form Page 1 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=10043&level=4 12/04/2006
8. Age range

9. Not reported

10. Specialty of learners in the study. (Check all that apply.)

Anesthesiology

Emergency Medicine 

Family Medicine

General Practice, not otherwise specified

Internal Medicine

Neurology

Obstetrics and Gynecology

Pediatrics

Primary Care Physician, not otherwise specified

Psychiatry

Surgery

Radiology

Ophthalmology

Pathology

Other (specify):

Not reported

Not applicable

Please indicate how the time interval from medical training is recorded for the study participants and then enter the 

mean, range, or other measure in years. (If reported by group, then record the range.)

11. 

learners' graduation from medical school

learners' completion of residency

learners' completion of fellowship

not reported

Clear Selection

12. If reported by total: 

Mean

Minimum of range

Maximum of range

Other

13. If reported by group: 

Mean

Minimum of range

Maximum of range

Other

14. How comparable were the study and control leaners? (Major factors include: sex, age/years in practice, 

profession/specialty, level of training, practice setting, and geographic region)

Learner groups were equivalent in all factors OR appropriate adjustments were made

Learner groups have minor differences in one or more major factors

Learner groups have large differences in one or more major factors

Learner characteristics not reported OR no statistical comparisons were made

Not applicable

Clear Selection

15. Type of practice of learner (check all that apply)

University/medical school faculty

Private practice

Hospital staff

Health plan

Administrator

Military/government (VA, Govt. agency, etc.)

Other (specify)

Not reported

SRS Form Page 2 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=10043&level=4 12/04/2006
Form took 0.671875 seconds to render

Participation of learners in this intervention was driven by (Only check factors reported by the authors.)

16. External factors

State licensing boards

Professional boards

Hospital credentialing

External audits

Academic advancement

Public demand/patient expectations

Quality improvement

Other regulatory requirements

Other (specify)

Not reported

17. Internal Factors

Monetary/financial gain

Reputation enhancement

Personal improvement (e.g., learning satisfaction or knowledge enhancement)

Other (specify)

Not reported

18. Mark the physical setting(s) where the educational intervention took place. (Check all that apply.)

Home/personal

Recreational/resort

School/institution

Practice setting

Other (specify)

Not reported

Not linked to a physical setting

19. Are there any external sponsors of the CME activity? (Check all that apply.)

Pharmaceutical industry

Medical device industry

Other industry (specify)

Professional society

Government

Insurance industry/health plan 

Not reported

20. Comments: 

Enlarge Shrink 

Thanks!

21. Reviewed by: 

Enlarge Shrink 

Save to finish later Submit Data

SRS Form Page 3 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=10043&level=4 12/04/2006
Previewing Only: You cannot submit data from this form

Previewing at Level 5 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy Decomposition Analysis Method, 

Chemistry, 2006 

State: Excluded, Level: 1

General Form Part 2

Save to finish later Submit Data

General Form Part 2

Please complete this form for ALL included articles.

READ THE INSTRUCTIONS PRIOR TO EACH QUESTION CAREFULLY.

 

1. What was the lowest response rate for the evaluation method for the intervention or control arm among those 

targeted for the evaluation?

>=80%

60-79%

40-59%

<40%

Not reported

Clear Selection

2. Which quantitative evaluation method was used? (Check all that apply.)

Participant questionnaire (self-assessment) or test

Observer assessment

Performance audit

Patient assessment

Other (specify)

Not reported

3. Was there a qualitative evaluation?

Yes

No

Not Reported

Clear Selection

 

If the evaluation involved a questionnaire or test, answer Q4, 5 and 6. Otherwise, skip to Q7. 

4. How was it administered? (Check all that 

apply)

Written

Computer

Oral

Other 

(specify)

Not 

reported

5. Did the questionnaire evaluate...? (Check all that 

apply)

Satisfaction with 

curriculum

Knowledge 

(self-reported)

Knowledge 

(objective)

Attitudes

Skill (selfassessed)

Behavior (selfassessed)

Intent to change 

6. What kind of response options was used?

Multiple-choice

Dichotomous: 

e.g., yes/no, 

true/false

Likert scale

Open-ended

Other (specify)

Not reported

SRS Form Page 1 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=10043&level=5 12/04/2006
behavior (selfassessed)

Clinical 

outcomes (selfassessed)

Other (specify:)

Not Reported

 

If the evaluation involved observer assessment, answer Q7, 8, and 9. Otherwise, skip to Q10.

7. What was the skill assessed? (Check all that apply)

Use of 

informatics

Procedural 

skills

Communication 

skills

Other (specify)

8. Who was observing? (Check all that apply)

Educational 

program 

representative

Employer 

Co-worker

Other 

(specify)

Not reported

9. With what format was the participant observed? 

(Check all that apply)

Live 

standardized 

patient

Computerized 

standardized 

patient/Roleplay

Videotaped 

interaction with real 

patient

Artificial model

Other (specify)

Not reported

 

If the evaluation involved a performance audit, answer Q10. Otherwise, skip to Q11.

10. How was the performance audit conducted? (Check all that apply)

On-the-job observer / rater questionnaire

Chart review

Health plan database

Other (specify)

Not reported

 

If the evaluation involved patient assessment, answer Q11. Otherwise, skip to Q12.

11. What did the questionnaire/test assess? (Check all that apply)

Patient knowledge, attitude, behavior

Provider knowledge, skill, behavior

Satisfaction with provider

Other (specify)

12. Was the evaluation conducted: (Check all that apply) 

pre and post intervention

post intervention only

13. Was the study design: (Check all that apply)

Randomized controlled trial

Non-randomized controlled trial

Observational study

Other (specify)

14. Comments: 

SRS Form Page 2 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=10043&level=5 12/04/2006
Form took 0.46875 seconds to render

Enlarge Shrink 

Thanks!

15. Reviewed by: 

Enlarge Shrink 

Save to finish later Submit Data

SRS Form Page 3 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=10043&level=5 12/04/2006
Previewing Only: You cannot submit data from this form

Previewing at Level 6 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy 

Decomposition Analysis Method, Chemistry, 2006 

State: Excluded, Level: 1

General Form Part 3

Save to finish later Submit Data

Please complete this form for ALL included articles.

 Group A Group B

Specify one 

intervention group 

and describe type 

of control

1. 

Intervention

Concurrent control

Historical control

Clear Selection

2. 

Intervention

Concurrent control

Historical control

Clear Selection

Number assigned to 

group

7. 

N

Not reported

Clear Selection

8. 

N

Not reported

Clear Selection

9. 

N

N

Clear S

N for the analysis 14. 

N

Not reported

Clear Selection

15. 

N

Not reported

Clear Selection

16. 

N

N

Clear S

Description 

of groups

21. 

Enlarge Shrink 

22. 

Enlarge Shrink 

Was the exposure 

to the CME 

activity...

27. 

One time

Multiple time or 

repetitive

Other (specify)

Not reported

Not applicable

Clear Selection

28. 

One time

Multiple time or 

repetitive

Other (specify)

Not reported

Not applicable

Clear Selection

29. 

One

Mult

repetitive

Othe

Not 

Not 

Clear Sel

Was the media 

utilized in the 

educational 

intervention (check 

all that apply)

33. 

Live

Computer-based 

off-line

Internet, real 

time (e.g., streaming)

34. 

Live

Computer-based 

off-line

Internet, real 

time (e.g., streaming)

35. 

Live

Com

off-line

Inte

time (e.g

SRS Form Page 1 of 5

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 12/04/2006
Internet, not real 

time

Video

Audio

Handheld

Print

Other (specify)

Not reported

Not applicable

Internet, not real 

time

Video

Audio

Handheld

Print

Other (specify)

Not reported

Not applicable

Inte

time

Vide

Aud

Han

Prin

Othe

Not 

Not 

What was the 

technique/ 

educational method 

of the educational 

intervention? 

(Check all that 

apply) (See below 

for definitions)

39. 

Academic 

detailing

Audience 

response systems

Case-based 

learning

Clinical 

experiences

Demonstration

Discussion group

Feedback

Lecture

Mentor/Preceptor

Point of care

Problem-based 

learning or teambased learning

Programmed 

learning

Readings

Role play

Simulation (other 

than standardized 

patient or role-play)

Standardized 

patient

Writing/authoring

Other (specify)

Not reported

Not applicable

40. 

Academic 

detailing

Audience 

response systems

Case-based 

learning

Clinical 

experiences

Demonstration

Discussion group

Feedback

Lecture

Mentor/Preceptor

Point of care

Problem-based 

learning or teambased learning

Programmed 

learning

Readings

Role play

Simulation (other 

than standardized 

patient or role-play)

Standardized 

patient

Writing/authoring

Other (specify)

Not reported

Not applicable

41. 

Aca

detailing

Aud

respons

Cas

learning

Clin

experien

Dem

Disc

Fee

Lect

Men

Poin

Prob

learning 

based le

Prog

learning

Rea

Role

Sim

than sta

patient o

Stan

patient

Writ

Othe

Not 

Not 

Was the 

intervention 

designed for 

individuals or 

practice 

settings/teams? 

(check all that 

45. 

Individuals

Practice setting/teams

46. 

Individuals

Practice settings/teams

SRS Form Page 2 of 5

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 12/04/2006
apply) Not reported

Not applicable

Not reported

Not applicable

Average time spent 

in CME activity. 

Please record in 

hours. If you 

cannot convert to 

hours, then mark 

"Not 

reported." (e.g., If 

CME activity was 4 

hours per day for 4 

days, then record 

16 hours. If CME 

activity was 3 days 

long, then mark 

"Not reported.")

51. 

hours

Not reported

Not applicable

Clear Selection

52. 

hours

Not reported

Not applicable

Clear Selection

53. 

ho

No

No

Clear S

Was the CME 

activity accredited?

57. 

Yes

No

Not reported

Not applicable

Clear Selection

58. 

Yes

No

Not reported

Not applicable

Clear Selection

Average duration of 

educational 

intervention (over 

what length of time 

administered)? 

INCLUDE UNITS

63. 

duration (include 

units)

Not reported

Not applicable

Clear Selection

64. 

duration (include 

units)

Not reported

Not applicable

Clear Selection

65. 

dura

units)

Not 

Not 

Clear Sel

Number of 

days/week exposed 

to activity

69. 

Once only

1 day/week

2 days/week

3 days/week

4 days/week

5 days/week

6-7 days/week

Not reported

Not applicable

Clear Selection

70. 

Once only

1 day/week

2 days/week

3 days/week

4 days/week

5 days/week

6-7 days/week

Not reported

Not applicable

Clear Selection

Length of time from 

end of CME to last 

evaluation INCLUDE 

UNITS

75. 

duration (include 

units)

Not reported

Not applicable

Clear Selection

76. 

duration (include 

units)

Not reported

Not applicable

Clear Selection

77. 

dura

units)

Not 

Not 

Not 

Clear Sel

Was the CME 

activity a part of a 

PI or QI 

(performance or 

quality 

improvement) 

81. 

Yes

No

82. 

Yes

No

SRS Form Page 3 of 5

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 12/04/2006
project?

Unclear

Not applicable

Clear Selection

Unclear

Not applicable

Clear Selection

 

Technique/educational method definitions: 

Academic detailing 

Detailing provided by an institution or hospital. 

Audience response systems

Addresses knowledge objectives. Used in combination with live lectures or discussion groups, 

these are computerized feedback tools that allow the teacher/instructor to pose a question to a 

large group and receive immediate feedback from each learner which is collated and presented 

on a screen. Instructor may choose to alter content based on audience response. 

Case-based learning

Addresses higher order knowledge and skill objectives. Actual or authored clinical cases are 

created to highlight learning objectives; clinical material is presented and followed with 

questions usually determined by the instructor. 

Clinical Experiences

Addresses skill, knowledge and attitudinal objectives. Generally refers to a preceptorship or 

observership with an expert, as in attending a specialty clinic or operating room. 

Demonstration

Addresses skill and or knowledge (knows how) objectives; can be presented live, or with video 

or audio. Teacher determines amount and pace of content. 

Discussion Group

Addresses knowledge, especially application or higher order knowledge, or affective objectives; 

usually requires preparation with readings, or another experience, such as viewing a videotape, 

or a role play. Can be facilitated by instructor, but group often determines content. 

Feedback 

The provision of information about an individual's performance to learners. 

Lecture 

Presentation of knowledge content; live, video, audio or slide presentation available online. 

Teacher/instructor determines amount and pace of content. 

Mentor/Preceptor 

Addresses higher order cognitive, skill and affective objectives. Learner is paired with a mentor 

who may observe, review documentation of performance, advise, coach, and facilitate learning. 

Point of Care

Addresses knowledge and higher order cognitive objectives (decision-making). Information 

which is provided at the time of clinical need, integrated into chart or electronic medical record 

Problem-based learning or Team-based learning

Addresses higher order knowledge objectives, metacognition and some skill (group work) 

objectives. A clinical scenario is presented to a team, who identify the learning objectives, 

assign information-seeking tasks, and return to share information and answer questions about the 

case. Can be facilitated or non-facilitated. 

Programmed learning

Addresses knowledge objectives. Content is delivered in sequential steps, which are tested with 

the learner, before moving to the next, usually more complicated step. Pace is determined by the 

learner, but objectives are set by the program (teacher). Can be delivered in text or online. 

Readings

Presentation of knowledge content or background for attitudinal objectives. Requires learner to 

complete; can be done at learner’s pace. Teacher/instructor directed or self-directed (e.g. 

journals, newsletters, searching online). 

Role Play

Addresses skill, knowledge and affective objectives. Learners assume role of patients and/ or 

clinicians in practicing focused encounters around training problems, usually when SPs are 

unavailable. Encounter may be recorded and reviewed or followed with a discussion group. 

Rarely used as sole method of education. 

Simulation (other than standardized patient or role-play)

Addresses knowledge and skill objectives; ability to simulate potentially addresses higher order 

integrative objectives, such as responding to an emerging clinical situation, understanding the 

SRS Form Page 4 of 5

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 12/04/2006
Form took 1.171875 seconds to render

unfolding of a protein structure, working in teams. Technology can be used for simulation 

training of procedures, as in endoscopy virtual reality trainers or anesthesia simulators. Includes 

also models, such as joint injection and suture. Requires active participation of learner; can use 

multiple learners in some scenarios. 

Standardized patient

Addresses skill and some knowledge and affective objectives. Usually used for communication 

skills training and assessment, the standardized patient (SP) or simulated patient (SP) is trained 

in a specific patient scenario and presentation of a clinical problem. Encounter may be audio or 

videotaped and timed. Review offers opportunity for reflection and “replay” of the scenario 

Writing/authoring

Addresses knowledge and affective objectives. Can include authoring test items (USMLE) and 

participation in test development. Journaling is used frequently for affective objectives, and may 

be followed with discussion groups or review with a mentor.

87. 

If the overall goals of the educational intervention are stated in this article, write the broad goals verbatim 

in the space provided. If the overall goals are not explicitly stated go to question 81.

Enlarge Shrink 

88. 

If the overall goals were alluded to but not explicitly stated, briefly summarize the broad educational goals

Enlarge Shrink 

89. Comments: 

Enlarge Shrink 

Thanks!

90. Reviewed by: 

Enlarge Shrink 

Save to finish later Submit Data

SRS Form Page 5 of 5

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 12/04/2006
Previewing Only: You cannot submit data from this form

Previewing at Level 15 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy 

Decomposition Analysis Method, Chemistry, 2006 

State: Excluded, Level: 1

Save to finish later Submit Data

Review of Systematic Reviews on the Effectiveness of Simulation in Medical 

Education

KQ3 General Form

Please complete this form for ALL articles that have been included for KQ3.

 

1. Please mark which simulation methods were included in this review? (check all that apply)

Full simulation (whole room or whole patient simulations (think of resusci-anne with computer 

integrated data and feedback)

Partial task simulation (the use of products to learn or practice a specific skill, such as intubation 

heads, central venous line chests, intraosseous line legs, umbilical artery cannulation trainers)

Computer simulation (the use of computer programs that allow the student to practice decision 

making skills, specific knowledge sets such as ACLS trainers and trauma management trainers)

Virtual reality (the use of advanced computerized technology to allow students to learn or practice 

how to perform cardiac catheterizations, colonoscopies, bronchoscopies, ureteroscopies, laparoscopic 

surgery, hysteroscopy, arthroscopy, ocular surgery, intravenous line placement, etc...)

Standardized patient (the use of individuals trained to play the roles of patients, family members, or 

others to allow students to practice physical exam skills, history taking skills, communication skills, etc...)

Other (specify:)

Other (specify:)

Other (specify:)

2. Which of the following types of health care professionals were included in the review? (check all 

that apply)

Medical students

Physicians-in-training (residents or fellows)

Fully trained physicians

Physician assistants

Nurse practitioners

Nurses

Pharmacists

Other (specify:)

Other (specify:)

Other (specify:)

Other (specify:)

SRS Form Page 1 of 4

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Other (specify:)

3. What study designs were included in the review? (check all that apply)

Randomized controlled trials

Nonrandomized controlled trials

Prospective studies

Case-control studies

Case series

Case reports

Other (specify:)

Other (specify:)

Other (specify:)

4. Was the review limited to studies that used blinding?

Yes

No

Not reported

Clear Selection

5. What comparisons were made? (check all that apply)

Simulation vs other simulation

Simulation vs other medical education

Simulation vs no education

Other (specify:)

Other (specify:)

Other (specify:)

6. What was the end date for the electronic search (month/year)?

Month

Year

7. How many original articles met eligibility criteria for inclusion in the review?

Enlarge Shrink 

8. Results

Were any subgroup analyses reported for: (check all that apply) 

Age

Gender

Practice setting

Stage of training

Personal motivation

Specialty

SRS Form Page 2 of 4

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Rewards/motivations

Learning satisfaction

Knowledge enhancement

International medical graduate vs. US graduate

Country of practice

Monetary/financial rewards

Provision of tools

Public demand/patient expectation

Other (specify:)

Other (specify:)

No subgroup analyses were reported

9. If there were subgroup analyses, list any main conclusions: 

Enlarge Shrink 

10. What sensitivity analyses were done? (check all that apply)

None

Excluding studies with low study quality

Excluding long or large studies

Excluding studies based on language of publication

Excluding studies based on source of funding

Excluding studies based on country

Other (specify:)

Other (specify:)

Other (specify:)

11. Did sensitivity analysis change results?

Yes, please specify

No

Clear Selection

12. What metaregressions were conducted? (check all that apply)

None

Trial quality

Other (specify:)

Other (specify:)

Other (specify:)

13. Did metaregression change results?

Yes (specify:)

No

Clear Selection

SRS Form Page 3 of 4

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Form took 0.4179688 seconds to render

14. Comments: 

Enlarge Shrink 

Save to finish later Submit Data

SRS Form Page 4 of 4

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Previewing Only: You cannot submit data from this form

Previewing at Level 16 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy 

Decomposition Analysis Method, Chemistry, 2006 

State: Excluded, Level: 1

Save to finish later Submit Data

Review of Systematic Reviews on the Effectiveness of Simulation in Medical 

Education

KQ3 Outcomes Form

Please complete this form for ALL articles that have been included for KQ3.

 

PLEASE RECORD ONLY THOSE RESULTS RELATED TO SIMULATION.

Please complete the following for each outcome related to simulation that is reported. Where possible: 

{ Summarize the results of similar outcome measures, 

{ Include numerical results (e.g., p-value or confidence intervals) and the number of items that are being summarized, and 

{ Summarize both within group differences and between group differences. 

Example 1: Haque S, Srinivasan S. A meta-analysis of the training effectiveness of virtual reality surgical simulators. IEEE Trans 

Inf Technol Biomed. 2006;10:51-58. 

Example 2: Sutherland LM, Middleton PF, Anthony A, Hamdorf J, Cregan P, Scott D, Maddern GJ. Surgical simulation: a 

systematic review. Ann Surg. 2006;243:291-300. 

Describe the 

outcome measure

(s) related to 

simulation.

Type of Objective (Check all that 

apply)

Were the 

results of 

individual 

studies 

pooled in a 

metaanalysis?

Were the 

results of 

the metaanalysis 

significant?

Briefly summarize the 

essence of the results 

for this objective or set 

of objectives.

Example: Haque 

2006

Effectiveness of 

virtual simulation 

in differentiating 

between 

experienced 

(those who have 

performed more 

than 50 such 

procedures) and 

novice (those who 

have performed 

less than 10 

procedures) 

trainees based on 

the "task 

completion 

time" (amount of 

time, in seconds 

or minutes, taken 

Example: Haque 2006

Skills

Example: 

Haque 2006

Yes

Example: 

Haque 2006

Yes

Example: Haque 2006

Virtual reality 

simulators help to 

differentiate between a 

novice and an 

experienced trainee as 

evidenced by the 

decreased amount of 

time taken by 

experienced trainees 

to complete a given 

surgical task in a 

virtual reality 

simulation 

environment. 

Standardized effect 

size = -1.059 (95%CI; -

1.331, -0.786) [fixedeffects model, p = 0.13 

for homogeneity; 

authors report 

SRS Form Page 1 of 9

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
by the trainee to 

complete task)

performing a random 

effects model with 

quite similar results.]

Example: Haque 

2006

Effectiveness of 

virtual simulation 

in differentiating 

between 

experienced 

(those who have 

performed more 

than 50 such 

procedures) and 

novice (those who 

have performed 

less than 10 

procedures) 

trainees based on 

the "error 

score" (the 

number of "wallstrikes" 

experienced by 

the trainee or the 

ratings provided 

by an external 

expert of the 

trainee’s 

performance)

Example: Haque 2006

Skills

Example: 

Haque 2006

Yes

Example: 

Haque 2006

Yes

Example: Haque 2006

Virtual reality 

simulators help to 

differentiate between a 

novice and an 

experienced trainee as 

evidenced by low error 

scores of the 

experienced trainees 

to complete a given 

surgical task in a 

virtual reality 

simulation 

environment. 

Standardized effect 

size = -1.325 (95%CI; -

2.125, -0.525) 

[Random-effects 

model, p <0.0001 for 

homogeneity]

Example: 

Sutherland 2006

Surgical trainees 

(surgeons, 

residents, medical 

students, and 

others) who were 

trained on 

computer 

simulation were 

compared with 

trainees without 

previous training. 

Outcomes 

measured were 

surgical task 

performance or 

satisfaction with 

surgical 

techniques or 

both. 

Example: Sutherland 2006

Skills

Example: 

Sutherland 

2006

No

Example: 

Sutherland 

2006

N/A (no 

metaanalysis 

conducted)

Example: Sutherland 

2006

Those who were 

trained on computer 

simulators performed 

better than those who 

received no training.

Example: 

Sutherland 2006

Surgical trainees 

(surgeons, 

residents, medical 

Example: Sutherland 2006

Skills

Example: 

Sutherland 

2006

No

Example: 

Sutherland 

2006

N/A (no 

metaExample: Sutherland 

2006

The computer 

simulation versus 

"standard" training 

SRS Form Page 2 of 9

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
students, and 

others) who were 

trained on 

computer 

simulation were 

compared with 

trainees who had 

standard training. 

Outcomes 

measured were 

surgical task 

performance or 

satisfaction with 

surgical 

techniques or 

both. 

 

analysis 

conducted)

comparisons varied, 

potentially confounded

by the different 

components of 

"standard" training, as 

well as by the different 

intensities of time 

allowed on the 

simulator in the 

computer simulation 

groups. 

Example: 

Sutherland 2006

Surgical trainees 

(surgeons, 

residents, medical 

students, and 

others) who were 

trained on 

computer 

simulation were 

compared with 

video simulationtrained trainees. 

Outcomes 

measured were 

surgical task 

performance or 

satisfaction with 

surgical 

techniques or both 

Example: Sutherland 2006

Skills

Example: 

Sutherland 

2006

No

Example: 

Sutherland 

2006

N/A (no 

metaanalysis 

conducted)

Example: Sutherland 

2006

Computer simulation 

showed mixed results, 

superior in some 

studies, but not others 

and was inferior to 

video simulation in 

one study.

Example: 

Sutherland 2006

Surgical trainees 

(surgeons, 

residents, medical 

students, and 

others) who were 

trained on 

computer 

simulation were 

compared with 

trainees who were 

trained on a 

physical trainer or 

model. Outcomes 

measured were 

surgical task 

performance or 

satisfaction with 

Example: Sutherland 2006

Skills

Example: 

Sutherland 

2006

No

Example: 

Sutherland 

2006

N/A (no 

metaanalysis 

conducted)

Example: Sutherland 

2006

Only one study was in 

this comparison and it 

showed computer 

simulation training to 

be superior to training 

on a physical trainer.

SRS Form Page 3 of 9

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
surgical 

techniques or both 

Example: 

Sutherland 2006

Surgical trainees 

(surgeons, 

residents, medical 

students, and 

others) who were 

trained on one 

type or level of 

computer 

simulation were 

compared with 

trainees on 

another type or 

level of computer 

simulation. 

Outcomes 

measured were 

surgical task 

performance or 

satisfaction with 

surgical 

techniques or 

both. 

Example: Sutherland 2006

Skills

Example: 

Sutherland 

2006

No

Example: 

Sutherland 

2006

N/A (no 

metaanalysis 

conducted)

Example: Sutherland 

2006

Two studies were 

included in this 

comparison. One 

showed that more 

demanding training 

may lead to 

better performance of 

surgical tasks on 

MIST-VR. Second 

study failed to show 

clear differences 

between massed and 

distributed practice on 

MIST-VR. 

Example: 

Sutherland 2006

Surgical trainees 

(surgeons, 

residents, medical 

students, and 

others) who were 

trained on video 

simulation were 

compared with 

trainees who had 

no previous 

training. 

Outcomes 

measured were 

surgical task 

performance or 

satisfaction with 

surgical 

techniques or 

both. 

Example: Sutherland 2006

Skills

Example: 

Sutherland 

2006

No

Example: 

Sutherland 

2006

N/A (no 

metaanalysis 

conducted)

Example: Sutherland 

2006

Video simulation 

groups did not show 

consistently better 

results than groups 

who did not receive 

training. 

 

Example: 

Sutherland 2006

Surgical trainees 

(surgeons, 

residents, medical 

students, and 

others) who were 

trained on video 

simulation were 

Example: Sutherland 2006

Skills

Example: 

Sutherland 

2006

No

Example: 

Sutherland 

2006

N/A (no 

metaanalysis 

conducted)

Example: Sutherland 

2006

no differences were 

seen between video 

box training and other 

forms of training such 

as bench models or 

standard training.

SRS Form Page 4 of 9

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
compared with 

trainees who had 

other forms of 

training (such as 

bench models or 

standard training). 

Outcomes 

measured were 

surgical task 

performance or 

satisfaction with 

surgical 

techniques or 

both. 

Example: 

Sutherland 2006

Surgical trainees 

(surgeons, 

residents, medical 

students, and 

others) who were 

trained on 

physical or model 

simulation were 

compared with 

trainees who had 

other forms of 

training, including 

no training. 

Outcomes 

measured were 

surgical task 

performance or 

satisfaction with 

surgical 

techniques or 

both. 

Example: Sutherland 2006

Skills

Example: 

Sutherland 

2006

No

Example: 

Sutherland 

2006

N/A (no 

metaanalysis 

conducted)

Example: Sutherland 

2006

Studies that were 

included in this 

comparison showed 

mixed results, model 

training may be better 

than no training and 

standard training such 

as instruction from 

mentors or manuals.

Example: 

Sutherland 2006

Surgical trainees 

(surgeons, 

residents, medical 

students, and 

others) who were 

trained on 

cadavers 

were compared 

with trainees who 

had standard 

training. 

Outcomes 

measured were 

surgical task 

performance or 

satisfaction with 

surgical 

techniques or 

Example: Sutherland 2006

Skills

Example: 

Sutherland 

2006

No

Example: 

Sutherland 

2006

N/A (no 

metaanalysis 

conducted)

Example: Sutherland 

2006

Only one study that 

reported this 

comparison found that 

the cadaver trained 

group received better 

scores than the 

standard training 

group, which learned 

independently from 

the manuals, for the 

global assessment of 

operative performance 

on cadavers.

SRS Form Page 5 of 9

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
both. 

Describe the 

outcome measure

(s) related to 

simulation.

Type of Objective (Check all that 

apply)

Were the 

results of 

individual 

studies 

pooled in a 

metaanalysis?

Were the 

results of 

the metaanalysis 

significant?

Briefly summarize the 

essence of the results 

for this objective or set 

of objectives.

1. 

Enlarge Shrink 

2. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

3. 

Yes

No

Unclear

N/A

Clear Selection

4. 

Yes

No

Unclear

N/A (no 

meta-analysis 

conducted)

Clear Selection

5. 

Enlarge Shrink 

6. 

Enlarge Shrink 

7. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

8. 

Yes

No

Unclear

N/A

Clear Selection

9. 

Yes

No

Unclear

N/A (no 

meta-analysis 

conducted)

Clear Selection

10. 

Enlarge Shrink 

11. 

Enlarge Shrink 

12. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

13. 

Yes

No

Unclear

N/A

Clear Selection

14. 

Yes

No

Unclear

N/A (no 

meta-analysis 

conducted)

Clear Selection

15. 

Enlarge Shrink 

16. 17. 18. 19. 20. 

SRS Form Page 6 of 9

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Enlarge Shrink 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

Yes

No

Unclear

N/A

Clear Selection

Yes

No

Unclear

N/A (no 

meta-analysis 

conducted)

Clear Selection

Enlarge Shrink 

21. 

Enlarge Shrink 

22. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

23. 

Yes

No

Unclear

N/A

Clear Selection

24. 

Yes

No

Unclear

N/A (no 

meta-analysis 

conducted)

Clear Selection

25. 

Enlarge Shrink 

26. 

Enlarge Shrink 

27. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

28. 

Yes

No

Unclear

N/A

Clear Selection

29. 

Yes

No

Unclear

N/A (no 

meta-analysis 

conducted)

Clear Selection

30. 

Enlarge Shrink 

31. 

Enlarge Shrink 

32. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

33. 

Yes

No

Unclear

N/A

Clear Selection

34. 

Yes

No

Unclear

N/A (no 

meta-analysis 

conducted)

35. 

Enlarge Shrink 

SRS Form Page 7 of 9

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

Clear Selection

36. 

Enlarge Shrink 

37. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

38. 

Yes

No

Unclear

N/A

Clear Selection

39. 

Yes

No

Unclear

N/A (no 

meta-analysis 

conducted)

Clear Selection

40. 

Enlarge Shrink 

41. 

Enlarge Shrink 

42. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

43. 

Yes

No

Unclear

N/A

Clear Selection

44. 

Yes

No

Unclear

N/A (no 

meta-analysis 

conducted)

Clear Selection

45. 

Enlarge Shrink 

46. Author conclusion/summary: 

Overall improvement after educational intervention

Partial improvement or mixed results

No improvement after educational intervention

Unclear

Other (specify:)

Clear Selection

47. 

Briefly summarize the main conclusions of the study (may cut and paste from abstract or article):

Example Haque 2006:

SRS Form Page 8 of 9

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Form took 2.078125 seconds to render

Virtual reality simulation can help to discriminate between the experienced and the inexperienced 

trainees.

Example: Sutherland 2006

Computer simulation generally showed better results than no training at all (and than physical 

trainer/model training in one RCT), but was not convincingly superior to standard training (such as 

surgical drills) or video simulation (particularly when assessed by operative performance). Video 

simulation did not show consistently better results than groups with no training at all, and there were 

not enough data to determine if video simulation was better than standard training or the use of 

models. Model simulation may have been better than standard training, and cadaver training may 

have been better than model training.

Enlarge Shrink 

48. Comments: 

Enlarge Shrink 

Save to finish later Submit Data

SRS Form Page 9 of 9

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Previewing Only: You cannot submit data from this form

Previewing at Level 17 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy 

Decomposition Analysis Method, Chemistry, 2006 

State: Excluded, Level: 1

Save to finish later Submit Data

Review of Systematic Reviews on the Effectiveness of Simulation in Medical 

Education

KQ3 Quality Form

Please complete this form for ALL articles that have been included for KQ3.

 

1. Did the authors clearly state the question addressed by the overview at the beginning of the article?

Yes. The authors stated a focused clinical question about the outcomes of treatment, AND specified a target population

Partially

No

Clear Selection

2. Did the authors describe the search methods used to find evidence (original research) on the primary 

question(s)?

Yes. Enough information was reported to permit replication

Partially

No

Clear Selection

3. Was the search for evidence reasonably comprehensive?

Yes. Search included MEDLINE (or other electronic database), hand-searching of select journals or reference lists, AND query of 1 or 

more experts.

Partially. Search included MEDLINE (or other electronic database), but did not include or did not report including hand-searching of 

journals or reference lists AND/OR did not include a query of experts.

No. Search did not include an electronic database of journals.

Clear Selection

4. Did the authors report on the criteria they used for deciding which studies to include in the 

systematic review?

Yes. Criteria were specified clearly enough to permit replication.

Partially. Criteria specified, but without enough detail to permit replication.

No. Criteria not specified.

Clear Selection

5. Were the inclusion criteria appropriate (aimed at avoiding bias in the included studies)?

Yes. Inclusion criteria are likely to capture all relevant studies (e.g., included countries other than the U.S.?).

Partially. 

No. Inclusion criteria likely to lead to biased sampling of studies.

Cannot tell. Inclusion criteria described inadequately.

SRS Form Page 1 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Clear Selection

6. Did the authors assess study quality?

Yes. Criteria to assess study quality were specified with adequate detail to permit replication.

Partially. Criteria to assess study quality not adequately described.

No.

Clear Selection

7. Was the quality assessment done appropriately?

Yes. Quality assessment was done using a validated instrument (with citation) or the authors demonstrated validity of their methods.

Partially. Authors used their own quality assessment instrument without validation, or another instrument with unknown measurement 

properties.

No.

Cannot tell. There was no quality assessment reported.

Clear Selection

8. Did the authors demonstrate that their methodology was reproducible?

Yes. The investigators mostly (>50% of the time) agreed on selection of articles, on quality assessment, AND on the data that was 

extracted.

Partially. Disagreement occurred the majority of the time either on the selection of articles, quality assessment, or data extraction (but 

not all 3).

No. Disagreement occurred the majority of the time on the selection of articles, quality assessment, AND data extraction.

Can’t tell. Authors didn’t comment on reproducibility.

Clear Selection

9. Did the authors discuss whether the variation in the results of the original research may be due to 

differences in study design or population?

Yes. Text or tables provide comparative information on most of following: study design, populations, interventions, and outcome

measures, AND the authors discuss possible sources of heterogeneity

Partially. 

No. 

Clear Selection

10. Were the results of the relevant studies combined appropriately relative to the primary question?

Yes. The overview included some assessment of the qualitative or quantitative heterogeneity of study results AND used an accepted 

qualitative or quantitative pooling method (i.e., more than simple addition, such as random effects vs fixed effects model for quantitative 

data)

Partially. 

No.

Cannot tell. No description of the methods used for combining studies. 

Clear Selection

11. Were the conclusions of the authors supported by the data and/or analysis reported in the overview?

Yes. 

Partially.

No.

Clear Selection

12. Were tests for publication bias conducted?

Yes

No

SRS Form Page 2 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Form took 0.5195313 seconds to render

Not reported

Clear Selection

13. If yes, was there publication bias?

Yes (statistical test with p<.05 or reported “yes” by author)

No

Not reported

Clear Selection

14. Additional comments: 

Enlarge Shrink 

Thank you!

Save to finish later Submit Data

SRS Form Page 3 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Previewing Only: You cannot submit data from this form

Previewing at Level 7 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy 

Decomposition Analysis Method, Chemistry, 2006 

State: Excluded, Level: 1

Outcomes Form

Save to finish later Submit Data

Outcomes Form

Please complete this form for ALL included articles.

 

Please complete the following for each relevant outcome that is reported. Where possible: 

{ Summarize the results of similar outcome measures, 

{ Include numerical results (e.g., p-value or confidence intervals) and the number of items that are 

being summarized, and 

{ Summarize both within group differences and between group differences. 

 Please see the example that relates to refid#6833 and 10859.

Describe the main 

outcome measure.

Type of Objective (check all that 

apply)

Were one or 

more of the 

learning 

objectives of the 

CME activity 

met? (Please 

use statistical 

significance to 

determine this. 

If not enough 

data is 

presented to 

determine this, 

then mark 

"Unclear.")

Briefly summarize the 

essence of the results for 

this objective or set of 

objectives.

Example (Refid 

#10859)

Physicians' 

perceptions of selfefficacy in 

cholesterol-lowering 

practices, as 

measured on Likert 

scale survey (14 

items) 

Example (Refid #10859)

Attitudes

Example (Refid 

#10859)

Unclear

Example (Refid #10859)

Overall posttest scores did 

not differ between 

intervention and control 

groups. Intervention groups 

(workshop and workshop + 

chart cue) had statistically 

significantly higher 

confidence on 1 of 14 

practices at posttest 1 

(dietary counseling) and 2 of 

14 practices at posttest 2 

(identifying patients to be 

screened and interpreting 

test results to patient). 

Results possibly due to 

multiple comparisons.

Example (Refid 

#6833)

Example (Refid #6833)

Knowledge or cognitive skills

Example (Refid 

#6833)

Example (Refid #6833)

Both enhanced and routine 

SRS Form Page 1 of 5

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Knowledge test Unclear group improved from pre to 

post test, but no comparison 

between groups was 

performed.

Example (Refid 

#6833)

Characteristics 

related to physician 

action such as use 

of a screening tool 

and optimal 

pharmacotherapy

Example (Refid #6833)

Practice behaviors

Example (Refid 

#6833)

Yes

Example (Refid #6833)

Enhanced group was more 

likely to utilize depression 

diagnostic tool, optimize 

drug therapy, and use a 

checklist for adverse events.

Example (Refid 

#6833)

Patient factors such 

as satisfaction, 

compliance, 

treatment outcome, 

and office visits

Example (Refid #6833)

Clinical outcomes

Example (Refid 

#6833)

No

Example (Refid #6833)

Patients were satisfied in 

both groups, but no between 

group comparison done, no 

differences were found in 

compliance or withdrawals. 

Patients treated by 

physicians in the enhanced 

group did make fewer office 

visits. Both patient groups 

improved with regard to self 

reported signs of depression 

(p<0.001) but the between 

group comparison showed 

no differences between 

groups. Patients treated by 

physicians in the enhanced 

group were less likely to be 

sent for adjunctive 

psychotherapy, (p <0.0001). 

1. 

Enlarge Shrink 

2. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

3. 

Yes

No

Unclear

Not applicable

4. 

Enlarge Shrink 

5. 

Enlarge Shrink 

6. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

7. 

Yes

No

Unclear

Not applicable

8. 

Enlarge Shrink 

SRS Form Page 2 of 5

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

9. 

Enlarge Shrink 

10. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

11. 

Yes

No

Unclear

Not applicable

12. 

Enlarge Shrink 

13. 

Enlarge Shrink 

14. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

15. 

Yes

No

Unclear

Not applicable

16. 

Enlarge Shrink 

17. 

Enlarge Shrink 

18. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

19. 

Yes

No

Unclear

Not applicable

20. 

Enlarge Shrink 

21. 22. 23. 24. 

SRS Form Page 3 of 5

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Enlarge Shrink 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

Yes

No

Unclear

Not applicable

Enlarge Shrink 

25. 

Enlarge Shrink 

26. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

27. 

Yes

No

Unclear

Not applicable

28. 

Enlarge Shrink 

29. 

Enlarge Shrink 

30. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

31. 

Yes

No

Unclear

Not applicable

32. 

Enlarge Shrink 

33. 

Enlarge Shrink 

34. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

35. 

Yes

No

Unclear

Not applicable

36. 

Enlarge Shrink 

SRS Form Page 4 of 5

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Form took 0.921875 seconds to render

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

37. 

Enlarge Shrink 

38. 

Knowledge 

or cognitive 

skills

Attitudes

Skills 

(psychomotor 

or procedural 

skills)

Practice 

behaviors

Clinical 

outcomes

Other 

(specify:)

39. 

Yes

No

Unclear

Not applicable

40. 

Enlarge Shrink 

41. Author conclusion/summary: 

Overall improvement after educational intervention

Partial improvement or mixed results

No improvement after educational intervention

Unclear

Other (specify:)

Clear Selection

42. Briefly summarize the main conclusions of the study (may cut and paste from abstract or article):

Enlarge Shrink 

43. Comments: 

Enlarge Shrink 

Thanks!

44. Reviewed by: 

Enlarge Shrink 

Save to finish later Submit Data

SRS Form Page 5 of 5

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Previewing Only: You cannot submit data from this form

Previewing at Level 8 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy 

Decomposition Analysis Method, Chemistry, 2006 

State: Excluded, Level: 1

Quality Form

Save to finish later Submit Data

Quality of Study Design Form

Please complete this form for ALL included articles. Please complete this 

form independently.

 

1. Was the study described as randomized (this includes the use of words such as randomly, random, and 

randomization)?

Yes

No

Not reported/Can't tell

Clear Selection

2. If yes to q1, was the randomization scheme described AND appropriate?

Yes: (1) appropriate randomization is if each study participant is allowed to have the same chance of receiving each intervention and the 

investigators could not predict which treatment was next

No: (-1) randomization described AND inappropriate (e.g. methods of allocation using date of birth, date of admission, hospital numbers, 

or alteration should not be regarded as appropriate

No: (0) randomization methods not described

Clear Selection

3. Was the evaluation of outcome blinded?

Yes

No

Not reported/Can't tell

Clear Selection

4. If yes to Q3, was the method of blinding described AND appropriate?

Yes: (1) appropriate blinding is if the person evaluating the outcome could not identify the intervention being assessed

No: (-1) the study was described as blind AND inappropriate

No: (0) no description of blinding available and unable to tell if appropriate or not

Clear Selection

5. Was there a description of withdrawals and drop-outs of learners?

Yes: (1) the number and the reasons for withdrawals in each group must be stated or state that there were no withdrawals. If subjects 

were not included in the analysis, they must state the number and reasons for not including them in the analysis

No (0)

Clear Selection

6. Did the authors describe their power analysis?

Yes

No or unclear

SRS Form Page 1 of 2

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Form took 0.296875 seconds to render

Clear Selection

7. If a power analysis is described, did the evaluation have enough power to show statistical 

significance?

Yes

No or unclear

Not applicable

Clear Selection

8. Comments: 

Enlarge Shrink 

Thanks!

9. Reviewed by: 

Enlarge Shrink 

Save to finish later Submit Data

SRS Form Page 2 of 2

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Form took 0.671875 seconds to render

Previewing Only: You cannot submit data from this form

Previewing at Level 1 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy 

Decomposition Analysis Method, Chemistry, 2006 

State: Excluded, Level: 1

Save to finish later Submit Data

1. Does this article potentially apply to any of our key questions?

Potentially eligible

Ineligible

Clear Selection

Save to finish later Submit Data

SRS Form Page 1 of 1

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=1... 09/21/2006
Previewing Only: You cannot submit data from this form

Previewing at Level 11 

Refid: 1, Fernandez, I. and Frenking, G., Direct Estimate of the Strength of Conjugation and Hyperconjugation by the Energy Decomposition Analysis Method, Chemistry, 2006

State: Excluded, Level: 1

KQ6 Form

Save to finish later Submit Data

KQ 6 Outcomes Form

Validity/Reliability of Tools

Please complete this form for ALL included articles. Please note: If a study mentions that they use a 

validated measure and provides a reference, mark "Yes" to Q1.

 

1. Does the study report the validity or reliability of any evaluation method? (Key words may include: content, concurrent, predictive, 

or construct validity; intra-rater, inter-rater, test-retest, inter-item reliability; percent agreement, correlation coefficient, kappa, Cronbach 

alpha)

Yes (Continue with form)

No (Please do NOT complete the rest of this form. Hit submit.)

Clear Selection

Instructions: Please complete a separate form for each evaluation method for which validity / reliability are reported. 

Wherever possible, you may report sub-scales and multiple statistical tests for the same method on the same form.

2. Briefly describe the evaluation method: 

Enlarge Shrink 

3. Does the study describe pilot testing / cognitive testing of this as a new evaluation method? 

Pilot testing involves administration of a method in a population prior to use; cognitive testing involves exploring the processes by which pilot populations answer questions. (Collins D, 

Quality of Life Research, 2003) 

Yes

No

Unclear

Clear Selection

4. Is this a method that was previously used?

Yes – Please write the reference number of the article that first used the method: 

No – Created new evaluation method for this study

Unclear

Clear Selection

5. Does the study change a previously validated / reliable evaluation method for this study?

Not Applicable (method not previously used)

Yes - questions

Yes - response options

Yes - mode of administration

No

Unclear

6. Did the study describe the validity of the evaluation method or tool (degree to which method truly measures what it is intended to 

measure)? Note: some authors may erroneously report inter-item reliability (Cronbach alpha) as validity; please report this under 

reliability.

No (Skip to Q22.)

Yes - but type of validity not specifically reported (Skip to Q22.)

Yes - Validity from previous study or in this study reported in this article (Continue)

Clear Selection

If statistics are reported from both prior and current study, record values for current study only. If only previous validity 

statistics are given, record them.

Type of Validity (Check all that apply) Comparison Method Statistics (Check all that apply) Source of 

validity

7. 8. Not applicable 9. 

SRS Form Page 1 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=10043&level=11 12/04/2006
Face / content: degree to which an instrument accurately 

represents the skill or characteristic it is designed to measure, 

based on people’s experience and available knowledge1

Experts/sources

Not reported

Clear Selection

Current 

study

Prior 

study

Clear 

Selection

10. 

Concurrent Criterion: degree to which an instrument 

produces the same results as another accepted or proven 

instrument that measures the same variable1

11. 

Comparison

Not reported

Clear Selection

12. 

Percentage 

agreement

Correlation 

coefficient

Sensitivity

Specificity

Other 

(specify:)

Statistic not 

reported

13. 

Current 

study

Prior 

study

Clear 

Selection

14. 

Predictive Criterion: degree to which a measure 

accurately predicts expected outcomes1

15. 

Comparison

Not reported

Clear Selection

16. 

Percentage 

agreement

Correlation 

coefficient

Sensitivity

Specificity

Other 

(specify:)

Statistic not 

reported

17. 

Current 

study

Prior 

study

Clear 

Selection

18. 

Other validity (e.g. 

construct, discriminant)

19. 

Comparison

Not reported

Clear Selection

20. 

Other 

(specify:)

Statistic not 

reported

21. 

Current 

study

Prior 

study

Clear 

Selection

22. Did the study describe the reliability of the evaluation method (consistency or reproducibility of measurements)? (check all that 

apply)

No (Skip to Q41.)

Yes – but type of reliability not specifically reported (Skip to Q41.)

Yes – Reliability from previous or current study reported in this article (Continue)

If statistics are reported from both prior and current study, record values for current study only. If only previous reliability 

statistics are given, record them.

Type of reliability Statistics (check all that apply) Source of 

reliability

23. 

Intra-rater reliability: degree to which measurements are the same when repeated by the same 

person1

24. 

Percentage 

agreement

Correlation 

coefficient

Kappa

Other (specify:)

Statistic not reported

25. 

Current study

Prior study

Clear Selection

26. 

Inter-rater reliability: degree to which measurements are the same when obtained by different 

persons1

27. 

Percentage 

agreement

Correlation 

coefficient

Kappa

Other (specify:)

Statistic not reported

28. 

Current study

Prior study

Clear Selection

29. 30. 31. 

SRS Form Page 2 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=10043&level=11 12/04/2006
Form took 0.796875 seconds to render

Test-retest reliability: degree to which the same test produces the same results when repeated 

under the same conditions1

Percentage 

agreement

Correlation 

coefficient

Kappa

Other (specify:)

Statistic not reported

Current study

Prior study

Clear Selection

32. 

Equivalence reliability: degree to which alternate forms of the same measurement instrument 

produce the same results1

33. 

Reliability coefficient

Coefficient of stability

Kappa

Other (specify:)

Statistic not reported

34. 

Current study

Prior study

Clear Selection

35. 

Internal consistency (inter-item) reliability: how well items reflecting the same construct yield 

similar results

36. 

Cronbach's alpha

Split-halves

Correlation 

coefficient

Kuder-Richardson 

method

Other (specify:)

Statistic not reported

37. 

Current study

Prior study

Clear Selection

38. 

Other reliability

39. 

Other (specify:)

Statistic not reported

40. 

Current study

Prior study

Clear Selection

1 Reed D, Price EG, Windish DM, et al. Challenges in systematic reviews of educational intervention studies. Ann Intern Med. 

2005;142:1080-1089.

41. Comments: 

Enlarge Shrink 

Thanks!

42. Reviewed by: 

Enlarge Shrink 

Save to finish later Submit Data

SRS Form Page 3 of 3

https://www.clinical-analytics.com/d2d/ul1/review.asp?mode=previewMode&articleid=10043&level=11 12/04/2006
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-1 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Adams, 1998125 RCT, post intervention 

only 

NA NA NA E: Provider 

knowledge, skill, behavior 

Allison, 2005147 RCT, pre and post 

intervention 

NA NA C: HEDIS NA 

Andersen, 199036 RCT, pre and post 

intervention 

A: Written E: Knowledge (objective) R: Multiple-choice, dichotomous, Likert scale 

NA NA NA 

Anderson, 1996103 RCT, pre and post 

intervention 

NA NA C: Prescription 

information collected through the Triplicate Prescription Program provided by the British Columbia College of Physicians and Surgeons 

NA 

Beaulieu, 2002140 RCT, post intervention 

only 

NA S: Procedural skills, 

communication skills O: Educational program representative F: Live standardized patient 

NA NA 

Beaulieu, 200444 Non-RCT, pre and post 

intervention 

A: NR E: Knowledge (selfreported), behavior (self-assessed) R: Multiple-choice 

NA C: Health plan 

database 

NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-2 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Bjornson, 199014 RCT, post intervention 

only 

A: Written E: Knowledge (selfreported), attitude, intent to change behavior (selfassessed) R: Likert scale 

NA C: Health plan 

database 

NA 

Block, 198837 Non-RCT, 

post intervention only 

A: Written E: Knowledge (objective), attitude, behavior (selfassessed) R: Multiple-choice, Likert scale 

NA NA NA 

Bloomfield, 200568

Overall trial does not appear to be randomized, but one aspect, the provision of prompts was randomized, pre and post intervention 

A: NR E: Attitude, behavior (self-assessed) R: NR 

NA C: Chart review NA 

Brown, 199972 RCT, pre and 

post intervention 

A: Written E: Satisfaction with curriculum, skill (selfassessed), behavior (self-assessed) R: NR 

NA NA E: Provider 

knowledge, skill, behavior, satisfaction with provider 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-3 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Brown, 2004135 RCT, pre and post 

intervention 

NA S: Procedural skills, 

communication skills O: Patients’ parents F: Patients' parents live observation of interaction with physician and symptoms of patients 

NA E: Provider 

knowledge, skill, behavior 

Browner, 1994122 RCT, post intervention 

only 

NA NA C: Chart review NA 

Bunting, 2004123 Non-RCT, pre and post 

intervention 

NA NA C: Health plan 

database 

NA 

Carney, 199588 RCT, post intervention 

only 

A: Written E: Knowledge (objective) R: NR 

NA NA E: Standardized 

patient assessment using performance checklist 

Casebeer, 1999131 RCT, pre and post 

intervention 

NA S: Communication 

skills O: Educational program representative F: Live standardized patient 

C: Lipid profile E: Patient knowledge

attitude, behavior 

Chan, 199921 RCT, pre and 

post intervention 

A: Computer E: Knowledge (selfreported) R: Multiple-choice 

NA NA NA 

Chassin, 1986106 RCT, pre and post 

intervention 

NA NA C: Chart review NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-4 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Cherkin, 199183

Observational study, pre and post intervention 

A: Written E: Knowledge (selfreported), attitude, skill (self-assessed), behavior (selfassessed) R: NR 

NA NA NA 

Chodosh, 200662 RCT, post intervention 

only 

A: Written E: Knowledge (objective), attitude R: Multiple-choice, dichotomous, Likert scale 

NA NA NA 

Chung, 200454 RCT, pre and post 

intervention 

A: NR E: Knowledge (selfreported), knowledge (objective) R: Multiple-choice 

NA NA NA 

Clark, 1998120 RCT, pre and 

post intervention 

A: Written E: Knowledge (selfreported), behavior (self-assessed) R: NR 

NA NA E: Provider 

knowledge, skill, behavior; parental knowledge of asthma management 

Clark, 200078 RCT, pre and 

post intervention 

A: Written, sent by mail E: Behavior (selfassessed), medication use, procedures encouraging selfmanagement for patients R: NR 

NA NA E: Provider 

knowledge, skill, behavior; health care utilization: emergency department visits; physician office visits; hospitalizations; communication regarding asthma management 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-5 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Cohn, 200219 Non-RCT, pre 

and post intervention 

A: NR E: Knowledge (objective), behavior (self-assessed) R: NR 

NA NA E: Provider 

knowledge, skill, behavior 

Costanza, 199235 Non-RCT, pre and post 

intervention 

A: Written E: Knowledge (objective), attitude, behavior (selfassessed) R: Likert scale, biographical data entered directly by physician 

NA NA NA 

Cummings, 1989112 RCT, pre and post 

intervention 

A: Written E: Attitude, behavior (self-assessed) R: Multiple-choice, Likert scale 

NA NA E: Patient knowledge

attitude, behavior; provider knowledge, skill, behavior 

Cummings, 1989113 RCT, pre and post 

intervention 

A: Written E: Satisfaction with curriculum, attitude, behavior (selfassessed) R: Multiple-choice, Likert scale 

NA NA E: Patient knowledge

attitude, behavior; provider knowledge, skill, behavior 

Cummings, 1989117 RCT, post intervention 

only 

NA NA NA E: Patient knowledge

attitude, behavior; provider knowledge, skill, behavior 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-6 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Curran, 200034 RCT, pre and post 

intervention 

A: Computer, by phone E: Satisfaction with curriculum, knowledge (selfreported), skill (selfassessed) R: Multiple-choice, dichotomous, Likert scale, open-ended 

NA NA NA 

Davis, 2004101 Non-RCT, pre 

and post intervention 

NA NA C: Health plan 

database 

NA 

Derebery, 2002153 Non-RCT, pre and post 

intervention 

A: Physicians answered questions about patient management as they were guided through cases and were given feedback about their responses E: Patient management R: NR 

NA C: Health plan 

database 

NA 

Des Marchais, 199061 RCT, pre and post 

intervention 

NA NA C: Interpersonal skill 

assessment scores 

NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-7 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Dietrich, 2000110 RCT, pre and post 

intervention 

A: Written E: Behavior (selfassessed) R: Likert scale 

S: Observations of office availability and use of tools O: Educational program representative F: Observations of office 

NA E: Provider 

knowledge, skill, behavior 

Dormuth, 200415 RCT, pre and post 

intervention 

NA NA C: Health plan 

database 

NA 

Doucet, 199840

Non-RCT, some evaluations conducted pre and post intervention and some conducted post intervention only 

A: Written E: Satisfaction with curriculum, knowledge (objective), clinical reasoning assessed by "Key Feature Problems" R: Multiple-choice, Likert scale 

NA NA NA 

Elliott, 199755 RCT, pre and 

post intervention 

A: Written, telephone E: Knowledge (objective), attitude R: Dichotomous, Likert scale 

NA NA E: Amount of pain 

Evans, 198656 RCT, post 

intervention only 

A: Written E: Knowledge (objective) R: Multiple-choice 

NA C: Chart review E: Patient knowledge

attitude, behavior; provider knowledge, skill, behavior; patient blood pressure 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-8 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Fordis, 200542 RCT, pre and 

post intervention 

A: Written, computer E: Knowledge (objective) R: Multiple-choice 

NA C: Chart review NA 

Frush, 200687 RCT, pre and 

post intervention 

NA S: (1) Dosing 

deviation from accepted dose range; (2) time requested to calculate medication doses during the management of the simulated pediatric stabilization scenario O: Pediatric emergency medicine specialist who served as the facilitator F: Artificial model 

NA NA 

Gerbert, 200286 RCT, pre and post 

intervention 

A: Computer E: Satisfaction with curriculum, knowledge (objective), attitude R: NR 

NA NA NA 

Gerrity, 199932 RCT, post intervention 

only 

A: Written E: Knowledge (objective) R: NR 

S: Communication skills, diagnostic and treatment skills O: Educational program representative F: Live standardized patient 

NA NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-9 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Gerstein, 199953 Non-RCT, pre and post 

intervention 

A: Written E: Knowledge (objective), attitude, behavior (selfassessed) R: Multiple-choice, dichotomous, Likert scale 

NA NA NA 

Gifford, 199646 RCT, post intervention 

only 

A: Written E: Knowledge (objective), intensity of CME R: Multiple-choice, Likert scale, openended 

NA NA NA 

Gifford, 1999142 RCT, post intervention 

only 

A: Written E: Knowledge (objective), behavior (self-assessed) R: NR 

NA C: Chart review NA 

Goldberg, 200193 RCT, pre and post 

intervention 

NA NA C: Health plan 

database 

NA 

Goldstein, 2005146 RCT, pre and post 

intervention 

NA NA C: Chart review NA 

Goldwater, 2001157

Non-RCT, post intervention only 

NA S: Clinical outcomes 

O: NR F: NR 

NA NA 

Gonzales, 1999111 Non-RCT, pre and post 

intervention 

NA NA C: Chart review, 

health plan database 

NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-10 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Grady, 199779 RCT, pre and 

post intervention 

A: Written E: Satisfaction with curriculum, behavior (self-assessed) R: Dichotomous, Likert scale 

NA C: Chart review NA 

Greenberg, 198525

Non-RCT, some evaluations conducted pre and post intervention and some conducted post intervention only 

A: Written E: Knowledge (objective) R: Multiple-choice 

S: Communication skills, decisionmaking skills O: Educational program representative F: Live standardized patient 

C: Chart review NA 

Gullion, 1988132 RCT, pre and post 

intervention 

NA NA C: Chart review E: Patient knowledge

attitude, behavior; provider knowledge, skill, behavior; measured clinical outcomes: blood pressure and weight 

Hagen, 2005126 Non-RCT, pre and post 

intervention 

NA NA C: Chart review, 

pharmacy database 

NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-11 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Harris, 200239 RCT, pre and 

post intervention 

A: NR E: Satisfaction with curriculum, knowledge (selfreported), attitude, skill (self-assessed), behavior (selfassessed) R: Likert scale 

NA NA NA 

Harris, 200575 RCT, some 

evaluations conducted pre and post intervention and some conducted post intervention only 

A: Written E: Satisfaction with curriculum R: Likert scale 

NA C: Chart review NA 

Heale, 198866 RCT, post 

intervention only 

A: Written E: Satisfaction with curriculum, knowledge (objective) R: Multiple-choice, Likert scale 

S: History, physician examination, investigations, communication with standardized patients in physician practice O: Standardized patients F: Live standardized patient 

NA NA 

Herbert, 2004102 RCT, pre and post 

intervention 

NA NA C: Health plan 

database 

NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-12 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Hergenroeder, 200260 RCT, pre and post 

intervention 

A: Written E: Satisfaction with curriculum, knowledge (objective), attitude R: Comfort level and evaluation of intervention rated using Likert scale, knowledge questions not specified 

S: Procedural skills O: Trained standardized patient at baseline; study coordinator at followup exam F: Live standardized patient 

NA NA 

Howe, 1997145 Non-RCT, pre 

and post intervention 

NA NA C: Chart review, 

Illinois Cancer Registry, and direct contact with treating physicians 

NA 

Jennett, 198876 RCT, pre and post 

intervention 

A: Written E: Satisfaction with curriculum R: Likert scale 

NA C: Chart review NA 

Juzych, 200592 Non-RCT, pre and post 

intervention 

NA NA C: Chart review NA 

Kemper, 200658

RCT, 2 x 2 factorial design, pre and post intervention 

A: NR E: Knowledge (objective), attitude, communication R: Multiple-choice, Likert scale 

NA NA NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-13 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Kiang, 200533 Non-RCT, pre 

and post intervention 

A: Written E: Knowledge (selfreported), attitude, clinical outcomes (self-assessed) R: Dichotomous, Likert scale 

NA NA NA 

Kim, 1999137 RCT, pre and 

post intervention 

NA NA C: Chart review E: Provider 

knowledge, skill, behavior; satisfaction with provider 

Kottke, 1989114 RCT, pre and post 

intervention 

A: Oral, phone survey E: Attitude, behavior (self-assessed), clinical outcomes (self-assessed) R: NR 

NA NA E: Provider 

knowledge, skill, behavior 

Kronick, 200371 RCT, pre and post 

intervention 

A: NR E: Satisfaction with curriculum, knowledge (selfreported), skill (selfassessed), behavior (self-assessed) R: NR 

NA NA NA 

Kutcher, 200265 Non-RCT, pre and post 

intervention 

A: Written E: Knowledge (objective) R: NR 

NA C: Chart review NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-14 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Labelle, 200467 Non-RCT, pre and post 

intervention 

A: Written E: Knowledge (objective), behavior (self-assessed) R: Multiple-choice, Likert scale, openended 

S: Procedural skills O: NR F: Live standardized patient 

NA NA 

Lane, 199169 RCT, pre and 

post intervention 

A: Written E: Knowledge (selfreported), skill (selfassessed), behavior (self-assessed) R: Dichotomous, Likert scale, self report of percentage of referrals 

NA NA NA 

Lane, 200145 Non-RCT, pre 

and post intervention 

A: Written E: Satisfaction with curriculum, knowledge (selfreported), behavior (self-assessed), clinical outcomes (self-assessed) R: NR 

NA C: Chart review NA 

Leopold, 200580 RCT, pre and post 

intervention 

A: Written E: Knowledge (selfreported), confidence R: Likert scale 

S: Procedural skills O: Educational program representative F: Artificial model 

NA NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-15 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Levinson, 1993138 RCT, pre and post 

intervention 

NA S: Communication 

skills O: Educational program representative F: Videotaped interaction with real patient 

NA NA 

Lewis, 199370 Non-RCT, pre 

and post intervention 

A: NR E: Satisfaction with curriculum, attitude R: NR 

NA NA E: Provider 

knowledge, skill, behavior 

Lin, 199773 Non-RCT, pre 

and post intervention 

A: NR E: Attitude R: NR 

NA C: Chart review, 

pharmacy records 

E: Patient knowledgeattitude, behavior; satisfaction with provider; measured symptoms of depression 

Lin, 2001133 RCT, pre and 

post intervention 

NA NA C: Health plan 

database 

NA 

LindsayMcIntyre, 1987115

RCT, post intervention only 

A: NR E: Satisfaction with curriculum R: Likert scale 

NA C: Chart review E: Patient knowledge

attitude, behavior; provider knowledge, skill, behavior 

Lockyer, 200264 Unclear, pre and post 

intervention 

A: NR E: Knowledge (objective), attitude, behavior (selfassessed) R: Multiple-choice, dichotomous 

NA NA NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-16 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Maclure, 199896 Non-RCT, pre and post 

intervention 

NA NA C: Prescription filling 

records and billing information 

NA 

Macrae, 200485 RCT, post intervention 

only 

A: NR E: Skill (selfassessed), critical appraisal skills measured objectively R: Likert scale, openended 

NA NA NA 

Maiman, 198847 RCT, post intervention 

only 

A: Written E: Satisfaction with curriculum, knowledge (selfreported), behavior (self-assessed) R: Dichotomous, Likert scale 

NA NA E: Patient knowledge

attitude, behavior 

Mann, 199752 RCT, pre and 

post intervention 

A: NR E: Knowledge (selfreported), knowledge (objective), attitude, behavior (selfassessed) R: Dichotomous, Likert scale, not reported for all subsets of test 

NA NA NA 

Margolis, 2004118 RCT, pre and post 

intervention 

A: NR E: NR R: NR 

NA C: Chart review NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-17 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Maxwell, 198457 Non-RCT, pre and post 

intervention 

A: Written E: Knowledge (objective) R: Multiple-choice 

NA C: Chart review NA 

Mazmanian, 199824 RCT, post intervention 

only 

A: Written E: Satisfaction with curriculum, attitude R: Likert scale 

NA NA NA 

Mazmanian, 2001161 RCT, post intervention 

only 

A: Written E: Behavior (selfassessed), intent to change behavior (self-assessed), clinical outcomes (self-assessed) R: Dichotomous, Likert scale, openended 

NA NA NA 

McBride, 2000144 RCT, pre and post 

intervention 

A: Written E: Knowledge (selfreported), attitude, behavior (selfassessed) R: NR 

NA C: Chart review E: Patient knowledge

attitude, behavior 

McClellan, 2003119 RCT, pre and post 

intervention 

NA NA C: Health plan 

database 

NA 

McMahon, 1988155 Non-RCT, pre and post 

intervention 

NA NA C: Hospital length of 

stay statistics 

NA 

Mehler, 200599 RCT, pre and post 

intervention 

NA NA C: Chart review NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-18 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Meredith, 200051 RCT, pre and post 

intervention 

A: Written E: Knowledge (objective) R: Likert scale 

NA NA NA 

Messina, 2002156 Non-RCT, pre and post 

intervention 

NA NA NA E: Patient knowledge

attitude, behavior 

Moran, 1996121 Non-RCT, pre and post 

intervention 

NA NA C: Chart review NA 

Mukohara, 200513 RCT, pre and post 

intervention 

A: Computer E: Amount or percentage devoted to finding answers or giving clinical guidance from published research evidence R: Multiple-choice, open-ended 

NA NA NA 

Myers, 2004148 RCT, pre and post 

intervention 

A: Written E: Clinical outcomes (self-assessed) R: Dichotomous, Open-ended 

NA C: Chart review, 

patient specific internal chart audit form 

NA 

Norris, 200081 RCT, pre and 

post intervention 

A: Written E: Knowledge (selfreported), knowledge (objective), attitude, skill (self-assessed), behavior (selfassessed) R: NR 

NA NA E: Patient knowledge

attitude, behavior; provider knowledge, skill, behavior 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-19 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Ockene, 1996124 RCT, post intervention 

only 

NA NA NA E: Provider 

knowledge, skill, behavior 

Ozer, 2005104 Non-RCT, pre 

and post intervention 

NA NA NA E: Provider 

knowledge, skill, behavior 

Pazirandeh, 2002128 Non-RCT, pre and post 

intervention 

NA NA NA E: Physician practices 

regarding clinical outcomes/ treatment 

Pereles, 1996163 RCT, post intervention 

only 

A: NR E: Behavior (selfassessed) R: NR 

NA NA NA 

Perera, 1983107 RCT, pre and post 

intervention 

NA NA C: Chart review NA 

Pimlott, 2003143 RCT, pre and post 

intervention 

NA NA C: Health plan 

database 

NA 

Pinto, 199874 RCT, pre and 

post intervention 

A: NR E: Satisfaction with curriculum, skill (selfassessed), behavior (self-assessed) R: Likert scale 

NA C: Physician selfreport E: Provider knowledge, skill, 

behavior; satisfaction with provider 

Premi J, 199338 RCT, pre and post 

intervention 

A: Written E: Knowledge (objective) R: NR 

NA NA NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-20 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Premi, 199441 Non-RCT, pre 

and post intervention 

A: Written E: Satisfaction with curriculum, knowledge (objective), behavior (self-assessed) R: Dichotomous 

NA NA NA 

Rabin, 199897 RCT, pre and 

post intervention 

A: Telephone E: Behavior (selfassessed) R: NR 

S: Communication skills, inclusion of appropriate STD/HIV history-taking and counseling O: Simulated patient instructor F: Live standardized patient 

NA NA 

Rahme, 2005100

RCT, 2 x 2 factorial design, pre and post intervention 

NA NA C: Chart review NA 

Ray, 198520 Non-RCT, pre 

and post intervention 

NA NA C: Medicaid 

prescription records 

NA 

Ray, 2001109 RCT, pre and 

post intervention 

A: Oral, by phone E: Patient pain, functioning, health R: Likert scale 

NA NA E: Patient pain, 

functioning, health 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-21 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Rodney, 198677 Non-RCT, pre and post 

intervention 

A: Written, phone interview E: Knowledge (objective), skill (selfassessed), behavior (self-assessed), clinical outcomes (self-assessed) R: NR 

NA NA NA 

Rosenthal, 200559 RCT, post intervention 

only 

NA NA NA E: Patient knowledge

attitude, behavior, satisfaction with provider 

Rost, 200122 RCT, pre and 

post intervention 

NA NA NA E: Patient knowledge

attitude, behavior; whether the patients had received psychiatric care or no

Roter, 199584 RCT, post 

intervention only 

A: Written E: Skill (selfassessed), behavior (self-assessed) R: Likert scale, openended 

S: Communication skills O: Visits were audiotaped and then coded F: Live standardized patient 

NA E: Patient knowledge

attitude, behavior 

Schectman, 1991139

Non-RCT, post intervention only 

A: NR E: Satisfaction with curriculum, behavior (self-assessed) R: NR 

NA NA NA 

Schectman, 1995130 RCT, pre and post 

intervention 

NA NA C: Health plan 

database 

NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-22 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Schectman, 199616 Non-RCT, pre and post 

intervention 

NA NA C: Health plan 

database 

NA 

Schectman, 2003108 RCT, pre and post 

intervention 

A: NR E: Satisfaction with curriculum R: NR 

NA C: Chart review E: Patient knowledge

attitude, behavior; satisfaction with provider; measures of clinical outcome 

Schroy, 199982 Non-RCT, pre and post 

intervention 

A: NR E: Attitude, behavior (self-assessed), cues to action (such as chart or computerized reminders) R: Dichotomous, Likert scale 

NA C: Review of 

appointment logs at the intervention neighborhood health centers (NHCs) as well as monitoring of off-site referrals for comparison NHCs 

NA 

Schwartzberg, 199798 Non-RCT, pre and post 

intervention 

A: Written, telephone E: Knowledge (selfreported), attitude R: Dichotomous, Likert scale, openended 

NA NA NA 

Sharif, 200294 Non-RCT, pre 

and post intervention 

NA NA C: Chart review NA 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-23 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Short, 200650 RCT, pre and 

post intervention 

A: Written E: Knowledge (selfreported), knowledge (objective), attitude, behavior (selfassessed) R: Multiple-choice, cases were linear with no alternative outcomes 

NA NA NA 

Sibley, 1982136 RCT, pre and post 

intervention 

A: Written E: Knowledge (objective) R: NR 

NA C: Chart review NA 

Slotnick, 199317

Non-RCT, post intervention only 

A: Written E: Knowledge (objective) R: NR 

NA NA NA 

Socolar, 1998134 RCT, pre and post 

intervention 

A: Written E: Knowledge (objective) R: Likert scale 

NA C: Chart review NA 

Solomon, 2004127 RCT, pre and post 

intervention 

NA NA C: Chart review NA 

Soumerai, 1987105 RCT, pre and post 

intervention 

NA NA C: Health plan 

database 

NA 

Stein, 200195 RCT, pre and 

post intervention 

NA NA NA E: Provider 

knowledge, skill, behavior; patient health and pain 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-24 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

Stewart, 200543 RCT, pre and post 

intervention 

A: Written E: Knowledge (objective) R: NR 

NA C: Chart review E: Provider 

knowledge, skill, behavior 

Stross, 1985116 RCT, pre and post 

intervention 

NA NA C: Chart review NA 

Terry, 198149 Non-RCT, pre 

and post intervention 

A: Written E: Knowledge (objective), attitude, behavior (selfassessed) R: Multiple-choice 

S: Communication skills, medical decision-making O: Educational program representative F: Live standardized patient, audiotaped standardized patient interaction 

NA NA 

Thom, 2000129 Non-RCT, pre 

and post intervention 

NA NA E: Satisfaction with 

provider 

Tziraki, 2000141 RCT, post intervention 

only 

A: Written E: Attitude, behavior (self-assessed) R: NR 

NA NA NA 

Wells, 2000151 RCT, pre and 

post intervention 

NA NA NA E: Patient knowledge

attitude, behavior; patient health outcomes (depression, QOL); utilization; quality of care; employment 
Evidence table 1. Characteristics of study design and evaluation methods in studies assessing the effectiveness of continuing 

F-25 

Author, year

Study design, when evaluation conducted

Participant questionnaire Administered (A) Evaluated (E) Response options (R)

Observer assessment Skill assessed (S) Observer (O) Format (F)

Performance audit Conducted (C)

Patient assessment Evaluated (E)

White, 198548 RCT, pre and 

post intervention 

A: Written E: Satisfaction with curriculum, knowledge (objective), attitude R: Multiple-choice, dichotomous 

NA C: Chart review NA 

White, 200463 RCT, pre and 

post intervention 

A: Written E: Knowledge (objective) R: Multiple-choice 

NA NA NA 

Wilson, 1988154 RCT, post intervention 

only 

NA NA C: Chart review E: Patient knowledge

attitude, behavior 

Winickoff, 198423 RCT, pre and post 

intervention 

NA NA C: Chart review, 

health plan database 

NA 

Worrall, 1999152 RCT, pre and post 

intervention 

NA NA NA E: NR 

Zuckerman, 200418 Non-RCT, pre and post 

intervention 

NA NA C: Health plan 

database 

NA 

CME = continuing medical education; HEDIS = Health Plan Employer Data and Information Set; NA = not applicable; NR = not reported; QOL = qu

randomized controlled trial; STD/HIV = sexually transmitted diseases/human immunodeficiency virus 
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-221

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Skills Objectives Met, Evaluation Duration Not ReportedFrush, 200687

Int: Internet, not real time, Video CC: NA 

Int: Lecture, case-based learning CC: NA 

Int: Education was available for 3 months CC: NA 

Time to determine dose 

Cognitive skills 

Yes We also found a 

significant decrease in the time it took clinicians to determine (prescribe) a dose in the education group after having received the intervention, as compared with the control group. 

The Web-based education prograproper use of the Broselow PediatrResuscitation Taimprove dosing aand reduce dosin

Kiang, 200533

Int: NA CC: NA Int: Many things were made available but it is not reported as to which groups used what methods, options included live, regional meetings, CDROMs, mailings, grandrounds CC: NA 

Int: NA CC: NA Int: Not clear CC: NA 

Int: NA CC: NA Int: Not clear CC: Not clear 

Responses to adult case scenarios for URI 

Cognitive skills 

Yes Both Minnesota and 

Wisconsin clinicians improved in their responses to the adult case scenarios for URI and bronchitis. The magnitude of improvement was greater for Wisconsin clinicians, but the improvement in Wisconsin was not significant after accounting for the secular trend in Minnesota. In the pediatric case scenarios, Wisconsin clinicians improved from 1999 to 2002 (p = 0.058), while the responses of Minnesota clinicians were essentially unchanged (p = 0.807). 

In conclusion, thisuggests that the campaign had at modest positive ethe knowledge adecision-making primary care clinWisconsin. 
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-222

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Kiang, 200533

Int: NA CC: NA Int: Many things were made available but it is not reported as to which groups used what methods, options included live, regional meetings, CDROMs, mailings, grandrounds CC: NA 

Int: NA CC: NA Int: Not clear CC: NA 

Int: NA CC: NA Int: Not clear CC: Not clear 

Responses to questions about nonpredictive clinical factors and social factors

Knowledge Cognitive skills 

Yes Yes 

In Wisconsin, significant improvement occurred in the responses to the 2 questions about nonpredictive clinical factors and the social factor that may increase the likelihood of prescribing antimicrobial agents. Overall, Wisconsin clinicians demonstrated significant improvement regarding the influence of purulent nasal discharge (p = 0.044) and productive cough (p = 0.010) after accounting for temporal changes in Minnesota. 

In conclusion, thisuggests that the campaign had at modest positive ethe knowledge adecision-making primary care clinWisconsin. 

Gerrity, 199932

Int: Live, Video, Audio, Print CC: NA 

Int: Clinical experiences, Discussion group, Feedback, Lecture, Readings, Role play CC: NA 

Int: Multiple time or repetitive CC: NA 

Communication skills for diagnosis and management of depression, as assessed by unannounced standardized patients' ratings 

Cognitive skills 

Yes Intervention physicians 

performed better than control physicians in 4 of 6 rated areas for female SP: assessing criteria of depression, suicidal ideation, and stress at home and considering diagnosis of depression. Intervention physicians performed better than control physicians in 2 of 6 rated areas for male SP: assessing stress at home and scheduling followup visit in 2 weeks. 

The Depression Education Prograchanged physiciabehavior and maimportant compothe efforts to impcare of depressepatients. 
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-223

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Andersen, 199036

Int: Live, Video CC: NA 

Int: Case-based learning, Lecture CC: NA 

Int: One time CC: NA 

Physicians' diagnostic accuracy for psychiatric conditions, as measured by pre and post-tests incorporating multiple clinical vignettes 

Knowledge Cognitive skills 

Yes Yes 

Compared with control, intervention physicians had significantly better post-test composite scores for affective and anxiety disorders; differences significant in 2 of 4 affective disorders (major depression and dysthymic disorder but not depression with psychotic features or bipolar), no specific anxiety disorders (0 of 4), and no somatic disorders. Pre-test scores negatively correlated with improvement in accuracy (more improvement on incorrect pre-test cases). 

A brief, single-seintervention can impact on physicpsychiatric diagnabilities. Additionphysicians particthe intervention wmore likely to refpsychiatric patien
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-224

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Skills Objectives Met, Evaluation Duration Greater Than 30 DaysMacrae, 200485

Int: Print CC: Print 

Int: Readings, Listserv discussion group with moderator CC: Readings 

Int: Multiple time or repetitive CC: Multiple time or repetitive 

Skills for critically appraising articles, as measured by examination 

Cognitive skills 

Yes In general, surgeons in 

the intervention group performed better on the test of critical appraisal than did those in the control group (with mean examination scores of 58%+-8 AND 50% +-8, respectively, t=3.92, p<0.0001). 

"The results of thsuggest that the intervention was in enhancing critappraisal skills. Tsize seen in this quite large. In thithe mean of the tgroup was a full deviation higher mean of the contgroup. Despite thnumber of potentsources of variatlarge proportion variance was accfor by the training condition (intervecontrol group) aloThus, enhancing appraisal skills in medical communlarge likely is of vand complementsources in ensurlevels of evidenccare. This study the first in the liteshow that it is poenhance the critiappraisal skills opracticing physica targeted, Internintervention that undertaken in onpractice setting." 
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-225

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Gerbert, 200286

Int: Internet, not real time CC: NA 

Int: Case-based learning, Feedback, Programmed learning, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Diagnosis and evaluation planning of skin cancer 

Cognitive skills 

Yes Overall diagnosis and 

evaluation planning showed significantly greater improvement from pretest to posttest I comparing the intervention to the control group. There was still greater improvement maintained by posttest II, but only in 5 of 9 outcomes. 

"The interventionphysicians in this significantly imprnine of the 14 ouincluding their ovdiagnosis and ovevaluation plann
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-226

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Doucet, 199840

Int: Live CC: Live 

Int: Problembased learning or team-based learning CC: Discussion group, Lecture 

Int: Multiple time or repetitive CC: One time 

The Key Feature Problems examination (evaluation of clinical reasoning skills) 

Knowledge Cognitive skills 

Yes Yes 

Whereas those enrolled in the lecture group had a mean score of 28 (SD = 5.23), participants of the PBL group had a mean examination score of 34.76 (SD = 5.96). This represents a 25% difference, deemed educationally significant. The difference in scores between the intervention and control groups was highly statistically significant (p = 0.001). 

"Tests of knowleacquisition and thtests of clinical reskills indicated thPBL group benefmore than the lecgroup. In additionphysicians particthe PBL sessions the interactive apand rated the promore highly. Partin the PBL group the program sessmore favorably ththeir counterpartlecture group acrseven of the nine program dimensiHowever, physicthe lecture group report that the prheld their interescontributed to theknowledge and sprovided content their practice. Phin the lecture groagreed that the fpresented the coeffectively, but winclined to agree discussion compthe lecture enhantheir learning." 
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-227

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Brown, 199972

Int: Live CC: NA 

Int: Clinical experiences, Discussion group, Lecture, Role play, Clinicians audiotaped interaction with patients and listened between workshops CC: NA 

Int: Multiple time or repetitive CC: NA 

Selfassessment of clinicians' communication skills, attitudes, and behavior, as measured by participant questionnaire 

Attitudes Skills (psychomotor or procedural skills) Practice behavior 

Yes Yes Yes 

Intervention group noted substantial improvements compared with control in 8 of 24 skills / attitudes / behaviors: awareness of and confidence in dealing with patients whom they found difficult; abilities to compliment patients’ efforts, ask open-ended questions, address psychosocial factors, express empathy and reassurance, and clarify expectations. 33% of clinicians reported that fewer than 5% of visits were frustrating after the program (compared with 21% of clinicians at baseline). Three months after the program, clinicians in the intervention group reported that it had improved communication with patients. On average, however, the scores for clinicians’ rating of improvement in patient satisfaction and improvement in clinicians’ personal satisfaction in their work decreased below the midpoint on the five-point scale (mean rating, 2.85 for both items). 

”Thriving in a BuPractice: PhysiciPatient Communa typical continuimedical educatioprogram geared developing cliniccommunication snot effective in imgeneral patient satisfaction. To imglobal visit satisfcommunication straining programneed to be longemore intensive, tbroader range of and provide ongoperformance fee
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-228

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Roter, 199584

Int: Live Int: Live CC: NA 

Int: Discussion group, Lecture, Role play, Standardized patient Int: Discussion group, Lecture, Role play, Standardized patient CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Physician use of emotion handling and problem defining skills measured from the audiotape analysis and simulated patients 

Cognitive skills 

Yes Emotion handling group 

used significantly more EH skills than control group; PD group fell in between. Problem defining group used significantly more PD behaviors than the control group with EH group falling between. 

Physicians' use ocommunication stheir practices chas a result of an CME program. Physicians trainespecific communskills recognized psychological protheir patients thauntrained physicTrained physiciashowed greater cproficiency in the management of a simulated patient compared with cgroup physicianspatients of trainephysicians compuntrained physicshowed greater rin emotional distas long as 6 montheir medical visi
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-229

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Roter, 199584

Int: Live Int: Live CC: NA 

Int: Discussion group, Lecture, Role play, Standardized patient Int: Discussion group, Lecture, Role play, Standardized patient CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Recognition & management of emotional problems 

Cognitive skills 

Yes Recognition of emotional 

problems and distress was higher among PD physicians than among control physicians. PD physicians also used more management strategies for emotional problems than control group (p=0.03). EH physicians were not significantly different than control physicians in terms of recognizing emotional problems & distress (p=0.18) and using any of the management strategies. 

Physicians' use ocommunication stheir practices chas a result of an CME program. Physicians trainespecific communskills recognized psychological protheir patients thauntrained physicTrained physiciashowed greater cproficiency in the management of a simulated patient compared with cgroup physicianspatients of trainephysicians compuntrained physicshowed greater rin emotional distas long as 6 montheir medical visi
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-230

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Roter, 199584

Int: Live Int: Live CC: NA 

Int: Discussion group, Lecture, Role play, Standardized patient Int: Discussion group, Lecture, Role play, Standardized patient CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Clinical proficiency 

Cognitive skills 

Yes PD physicians had 

significantly higher scores than the control group. EH physicians had slightly higher scores than control, but the difference was not statistically significant. 

Physicians' use ocommunication stheir practices chas a result of an CME program. Physicians trainespecific communskills recognized psychological protheir patients thauntrained physicTrained physiciashowed greater cproficiency in the management of a simulated patient compared with cgroup physicianspatients of trainephysicians compuntrained physicshowed greater rin emotional distas long as 6 montheir medical visi
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-231

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Terry, 198149

Int: Live, Audio, Print Int: Live, Audio, Print Int: Audio, Print Int: Live, Audio, Print CC: Audio, Print 

Int: Case-based learning, Discussion group, Feedback Lecture, Readings Int: Case-based learning, Discussion group, Lecture, Readings Int: Case-based learning, Feedback, Lecture, Readings Int: Case-based learning, Lecture, Readings CC: Case-based learning, Lecture, Readings 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive CC: Multiple time or repetitive 

Knowledge and judgment in COPD management, as measured by selfassessment questionnaires at baseline, 8 months, and 18 months 

Knowledge Cognitive skills 

Yes Yes 

Intervention groups showed a 23% significant increase in scores on posttest 1 to match those of pulmonologists, while there was no improvement for control groups (p>0.05 for group differences). Intervention groups receiving feedback had similar scores on posttest 2 to intervention groups not receiving feedback. Among intervention physicians who did not agree to participate in standardized patients, posttest scores dropped to baseline. 

Physicians comphome study AV pincreased knowleabout diagnosis treatment of COPtheir behavior in simulated exercisnot different from controls. Experimgroup physicians more patient-eduand smoking cesinformation durinvisits. Group meeneeds assessmefeedback (given after tests) had napparent effect bthe audiovisual m

Skills Objectives Not Met, Evaluation Duration Not ReportedMukohara, 200513

Int: Computerbased off-line CC: Computerbased off-line 

Int: Readings CC: Readings 

Int: Multiple time or repetitive CC: Multiple time or repetitive 

Frequency of finding answers to clinical questions 

Cognitive skills 

No By the end of the 3-

month trial, there were no significant differences within or between the WBJC and control groups in the frequency of finding an answer to clinical questions. 

While doctors appreciated thessummaries, whicimproved their reefficiency, the intervention had impact on their uresearch evidencpractice. 
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-232

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Skills Objectives Not Met, Evaluation Duration Greater Than 30 DaysRodney, 198677

Int: Live, Video Int: Live, Video CC: NA 

Int: Demonstration, Lecture, Simulation (other than standardized patient or roleplay) Int: Demonstration, Lecture, Simulation (other than standardized patient or roleplay) CC: NA 

Int: One time Int: One time CC: NA 

Behavior related to flexible sigmoidoscopy use, as measured by phone or written survey 

Attitudes Cognitive skills Practice behavior 

Yes No Yes 

Small group learners were more likely to acquire additional training and teaching attachments for their sigmoidoscopes, and less likely to use small (35 cm) scopes; there was no difference in biopsy utilization. 90% of large group learners acquired scopes after training vs. 40-56% of small groups. Small groups were associated with shorter procedure times (p<0.05) for first 10 procedures, but otherwise no differences in times, insertion depths, or number of exams performed. Compared to a randomly surveyed group of physicians, those with CME were significantly more likely to perform flexible sigmoidoscopy. 

Physicians who participate in couflexible sigmoidohave a higher proof office utilizatiothese skills than who do not take Minimal differencbetween large angroup CME form
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-233

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Skills Objectives With Mixed Results, Evaluation Duration Greater Than 30 DaysCarney, 199588

Int: Live, Video CC: NA 

Int: Demonstration, Discussion group, Feedback, Lecture Programmed learning, Role play CC: NA 

Int: One time CC: NA 

Which CME techniques had the greatest effect on physician cancer screening and prevention counseling skills 

Cognitive skills Practice behavior 

MixedMixed

Performance of intervention physicians was consistently better though only 4 of 19 measured endpoints of the physicians' behavior by the SP reached statistical significance. 

Performance bastechniques have positive influence physicians' perfo(especially in skitraining). Using unannounced standardized patfeasible method assess performawithin the practicenvironment. 
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-234

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Skills Objectives With No Control Group, Evaluation Duration Not ReportedLeopold, 200580

Int: Print Int: Computerbased off-line Int: Live 

Int: Readings Int: Demonstration, Readings Int: Demonstration, Feedback, Mentor/Preceptor, Simulation (other than standardized patient or roleplay) 

Int: NA Int: NA Int: One time 

Competence in task 

Skills (psychomotor or procedural skills) 

No control group 

Before the instruction, competence skills did not differ between the three randomized groups. After the instruction, competence skills increased for all groups. 

Even low-intensitof instruction impindividuals’ conficompetence, and assessment of thin performing the straightforward psychomotor tassimulated knee inHowever, men aphysicians disproportionateloverestimated thboth before and training, a finding worsened as conincreased. The inrelationship betwconfidence and competence that observed before educational interas well as the demographic diffthat we noted shraise questions ahow complex new procedures shouintroduced and wtrained practitionshould begin to pthem. 
Evidence table 10. Effectiveness Of continuing medical education on short-term and long-term skill outcomes 

F-235

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Hergenroeder, 200260

Int: Video, Print Int: Live, Video, Print 

Int: Demonstration Int: Demonstration, Feedback, Simulation (other than standardized patient or roleplay) 

Int: One time Int: One time 

Physician skills in performing a physical exam 

Skills (psychomotor or procedural skills) 

No control group

There was a statistically significant increase between baseline and followup in clinical skills assessment examination scores for both intervention groups, and there was a significant difference between the groups at followup. 

"This study demothat improvemenphysicians' knowand skills in perfoankle and knee pexaminations weassociated with tphysicians' particin either intervenThe improvemenphysicians' knowand skills in the aknee examinatiogreater in the vidplus skills intervegroup than in the videotape-alone 

AV = audiovisual; CC = concurrent control; CME = continuing medical education; COPD = chronic obstructive pulmonary disease; EH = emotion ha

group; KFP = Key Features Problems; NA = not applicable; NR = not reported; PBL = problem-based learning; PD = problem defining; SD = standar

standardized patient; URI = upper respiratory infection; WARN = Wisconsin Antibiotic Resistance Network 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-236 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Practice Behavior Objectives Met, Evaluation Duration Not ReportedFrush, 200687

Int: Internet, not real time, Video CC: NA 

Int: Lecture, Simulated scenario CC: NA 

Int: Education was available for 3 months CC: NA 

Dosing deviation from accepted dose range 

Practice behavior 

Yes In this randomized, 

controlled clinical trial, we found a significant reduction in medication dosing deviation when emergency providers received education about proper use of one resuscitation aid (the Broselow Pediatric Resuscitation Tape) as compared with no education being offered. 

The Web-based education prograproper use of the Broselow PediatResuscitation Taimprove dosing aand reduce dosin
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-237 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Juzych, 200592

Int: Live, Print CC: NA 

Int: Case-based learning, Lecture, Readings CC: NA 

Int: Single half day session CC: NA 

Prescription rates of antibiotics 

Practice behavior 

Yes Among physicians in the 

intervention population, the overall prescribing rates of antibiotics declined 24.6% from 49.9% to 37.6% (P<.0001). Between the control and intervention groups, the decline in prescribing rates for antibiotics was significant (P<.0001). Statistically significant declines in prescribing of antibiotics were seen among physicians in the intervention group for pharyngitis (P=.007), otitis media (P=.001), and URIsnos (P=.001). For bronchitis, there was no significant change in prescribing of antibiotics in the intervention group (P=.84). 

This study suggean educational intervention on appropriate antibreduced antibiotitreating uncomplURIs. A significadifference in chaprescribing rates the intervention acontrol populatiofound only for URThis confirms prestudies of educainterventions to rantimicrobial use treating URIs and demonstrates thameaningful reducachievable when program is appliestaff members at Medicaid HMO. Educational efforimprove prescribantibiotics for treuncomplicated vshould be expanshould include acare staff. Such should provide pand staff with the diagnose URIs appropriately and them how to conpatients why antimicrobials maappropriate. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-238 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Bloomfield, 200568

Int: Live, Print, Patient letter prompting patient to discuss with provider about letter contents (treatment) Int: Live, Print, Chart reminders appearing on cover page of patient's computerized medical record Int: Live, Print, Progress notes reminding primary care clinician about appropriate approach CC: NA 

Int: Discussion group, Lecture, Patient informs provider Int: Discussion group, Lecture, Point of care Int: Discussion group, Lecture, Point of care CC: NA 

Int: One time Int: One time Int: One time CC: NA 

Prescription rates 

Practice behavior 

Yes The prescription rate at 

the control sites did not change between the two time periods (preintervention period 18.9%, intervention period, 17.7%, p=0.19), but it increased from 8.3% to 39.1% (OR=6.5, 95% CI 5.2 to 8.2, p=0.0001) at the intervention sites. There was a significant interaction between group and time period (p=0.0001). The adjusted odds of receiving a prescription during the intervention period was 3.1 times higher at the intervention sites than at the control sites (95% CI 2.1 to 4.7). Overall, there was no statistically significant difference in prescription rates among the three prompt groups (40.7% for progress notes, 36.9% for patient letters, and 39.4% for reminders, p=0.60). However, there was a significant interaction between group and site, indicating that the efficacy of the prompts differed by site.

In conclusion, thshows that a relasimple interventioeducational workopinion leader inand prompts) batheoretical modeprovider behaviois designed to adempirically identibarriers, can ressubstantial improin provider prescbehavior. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-239 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Dormuth, 200415

Int: Print CC: Print 

Int: Readings CC: Readings 

Int: Multiple time or repetitive CC: Multiple time or repetitive 

Proportion receiving drugs recommended from letter 

Practice behavior 

Yes A significant change was 

observed in the proportion of newly treated patients receiving the analysis drugs as first-line therapy. The preference for the analysis drugs was 1.3 times more in the predicted direction in the intervention group of physicians than in the control group (95% CI: 1.13–1.52). 

The results of thirandomized conttrial demonstrate significant changprescribing to netreated patients wimpact of a serieletters was subjecombined analysWe conclude thaletters distributed ongoing series frcredible and trussource can have clinically significaimpact on prescrnewly treated pa

Goldberg, 200193

Int: Live, Video, Print CC: NA 

Int: Academic detailing, Discussion group, Lecture CC: NA 

Int: Multiple time or repetitive CC: NA 

Quarterly observations of surgical rates 

Practice behavior 

Yes After the intervention, the 

rates of surgery went down in the intervention group practices, while rates went up in the control group practices. Reduction in surgical rate: 8.9% (p=0.01) 

After implementathe intervention, rates declined in intervention combut increased sligthe control commThe net effect of intervention is esto be a decline ooperations per 10relative reduction (P = 0.01). ConcWe were able to scientific evidencengender voluntachange in back ppractice patterns entire communiti
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-240 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Gerrity, 199932

Int: Live, Video, Audio, Print CC: NA 

Int: Clinical experiences, Discussion group, Feedback, Lecture, Readings, Role play CC: NA 

Int: Multiple time or repetitive CC: NA 

Participatory decision making 

Practice behavior 

Yes Intervention physicians 

scored higher than control physicians on the Participatory DecisionMaking Scale (p=.017 and p=.014 in two SP cases). 

The Depression Education Prograchanged physiciabehavior and maimportant compothe efforts to impcare of depressepatients. 

Grady, 199779

CC: Live, Print Int: Live, Print Int: Live, Print 

CC: Lecture, Readings Int: Lecture, Point of care, Readings Int: Feedback, Lecture, Point of care, Readings Financial reward for compliance 

CC: One time Int: Multiple time or repetitive Int: Multiple time or repetitive 

Effect of intervention on mammography referral, completion, and compliance rates at the practice level

Practice behavior 

Yes Mammography referral, 

completion and compliance were all higher at one year at the experimental groups (i.e. cue) than at the control groups. However, no significant difference was seen between the two experimental groups (i.e. no extra effect from feedback and rewards). 

Cueing (posters chart stickers) hapositive impact omammography rcompletion, and compliance abovbeyond educatioHowever, there wadded benefit frofeedback and finrewards for compIn addition, physacceptance of thinterventions was marginal. There appears to be a cimpact of the phycharacteristics oresults of the inte
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-241 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Grady, 199779

CC: Live, Print Int: Live, Print Int: Live, Print 

CC: Lecture, Readings Int: Lecture, Point of care, Readings Int: Feedback, Lecture, Point of care, Readings Financial reward for compliance 

CC: One time Int: Multiple time or repetitive Int: Multiple time or repetitive 

Impact of interventions at the physician level 

Practice behavior 

Yes Same result as above-the 

pattern was the same for physicians 

Cueing (posters chart stickers) hapositive impact omammography rcompletion, and compliance abovbeyond educatioHowever, there wadded benefit frofeedback and finrewards for compIn addition, physacceptance of thinterventions was marginal. There appears to be a cimpact of the phycharacteristics oresults of the inte

Costanza, 199235

Int: Live, Print CC: NA 

Int: Discussion group, Lecture, Point of care, Simulation (other than standardized patient or roleplay) CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician's self report of screening practice changes 

Practice behavior 

Yes An adjusted odds ratio of 

7.85 was obtained when comparing the improvement in selfreported screening behavior among physicians in the intervention group to that in the control group. 

This study demothat primary care physicians will chtheir screening presponse to interaimed at altering regarding mammbenefits or to bartheir sense of codevelopment. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-242 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Lane, 199169

Int: Live, Print Int: Live, Print Int: NA CC: NA 

Int: Audience response systems, Clinical experiences, Demonstration, Discussion group, Feedback, Lecture, Mentor/Preceptor, Readings Int: Audience response systems, Clinical experiences, Demonstration, Discussion group, Feedback, Lecture, Mentor/Preceptor, Readings Int: NA CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: No CME intervention, just free mammography CC: NA 

Change in mammography referrals 

Practice behavior 

Yes Physicians across 

specialties reported an increase in mammography screening referrals. 

Physicians in the interventions rep(self) an increase number of mammreferrals. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-243 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Andersen, 199036

Int: Live, Video CC: NA 

Int: Case-based learning Lecture CC: NA 

Int: One time CC: NA 

Physicians' treatment recommendations for psychiatric conditions, as measured by pre- and post-tests incorporating multiple case vignettes 

Attitudes Practice behavior 

Yes Yes 

Compared with control, intervention physicians were significantly more inclined to refer patients to mental health professional and less inclined to treat them in primary care. Referral was high on pretest for both groups, but experimental group showed increase for 6 of 7 (1 tie) disorders while control did for 2 of 5 (3 ties). For medication, there were no significant effects. For behavioral therapy, there were significant increases for intervention group. 

A brief, single-seintervention can impact on physicpsychiatric diagnabilities. Additionphysicians particthe intervention wmore likely to refpsychiatric patien

Practice Behavior Objectives Met, Evaluation Duration Less Than or Equal to 30 DaysSharif, 200294

Int: Live CC: NA 

Int: Lecture, Role play CC: NA 

Int: One time CC: NA 

Smoking cessation behavior and documentation, as measured by standardized chart abstraction of medical records for all scheduled visits 3 weeks before and 3 weeks after workshop training 

Practice behavior 

Yes Posttraining, workshop 

attendees were significantly more likely to do 3 of 6 behaviors: inquire about parental smoking status (RR 3.06, 1.86-4.87), to identify smokers (RR 8.89, 2.79-28.40), and to offer advice about the effects of environmental tobacco smoke exposure (RR 2.33, 1.15-4.72). 

AAPP physician in smoking cessacounseling was ein changing physbehavior at this i
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-244 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Practice Behavior Objectives Met, Evaluation Duration Greater Than 30 DaysStein, 200195

Int: Live, Handheld, By phone CC: NA 

Int: Readings, study physician visit, algorithm CC: NA 

Int: Multiple time or repetitive CC: NA 

Amount of NSAID use before and after the intervention 

Practice behavior 

Yes The intervention resulted 

in fewer NSAIDS and more acetaminophen being used in the intervention homes (p=0.0001). 

An educational intervention effecreduced NSAID nursing homes wworsening of arthpain. 

Maclure, 199896

CC: NA Int: Live, Video CC: NA Int: Live CC: NA Int: Print 

CC: NA Int: Lecture, teleconference CC: NA Int: Discussion group, Lecture CC: NA Int: Readings 

CC: NA Int: One time CC: NA Int: NR CC: NA Int: Multiple time or repetitive 

Pooled impact of all three interventions

Practice behavior 

Yes There was a statistically 

significant decrease in the prescription of calcium channel blockers (and an increase in the prescription of diuretics) Increase in preference for thiazides by a factor of 1.28 (95% CI 1.02-1.61), and decrease in preference for CCBs by 0.64 (95% CI 0.41-1.0). 

There was a gentrend toward the effect by the eduinterventions. 

Rabin, 199897

CC: NA Int: Print Int: Live, Print 

CC: NA Int: Readings Int: Readings, Standardized patient 

CC: NA Int: NR Int: Multiple time or repetitive 

Simulated patient evaluation of physician practice 3 months after the intervention 

Practice behavior 

Yes The simulated patient 

assessment showed significant improvement in physician behaviors related to STD risk in patients in the physician group who received simulated patient instruction during the intervention. There was no significant difference between the education material only group and the control group. 

Simulated patieninstructors did imperformance relano intervention aeducational mateonly. However, twere still deficienperformance whithe subsequent ttrainers interventwas the focus of second part of th
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-245 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Rabin, 199897

CC: NA Int: Print Int: Live, Print 

CC: NA Int: Readings Int: Readings, Standardized patient 

CC: NA Int: NR Int: Multiple time or repetitive 

Physician self-report of practice behaviors 

Practice behavior 

Yes Physicians in the 

educational materials only group did not improve relative to controls. However, physicians who were in the simulated patient groups did show significant improvement in self-rated behaviors relative to the other two groups. 

Simulated patieninstructors did imperformance relano intervention aeducational mateonly. However, thwere still deficienperformance whithe subsequent ttrainers interventwas the focus of second part of th

Schwartzberg, 199798

CC: NA Int: Live, Print 

CC: NA Int: Case-based learning, Lecture, Readings 

CC: NA Int: One time 

Physician self-report of home care activity in their practice

Practice behavior 

Yes Physicians in the IG were 

statistically significantly more likely to report making house calls, referred to community agencies, and changed their office practice at 3 months. 

The seminars imphysicians' attitutoward home carprovided a changpractice behavioregarding home months relative tcontrols. 

Lewis, 199370

Int: Live, Video Int: Print CC: NA 

Int: Discussion group, Lecture Int: Readings CC: NA 

Int: One time Int: One time CC: NA 

Percentages of new patients reporting they were asked sexual history questions by their physicians (internists) 

Practice behavior 

Yes There was a statistically 

significant increase in the proportion of new patients seen by internists at the experimental site who reported being asked sexual history questions. 

These results suthat a medical edprogram that goebeyond standard and incorporates interactive formachange physician behaviors. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-246 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Mehler, 200599

Int: Internet, not real time, Print Int: Live, Print CC: NA 

Int: Academic detailing, Readings Int: Academic detailing, Lecture CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Likelihood of lipid testing 

Practice behavior 

Yes They observed a 3.0 

(1.62-5.66) times greater likelihood of lipid testing in the electronic group compared with the control group, adjusting for type of site and race. There was a moderate difference between the direct group and control group (P=.09); however, there was no evidence of a difference between the electronic and direct groups (P=.17). Last, the change in the proportion of patients experiencing at least one of the favorable provider actions for the pre- and post-intervention periods was significantly greater in the combined direct and electronic groups versus control (P=.01). 

A simple educatiintervention seempositively influenprovider behavioarea of lipid manin diabetes mellitelectronic and didetailing seem to viable approacheFuture studies to determine optimaeducational comthat facilitate appprovider actions or intensify lipid tseem warranted the burgeoning population of diapatients at risk focoronary heart dmorbidity and mo
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-247 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Rahme, 2005100

Int: Live, Print Int: Live Int: Print CC: NA 

Int: Discussion group, Lecture, Readings Int: Discussion group, Lecture Int: Readings CC: NA 

Int: One time Int: One time Int: NA CC: NA 

Evaluation of NSAIDS, COX_2 inhibitors and acetaminophen dispensed prescriptions

Practice behavior 

Yes There was a greater 

improvement in scores for both NSAIDs and COX-2 inhibitors and an increase in the number of acetaminophen prescriptions in the workshop groups over the control & decision tree groups. The Bayesian hierarchical model showed that there were improvements in all groups over time, with stronger evidence in the workshop & tree group. The adjusted odds (95% CI) of adequate prescribing in the postversus pre-intervention periods in the workshop & tree group was 1.8 (1.3, 2.4). This odds was higher than that found in the control group 1.3 (0.9, 1.8); but, the odds ratio, 1.5 (0.9, 2.3), indicated only moderate evidence for the workshop & tree group over control. 

In summary, this found some weaevidence that an educational packwhich the generapractitioner showopportunity to disrelated issues camore effective in changing prescripractices compathe distribution oguidelines. Physattendance is, pekey to the succesinteractive interv
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-248 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Davis, 2004101

Int: Live, Video, Audio, Audio teleconferencing CC: NA 

Int: Case-based learning, Discussion group, Problembased learning or team-based learning CC: NA 

Int: Multiple time or repetitive CC: NA 

Change in prescribing patterns with respect to use of LTRAs 

Practice behavior 

Yes A statistically significant 

(p<0.005) increase in LTRAs were prescribed by study physicians after the intervention compared to before the intervention while no statistically significant change in prescribing patterns of LTRAs was experienced by control physicians. 

We report the firsstyle teleconfereseries to demonssignificant changphysician prescrhabits toward beconcordance witasthma guidelinestudy demonstrapowerful impact PBL format has ophysician learninactually translateaction—a measuchange in beliefs behavior that led changes in the nprescriptions writthe participants. standard, traditiodidactic lectures the norm for CMthe US and credgiven for attendamany studies hashown that PBL actually more effenhancing physiclearning. In the Uoffices should plasmall-group workteleconferences integrating the Pformat, with skillefacilitators trainedevelop the casemoderate the sesultimately to impquality of CME in 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-249 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Herbert, 2004102

Int: Live Int: Live Int: Live CC: Live 

Int: Case-based learning, Clinical experiences, Demonstration, Discussion group, Problembased learning or team-based learning, Readings Int: Clinical experiences, Feedback, Readings Int: Case-based learning, Clinical experiences, Demonstration, Discussion group, Problembased learning or team-based learning CC: NA 

Int: One time Int: One time Int: One time CC: One time 

Physician prescribing preference. 

Practice behavior 

Yes In the 6 months following 

the intervention module, physicians' preference for thiazides increased by 6.8% (95% CI: 2.2%, 11.5%) after controlling for the control group's preference. The portrait was similarly associated with a 6.5% increase in preference for thiazides (95% CI: 1.8%, 11.1%). The group of physicians who received both interventions had an 11.5% increase in preference (95% CI: 4%, 18.9%). 

 "This study demonstrated thaeducational interwith clear messalead to meaningfchanges in physprescribing behathis trial, both interventions weassociated with significant absoluincreases in the thiazides as first-therapy for hypertension—m(6.8%) and portra(6.5%). The comintervention, conboth the educatiomodule and portthe strongest imp(11.5%)." 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-250 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Beaulieu, 200444

Int: Live, Print Int: Live Int: Print CC: NA 

Int: Case-based learning, Discussion, group, Problembased learning or team-based learning, Programmed learning, Readings Int: Case-based learning, Discussion group, Problembased learning or team-based learning, Readings Int: Discussion group, Programmed learning, Readings CC: NA 

Int: One time Int: One time Int: One time CC: NA 

Medication prescriptions

Practice behavior 

Yes In ITT analysis, there was 

an adjusted statistically significant 8% improvement (p=0.003) on mean scores (scores to evaluate impact on medication prescription) between both the control group and the workshop and workshop and decision tree groups combined. (PP analysis showed a 12% improvement (p=0.008)). 

"The results of thevaluations have demonstrated thaevidence-based interventions wesuccessful not onimproving physicknowledge regardiagnosis and management of Oalso—more impoin changing their to make more aptherapy choices patients. The obsmodification of thprescription pattereflects an improin their medical pwhich may lead tpatient outcomes generate greater efficiencies for thcare system." 

Cohn, 200219

Int: Live, Print Int: Live, Print CC: NA 

Int: Academic detailing Int: Academic detailing CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Provider practices 

Practice behavior 

Yes Data show evidence that 

the intervention group providers improved their practices about asking about DES while the control group didn't improve. No mention of statistical significance was made. 

"Academic detailincrease DES knand history takinprimary care pro
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-251 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Lane, 200145

Int: Live CC: NA 

Int: Lecture, Standardized patient CC: NA 

Int: One time CC: NA 

One main outcome measure was whether or not physicians referred 90-100% of women over age 50 to get a mammogram every 1-2 years. The other main outcome was a reduction in the score of needing the CME activity (improved knowledge and understanding) 

Knowledge Practice behavior 

Yes Yes 

There was improvement in the need for CME scores of more physicians in the intervention group than the control group. The intervention significantly improved knowledge and behaviors about breast cancer screening practices. 

Participation in thactivity improved physicians' awarknowledge, and behaviors regardclinical breast exbreast cancer scpractices, as coma control group. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-252 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Norris, 200081

Int: Live, Print, F/U phone calls about protocol CC: NA 

Int: Lecture, Point of care, Opinion-leader from clinic teaching CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician behavior in counseling patients about physical activity, as reported on patient questionnaire 

Practice behavior 

Yes Both intervention and 

control physicians asked and counseled about physical activity more frequently at 6-month follow-up, but the increase was significantly higher for intervention providers (p=0.001). Inquiring change from baseline to f/u: intervention 48->79% vs. control 64->64% inquiring. Counseling change from baseline to followup: intervention 42->94% vs. control 55->81%. 

"A one-time PACcounseling sessiminimal reinforcea setting with higbaseline levels odoes not further activity ... Contemadvanced in stagbehavior change
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-253 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Brown, 199972

Int: Live CC: NA 

Int: Clinical experiences, Discussion group, Lecture, Role play, Clinicians audiotaped interaction with patients and listened between workshops CC: NA 

Int: Multiple time or repetitive CC: NA 

Selfassessment of clinicians' communication skills, attitudes, and behavior, as measured by participant questionnaire 

Attitudes Skills (psychomotor or procedural skills) Practice behavior 

Yes Yes Yes 

Intervention group noted substantial improvements compared with control in 8 of 24 skills / attitudes / behaviors: awareness of and confidence in dealing with patients whom they found difficult; abilities to compliment patients’ efforts, ask open-ended questions, address psychosocial factors, express empathy and reassurance, and clarify expectations. 33% of clinicians reported that fewer than 5% of visits were frustrating after the program (compared with 21% of clinicians at baseline). Three months after the program, clinicians in the intervention group reported that it had improved communication with patients. On average, however, the scores for clinicians’ rating of improvement in patient satisfaction and improvement in clinicians’ personal satisfaction in their work decreased below the midpoint on the five-point scale (mean rating, 2.85 for both items). 

”Thriving in a BuPractice: PhysiciPatient Communa typical continuimedical educatioprogram geared developing cliniccommunication snot effective in imgeneral patient satisfaction. To iglobal visit satisfcommunication straining programneed to be longemore intensive, tbroader range of and provide ongoperformance fee
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-254 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Gifford, 199646

Int: Video, Print CC: NA 

Int: Programmed learning, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Neurologists adherences to practice recommendations for movement disorders, as measured by test 

Practice behavior 

Yes For 9 of the 16 

recommendations, a higher proportion of the intervention group compared with controls reported clinical decisions that were adherent to the practice recommendations; for all 9 differences p<.05. Range of adherence for intervention was 41-98% and for control 22-97%. 

The educational improved neuroloreported decisionmaking. 

Anderson, 1996103

Int: Live, Video CC: NA CC: NA 

Int: Case-based learning, Discussion group, Lecture CC: NA CC: NA 

Int: One time CC: NA CC: NA 

Change in physician prescribing patterns of regulated drugs 

Practice behavior 

Yes There was a 33% 

reduction in the number of prescriptions written in the education plus notification group, a 25% reduction in the notification only group, and a slight increase in the control group. Both the education plus notification and notification only groups wrote significantly fewer prescriptions than the control group (p<0.003 and p<0.008 respectively). There was not a significant difference between the number of prescriptions written by the education plus notification group and the notification only groups (p<0.719). 

Notification that tnumber of prescthey had written abnormally high effective in signifreducing the numprescriptions writduring the subsemonths as notificcombined with a designed groupeducation activity
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-255 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Maiman, 198847

Int: Live, Print Int: Print CC: NA 

Int: Discussion group, Lecture, Readings Int: Readings CC: NA 

Int: Multiple time or repetitive Int: One time CC: NA 

Mother report of quantity of physician compliance enhancing behavior 

Practice behavior 

Yes Significant association 

between study group and extent of behaviors reported by mothers (patients). 

CME increased pknowledge and compliance-enhapractices and resimprovement in madherence to the

White, 198548

Int: Live CC: Live, NA Int: Live Int: Live 

Int: Case-based learning, Discussion group, Lecture CC: NA Int: NR Int: NR 

Int: One time CC: NA Int: One time Int: One time 

Desired patient care practices in hospital care of patients with acute myocardial infarction, as measured by chart review 

Practice behavior 

Yes Intervention group had 

statistically significantly higher percent of desired care practices (60%, 56.6-63.3) vs. control group (46.3%, 40.4-52.3)[p-value <0.001 for post-test comparisons). Intervention group had statistically significant gains in 3 of 8 objectives: prophylactic use of lidocaine, avoidance of intramuscular medications, and appropriate length of stay. 

"A carefully concand executed traCME program canot only in signifiincreases in physknowledge but arelated changes patient care prac"both knowledge behavioral changpersist for at leasmonths" 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-256 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Terry, 198149

Int: Live, Audio, Print Int: Live, Audio, Print Int: Audio, Print Int: Live, Audio, Print CC: Audio, Print 

Int: Case-based learning, Discussion group, Feedback, Lecture, Readings Int: Case-based learning, Discussion group, Lecture, Readings Int: Case-based learning, Feedback, Lecture, Readings Int: Case-based learning, Lecture, Readings CC: Case-based learning, Lecture, Readings 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive CC: Multiple time or repetitive 

Skill and behavior in COPD management, as measured by analysis of audiotaped standardized patient (SP) interaction 

Practice behavior 

Yes Physicians were not 

blinded due to audiotaping. Intervention and control groups did not differ on diagnostics / therapeutics scores, but intervention group used strategies for patient understanding and compliance more frequently (37% vs. 22%, p<0.01)[9 of 16 specific tasks significantly improved - explaining how medications would help, instructing on use of medications, repeating / summarizing / answering, discussing benefits of smoking cessation]. 

Physicians comphome study AV pincreased knowleabout diagnosis treatment of COPtheir behavior in simulated exercisnot different from controls. Experimgroup physicians more patient-eduand smoking cesinformation durinvisits. Group meeneeds assessmefeedback (given after tests) had napparent effect bthe audiovisual m
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-257 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Chodosh, 200662

Int: Live, Internet, not real time, Print CC: NA 

Int: Discussion group, Lecture CC: NA 

Int: Multiple time or repetitive CC: NA 

Quality of care, as measured by adherence to guidelines (as reported in prior manuscript) 

Practice behavior 

Yes Intervention group had 

significantly higher rates of guideline-adherent care for 21 of 29 care processes (e.g., management of behavior problems, use of respite, development of ongoing treatment plan) than usual care group (all p<0.05) at 1-year followup. 

Despite a succesintervention demonstrating siimprovements in of care for patiendementia, providknowledge and awere minimally a

Ozer, 2005104

Int: Live, Print CC: NA 

Int: Discussion group, Lecture, Role play, Screening and charting tools CC: NA 

Int: One time CC: NA 

Screening rates 

Practice behavior 

Yes Screening rates for 

different health behaviors (tobacco use, alcohol use, drug use, sexual behavior, seatbelt and helmet use) increased after the first intervention but remained stable after the second intervention (of screening and charting tools) compared to the comparison group.

The results of the ANCOVAs demothat (1) screening counseling rates significantly highintervention grouthe comparison gafter the full implementation ointervention (T2)screening and corates were signifhigher in the integroup than in the comparison grouthe training compalone (T1); and (screening and corates did not incsignificantly in thintervention grourelation to the comparison grouthe addition of thcomponent. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-258 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Ozer, 2005104

Int: Live, Print CC: NA 

Int: Discussion group, Lecture, Role play, Screening and charting tools CC: NA 

Int: One time CC: NA 

Counseling rates 

Practice behavior 

Yes Counseling rates for 

different health behaviors (tobacco use, alcohol use, drug use, sexual behavior, seatbelt and helmet use) increased after the first intervention but remained stable after the second intervention (of screening and charting tools) compared to the comparison group.

The results of the ANCOVAs demothat (1) screening counseling rates significantly highintervention grouthe comparison gafter the full implementation ointervention (T2)screening and corates were signifhigher in the integroup than in the comparison grouthe training compalone (T1); and (screening and corates did not incsignificantly in thintervention grourelation to the comparison grouthe addition of thcomponent. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-259 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Soumerai, 1987105

Int: Live, Print Int: Print CC: NA 

Int: Academic detailing, Readings, Brochures Int: Readings, Brochures CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

To reduce inappropriate prescribing of 3 target drugs. 

Practice behavior 

Yes Reduction in prescribing 

was highly significant -8 to -20%, p<.025) in 11 of the 14 physician groups. The presence of a followup reinforcement visit was strong independent predictor of prescribing change (p<0.05) 

Face to face pharmaceutical eis an effective quassurance and ccontainment straa wide variety of physicians, regatheir background baseline prescribpractices. Reinforcement vbe necessary to economically or cimportant improvin physician drug prescribing patteincrease from on2 visits was assowith an approximdoubling of the sprogram effect. Long visits with physicians (more 10-15 minutes) dappear to increasprescribing improvements. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-260 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Chassin, 1986106

Int: Live, Print CC: NA 

Int: Feedback, Lecture, 16 of 64 hospitals received only print mailings to physicians with privileges, without lecture CC: NA 

Int: One time CC: NA 

Pelvimetry rates, as measured by chart abstraction from radiology and obstetrical delivery logs

Practice behavior 

Yes Intervention group 

hospitals had 86% reduction in pelvimetry rate compared with 53% in control group (p<0.00001). Findings slightly tempered by increased baseline pelvimetry rate in control group (p<0.001) and general trend toward decreased pelvimetry in both groups prior to intervention. 

An educational pfollowed by feeddata can markedimprove physiciaperformance by decreasing inapppelvimetry at a hlevel. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-261 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Perera, 1983107

Int: Live Int: Live CC: NA 

Int: Clinical experiences, Demonstration Int: Clinical experiences, Demonstration CC: NA 

Int: One time Int: One time CC: NA 

Sigmoidoscopy rate and sigmoidoscopy: barium enema ratio, as measured by clinic nurses before and after training

Practice behavior 

Yes Baseline rates were 

similar between two intervention groups and control. Intervention group had significantly higher post-intervention sigmoidoscopy rate (13.5/1000 vs. 9.8/1000 pre-intervention, p <0.05). The rate in the intervention group declined to 11.1/1000 by 7-10 months. The group receiving training later increased from 6.2 to 8.4/1000, and the control decreased from 8.5 to 7.4/1000. The sigmoidoscopy to barium enema ratio significantly increased in first study group from 0.62 to 0.90 (p < .05). The second study group increased from 0.59 to 0.79, and the control group decreased from 0.69 to 0.67. 

"program increasrate of sigmoidosdone by physiciaimproves the promedical care (at over the short te
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-262 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Schectman, 2003108

Int: Live, Video, Print CC: Video, Print, NA 

Int: Lecture, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician behaviors: the adherence of utilization services to guidelines for low back pain. 

Practice behavior 

Yes There was a significant 

increase in guidelineconsistent behavior among intervention physicians while there was a decrease in the control group. 

"An intervention accepted strategphysician educatpractice audit witperformance feeand peer opinion use produced a mbut significant incphysician actions consistent with a guideline for the acute low back p

Ray, 2001109

Int: Live, Print CC: NA 

Int: Academic detailing, Readings, Chart reminders CC: NA 

Int: Multiple time or repetitive CC: NA 

Change between baseline andfollow-up years in: days of prescribed NSAIDs, acetaminophen, other drugs for musculoskeletal disorders, and GI drugs; outpatient visits and inpatient days of stay 

Practice behavior 

Yes Intervention-attributable 

reduction of 7% (95% CI, 3% to 11%) in days of prescribed NSAIDs use with concomitant increase in acetaminophen use. 

The educational modestly reduceexposure in comdwelling elderly pwithout undesirasubstitution of otmedications or dworsening of musculoskeletal symptoms. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-263 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Dietrich, 2000110

Int: Live, Print, Telephone call CC: NA 

Int: Academic detailing, Lecture, Problem-based learning or teambased learning, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Percent of parents counseled on sun protection by clinicians, as measured by parent selfreport survey

Practice behavior 

Yes Intervention community 

parents reported similar rates of counseling to control parents (25% and 26%, p=0.68), but significantly higher counseling after intervention (34% vs. 27%, p=0.03). 

The SunSafe pricare intervention increased sun prcounseling activiparticipating cliniConclusions limitparticipation rate survey (69% for survey; of intervecommunity clinicparticipated in baonly 59% on f/u sAlso unclear whacomponent of pareport may relate community intervrather than practintervention. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-264 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Dietrich, 2000110

Int: Live, Print, Telephone call CC: NA 

Int: Academic detailing, Lecture, Problem-based learning or teambased learning, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Improvement from baseline in provision of sun protection services to patients / families, as measured by clinician selfreport 

Practice behavior 

Yes Greater improvement for 

intervention vs. control clinicians in the availability of educational materials (91% vs. 43%, p=0.02), distribution (74% vs. 0%, p=0.001) of educational materials, and distribution of sunscreen samples (69% vs. 13%, p=0.02). Unclear if intention-totreat analysis done for providers in intervention communities who declined to participate in practice intervention. Consideration of baseline practices unclear in the calculation of "improvement from baseline." 

The SunSafe pricare intervention increased sun prcounseling activiparticipating cliniConclusions limitparticipation rate survey (69% for survey; of intervecommunity clinicparticipated in baonly 59% on f/u sAlso unclear whacomponent of pareport may relate community intervrather than practintervention. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-265 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Schroy, 199982

Int: Live CC: NA 

Int: Academic detailing, Discussion group, Lecture CC: NA 

Int: One time CC: NA 

Provider utilization of screening sigmoidoscopy as determined by appointment schedules 

Practice behavior 

Yes Use of on-site 

sigmoidoscopy was monitored by review of appointment schedules every 3 months after initiation of the program (for the intervention sites). Use of outside sigmoidoscopy services was monitored through the NHC's scheduling secretary and/or contact with all off-site endoscopists identified by referring primary care providers. There was substantial agreement between self-reported compliance rates and actual utilization; namely, 78% of those reporting compliance with sigmoidoscopy recommendations actually referred 1 or more asymptomatic average-risk persons for screening examinations. 90% of those reporting noncompliance were not referring patients for screening. Overall compliance, using utilization as the outcome of interest, was 47% in the intervention group, vs. 4% in the comparison group at year 1 (p<0.001). 

"In summary, this clearly shows thaacademic detailinform of an outreadidactic educatioseminar followed implementation osigmoidoscopy sis an effective strenhancing providcompliance with screening guidel
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-266 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Gonzales, 1999111

Int: Live, Print Int: Print CC: NA 

Int: Academic detailing, Demonstration, Lecture, Point of care Int: Academic detailing, Point of care CC: NA 

Int: One time Int: NA CC: NA 

Antibiotic prescriptions for incident visits 

Practice behavior 

Yes Full intervention 

demonstrated a significant decrease in the number of prescriptions. Other groups did not show a significant difference. 

Antibiotic treatmeadults diagnosed having uncomplicacute bronchitis safely reduced ucombination of pand practitioner interventions. 

Cummings, 1989112

Int: Live, Video, Print CC: NA 

Int: Demonstration, Discussion group, Lecture, Role play CC: NA 

Int: Multiple time or repetitive CC: NA 

Effectiveness of smoking cessation counseling 

Practice behavior 

Yes Physicians participating 

in the intervention increased their effectiveness of smoking cessation counseling. Specifically, they discussed cessation with patients more often and longer, set more quit dates, and gave out more self-help materials. 

Intervention physdiscussed smokigreater rate than control physicianIntervention physspent more time discussing smoktheir patients. Mosmoking patients intervention physset quit dates anmore follow-up wphysicians. Ratesmoking cessatioamong patients ayear were no diffbetween control intervention phys
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-267 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Cummings, 1989113

Int: Live, Video, Print CC: NA 

Int: Case-based learning, Demonstration, Discussion group, Lecture, Role play CC: NA 

Int: Multiple time or repetitive CC: NA 

Effectiveness of smoking cessation counseling 

Practice behavior 

Yes Physicians involved in 

the intervention improved aspects of their smoking cessation counseling. Specifically, they were more likely to set quit dates, were more likely to use nicotine gum appropriately, spent more time discussing smoking cessation with patients, and gave out more selfhelp brochures. 

Intervention physdiscussed smokicessation at a higthan did control physicians. They more time discussmoking overall, more of their patquit dates. Overasmoking cessatioof patients at one did not differ betwgroups. 

Kottke, 1989114

Int: Live, Print Int: Print CC: NA 

Int: Demonstration, Discussion group, Lecture, Readings Int: Readings CC: NA 

Int: One time Int: Amount of exposure determined by participant CC: NA 

Physician behaviors regarding counseling for smoking cessation 

Practice behavior 

Yes Patients of physicians in 

the workshop group and materials-only group reported their physicians were more likely to have asked them to quit smoking (p < 0.025)and were more likely to report agreeing to quit smoking (p < 0.025) when compared to the control group. Patients of physicians in the workshop group report they were asked to set a quit date (p < 0.005) compared to materialsonly group and control group. There was no difference between groups regarding patient reports of being asked if they smoked and patients being given follow-up appointments. 

A brief training pand/or the distribeducation materimarginally increasmoking cessatiobehaviors among volunteer physicdid not result in ismoking cessatioat one year. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-268 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Jennett, 198876

Int: Live, Print, Teleconference Int: Live, Print, Teleconference CC: NA 

Int: Discussion group, Feedback, Mentor/Preceptor, Readings Int: Discussion group, Feedback Mentor/Preceptor, Readings CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Percentage of recommended behaviors in cancer medicine carried out at the specified time period 

Practice behavior 

Yes The performance of study 

offices improved 6 months following the education by 35%. 

A carefully plannprogram, adherinessential learning principles, was ein changing office practice of volundoctors as long amonths after the intervention. 

Jennett, 198876

Int: Live, Print, Teleconference Int: Live, Print, Teleconference CC: NA 

Int: Discussion group, Feedback, Mentor/Preceptor, Readings Int: Discussion group, Feedback Mentor/Preceptor, Readings CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Performance of recommended behaviors in cardiovascular medicine carried out at the specified time period 

Practice behavior 

Yes Performance of 

recommended behaviors in study offices improved 44.5% as compared to controls, 4.0%. 

A carefully plannprogram, adherinessential learning principles, was ein changing office practice of volundoctors as long amonths after the intervention. 

Harris, 200575

Int: Live, Audio, Print, Teleconference CC: NA 

Int: Case-based learning, Lecture, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Documentation of tests performed 

Practice behavior 

 Yes Significantly more patient 

records in the intervention group had documentation of BMI, eye exams, communication of a treatment plan, and used a flow sheet. 

CME delivered bteleconference wfeasible, well attewell received by participants, and improved some kdiabetes managepractices and oualthough primary improving HbA1C not achieved. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-269 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Lindsay-McIntyre, 1987115

Int: Print Int: Live, Video, Print CC: NA 

Int: Point of care Int: Demonstration, Lecture, Point of care, Standardized patient CC: NA 

Int: Multiple time or repetitive Int: Training 1 time, cueing with each patient CC: NA 

Smoking cessation counseling behaviors by family physicians, as measured by chart review using intervention flowsheet (intervention) or medical records (control) 

Practice behavior 

Yes Physician discussed 

smoking in 85.4% of patients of trained physicians, vs. 70.2% of gum-only physicians and 31.1% of usual care physicians. Gum offered by >60% of gum-only and trained physicians vs. 9% for usual care physicians. Trained physicians reported patients wanted to see them again more often and to offer takehome materials compared with gum only or usual care. (p-values NR) 

Nicotine gum witcues helped stimpatient smoking cessation attempphysician training counseling with a intensive flowsheproduced more successful short-cessation. 

Stross, 1985116

Int: Live, Print, "Audiovisual materials" CC: NA 

Int: Feedback, Readings, Use of "educationally influential" physician CC: NA 

Int: Multiple time or repetitive CC: NA 

Inpatient care of osteoarthritis patients, as measured by chart audit of hospital and outpatient data 

Practice behavior 

Yes Intervention hospitals had 

statistically significant (all p<0.05) increased use of intraarticular corticosteroids (40% vs. 11%), decreased use of systemic steroids (3% vs. 22%), and increased preop PT (97% vs. 40%) compared with control; non-significant in 8 other areas, including pre-op / post-op management and length of stay. Use of intraarticular corticosteroids also increased in outpatients in intervention vs. control (23.9% vs. 14.6%, p<0.05). 

Educationally infphysician peers influence use of intraarticular and systemic steroids osteoarthritis management witcommunities. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-270 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Cummings, 1989117

Int: Live, Video, Patient education materials CC: NA 

Int: Case-based learning, Demonstration, Role play, Selfreflection CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician smoking cessation counseling behaviors, as measured by patient interviews 

Practice behavior 

Yes Intervention group was 

more likely to discuss smoking with patients who smoked (64% vs. 44%), spent more time counseling smokers about quitting (7.5 vs. 5.2 minutes), helped set more quit dates (29% vs. 5%), gave out more booklets (37% vs. 9%), and more likely to make followup appointments about quitting (19 vs. 11%). 

3-hour continuing education progracombined with sumaterials for officchanged the way physicians in privpractice counselesmoking, but had statistically signifeffects on patienoutcomes. Some significant differephysicians and pintervention vs. cgroup related to but adjustment fodid not affect out
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-271 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Schroy, 199982

Int: Live CC: NA 

Int: Academic detailing, Discussion group, Lecture CC: NA 

Int: One time CC: NA 

Provider attitudes or practices related to colorectal cancer screening as determined from a preeducational vs. posteducational survey instrument, based on an instrument that was similar to one used by the American Cancer Society in 1989 

Attitudes Practice behavior 

Unclear Yes 

At year 1, there were no significant differences in concern about patient fear and discomfort, time, procedural skills, or cost. However, the mean scores for equipment availability, efficacy, and yield were all significantly higher in the noncompliant group than the compliant group (p = 0.001). Most of the compliant providers were at the intervention sites, which indicate that availability of on-site screening may trump perceived barriers for sigmoidoscopy adherence. At baseline, 24% of providers at intervention sites, and 19% at comparison sites, reported recommending screening sigmoidoscopy. Significant differences in compliance were observed at the year 1 follow-up survey. Overall self-reported compliance rates increased by 36% at the intervention site, vs. 7% at the comparison site (p = 0.001). 

"In summary, this clearly shows thaacademic detailinform of an outreadidactic educatioseminar followed implementation osigmoidoscopy sis an effective strenhancing providcompliance with screening guidel
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-272 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Margolis, 2004118

Int: Live, Organized set of tools otherwise unspecified CC: NA 

Int: Academic detailing, Feedback, Lecture, Unspecified organized set of tools CC: NA 

Int: Multiple time or repetitive CC: NA 

Proportion of children who received all four preventive services (immunizations and screening for anemia, lead, and tuberculosis)

Practice behavior 

Yes The change in the 

prevalence of all four services between the beginning and end of the study was 4.6 fold greater (95% CI 1.6 to 13.2) in the intervention practices than in the control practices. 

A continuing eduprogram designeassist primary capractices in testinimplementing offsystems for prevhealth care prodclinically and stasignificant improvin rates of prevencare for children. 

McClellan, 2003119

Int: Video, Print CC: NA 

Int: Feedback, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

HbA1c tests ordering 

Practice behavior 

Yes Statistically significant 

increase 

"The main result study was that a population-based improvement intebased on applyinHCQIP model to ambulatory care associated with iprocesses of cardiabetes mellitus
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-273 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

McClellan, 2003119

Int: Video, Print CC: NA 

Int: Feedback, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Usage related to physician behavior in monitoring diabetic patients 

Practice behavior 

Yes Usage of HbA1c testing 

increased among both the intervention and control groups; usage among the intervention group was statistically significantly greater than the control group (p=0.02 unadjusted, p=0.03 adjusted). Other indicators (eye examinations, urine testing) also increased in both groups, though the differences between intervention and control were positive, but not statistically significant. 

"The main result study was that a population-based improvement intebased on applyinHCQIP model to ambulatory care associated with iprocesses of cardiabetes mellitus
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-274 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Mann, 199752

Int: Live, Video, Print Int: Live, Video, Print Cue stickers on medical chart CC: NA 

Int: Case-based learning, Demonstration, Discussion group, Readings, Simulation (other than standardized patient or roleplay) Int: Case-based learning, Demonstration, Discussion group, Point of care, Readings, Simulation (other than standardized patient or roleplay) CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NR 

Physicians dietary counseling practices, as measured by self-report. 

Practice behavior 

Yes Physicians in both 

intervention groups scored statistically significantly higher than the controls in dietary counseling practice evaluation (p=0.0005). 

Educational trainworkshops appeeffective in changphysician behavishould thus, be calong with additioresearch on the mechanisms of wbehavior change 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-275 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Clark, 1998120

CC: NA Int: Live, Video 

CC: NA Int: Case-based learning, Clinical experiences, Lecture 

CC: NA Int: Multiple time or repetitive 

Impact if intervention on physician self-report of practice behaviors at 5 months 

Practice behavior 

Yes Physicians in the 

intervention group were significantly more likely to use anti-inflammatory agents, address fears about meds, give written instructions, how to adjust meds in flair. Interestingly, intervention physicians also reported spending less time with asthma patients. 

The intervention positive impact ophysician and pareported behaviopatient outcomes also positively imby the interventioaddition, the impappeared to be mthan just that resfrom the increaseanti-inflammatory medication and tchange in practicbehavior and the management moseems to be impwell. 

Clark, 1998120

CC: NA Int: Live, Video 

CC: NA Int: Case-based learning, Clinical experiences, Lecture 

CC: NA Int: Multiple time or repetitive 

Parent (patient) views of physician behavior 

Practice behavior 

Yes Intervention physicians 

were statistically significantly more likely to be reassuring and encouraging, stating a goal that the child be active, were more likely to have prescribed antiinflammatory medication, and to have provided a written plan. 

The intervention positive impact ophysician and pareported behaviopatient outcomes also positively imby the interventioaddition, the impappeared to be mthan just that resfrom the increaseanti-inflammatory medication and tchange in practicbehavior and the management moseems to be impwell. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-276 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Moran, 1996121

Int: Live CC: Live 

Int: Case-based learning Discussion group CC: Case-based learning Discussion group 

Int: Multiple time or repetitive CC: Multiple time or repetitive 

Changes in score for dimensions of care (clinical patient care), charting, prevention, and drug use

Practice behavior 

Yes Average scores 

increased significantly and approached average Manitoba scores by 18 months. 

This study demothe feasibility of improving physicperformance throform of CME thadesigned to be supportive, non-pand learner-focu

Browner, 1994122

Int: Live, Print Int: Live, Print CC: NA 

Int: Lecture, Readings Int: Academic detailing, Casebased learning, Lecture, Point of care, Readings CC: NA 

Int: One time Int: Multiple time or repetitive CC: NA 

Proportion of patients with high cholesterol management in compliance with NCEP recommendations 

Practice behavior 

Yes Trend toward better 

compliance in the two CME groups (p=.07). Compliance rates were 6% higher (95% CI 1% -11%, p=.02) in the intensive CME group than in the control group.

CME was not paeffective in improcompliance with guidelines amonuniversity affiliatecommunity physwho practice primcare. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-277 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Rodney, 198677

Int: Live, Video Int: Live, Video CC: NA 

Int: Demonstration, Lecture, Simulation (other than standardized patient or roleplay) Int: Demonstration, Lecture, Simulation (other than standardized patient or roleplay) CC: NA 

Int: One time Int: One time CC: NA 

Behavior related to flexible sigmoidoscopy use, as measured by phone or written survey 

Attitudes Cognitive skills Practice behavior 

Yes No Yes 

Small group learners were more likely to acquire additional training and teaching attachments for their sigmoidoscopes, and less likely to use small (35 cm) scopes; there was no difference in biopsy utilization. 90% of large group learners acquired scopes after training vs. 40-56% of small groups. Small groups were associated with shorter procedure times (p<0.05) for first 10 procedures, but otherwise no differences in times, insertion depths, or number of exams performed. Compared to a randomly surveyed group of physicians, those with CME were significantly more likely to perform flexible sigmoidoscopy. 

Physicians who participate in couflexible sigmoidohave a higher proof office utilizatiothese skills than who do not take Minimal differencbetween large angroup CME form
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-278 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Bunting, 2004123

Int: Live, Print CC: NA 

Int: Feedback, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician laboratory test ordering behavior 

Practice behavior 

Yes Though both intervention 

and control group test utilization rate declined soon after the intervention, both also showed trends back towards baseline values with increasing time post intervention. However, the intervention group declined to a greater degree than the control group (which had nonsignificant reductions compared to baseline), and a 7.9% (p<0.0001) relative reduction resulted from the intervention. 

"We found that a multifaceted eduand feedback strsignificantly and persistently decrlaboratory utilizaamong practicing community physIn conclusion, thshowed a statistisignificant effect simple feedback education intervethe ordering of latests by high-volcommunity phys
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-279 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Clark, 200078

Int: Live, Video CC: NA 

Int: Case-based learning, Demonstration, Lecture CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician behavior change in teaching patients and increase in communication skills 

Practice behavior 

Yes Intervention physicians 

were more likely to use protocols for delivering asthma education (OR 4.9, p=.2), write down for patients how to adjust medicines when symptoms change (OR 5.8, p=.05), and provide more guidelines for modifying therapy (OR 3.8, p=0.06). Increase in communication skills in the intervention group: more wrote down dose adjustment and timing info for patients; provided guidelines for patients to use when conditions change. No difference in the amount of time physicians spent with patients between groups and no difference in the proportion of physicians prescribing antiinflammatories. 

Participating phyreported that thecommunicated ataught patients in sophisticated waParents of intervpatients reported physicians used of communicatioeducation strategenhance patient and satisfaction. Intervention patieshowed a decreahospitalization. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-280 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Gerstein, 199953

Int: Live CC: NA 

Int: Case-based learning, Discussion group CC: NA 

Int: One time CC: NA 

Knowledge, attitudes, and practice behavior regarding diabetes care, based on participant questionnaire 

Knowledge Attitudes Practice behavior 

Yes Yes Yes 

After 40 days, participants' overall scores improved significantly while there was no change in the controls' scores (F =24.14; p<0.0001). Significant improvement was also noted in domains of attitude (F=31.75; p<0.0001), knowledge (F=4.23; P =0.041), and practice behavior (F=10.43; p=0.0014). However, improvement was not apparent after a year. Participants who completed 425-day assessment scored lower in attitude subscale compared to controls, despite having initially scored higher at the 40-day assessment. 

An interactive, smgroup, diabetes continuing educaprogram effectivedisseminates praguidelines to famphysicians. The such a program dafter 1 year. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-281 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Ockene, 1996124

CC: NA Int: Live, Video Int: Live, Video 

CC: NA Int: Feedback, Lecture, Role play, Standardized patient Int: Feedback, Lecture, Role play, Standardized patient 

CC: NA Int: 2 types of training sessions, each provided once Int: 2 types of training sessions, each provided once 

Mean number of physician nutrition counseling steps 

Practice behavior 

Yes Physicians who received 

both the nutrition counseling training and the practice management system performed, on average, more counseling steps (adjusted mean = 6.28 out of a possible 10 steps, p<0.0001). Those who received no intervention or who received nutrition counseling training only performed fewer steps. 

Primary care intewhen provided wtraining in counstechniques and a supportive office environment, wilout patient counsappropriately. Talone, however, sufficient and macounterproductiv

Adams, 1998125

CC: Live Int: Live 

CC: Lecture Int: Lecture, Mentor/Preceptor, Office system: cuing 

CC: NR Int: Multiple time or repetitive 

Impact of educational intervention on patient exit interview rating of physician counseling behaviors 

Practice behavior 

Yes There was a statistically 

significant increase in physician counseling ratings by patients in the intervention group. The mean number of patient exit interview steps completed (total possible 15) was 2.4 in Usual Care group vs. 10.3 in Special Intervention group (p = .0001) In addition, this effect remained constant for the full 32 months that patients exit interviews continued to be monitored. 

The educational intervention did ipatient-rated phyperformance of counseling on alcuse. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-282 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Myers, 2004148

Int: Live, Audio, Print CC: NA 

Int: Academic detailing, Feedback, Lecture, Readings CC: Readings 

Int: Multiple time or repetitive CC: NA 

Complete diagnostic exam (CDE) performance rates 

Practice behavior 

Yes CDE performance rates 

for eligible patients increased for both intervention and control group, but only to a statistically significant degree among the intervention group (adjusted post/pre OR = 1.71, p<0.03). Intervention to control group OR of CDE recommendation was 1.63, p=0.03. 

"Use of a physicioriented intervensubstantially and significantly increCDE recommendand performance Intervention Groupractices as comControl Group prThese findings demonstrate that targeting PCPs fdelivery of a comCDE reminder feand educational intervention can meaningful impaphysician behavipatient followup colorectal cancescreening. It is nthat the magnitudintervention impagreater for CDE recommendation CDE performancThe intervention a statistically sigimpact on CDE recommendation when the combintimeframe (Perio3) was compared Period 1." 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-283 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Myers, 2004148

Int: Live, Audio, Print CC: NA 

Int: Academic detailing, Feedback, Lecture, Readings CC: Readings 

Int: Multiple time or repetitive CC: NA 

Complete diagnostic exam (CDE) recommendation rates for eligible patients 

Practice behavior 

Yes CDE recommendation 

rates for eligible patients increased for patients whose physicians received the intervention (adjusted post/pre OR = 2.11, p <0.001), but not for those in the control group. Intervention to control group OR of CDE recommendation was 2.28, p=0.002. 

"Use of a physicioriented intervensubstantially and significantly increCDE recommendand performance Intervention Groupractices as comControl Group prThese findings demonstrate that targeting PCPs fdelivery of a comCDE reminder feand educational intervention can meaningful impaphysician behavipatient followup colorectal cancescreening. It is nthat the magnitudintervention impagreater for CDE recommendation CDE performancThe intervention a statistically sigimpact on CDE recommendation when the combintimeframe (Perio3) was compared Period 1.” 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-284 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Practice Behavior Objectives Not Met, Evaluation Duration Not Reported

Mukohara, 200513

Int: Computerbased off-line CC: Computerbased off-line 

Int: Readings CC: Readings 

Int: Multiple time or repetitive CC: Multiple time or repetitive 

Proportion of patients for whom physicians incorporated or read published evidence for medical decision making 

Practice behavior 

No There were no 

significant differences within or between groups in the proportion of patients for whom they incorporated or read published evidence for medical decision making. 

While doctors appreciated thessummaries, whicimproved their reefficiency, the intervention had impact on their uresearch evidencpractice. 

Schectman, 1991139

Int: Print CC: NA CC: NA 

Int: Readings,: Directed physicians to use ATPs CC: NA CC: NA 

Int: NR CC: NA CC: NA 

Prescribing patterns of antihistamines 

Practice behavior 

No There were no 

significant differences between the three groups in the rate of prescribing more expensive antihistamines compared to the preferred antihistamines in the study. 

The intervention met favorably by physicians, but thwas no impact of intervention on antihistamine prepatterns 

Maxwell, 198457

Int: Live CC: NA 

Int: Case-based learning, Discussion group CC: NA 

Int: Multiple time or repetitive CC: NA 

Practice behavior on key topics, as measured by chart review and qualitative interviews 

Practice behavior 

No Unsuccessful data 

collection due to insufficient numbers. Anecdotal evidence of behavior change from stories from 12 committee members. 

Medical care evameetings have educational value 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-285 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Practice Behavior Objectives Not Met, Evaluation Duration Less Than or Equal to 30 DaysZuckerman, 200418

Int: Print Int: Print CC: NA 

Int: Readings Int: Readings CC: NA 

Int: One time Int: One time CC: NA 

Patient factors: percent filling prescription for betablocker within 7-30 days. 

Practice behavior 

No There was an increase in 

post-AMI hospitalization beta-blocker prescribing after the intervention, though this increase was not statistically significant (p=0.13 at 30 days, p=0.12 at 7 days). 

"The systemwide physician educatprogram in PennMedicaid program increased beta-bprescribing after hospitalization by increasing physicawareness of the guidelines for treof AMI survivors. educational interalso improved pacompliance with blocker therapy. effects are likely to AMI patients wbeyond the study population. Besidclinical effects, thintervention progled to cost savingPennsylvania Meprogram, as well avoidance of a fedeaths." 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-286 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Levinson, 1993138

Int: Live CC: NA 

Int: Case-based learning, Lecture CC: NA 

Int: One time CC: NA 

Change scores in 34 categories of physicianpatient communication 

Practice behavior 

No There was no evidence 

of an effect from the short program training in any of the RIAS content analyses, looking at differences in posttest scores between groups, controlling for pretest scores as a covariate. 

The study concluthe long program in significant chaphysicians and pcommunication, at pre/post componly (no control). authors wrote, Tdemonstrates sopotentially imporchanges in physand patients' communication a2.5 day CME proThe changes demonstrated in content and affechave important inon both biologic and physician ansatisfaction. Howthe study that waabstracted (i.e. thwith a control groimpact of the shointervention was 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-287 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Practice Behavior Objectives Not Met, Evaluation Duration Greater Than 30 DaysMehler, 200599

Int: Internet, not real time, Print Int: Live, Print CC: NA 

Int: Academic detailing, Readings Int: Academic detailing, Lecture CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Proportion of patients receiving lipid testing 

Practice behavior 

No A favorable trend was 

observed for a larger increase in frequency of lipid testing at intervention sites (+23% combination of electronic and direct detailing) compared with control sites (+11%), although the difference was not statistically significant (p = 0.06). 

A simple educatiintervention seempositively influenprovider behavioarea of lipid manin diabetes mellitelectronic and didetailing seem to viable approacheFuture studies to determine optimaeducational comthat facilitate appprovider actions or intensify lipid tseem warranted the burgeoning population of diapatients at risk focoronary heart dmorbidity and mo

Maclure, 199896

CC: NA Int: Live, Video CC: NA Int: Live CC: NA Int: Print 

CC: NA Int: Lecture, Teleconference CC: NA Int: Discussion group, Lecture CC: NA Int: Readings 

CC: NA Int: One time CC: NA Int: NR CC: NA Int: Multiple time or repetitive 

Impact of teleconference on prescribing pattern 

Practice behavior 

No There was a trend toward 

increased use of calcium channel blockers, but it was not statistically significant. Participants in the teleconference showed a shift from ACEI to thiazides, compared with matched controls. 

There was a gentrend toward the effect by the eduinterventions. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-288 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Maclure, 199896

CC: NA Int: Live, Video CC: NA Int: Live CC: NA Int: Print 

CC: NA Int: Lecture, Teleconference CC: NA Int: Discussion group, Lecture CC: NA Int: Readings 

CC: NA Int: One time CC: NA Int: NR CC: NA Int: Multiple time or repetitive 

Impact of newsletters on prescribing pattern 

Practice behavior 

No There was a trend to 

decreased prescribing of calcium channel blockers, (shift from preference for first-line CCBs to first-line thiazides) but it was not statistically significant 

There was a gentrend toward the effect by the eduinterventions. 

Maclure, 199896

CC: NA Int: Live, Video CC: NA Int: Live CC: NA Int: Print 

CC: NA Int: Lecture, Teleconference CC: NA Int: Discussion group, Lecture CC: NA Int: Readings 

CC: NA Int: One time CC: NA Int: NR CC: NA Int: Multiple time or repetitive 

Impact of small group on prescribing pattern 

Practice behavior 

No There was a trend to 

decreased prescribing of calcium channel blockers, (a shift from preference for first-line CCBs to first-line thiazides) but it was not statistically significant 

There was a gentrend toward the effect by the eduinterventions. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-289 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Lewis, 199370

Int: Live, Video Int: Print CC: NA 

Int: Discussion group, Lecture Int: Readings CC: NA 

Int: One time Int: One time CC: NA 

Percentages of new patients reporting they were asked sexual history questions by their physicians (family physicians) 

Practice behavior 

No The program had no 

apparent impact on family physicians. 

These results suthat a medical edprogram that goebeyond standard and incorporates interactive formachange physician behaviors. 

Lewis, 199370

Int: Live, Video Int: Print CC: NA 

Int: Discussion group, Lecture Int: Readings CC: NA 

Int: One time Int: One time CC: NA 

Percentages of new patients reporting they were asked sexual history questions by their physicians (obstetriccians/gynecologists) 

Practice behavior 

No Practitioners of obstetrics 

and gynecology at all sites demonstrated a relatively high frequency of asking sexual history questions before the program, but these rates did not change. 

These results suthat a medical edprogram that goebeyond standard and incorporates interactive formachange physician behaviors. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-290 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Hagen, 2005126

Int: Live, Print, laminated copies to be posted at stations and in charts CC: NA 

Int: Lecture, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Dose of benzodiazepines 

Practice behavior 

No The total doses of 

benzodiazepines remained largely unchanged over time in control and experimental facilities, although the control group experienced two decreases postintervention at Times 4 and 6. R-ANOVA conducted for 4 months postintervention (Time 3 vs. Time 5) found no significant effects for either time (F (1.734)=2.08, p=0.15) or time by facility (F (1.734)=3.00, p=0.083). R-ANOVA conducted for 6 months postintervention (Time 3 vs. Time 6) revealed significant effects for time by facility (F (1.699)=4.77, p=0.029) but not for time alone (F (1.699)=0.18, p=0.67). 

Our finding that tintervention did nreduce the perceresidents on neuor benzodiazepinthe dosages administered, is inconsistent with findings of Avorn (1992), who repoprovider educatioreduced the use psychotropic drunursing homes. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-291 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Hagen, 2005126

Int: Live, Print, laminated copies to be posted at stations and in charts CC: NA 

Int: Lecture, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Effect of educational intervention upon use of psychotropics 

Practice behavior 

No The primary hypothesis, 

that the intervention facilities would show less postintervention use of psychotropics than the control facilities, was not supported. Rather, both control and experimental LTC (long-term care) facilities experienced a small rise in the percentage of residents receiving neuroleptics after the education intervention between Time 3 and Time 4, although the increases were only significant in the control group. 

Our finding that tintervention did nreduce the perceresidents on neuor benzodiazepinthe dosages administered, is inconsistent with findings of Avorn (1992), who repoprovider educatioreduced the use psychotropic drunursing homes. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-292 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Davis, 2004101

Int: Live, Video, Audio, audio teleconferencing CC: NA 

Int: Case-based learning, Discussion group, Problembased learning or team-based learning CC: NA 

Int: Multiple time or repetitive CC: NA 

Change in prescribing patterns with respect to use of SABAs 

Practice behavior 

No Neither the study nor the 

control group demonstrated a statistically significant change in the use of SABAs before and after the intervention. (Study group prescriptions decreased between 6 months before and 6 months after, while control group prescriptions of SABAs increased. No level of statistical significance was given for these changes.) 

"We report the firstyle teleconfereseries to demonssignificant changphysician prescrhabits toward beconcordance witasthma guidelinestudy demonstrapowerful impact PBL format has ophysician learninactually translateaction—a measuchange in beliefs behavior that led changes in the nprescriptions writthe participants. standard, traditiodidactic lectures the norm for CMthe US and credgiven for attendamany studies hashown that PBL actually more effenhancing physiclearning. CME ofshould plan more group workshops teleconferences integrating the Pformat, with skillefacilitators trainedevelop the casemoderate the sesultimately to impquality of CME." 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-293 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Solomon, 2004127

Int: Live, Print CC: NA 

Int: Discussion group, Lecture, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Osteoporosis management: proportion who underwent bone densitometry

Practice behavior 

No After the intervention, the 

proportion of patients undergoing bone densitometry was not statistically different between the groups (p=0.9). 

"In conclusion, wconducted a randcontrolled trial of multifaceted intefor GIOP. This intervention was associated with aincrease in use oosteoporosis meor bone densitomover a 6-month postintervention observation perioBased on severapotential explanathese findings, wconsidering futurinterventions thadirect-to-patient educational mailwell as the use orheumatology nurecognize at-risk and initiate a diaworkup." 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-294 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Solomon, 2004127

Int: Live, Print CC: NA 

Int: Discussion group, Lecture, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Osteoporosis management: proportion prescribed a medication for osteoporosis

Practice behavior 

No Although the proportion 

of pts who were prescribed a medication for osteoporosis did increase (between 6 months before and 6 months after the intervention) in both the intervention and control groups, neither of them increased in a statistically significant fashion, nor were they statistically significantly different from one another (p=0.3). 

"In conclusion, wconducted a randcontrolled trial of multifaceted intefor GIOP. This intervention was associated with aincrease in use oosteoporosis meor bone densitomover a 6-month postintervention observation perioBased on severapotential explanathese findings, wconsidering futurinterventions thadirect-to-patient educational mailwell as the use orheumatology nurecognize at-risk and initiate a diaworkup." 

Pazirandeh, 2002128

Int: Live, Print CC: NA 

Int: Lecture, Readings, Q & A period CC: NA 

Int: One timeCC: NA 

Physician initiation of osteoporosis discussion and screening orders 

Practice behavior 

No The discussion of 

osteoporosis and orders for bone mineral density tests increased for both control and intervention group. However, this is likely a result of the patient education intervention, not the physician intervention which did not show evidence of affecting practice patterns. 

"In summary, in tnonrandomized controlled study, was some evidensupporting the gerecognized beliedidactic lectures effective in modifphysician practicpatterns." 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-295 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Thom, 2000129

Int: Live, Video CC: NA 

Int: Demonstration, Discussion group, Lecture, Role play CC: NA 

Int: One timeCC: NA 

Modification of physician behaviors that affect patient trust 

Practice behavior 

No Five behaviors correlated 

with patient trust: "letting you tell the story"; listening carefully; never interrupting; answering questions clearly showed the least net difference between groups, but are very important to patients. 

This study did noany effect from a physician training intervention on pphysician trust, por patient satisfautilization, continadherence. 

Schectman, 199616

Int: Print, Samples of first generation antihistamines to use with patients CC: NA 

Int: Readings CC: NA 

Int: One timeCC: NA 

Antihistamine prescribing costs per member per month, as measured from health plan database. 

Practice behavior 

No Education and providing 

drug samples had no substantial effect on prescribing behavior between intervention and control sites. Both groups decreased costs 2% in the study period. 

Education and psamples did not decreased antihiprescribing costseffective intervenwould require tarmajor patient-relabarriers (e.g. patexpectations) to behavior change

Bjornson, 199014

Int: Print CC: NA 

Int: Readings CC: NA 

Int: One time CC: NA 

Change in physician prescribing patterns 

Practice behavior 

No There was no statistical 

difference between the number of physicians in the intervention and control groups who made full changes (switching patients to both hydralazine and isosorbide), but the number who made full and partial changes approached significance (p=0.07). 

The intervention successful in mothe prescribing pof physicians witto CHF patients. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-296 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Stewart, 200543

Int: Internet, not real time, CC: NA 

Int: Case-based learning, Discussion group, Reading CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician behaviors 

Practice behavior 

No Scores on the 

standardized patient checklist were not significantly different in the intervention group than in the control group.

The case-based discussion demoa mixed effect, wsignificant differeonly one of two cand for only two three outcomes (physicians’ knowand quality of praThe study identifpromising contineducation format based, on-line leas well as questifuture research rthe content and ocases presented line education. 

Lin, 199773

CC: NA Int: Live, Video, Print 

CC: NA Int: Academic detailing, Feedback, Lecture, Readings, Role play 

CC: NA Int: Multiple time or repetitive 

Change in depression practice behaviors as a result of the intervention 

Practice behavior 

No There was no difference 

between the two groups in number of follow-up visits for depression, or in the number of educational messages received by patients. 

The results do nosupport the concthis complex and aggressive interveffected a sustaiin practice behavmonths after the intervention. In fsome positive chwere noted immeafter the intervenprescribing pattewere lost 6 montthe intervention w
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-297 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Schectman, 1995130

Int: Print Int: Print Int: Print CC: Print CC: Print CC: Print 

Int: Feedback, Readings Int: Feedback, Readings Int: Feedback, Readings CC: Readings CC: Readings CC: Readings 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive CC: Multiple time or repetitive CC: Multiple time or repetitive CC: Multiple time or repetitive 

Change in prescribing patterns of H2 blockers 

Practice behavior 

No Education: memo alone 

v. memo plus feedback; no difference across groups. Group model physicians did respond to the intervention and network physicians did not. 

A simple educatiintervention to chphysician prescrpatterns can be meffective. In this sonly group-modephysicians changbehavior and indfeedback did not higher percentagphysicians respothe intervention athe total study po

Casebeer, 1999131

Int: Audio, Telephone conferences CC: NA 

Int: Case-based learning, Discussion group, Point of care CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician strategies in enhancing adherence to hypercholesterolemic treatment, as measured by standardized patients preand 3 months postintervention 

Practice behavior 

No No significant differences 

between groups in use of seven communication strategies. Intervention group had significant preto post- improvements in facilitating patients' understanding of hypercholesterolemia (p=0.009) and interpersonal interaction (p=0.28). 

Combining a serinteractive case aconferences with reminders shows in increasing phyadherence-enhastrategies. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-298 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Pinto, 199874

CC: Prin,t Int: Live, Print 

CC: Readings, Int: Discussion group, Readings, Role play 

CC: Amount of exposure up to the physicians Int: One time 

Report of actual counseling activities by physicians 

Practice behavior 

No The intervention group 

had a slight increase in actual self-reported counseling by physicians, but it was not statistically significant over the control group. 

The program impphysician confidecounseling and psatisfaction, but dincrease the phyreports of exerciscounseling providpatients. 

Gullion, 1988132

Int: Print, Telephone conference Int: Print, Telephone conference Int: Print, Telephone conference CC: NA 

Int: Discussion group, Feedback, Readings Int: Discussion group, Feedback, Readings Int: Discussion group, Feedback, Readings CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Physician behavior regarding behavioral management of hypertension, as assessed by chart review 

Practice behavior 

No Using adjusted p-value 

(p<0.0025) for multiple comparisons, there were no significant differences. Patients of physicians who received behavioral education had higher weight scores (p<0.005) and were slightly more likely to be heavy drinkers (p<0.09) compared with physicians who did not receive behavioral education. They were also more likely to have received advice about how to take medications (p<0.04), about side effects (p<0.006), and about sodium intake (p<0.04). Patients who were most overweight lost weight if physicians received behavioral education and gained weight if not; slightly overweight patients gained weight regardless of MD training. 

An education procombining an individualized feereport of performpeer-reviewed syand an educationsession in the fotelephone conferhad no significanon patients' hype
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-299 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Gullion, 1988132

Int: Print, Telephone conference Int: Print, Telephone conference Int: Print, Telephone conference CC: NA 

Int: Discussion group, Feedback, Readings Int: Discussion group, Feedback, Readings Int: Discussion group, Feedback, Readings CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Physician behavior regarding medication management for hypertension, as assessed by chart review 

Practice behavior 

No Using adjusted p-value 

(p<0.0056) for multiple comparisons, there were no significant differences. Physicians receiving medication education were more likely to take correct actions with abnormal labs compared with no medication education groups (p<0.03), but no other significant differences on other 8 measures. 

An education procombining an individualized feereport of performpeer-reviewed syand an educationsession in the fotelephone conferhad no significanon patients' hype
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-300 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Lin, 2001133

Int: Live CC: Live NA 

Int: Academic detailing, Casebased learning, Demonstration, Discussion group, Role playCC: NA 

Int: Multiple time or repetitive CC: Multiple time or repetitive 

Characteristics related to physician practices regarding diagnosis and treatment of depression 

Practice behavior 

No While both groups slightly 

increased in both their rate of diagnosing depression and the rate of new Rx's of antidepressants for new diagnoses, the intervention group decreased slightly while the usual care group increased slightly in their rate of antidepressant prescriptions. However, pre-to-post changes were not statistically significantly different between intervention and usual care group when it came to any of these 3 outcome measures (rate of diagnosing depression, Rx rate of antidepressant meds, or rate of new Rx of antidepressants for new diagnoses of depression). 

"After education optimal managemdepression, interphysicians did nofrom their usual ccolleagues in dediagnosis or pharmacotherap

Socolar, 1998134

CC: NA Int: Print 

CC: NA Int: Feedback, Readings 

CC: NA Int: Multiple time or repetitive 

Impact of the intervention on history and physical exam scores from the chart audit 

Practice behavior 

No There was no difference 

in charting of the history and physical exam between the groups. Also, there was no difference in the pre and post knowledge tests between the groups. 

The intervention to have no effectHowever, having credit in child abassociated with idocumentation wmay indicate thachart audit technproviding feedbaeducation may noptimal method oproviding educat
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-301 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Socolar, 1998134

CC: NA Int: Print 

CC: NA Int: Feedback, Readings 

CC: NA Int: Multiple time or repetitive 

Variables that did correlate with improved documentation 

Practice behavior 

No There was an overall 

improvement in documentation by both groups during the study, so correlates of this improvement were sought as the intervention itself seemed to have no impact. The statistically significant associations were having obtained CME credit in child abuse evaluation, a structured medical record, and female gender. 

The intervention to have no effectHowever, having credit in child abassociated with idocumentation wmay indicate thachart audit technproviding feedbaeducation may noptimal method oproviding educat

Brown, 2004135

Int: Live, Video, Print CC: NA 

Int: Case-based learning, Demonstration, Lecture, Problem-based learning or teambased learning, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

see comments regarding parents and patients 

Practice behavior 

No Treatment group parents 

more often reported that their children were prescribed antiinflammatory therapy from physicians compared to parents in the intervention group, though these differences were not statistically significant. However, other measures of clinical outcomes, such as urgent health care services were compared, but level of statistical significance was only given within groups (from baseline to follow-up), and not for comparing between intervention and control groups. 

"The physician’s interactive seminbeen shown to easthma care and outcomes. The imthe program is noreserved merely patients with moresources. The gdecline in emergdepartment use wchildren from lowfamilies." 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-302 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

McClellan, 2003119

Int: Video, Print CC: NA 

Int: Feedback, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Eye exams ordering 

Practice behavior 

No not statistically significant "The main result 

study was that a population-based improvement intebased on applyinHCQIP model to ambulatory care associated with iprocesses of cardiabetes mellitus

McClellan, 2003119

Int: Video, Print CC: NA 

Int: Feedback, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Quantitative urine ordering 

Practice behavior 

No not statistically significant "The main result 

study was that a population-based improvement intebased on applyinHCQIP model to ambulatory care associated with iprocesses of cardiabetes mellitus

Browner, 1994122

Int: Live, Print Int: Live, Print CC: NA 

Int: Lecture, Readings Int: Academic detailing, Casebased learning, Lecture, Point of care, Readings CC: NA 

Int: One time Int: Multiple time or repetitive CC: NA 

Proportion of patients screened for high cholesterol 

Practice behavior 

No No differences between 

control and intervention groups, p>.25. 

CME was not paeffective in improcompliance with guidelines amonuniversity affiliatecommunity physwho practice primcare. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-303 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Browner, 1994122

Int: Live, Print Int: Live, Print CC: NA 

Int: Lecture, Readings Int: Academic detailing, Casebased learning, Lecture, Point of care, Readings CC: NA 

Int: One time Int: Multiple time or repetitive CC: NA 

Proportion of screened patients who were managed in compliance with NCEP recommendations 

Practice behavior 

No No difference between 

groups, p>.25. 

CME was not paeffective in improcompliance with guidelines amonuniversity affiliatecommunity physwho practice primcare. 

Sibley, 1982136

Int: Audio, Print CC: NA 

Int: Case-based learning, Lecture, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Quality of care processes on common medical conditions, as measured by chart audit 

Practice behavior 

No No significant difference 

in the improvement of proportion of physicians providing adequate or superior care comparing intervention (5% improvement) vs. control (2%). Preplanned analysis revealed equivalent improvements in high-preference topics (6% both groups), but statistically significantly higher improvement for low-preference topics in study (10% improvement) vs. control (1% worsening)[p=0.01]. 

"Despite statisticsignificant gains knowledge of hoevaluate and mavariety of indicatoconditions, the stphysicians in this neither clinically important nor stasignificant improvin the documentequality of care ... continuing educaabout a high-precondition was as getting it; it workeit was not particuwanted; and its edid not extend bethe topics covere
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-304 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Evans CE, 198656

Int: Print CC: NA 

Int: Readings, Chart cue materials offered, but not necessarily implemented CC: NA 

Int: Multiple time or repetitive CC: NA 

Proportion of patients prescribed antihypertensive medications by a study physician, as assessed by patient interview and chart audit 

Practice behavior 

No Proportion rose from 57 

to 76% in the study group and from 54 to 79% in the control group (p<0.01 overall), but difference between groups not significant. 

"Our study demono influence of a continuing mediceducation prograpractices of physon the control of pressure of hypepatients referred community survethese physicians program was beg

Kim, 1999137

Int: Live, Print CC: Print 

Int: Academic detailing, Feedback, Readings CC: Readings 

Int: Multiple time or repetitive CC: Multiple time or repetitive 

Preventative services, as observed in chart review 

Practice behavior 

No No significant increases 

in offered procedures and actual declines in mammography and clinical breast exam in both groups; discrepancies noted between patient selfreport and chart review 

A physician-targeapproach of educpeer-comparison feedback, and acdetailing has moeffects on patiensatisfaction and pon the offering of selected preventcare services. Thagreement betwepatient reports amedical records raises concerns current methods ascertaining comwith guidelines fopreventative care
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-305 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Practice Behavior Objectives With Mixed Results, Evaluation Duration Greater Than 30 DaysKronick, 200371

Int: Live CC: NA 

Int: Case-based learning, Demonstration CC: NA 

Int: One timeCC: NA 

Physician comfort level with accessing medical information via internet. 

Attitudes Practice behavior 

Yes Mixed

Intervention group physicians increased their frequency of and comfort with accessing Internet medical information. Statistically significant differences in change from baseline between intervention and control group were seen with frequency of use of www to address patient related questions (p=.009), in the comfort level using online databases (p=.032) and in the frequency of accessing online databases (p=.044). Non-statistically significant differences were seen in frequency of accessing email to answer pt-related questions and comfort in using email, the internet, opinion of the value of the internet and in accessing online full-text journals. 

"Rural physicianscomfort and comin use of computaddress patient pcan be improved individualized 3-htraining session. data suggest thaphysicians distanmedical libraries have excellent acevidence-based resources; as coto the Internet befaster, more uniforeliable across communities, the will become easideliver." 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-306 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Fordis, 200542

Int: Live, Print, Risk calculator Int: Internet, real time (e.g., streaming), Internet, not real time, Print, Risk calculator CC: NA 

Int: Case-based learning, Lecture, Programmed learning, Readings Int: Case-based learning, Lecture, Programmed learning, Readings CC: NA 

Int: One time Int: Multiple time or repetitive CC: NA 

Appropriate screening and management of high-risk patients, as measured by chart review 

Practice behavior 

Mixed For appropriate 

screening for lipid abnormalities, there were high baseline screening rates >=93% with no significant postintervention change. The online CME group did not differ significantly from the live CME and control groups (P = .24). The live CME group did not differ from the control group (P=.16). Regarding drug treatment for patients at high risk, there was a statistically significant though relatively small increase (5.0% (95% CI, 1.0%-9.1%); pre 85.3% -> 90.3% post) in the percentage of patients appropriately treated by the online CME group when compared with the live CME and control groups (partial w2=0.16, P=.04). The live CME and control groups did not differ significantly in treatment of patients. 

Appropriately deevidence-based CME can producobjectively measchanges in behawell as sustained knowledge that acomparable or suthose realized froeffective live acti
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-307 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Beaulieu, 2002140

Int: Live CC: NA 

Int: Case-based learning, Problem-based learning or teambased learning, Readings, Role play CC: 

Int: One time CC: NA 

Guidelineappropriate screening in the periodic health exam, as measured by number of screening items ordered. 

Practice behavior 

Mixed Exposure to the 

workshop had no impact on the total number of items recommended for inclusion in periodic health exam, but did impact the total number of items recommended for exclusion as well as the secondary outcome of total number of other tests ordered. 

"Results suggest short interactive workshop can dethe ordering of unnecessary scrtests for adults wconsult for check

Tziraki, 2000141

Int: Live, Print Int: Print CC: NA 

Int: Discussion group, Lecture, Problem-based learning or teambased learning, Role play Int: Readings CC: NA 

Int: One time Int: One time CC: NA 

Adherence to NCI manual's recommendations for nutrition counseling, as measured by physician / office staff questionnaire and blinded observation of practices and charts 

Practice behavior 

Mixed Training group had 

significantly higher adherence (28.5%) to office organization recommendations (3.8% more than manualonly[p=0.005] and 5.5% more than control[p<0.001]). Training group had significantly higher adherence to nutrition screening (2.5% more than manualonly[p=0.046] and 3% more than control[p=0.012]). But no differences in advice/referral (52.3-57% adherence) or patient follow-up (13.6-14.6% adherence). 

"The impact of thtraining was modand not statisticasignificant for nuadvice/referral or follow-up, which important in achilong-term dietary in patients." 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-308 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Gifford, 1999142

HC: NA CC: NA Int: Live, Print 

HC: NA CC: NA Int: Case-based learning, Discussion group, Lecture, Readings, mailings 

HC: NA CC: NA Int: One time 

Neurologists' adherence to guidelines, as measured by participant survey using clinical scenarios 

Practice behavior 

Mixed Compared with the 

combined baseline/ control group, the intervention group was significantly more likely to adhere to 3 of 6 practice recommendations: use of neuroimaging (adjusted OR 4.3; 95% CI: 1.9-9.8), referral to the Alzheimer’s Association (2.7; 95% CI: 1.5-4.7), and referral to the Safe Return Program (9.7; 95% CI: 3.1-30.5). For the other recommendations, adherence did not differ between groups. For 2 of these, adherence was high (>80%) in all 3 groups. The intervention group had higher adherence with 1 of 3 recommendations designed to reduce overuse (significant for use of neuroimaging but not for use of EEG and APOE genotype testing) and 2 of 3 recommendations designed to increase use (significant for referral to the Alzheimer’s Association and the Safe Return Program but not for diagnosis and treatment of depression).

A multifaceted educational progimprove physiciaadoption of practguidelines. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-309 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Lin, 199773

CC: NA Int: Live, Video, Print 

CC: NA Int: Academic detailing, Feedback, Lecture, Readings, Role play 

CC: NA Int: Multiple time or repetitive 

Pattern of antidepressant prescriptions

Practice behavior 

Mixed There was an increase in 

imipramine and a decrease in amitryptilline/doxepin use during the intervention (a goal of the study), but this effect was lost 6 months after the intervention. There was no difference in the use of Prozac between the control and intervention. There was also no difference in adequacy of antidepressant medication use at 6 months between the two groups. 

The results do nosupport the concthis complex and aggressive interveffected a sustaiin practice behavmonths after the intervention. In fsome positive chwere noted immeafter the intervenprescribing pattewere lost 6 montthe intervention w

Carney, 199588

Int: Live, Video CC: NA 

Int: Demonstration, Discussion group, Feedback, Lecture, Programmed learning, Role play CC: NA 

Int: One timeCC: NA 

Which CME techniques had the greatest effect on physician cancer screening and prevention counseling skills 

Cognitive skills Practice behavior 

MixedMixed

Performance of intervention physicians was consistently better though only 4 of 19 measured endpoints of the physicians' behavior by the SP reached statistical significance. 

Performance bastechniques have positive influence physicians' perfo(especially in skitraining). Using unannounced standardized patfeasible method assess performawithin the practicenvironment. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-310 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Ray, 198520

CC: NA Int: Live Int: Live 

CC: NA Int: Academic detailing Int: Academic detailing 

CC: NA Int: One time Int: One time 

Prescribing of contraindicated antibiotics and oral cephalosporins, as measured by Medicaid data 

Practice behavior 

Mixed Drug-educator group 

produced modest improvement in prescribing in year 1, but not year 2. Physiciancounselor group had significant improvements in both drug classes in years 1 and 2 by "average prescription change index," but not by proportion of doctors. Improvements in "average patient change index" in physiciancounselor group were attenuated in year 2 for contraindicated antibiotics, but not oral cephalosporins. 

"the beneficial imprescribing of a sphysician-counseof 15 minutes wastrong and lastin"although some attenuation of effseen"; "no lasting beneficial effect ovisit by the drug educator" 

Kim, 1999137

Int: Live, Print CC: Print 

Int: Academic detailing, Feedback, Readings CC: Readings 

Int: Multiple time or repetitive CC: Multiple time or repetitive 

Preventative care services offered, as reported by patient questionnaire 

Practice behavior 

Mixed Proportions of patients 

reporting being offered services increased in both groups for influenza, pneumococcal, and tetanus immunization; in educational only group for mammography and clinical breast exam; in comprehensive group for exercise counseling; and for neither in smoking cessation. 

A physician-targeapproach of educpeer-comparison feedback, and acdetailing has moeffects on patiensatisfaction and pon the offering of selected preventcare services. Thagreement betwepatient reports amedical records raises concerns current methods ascertaining comwith guidelines fopreventative care
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-311 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Unclear if Practice Behavior Objectives Met, Evaluation Duration Greater Than 30 DaysDavis, 2004101

Int: Live, Video, Audio, audio teleconferencing CC: NA 

Int: Case-based learning, Discussion group, Problembased learning or team-based learning CC: NA 

Int: Multiple time or repetitive CC: NA 

Change in prescribing patterns with respect to use of ICSs 

Practice behavior 

Unclear 

Most study group physicians increased their use of ICSs, all more so did the control group. No overall increase in the study group was given to compare to the control group's overall 24% increase in use (6 months before compared to 6 months after intervention), nor was any indication of statistical significance given (for between or with group differences). 

"We report the firstyle teleconfereseries to demonssignificant changphysician prescrhabits toward beconcordance witasthma guidelinestudy demonstrapowerful impact PBL format has ophysician learninactually translateaction—a measuchange in beliefs behavior that led changes in the nprescriptions writthe participants. standard, traditiodidactic lectures the norm for CMcredits are given attendance, manhave now shown PBL is actually meffective at enhaphysicians learnioffices should plasmall-group workteleconferences integrating the Pformat, with skillefacilitators trainedevelop the casemoderate the sesultimately to impquality of CME." 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-312 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Practice Behavior Objectives With No Control Group, Evaluation Duration Not ReportedGoldstein, 2005146

Int: Live, Print Int: Live, Print 

Int: Discussion group, Readings, packet which had patientspecific information form Int: Discussion group, Readings, packet which had patientspecific information form and advisory about guideline concordance of patient's antihypertensive drug regimen 

Int: Not specified Int: Not specified 

Guideline concordance of clinician prescribing 

Practice behavior 

No control group

Concordance with the drug therapy guidelines for hypertension improved in both study groups, with substantially more improvement in the individualized intervention group compared with the general-intervention group. Concordance improved almost 11% with the individualized intervention compared with 4% with the general intervention (t=2.796, P=.008); this absolute increase of 11% represents a 26% relative improvement over baseline nonconcordance in the individualized group versus 7% in the general group. Bootstrap analysis showed that being in the individualizedintervention group increased the odds of concordance 1.5-fold (2-sided P=.03, z=2.23; 95% CI: 0.05, 2.12). 

Individualized recommendationdrug therapy for hypertension preto clinicians at tha patient visit are effective in changprescribing to achigher rates of gadherence. Providing individrecommendationclinicians can be healthcare systeelectronic pharmdiagnostic data, the absence of a complete electrohealth record. Generation and distribution of recommendationdone efficiently if activities are intewith existing proc
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-313 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Rosenthal, 200559

Int: Live, Handheld, Review of data CC: Review of data 

Int: Lecture, Readings, Review of practice's data CC: Review of practice's data 

Int: Multiple time or repetitive CC: One time 

Parent reports of anticipatory guidance 

Practice behavior 

No control group

The adjusted proportion of families of 1-montholds who received 4 of the 5 age-appropriate anticipatory guidance items changed from 15.9% (95% CI: 8.9%-26.7%) to 10.0% (95% CI: 5.1%-18.8%) in the control practices and from 7.3% (95% CI: 4.1%-12.9%) to 24.0% (95% CI: 14.6%-36.9%) in the intervention practices (difference between two differences, p=.002). The adjusted proportion of families of 6-montholds who received all of the age-appropriate anticipatory guidance changed from 8.2% (95% CI, 3.6%-17.8%) to 5.4% (95% CI, 2.8%-10.2%) in the control practices and from 2.2% (95% CI, 0.8%-5.9%) to 18.1% (95% CI, 10.3%-29.9%) in the intervention practices (difference between two differences, P=.001). 

An office system intervention imprparent reports of of anticipatory gubut did not changknowledge or pabehavior. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-314 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Allison, 2005147 CC: Computer- based off-line 

Int: Computerbased off-line 

CC: Readings Int: Feedback (chlamydia screening rates for the physician's entire office), Readings 

CC: Multiple time or repetitive Int: Multiple time or repetitive 

Screening rates 

Practice behavior 

No control group

The mean screening rates before, during, and after the intervention for the comparison offices were 18.9%, 13.0%, and 12.4%, respectively, and for the intervention offices were 16.2%, 13.3%, and 15.5%, respectively, (p=0.044 for post-intervention differences after adjusting for baseline performance). The difference in postintervention screening rates by study group remained significant when adjusting for both pre-intervention and intra-intervention screening rates using repeated-measures analysis (p=0.009). 

This randomized multicomponent, based CME interfound an attenuadecrease in screrates for offices eto the interventiodemonstrated impractice patterns with low intervenintensity increasepotential importamCME as a learnmethod that is eadisseminate. The appropriateness mCME for other conditions and semust be assesse
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-315 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Practice Behavior Objectives With No Control Group, Evaluation Duration Greater Than 30 DaysLockyer, 200264

Int: Live Int: Live 

Int: Case-based learning, Discussion group, Lecture, Role play Int: Case-based learning, Discussion group, Lecture 

Int: One time Int: One time 

Physician practice behaviors: involvement in patient care 

Practice behavior 

No control group

Both track 1 (introductory course) and track 2 (advanced course) participants improved statistically significantly between pre and posttest scores with a moderate effect size difference (0.7 & 0.4, respectively). Between tracks comparisons showed statistically significant differences between tracks for both precourse and postcourse assessment of involvement. 

Track 1 (introduccourse) physiciaimproved moderawhile track 2 (adcourse) physiciashowed a small onegligible change knowledge, comfinvolvement in pacare for dementia patients. Trackin- assigning physcourses based ocourse ability, intskill - needs furth

Mazmanian, 199824

CC: Live Int: Live 

CC: Lecture Int: Lecture 

CC: One time Int: One time 

Impact of CME content including barriers to practice change on actual selfreported practice change by physicians 

Practice behavior 

No control group

There was no difference between the control and intervention group and their self-reported practice change. 

Including contenbarriers to changCME lecture matnot make physicmore likely to chatheir practice. Hphysicians who rintent to change likely to actually their practice patthan those who d
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-316 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Kutcher, 200265

Int: Live, Print Int: Live, Print 

Int: Academic detailing, Discussion group, Lecture, Readings Int: Discussion group, Lecture, Readings 

Int: One time Int: One time 

Characteristics related to physician action such as use of a screening tool and optimal pharmacotherapy 

Practice behavior 

No control group

Enhanced intervention group was more likely to utilize depression diagnostic tool, optimize drug therapy, and use a checklist for adverse events (p<0.0001). 

"A well-designeddirectional, brief, and low-cost eduprogram can incrfamily physiciansknowledge of deimprove their diaskills, and optimitreatment of dep

Pimlott, 2003143

Int: Print CC: Print 

Int: Feedback, Readings CC: Feedback, Readings 

Int: Multiple time or repetitive CC: Multiple time or repetitive 

Characteristics of physician prescribing (of benzodiazepines to elderly) 

Practice behavior 

No control group

Though a small reduction in the prescribing of benzodiazepines was observed among the intervention group, and the mean change in percent from baseline to the end was significantly different (p=0.036) for the intervention and control groups, the intervention still had no significant impact on the proportion of seniors who received long-term benzodiazepine therapy (alone or in combination).

"We found that aintervention that reasonable succchanging antibioprescribing patteunsuccessful whapplied to benzodiazepine prescribing." 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-317 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

McBride, 2000144

Int: Live, Print Int: Live, Print Int: Live, Print CC: Live, Print 

Int: Academic detailing, Discussion group, Feedback, Lecture, Point of care, Problembased learning or team-based learning Int: Lecture, Point of care, Problem-based learning or teambased learning, Patient education Int: Academic detailing, Discussion group, Feedback, Lecture, Problem-based learning or teambased learning, Patient education CC: Lecture, Point of care 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive CC: One time 

Changes in documentation of screening of cardiovascular risk factors, as measured by medical record audit at baseline, 12 months, and 18 months 

Practice behavior 

No control group

Tripled in combined intervention group, with significance at 18 months. Prevention coordinator group had significant increase at 12 months and slight decrease at 18 months. Consultation group increases did not reach significance. No changes in conference-only group. 

Practices can semake changes in organization for prevention servicincrease risk factscreening and management documentation. Consultation pramore goals, but prevention coordpractices achievegreater increases use of medical retools and documof screening and management. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-318 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

McBride, 2000144

Int: Live, Print Int: Live, Print Int: Live, Print CC: Live, Print 

Int: Academic detailing, Discussion group, Feedback, Lecture, Point of care, Problembased learning or team-based learning Int: Lecture, Point of care, Problem-based learning or teambased learning, Patient education Int: Academic detailing, Discussion group, Feedback, Lecture, Problem-based learning or teambased learning, Patient education CC: Lecture, Point of care 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive CC: One time 

Improved use of cardiovascular risk assessment tools, as measured by medical record audit at baseline, 12 months, and 18 months 

Practice behavior 

No control group

Increases in cardiovascular disease risk documentation were greatest in combined intervention group at 12 months (21-35% in different locations in chart) and maintained at 18 months. Prevention coordinator group had largest increases flowsheet (22%) and second-largest in patient questionnaire, problem list, and chart label (10-22%), with most changes maintained at 18 months. Documentation increased to lesser degree in consultation group. Conference-only group demonstrated small increase in questionnaires, flowhseets, but not in chart labels or problem list. 

Practices can semake changes in organization for prevention servicincrease risk factscreening and management documentation. Consultation pramore goals, but prevention coordpractices achievegreater increases use of medical retools and documof screening and management. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-319 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

McBride, 2000144

Int: Live, Print Int: Live, Print Int: Live, Print CC: Live, Print 

Int: Academic detailing, Discussion group, Feedback, Lecture, Point of care, Problembased learning or team-based learning Int: Lecture, Point of care, Problem-based learning or teambased learning, Patient education Int: Academic detailing, Discussion group, Feedback, Lecture, Problem-based learning or teambased learning, Patient education CC: Lecture, Point of care 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive CC: One time 

Goal setting by practices, as reported by physician and staff questionnaires 

Practice behavior 

No control group

Combined intervention and consultation groups set average of 7 goals, while coordinator group set 5 and conferenceonly group set 3. Conference-only group less likely to set screening tool goal (42% vs. 82-100% in other groups, p<0.05). 

Practices can semake changes in organization for prevention servicincrease risk factscreening and management documentation. Consultation pramore goals, but prevention coordpractices achievegreater increases use of medical retools and documof screening and management. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-320 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

McBride, 2000144

Int: Live, Print Int: Live, Print Int: Live, Print CC: Live, Print 

Int: Academic detailing, Discussion group, Feedback, Lecture, Point of care, Problembased learning or team-based learning Int: Lecture, Point of care, Problem-based learning or teambased learning, Patient education Int: Academic detailing, Discussion group, Feedback, Lecture, Problem-based learning or teambased learning, Patient education CC: Lecture, Point of care 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive CC: One time 

Changes in documentation of management of cardiovascular risk factors, as measured by medical record audit at baseline, 12 months, and 18 months 

Practice behavior 

No control group

Appropriate management documented for 65% of patients at baseline, with significant increase in combined intervention group (72%) at 12 months compared with conference-only group (64%, p<0.05). At 18 months, all intervention groups (88-95%) had increased documentation compared with control (71%), with significant difference in coordinator and consultation groups. 

Practices can semake changes in organization for prevention servicincrease risk factscreening and management documentation. Consultation pramore goals, but prevention coordpractices achievegreater increases use of medical retools and documof screening and management. 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-321 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Cherkin, 199183

Int: Live, Video Int: Live, Video 

Int: Demonstration, Discussion group, Lecture Int: Demonstration, Discussion group, Lecture 

Int: Multiple time or repetitive Int: One time 

Ordering spine radiographs 

Practice behavior 

No control group

Percentage of providers indicating they would order lumbosacral spine radiographs fell from 67% to 43% (not statistically significant). 

Both HMO and ppractice achieved goal. As a result intervention abouthe physicians feconfident in their manage back pa

Heale, 198866

Int: Live Int: Live Int: Live 

Int: Lecture Int: Case-based learning, Discussion group Int: Problembased learning or team-based learning 

Int: One timeInt: One time Int: One time 

Performance by participants with SPs in the office in 7 months 

Practice behavior 

No control group

There were no differences between groups on physician performance in test cases, but there were differences between cases, with physicians performing significantly lower (p<.001) on one case than on the other two cases. The small group problem-based group participants performed better on one item related to one case than other participants (p<.05). 

Within a one day course in family medicine, the leaformat had no efacquired or retaiknowledge or on physician performthree patient proPhysicians rated small group probbased format hig
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-322 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Greenberg, 198525

Int: Live Int: Live 

Int: Lecture Int: Case-based learning 

Int: One time Int: One time 

Documentation of appropriate management of common pediatric problems 

Practice behavior 

No control group 

No statistically significant difference in recording diagnoses (case-based 70% vs. lecture-based 38%) or in accuracy of diagnosis (79 vs. 78%). Significantly more casebased learners recorded a plan than lecture-based learners (80 vs. 39%, p<0.02). Plans were appropriate in 63% casebased vs. 44% lecturebased learners, but no significance test was reported. 

Case-based learcompared with lebased learning, wassociated with ssignificant advanteaching skills anbehavior in manaof common pediaproblems. 

Labelle, 200467

Int: Live, Print Int: Live 

Int: Case-based learning, Demonstration, Discussion group, Lecture, Problem-based learning or teambased learning, Role play, Standardized patient Int: Case-based learning, Role play, Standardized patient 

Int: Multiple time or repetitive Int: Multiple time or repetitive 

Impact on prescribing practice, as assessed by OSCE 

Practice behavior 

No control group

Though the scores increased somewhat for both groups (reflecting increase in WAP use in practice), the change was marginally not significantly different between the intervention groups (p=0.052). 

"This study demoa positive impact combination of a based, interactivworkshop featuripreformatted toodrafting of WAPsreinforcing OSCE months post-woron GP knowledgself-reported use WAPs. These resupport the concreviews of CME programs that intand sequential educational activproviding opportupractice appear promising in chaphysician practic
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-323 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Winickoff, 198423

Int: Print Int: Print 

Int: Feedback Int: Feedback 

Int: Multiple time or repetitive Int: Multiple time or repetitive 

Compliance with colorectal screening, as measured by chart review / health plan records 

Practice behavior 

No control group 

Intervention period 1: both intervention and control groups improved significantly (p<0.001) from rates of 66 and 67.5% to 79.9 and 76.6%, respectively. Intervention period 2: group 1 stabilized at 79.8%, while group 2 continued to improve to 84.1% (p<0.001 for improvement, p<0.025 compared with group 1). Rates sustained at 6 and 12 months postintervention. 

Peer comparison feedback may imcompliance with improvement 
Evidence table 11. Effectiveness of continuing medial education on short-term and long-term practice behavior outcomes 

F-324 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Howe, 1997145

Int: NR Int: NR Int: NR Int: NR 

Int: Feedback, Point of care, Readings Int: Feedback, Lecture, Point of care, Readings, (the major difference between groups was the intensity of the intervention with frequent seminars and more intensive outreach but with greater access to experts in breast cancer) Int: Feedback, Point of care, Readings Int: Feedback, Point of care, Readings 

Int: NR Int: Multiple time or repetitive Int: Multiple time or repetitive Int: NR 

Changes in breast cancer management practices before and after the intervention 

Practice behavior 

No control group 

Statistically significant urban-rural discrepancies were eliminated by both interventions for many practices. 

The intervention some impact on individuals measoverall it did not measurable impathe rural-urban d

AAP = American Academy of Pediatrics; ACEI = angiotensin-converting enzyme inhibitor; AMI = acute myocardial infarction; ANCOVA = analysis allergy trial pack; CC = concurrent control; CCBs = calcium channel blocker; CDE = complete diagnostic exam; CHF = congestive heart failure; CI = 

continuing medical education; COPD = chronic obstructive pulmonary diseases; COX-2 = cyclooxygenase-2; DES = diethylstilbestrol; GIOP = gluco

osteoporosis; HC = historical control; HCQIP = Health Care Quality Improvement Program; IG = intervention group; Int =intervention group; LTC = 

leukotriene antagonists; mCME = multicomponent Internet continuing medical education; NA = not applicable; NCEP = National Cholesterol Educatineighborhood health center; NR = not reported; NSAID = non-steroidal anti-inflammatory drugs; OA = osteoarthritis; OR = odds ratio; OSCE = objec

examination; PBL = problem-based learning; PCP = primary care provider; RCT = randomized controlled trial; RR = relative risk; R-ANOVA = repea

variance; SABA = short acting bronchodilators; SD = standard deviation; SP = standardized patient; URI = upper respiratory infection; WAP = writte
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-325

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Clinical Outcomes Objectives Met, Evaluation Duration Not ReportedKiang*, 200533

Int: NA CC: NA Int: many things were made available but it is not reported as to which groups used what methods, options included live, regional meetings, CDROMs, mailings, grandrounds CC: NA 

Int: NA CC: NA Int: not clear CC: NA 

Int: NA CC: NA Int: not clear CC: not clear 

Perception of prescription and prescription rates of antimicrobial drug use for upper respiratory infections among pediatric patients 

Clinical outcome 

Yes In both states, a decline 

was noted in the perceived parental demand for antimicrobial agents to treat pediatric respiratory illness. The temporal change was significant in Wisconsin (p = 0.004) and approaching significance in Minnesota (p = 0.064). The median reported percentage of parents who requested an antimicrobial agent decreased from 25% in 1999 to 20% in 2002 in both states, but the distribution around the medians differed significantly between the states. 

In conclusion, thsuggests that the campaign had at modest positive the knowledge adecision-making primary care clinWisconsin. 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-326

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Kiang*, 200533

Int: NA CC: NA Int: many things were made available but it is not reported as to which groups used what methods, options included live, regional meetings, CDROMs, mailings, grandrounds CC: NA 

Int: NA CC: NA Int: not clear CC: NA 

Int: NA CC: NA Int: not clear CC: not clear 

Perception of prescription and prescription rates of antimicrobial drug use for upper respiratory infections among adult patients 

Clinical outcome 

Yes In 2002, Wisconsin 

clinicians perceived less demand for antimicrobial agents among adult patients compared with 1999 (p<0.001). Based on clinician estimates, the median percentage of patients who requested an antimicrobial agent for cough, cold, or flu symptoms decreased from 50% in 1999 to 30% in 2002. Minnesota clinicians also perceived a decrease in the percentage of patients who requested antimicrobial agents, but the difference was not significant (p = 0.152); the median percentage of Minnesota patients requesting antimicrobial agents decreased from 40% in 1999 to 30% in 2002. 

In conclusion, thsuggests that the campaign had at modest positive the knowledge adecision-making primary care clinWisconsin. 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-327

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Wells, 2000151

Int: Live, Print CC: NA 

Int: Academic detailing, Discussion group, Feedback, Lecture, Point of care, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Depression, as measured by patient survey (CES-D) 

Clinical outcome 

Yes At 6 months, 39.9% of QI 

patients and 49.9% of controls still met criteria for probable depressive disorder (P = .001), with a similar pattern at 12 months (41.6% vs. 51.2%; P = .005). Intervention patients were less likely to have probable depression at 6-and 12-month follow-up by 7 to 10 percentage points. QI-therapy and QI-meds patients, each compared with controls, were 8 to 10 percentage points less likely to have probable disorder at 6 and 12 months (P = .03). 

When these manprimary care praimplemented QI programs that imopportunities for depression treatmwithout mandatinquality of care, mhealth outcomesretention of empof depressed patimproved over a while medical visnot increase ove
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-328

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Wells, 2000151

Int: Live, Print CC: NA 

Int: Academic detailing, Discussion group, Feedback, Lecture, Point of care, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Health care utilization, as measured by patient survey 

Clinical outcome 

Yes There were no 

differences in probability of having any medical visit at any point (each P.21). At 6 months, 47.5% of QI patients and 36.6% of controls had a medical visit for mental health problems (P = .001), and QI patients were more likely to see a mental health specialist at 6 months (39.8% vs. 27.2%; P<.001) and at 12 months (29.1% vs. 22.7%; P = .03). At each follow-up, QI-meds patients had higher rates than controls of any specialty counseling (by 10-12 percentage points, P = .003). 

When these manprimary care praimplemented QI programs that imopportunities for depression treatmwithout mandatinquality of care, mhealth outcomesretention of empof depressed patimproved over a while medical visnot increase ove
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-329

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Wells, 2000151

Int: Live, Print CC: NA 

Int: Academic detailing, Discussion group, Feedback, Lecture, Point of care, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Appropriate management of depression, as measured by patient survey 

Clinical outcome 

Yes At 6 months, 50.9% of QI 

patients and 39.7% of controls had counseling or used antidepressant medication at an appropriate dosage (P<.001), with a similar pattern at 12 months (59.2% vs. 50.1%; P = .006). At 6 months, rates of appropriate care were greater by 14% for QImeds than controls (P<.001) and by 8% for QI-therapy patients than controls (P = .002). At 12 months, QI-meds patients had higher rates of appropriate care than did controls (P<.001) or QI-therapy patients (P = .02), who did not differ significantly from controls. At each followup, QI-meds patients had higher rates than controls of appropriate medication use (by 10-14 percentage points, P = .001). 

When these manprimary care praimplemented QI programs that imopportunities for depression treatmwithout mandatinquality of care, mhealth outcomesretention of empof depressed patimproved over a while medical visnot increase ove
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-330

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Clinical Outcomes Objectives Met, Evaluation Duration Less Than or Equal to 30 DaysZuckerman, 200418

Int: Print Int: Print CC: NA 

Int: Readings Int: Readings CC: NA 

Int: One time Int: One time CC: NA 

Patient factors: compliance of betablocker users. 

Clinical outcome 

Yes There was an 8.3% 

(p=0.02) increase in compliance among betablocker users from pre to post-intervention. 

"The system widphysician educatprogram in PennMedicaid program increased beta-bprescribing after hospitalization by increasing physicawareness of the guidelines for treof AMI survivors. educational interalso improved pacompliance with blocker therapy. These effects are apply to AMI patbeyond the study population. Besidclinical effects, thintervention progled to cost savingPennsylvania Meprogram, as well avoidance of a fedeaths." 

Clinical Outcomes Objectives Met, Evaluation Duration Greater Than 30 DaysStein, 200195

Int: Live, Handheld by phone CC: NA 

Int: Readings study, physician visit, algorithm CC: NA 

Int: Multiple time or repetitive CC: NA 

Among patients: scores on measures of pain, functioning, and disability before and after intervention. 

Clinical outcome 

Yes Arthritis pain showed a 

similar profile in both intervention and control homes: for about 1/3 of subjects, pain worsened in both groups (p=0.81). 

An educational intervention effecreduced NSAID nursing homes wworsening of arthpain. 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-331

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Stewart, 200543

Int: Internet, not real time CC: NA 

Int: Case-based learning, Discussion group, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Quality of practice 

Clinical outcome 

Yes Participation in the 

intervention was associated with quality of practice on the prevention topic; the mean quality of practice score at 6 months for the intervention group was significantly greater than for the control group. However, there were no differences on the diabetes topic. In a multiple regression analysis assessing the relationship of the intervention on quality of practice at 2 months and 6 months, taking knowledge into account, we found that after controlling for 2-month knowledge (as well as solo/group and baseline knowledge), the quality of practice on the prevention topic was significantly better in the intervention group than in the control group at both 2 months (b=1.27, p=.028) and 6 months (b=1.25, p=.016). 

The case-based discussion demoa mixed effect, wsignificant differeonly one of two cand for only two three outcomes (physicians’ knowand quality of praThe study identifpromising contineducation format based, on-line leas well as questifuture research rthe content and ocases presented line education. 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-332

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Rost, 200122 Int: by phone 

CC: NA 

Int: conference calls CC: NA 

Int: Multiple time or repetitive CC: NA 

Improvement in patient depressive symptoms (for patients starting a new treatment episode) based on increased use of psychotherapy or medication and patient satisfaction with care. 

Clinical outcome 

Yes Patients in the 

intervention practices starting a new treatment episode reported improvements of depressive symptoms, versus patients in the control practices (p=0.04). Patients who claimed that psychopharmacology was acceptable to use showed more improvement overall (p=0.007) and patients in the intervention group practices were more satisfied with their care than those in the control group practices (p=0.02), among those for whom medication was acceptable. However, patients who were already in treatment did not report improvements with the intervention. 

Patients starting treatment episoddepression in the intervention prac(which were traindetermine practicroles) were signimore likely to shimprovements of symptoms (mentphysical) than wepatients in the copractices. Those were already in tshowed no signifimprovements, regardless of gro

Worrall, 1999152

Int: Live CC: Print 

Int: Case-based learning, Discussion group, Lecture CC: Readings 

Int: One time CC: One time 

Number and percent of patients taking medication at 6 month follow-up 

Clinical outcome 

Yes Significant difference with 

more patients in intervention group taking medication at 6 months, p<.05 

"The educationahad a modest beeffect on the outcpatients with depbut there are still concerns regardlow rates of drug treatment and remental health professionals by physicians." 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-333

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Worrall, 1999152

Int: Live CC: Print 

Int: Case-based learning, Discussion group, Lecture CC: Readings 

Int: One time CC: One time 

The gain score, defined as the difference in the scores between the initial threshold score of depression on the Center for Epidemiologic Studies Depression Scale (CESD, a 20-item self-report questionnaire) and the same score obtained 6 months later 

Clinical outcome 

Yes Before treatment, the 

mean CES-D score for patients didn't differ. The mean scores at 6 months were lower overall, but the difference between the intervention and control groups was not statistically significant. However, the mean gain score for patients in the intervention group was significantly higher (p <0.05). 

"The educationahad a modest beeffect on the outcpatients with depbut there are still concerns regardlow rates of drug treatment and remental health professionals by physicians." 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-334

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Worrall, 1999152

Int: Live CC: Print 

Int: Case-based learning, Discussion group, Lecture CC: Readings 

Int: One time CC: One time 

The gain score, which is the difference between the score obtained from the initial subjective diagnosis of depressive symptoms (based on a 4-point ordinal scale where 1 = absence of depressive symptoms and 4= severely depressed) and the same measure taken 6 months later 

Clinical outcome 

Yes A statistically significant 

difference between the intervention and control groups' mean gain scores for physicians' ratings of depression severity was observed (p = 0.02). 

"The educationahad a modest beeffect on the outcpatients with depbut there are still concerns regardlow rates of drug treatment and remental health professionals by physicians." 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-335

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Roter, 199584

Int: Live Int: Live CC: NA 

Int: Discussion group, Lecture, Role play, Standardized patient Int: Discussion group, Lecture, Role play, Standardized patient CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Effect on patients' emotional distress 

Clinical outcome 

Yes There was a reduction in 

emotional distress for all patient groups. However, patients of PD physicians showed significantly greater reduction in distress at 2 weeks, 3 months, and 6 months (p<0.05). Although their scores were lower, the patients of EH physicians did not have significantly greater reductions in emotional distress than the patients of control physicians. 

Physicians' use ocommunication stheir practices chas a result of an CME program. Physicians trainespecific communskills recognized psychological protheir patients thauntrained physicTrained physiciashowed greater cproficiency in the management of simulated patiencompared with cgroup physicianspatients of trainephysicians compuntrained physicshowed greater rin emotional distas long as 6 montheir medical visi

Maiman, 198847

Int: Live Print Int: Print CC: NA 

Int: Discussion group, Lecture, Readings Int: Readings CC: NA 

Int: Multiple time or repetitive Int: One time CC: NA 

Mothers (patient) compliance with antibiotics 

Clinical outcome 

Yes Educational interventions 

were associated with increased adherence as measured by liquid/pill counts in the home, and mothers' report of missed doses. 

CME increased pknowledge and compliance-enhapractices and resimprovement in madherence to the
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-336

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Pinto, 199874

CC: Print Int: Live, Print 

CC: Readings Int: Discussion group, Readings, Role play 

CC: Given materials to readexposure up to the physicians Int: One time 

Patient satisfaction with care 

Clinical outcome 

Yes Patients in the IG were 

statistically more likely to report an increase in their satisfaction with care than controls (t =4.55, df =255, p<.01). 

The program impphysician confidecounseling and psatisfaction, but increase the phyreports of exerciscounseling provipatients. 

Derebery, 2002153

Int: Live, Print CC: Print 

Int: Demonstration, Discussion group, Lecture, Problem-based learning or teambased learning, Readings, Simulation (other than standardized patient or roleplay) CC: Readings 

Int: In addition to workshop, physicians received updated LBP manual to read on their own within 2-3 months for additional CME credit CC: NA 

Characteristics related to physician management of lower back pain including restricted duty rate, duration of restricted work, offduty rate, number of therapy visits, number of doc visits, & case duration. 

Clinical outcome 

Yes There was a significant 

decrease in the outcomes (physician lower back pain management practices) of the study group while the control group experienced no significant changes between pre and posttraining period. 

"All the outcomemeasured decreasignificantly in thgroup while the cgroup experiencesignificant changthe pretraining pthe posttraining pwith the exceptiosignificant increacase duration." 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-337

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Ray, 2001109

Int: Live, Print CC: NA 

Int: Academic detailing, Readings, chart reminders CC: NA 

Int: Multiple time or repetitive CC: NA 

Change between baseline and follow-up years in: SF36 measures of general health, physical function, and bodily pain (from 40% random patient sample) 

Clinical outcome 

Yes No change over time of 

pain, health, or functioning scores for either group. 

The educational modestly reduceexposure in comdwelling elderly pwithout undesirasubstitution of otmedications or dworsening of musculoskeletal symptoms. 

Gonzales, 1999111

Int: Live, Print Int: Print CC: NA 

Int: Academic detailing, Demonstration, Lecture, Point of care Int: Academic detailing, Point of care CC: NA 

Int: One time Int: NA CC: NA 

Incidence of adult office visits (incident visits) for uncomplicated acute bronchitis and changes in return visit rates 

Clinical outcome 

Yes No increase across sites. Antibiotic treatme

adults diagnosed having uncompliacute bronchitis safely reduced ucombination of pand practitioner interventions. 

Wilson, 1988154

CC: NA CC: NR Int: Live 

CC: NA CC: NR Int: NR 

CC: NA CC: MDs were told to offer nicotine gum Int: One time 

Smoking cessation quit rates 

Clinical outcome 

Yes Patients of physicians in 

intervention group had higher rates of smoking cessation (8.8%) at 3 months than patients of non-intervention physicians (4.4% for usual care and 6.1% for gum only). 

Patients of physiintervention grouhigher rates of smcessation (8.8%) months than patinon-intervention physicians (4.4% usual care and 6gum only). 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-338

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Lindsay-McIntyre, 1987115

Int: Print Int: Live, Video, Print CC: NA 

Int: Point of care Int: Demonstration, Lecture, Point of care, Standardized patient CC: NA 

Int: Multiple time or repetitive Int: Training 1 time, cueing with each patient CC: NA 

Patient smoking cessation behaviors, as measured by patient questionnaire at 2 months and 1 year 

Clinical outcome 

Yes 17.8% of gum plus 

training patients reported successful smoking cessation at 2 months, vs. 4% of usual care and 7% of gum only (p=0.05), with preliminary 1-year analysis showing narrower but still statistically significant differences. No significant differences in attempts to quit between gum only group (62.8%) and gum plus training group (76.5%). 

Nicotine gum witcues helped stimpatient smoking cessation attempphysician training counseling with a intensive flow shproduced more successful short-cessation. 

Clark, 1998120

CC: NA Int: Live, Video 

CC: NA Int: Case-based learning, Clinical experiences, Lecture 

CC: NA Int: Multiple time or repetitive 

Impact of the intervention independent of the prescription of inhaled antiinflammatory devices. 

Clinical outcome 

Yes This was a subgroup 

analysis of 72 children (combined intervention and control groups) who were started on antiinflammatory devices during the study. Intervention group patients were statistically significantly more likely to have fewer symptomatic asthma days in spring, summer, and winter (but not in the fall). 

The intervention positive impact ophysician and pareported behaviopatient outcomes also positively imby the interventioaddition, the impappeared to be mthan just that resfrom the increaseanti-inflammatory medication and tchange in practicbehavior and the management moseems to be impwell 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-339

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

McMahon, 1988155

Int: Live Int: Live CC: Live CC: NA 

Int: Discussion group, Feedback, Lecture Int: Discussion group, Lecture CC: Discussion group, Lecture CC: NA 

Int: Multiple time or repetitive Int: One time CC: One time CC: NA 

Length of stay 

Clinical outcome 

Yes LOS for patients 

discharged from the study unit dropped significantly (p<.001) when compared with preintervention. Between group differences were not measured/ only pre and post intervention. 

The length of stadecreased signifthe three targetespecialty service2 pronged Int: 1 meeting with medirector, and a Cgroup meeting. 

Clark, 200078

Int: Live, Video CC: NA 

Int: Case-based learning, Demonstration, Lecture CC: NA 

Int: Multiple time or repetitive CC: NA 

Patient use of healthcare for asthma 

Clinical outcome 

Yes Intervention patients had 

fewer hospitalizations (p=.03). Intervention patients did not show less utilization of ED use, but high baseline users of the ED did show a decrease in use (p=.03). Number of office visits was not affected. 

Participating phyreported that thecommunicated ataught patients in sophisticated waParents of intervpatients reported physicians used of communicatioeducation strategenhance patient and satisfaction. Intervention patieshowed a decreahospitalization. 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-340

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Kim, 1999137

Int: Live, Print CC: Print 

Int: Academic detailing, Feedback, Readings CC: Readings 

Int: Multiple time or repetitive CC: Multiple time or repetitive 

Preventative services used by patients, as reported by patient questionnaire 

Clinical outcome 

Yes Comprehensive group 

reported greater use of pneumococcal and tetanus vaccination, (p=.02 and p<.01, respectively) and decline in mammography not seen in comprehensive group but seen in education only group (p<.01) 

A physician-targeapproach of edupeer-comparison feedback, and acdetailing has moeffects on patiensatisfaction and on the offering of selected preventcare services. Thagreement betwepatient reports amedical records raises concerns current methods ascertaining comwith guidelines fopreventative care
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-341

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Clinical Outcomes Objectives Not Met, Evaluation Duration Greater Than 30 DaysMehler, 200599

Int: Internet, not real time, Print Int: Live, Print CC: NA 

Int: Academic detailing, Readings Int: Academic detailing, Lecture CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Percent change in LDLcholesterol levels 

Clinical outcome 

No They observed a 

decrease in LDLcholesterol levels in all groups (electronic 111 to 97 mg/dL, direct 115 to 104 mg/dL, and control 109 to 101 mg/dL), although there were no differences in magnitude between the groups (P =.4). 

A simple educatiintervention seempositively influenprovider behavioarea of lipid manin diabetes mellitelectronic and didetailing seem to viable approacheFuture studies to determine optimaeducational comthat facilitate appprovider actions or intensify lipid tseem warranted the burgeoning population of diapatients at risk focoronary heart dmorbidity and mo

Norris, 200081

Int: Live, Print F/U phone calls about protocol CC: NA 

Int: Lecture, Point of care, Opinion-leader from clinic teaching CC: NA 

Int: Multiple time or repetitive CC: NA 

Patient selfreported physical activity, as reported on telephone questionnaire at 6-month f/u 

Clinical outcome 

No No differences in energy 

expenditure at 6 months (p=0.77) or other measures of physical activity, although intervention patients had higher stages of change scores among subset of patients classified as Contemplators (p=0.03). 

"a one-time PACcounseling sessiminimal reinforcea setting with higbaseline levels odoes not further activity of Contemadvanced in stagbehavior change
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-342

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Brown, 199972

Int: Live CC: NA 

Int: Clinical experiences, Discussion group, Lecture, Role play, Clinicians audio taped interaction with patients and listened between workshops CC: NA 

Int: Multiple time or repetitive CC: NA 

Patient satisfaction scores as measured by change in mean score on Art of Medicine survey during 6-month periods preand postworkshop 

Clinical outcome 

No The mean score on the 

Art of Medicine survey improved more in the control group (0.072[0.030 + 0.042,CI, -0.010 to 0.154]) than in the intervention group (0.030[CI, -0.060 to 0.120]). The difference in these changes was 0.042 (CI, -0.080 to 0.164). Exposure to the communication skills program therefore did not affect changes in patient satisfaction scores. 

”Thriving in a BuPractice: PhysiciPatient Communa typical continuimedical educatioprogram geared developing cliniccommunication snot effective in imgeneral patient satisfaction. To iglobal visit satisfcommunication straining programneed to be longemore intensive, tbroader range of and provide ongoperformance fee

Lin, 199773

CC: NA Int: Live, Video, Print 

CC: NA Int: Academic detailing, Feedback, Lecture, Readings, Role play 

CC: NA Int: Multiple time or repetitive 

Patient satisfaction 

Clinical outcome 

No There was no difference 

in patient satisfaction with their PCP or in the care they received. 

The results do nosupport the concthis complex and aggressive interveffected a sustaichange in practicbehaviors 6 monthe intervention. some positive chwere noted immeafter the intervenprescribing pattewere lost 6 montthe intervention w
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-343

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Casebeer, 1999131

Int: Audio Telephone conferences CC: NA 

Int: Case-based learning, Discussion group, Point of care CC: NA 

Int: Multiple time or repetitive CC: NA 

Patient health status as measured by SF-12 

Clinical outcome 

No No difference in physical 

health status or mental health status between intervention and control groups. 

Combining a serinteractive case conferences with reminders shows in increasing phyadherence-enhastrategies. 

Gullion, 1988132

Int: Print, Telephone conference Int: Print, Telephone conference Int: Print, Telephone conference CC: NA 

Int: Discussion group, Feedback, Readings Int: Discussion group, Feedback, Readings Int: Discussion group, Feedback, Readings CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Change in patients' diastolic blood pressure (DBP), from pre- to postintervention (up to 11 months postphone call) 

Clinical outcome 

No DBP improved for all 

groups (1.32 mmHg, p<0.0001), with no significant differences between control group and the intervention groups. 

An education procombining an individualized feereport of performpeer-reviewed syand an educationsession in the fotelephone conferhad no significanon patients' hype

Harris, 200575

Int: Live, Audio, Print, teleconference CC: NA 

Int: Case-based learning, Lecture, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Overall mean fasting plasma glucose values 

Clinical outcome 

No The overall mean fasting 

plasma glucose values were not significantly affected (control, 8.5 mmol/L, versus intervention, 8.4 mmol/L; p = .74). 

CME delivered bteleconference wfeasible, well attewell received by participants, and improved some kdiabetes managepractices and oualthough primary improving HbA1C not achieved. 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-344

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Harris, 200575

Int: Live, Audio, Print, teleconference CC: NA 

Int: Case-based learning, Lecture, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

HbA1c levels Clinical 

outcome 

No The intervention did not 

significantly affect the primary outcome measure (control, 0.076, versus intervention, 0.073; p = .29). The intervention did, however, significantly (p < .04) alter the distribution of patients (n = 369) by level of glycemic control as defined by 1998 Canadian diabetes guidelines and as taught in the TED CME. In the intervention group, fewer patients had inadequate glycemic control (HbA1c levels > 0.084) than in the control group (15.8% versus 23.9%). Significantly more patients in the intervention group were taking insulin (alone or in combination with oral agents), whereas significantly more patients in the control group were prescribed oral agents only. Significantly more patient records in the intervention group had documentation of BMI, eye exams, communication of a treatment plan, and used a flow sheet. 

CME delivered bteleconference wfeasible, well attewell received by participants, and improved some kdiabetes managepractices and oualthough primary improving HbA1C not achieved. 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-345

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Cummings, 1989112

Int: Live, Video, Print CC: NA 

Int: Demonstration, Discussion group, Lecture, Role play CC: NA 

Int: Multiple time or repetitive CC: NA 

Long-term smoking cessation rates of patients 

Clinical outcome 

No There was no difference 

between intervention and control groups regarding long-term smoking cessation rates among patients at one year. 

Intervention physdiscuss smoking greater rate than control physicianIntervention physspent more time discussing smoktheir patients. Mosmoking patients intervention physset quit dates anmore follow-up wphysicians. Ratesmoking cessatioamong patients ayear were no diffbetween control intervention phys

Cummings, 1989113

Int: Live, Video, Print CC: NA 

Int: Case-based learning, Demonstration, Discussion group, Lecture, Role play CC: NA 

Int: Multiple time or repetitive CC: NA 

Patient smoking cessation rates at one year 

Clinical outcome 

No Although patients in the 

intervention group received more counseling, their patients did not have a higher rate of smoking cessation at one year. 

Intervention physdiscussed smokicessation at a hithan did control physicians. They more time discussmoking overall, more of their patquit dates. Overasmoking cessatioof patients at one did not differ betwgroups. 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-346

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Kottke, 1989114

Int: Live, Print Int: Print CC: NA 

Int: Demonstration, Discussion group, Lecture, Readings Int: Readings CC: NA 

Int: One time Int: determined by participant CC: NA 

Smoking cessation rates at one year. 

Clinical outcome 

No Smoking cessation rates 

were not significantly different among the three groups (workshop, materials-only, and control) 

A brief training pand/or the distribeducation materimarginally increasmoking cessatiobehaviors among volunteer physicdid not result in ismoking cessatioat one year. 

Cummings, 1989117

Int: Live, Video, Patient education materials CC: NA 

Int: Case-based learning, Demonstration, Role play, Selfreflection CC: NA 

Int: Multiple time or repetitive CC: NA 

Smoking cessation by patients, as measured by self-report and biochemical validation 

Clinical outcome 

No No statistical differences 

in attempt to quit (39.7 vs. 36.6%), self-report of abstinence, or biochemically validated long-term (>=9 month) abstinence were similar between groups (3.2 vs. 2.5%, difference -1.7 to +3.1). 

3-hour continuing education progracombined with sumaterials for officchanged the way physicians in privpractice counselsmoking, but had statistically signifeffects on patienoutcomes. Some significant differephysicians and pintervention vs. cgroup related to but adjustment fodid not affect out
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-347

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Elliott, 199755

CC: NA Int: Live, Print 

CC: NA Int: Case-based learning, Clinical experiences, Discussion group, Lecture, Readings 

CC: NA Int: Multiple time or repetitive 

Attitude score of patients and caregivers 

Clinical outcome 

No Attitude score declined in 

patients and caregivers in the intervention group but not statistically significant. 

With the exceptioattitudes of patiecaregivers, there trend toward improvement in pratings as well as provider attitudes knowledge but thwas minor and thresults were not overwhelmingly convincing 

Elliott, 199755

CC: NA Int: Live, Print 

CC: NA Int: Case-based learning, Clinical experiences, Discussion group, Lecture, Readings 

CC: NA Int: Multiple time or repetitive 

Pain scores of patients 

Clinical outcome 

No Improved in the 

intervention group but not statistically significant. 

With the exceptioattitudes of patiecaregivers, there trend toward improvement in pratings as well as provider attitudes knowledge but thwas minor and thresults were not overwhelmingly convincing. 

Evans, 198656

Int: Print CC: NA 

Int: Readings, Chart cue materials offered, but not necessarily implemented CC: NA 

Int: Multiple time or repetitive CC: NA 

Average blood pressure change from baseline to postintervention, as measured by home visit to patient 

Clinical outcome 

No Blood pressure improved 

significantly in both intervention and control groups (SBP 12.2 and 13 mmHg lower, DBP 10.4 and 10.6 mmHg lower), but no difference between study and control groups. 

"Our study demono influence of a continuing mediceducation prograpractices of physon the control of pressure of hypepatients referred community survethese physicians program was beg
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-348

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Kim, 1999137

Int: Live, Print CC: Print 

Int: Academic detailing, Feedback, Readings CC: Readings 

Int: Multiple time or repetitive CC: Multiple time or repetitive 

Patient satisfaction 

Clinical outcome 

No Satisfaction high at 

baseline with no significant changes, but comprehensive intervention patients felt quality of care improved. Patient satisfaction reporting preventative services offered reported higher satisfaction. 

A physician-targeapproach of edupeer-comparison feedback, and acdetailing has moeffects on patiensatisfaction and on the offering of selected preventcare services. Thagreement betwepatient reports amedical records raises concerns current methods ascertaining comwith guidelines fopreventative care

Clinical Outcomes Objectives With Mixed Results, Evaluation Duration Not ReportedWells, 2000151

Int: Live, Print CC: NA 

Int: Academic detailing, Discussion group, Feedback, Lecture, Point of care, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Patient employment status, as measured by patient survey 

Clinical outcome 

Mixed Among patients initially 

employed, 89.7% of intervention patients and 84.7% of control patients worked at 12 months (P = .05). Among those initially not working, 16.4% of intervention patients and 11.4% of control patients were working at 6 months (P>.10); by 12 months, 17% to 18% of intervention and control patients started working. 

When these manprimary care praimplemented QI programs that imopportunities for depression treatmwithout mandatinquality of care, mhealth outcomesretention of empof depressed patimproved over a while medical visnot increase ove
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-349

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Wells, 2000151

Int: Live, Print CC: NA 

Int: Academic detailing, Discussion group, Feedback, Lecture, Point of care, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Healthrelated quality of life, as measured by patient survey (SF12) 

Clinical outcome 

Mixed Intervention patients 

improved more on mental HRQOL by 1 to 2 points at each time point, but not global physical health or physical limitations. 

When these manprimary care praimplemented QI programs that imopportunities for depression treatmwithout mandatinquality of care, mhealth outcomesretention of empof depressed patimproved over a while medical visnot increase ove

Gerrity, 199932

Int: Live, Video, Audio, Print CC: NA 

Int: Clinical experiences, Discussion group, Feedback, Lecture, Readings, Role play CC: NA 

Int: Multiple time or repetitive CC: NA 

Patient satisfaction 

Clinical outcome 

Mixed Patient satisfaction in the 

female case was higher for intervention physicians than control (p=.014), and higher in the male patient, but not at a significant level. 

The Depression Education Prograchanged physiciabehavior and maimportant compothe efforts to impcare of depressepatients. 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-350

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Clinical Outcomes Objectives With Mixed Results, Evaluation Duration Greater Than 30 DaysLin, 199773

CC: NA Int: Live, Video, Print 

CC: NA Int: Academic detailing, Feedback, Lecture, Readings, Role play 

CC: NA Int: Multiple time or repetitive 

Depression outcome measures 

Clinical outcome 

Mixed The main depression 

outcome measure (the IDS score) did not show any change as a result of the intervention (as expected it decreased with treatment but the amount of decrease was not higher as a result of the intervention). Another measure called the SCL-90 did show a significantly greater decrease of scores after the intervention but the overall result was mixed. 

The results do nosupport the concthis complex and aggressive interveffected a sustaichange in practicbehaviors 6 monthe intervention. some positive chwere noted immeafter the intervenprescribing pattewere lost 6 montthe intervention w

Casebeer, 1999131

Int: Audio Telephone conferences CC: NA 

Int: Case-based learning, Discussion group, Point of care CC: NA 

Int: Multiple time or repetitive CC: NA 

Patient knowledge and behavior on hypercholesterolemia, as measured by self-reported patient questionnaire and lipid levels 

Clinical outcome 

Mixed Intervention group had 

significantly higher knowledge score (p=0.008) and selfreported consumption of dietary fats (p=0.002), but no significant difference in 8 other items. 

Combining a serinteractive case conferences with reminders shows in increasing phyadherence-enhastrategies. 

Casebeer, 1999131

Int: Audio Telephone conferences CC: NA 

Int: Case-based learning, Discussion group, Point of care CC: NA 

Int: Multiple time or repetitive CC: NA 

Patient lipid levels, as measured by laboratory testing 

Clinical outcome 

Mixed Significantly lower serum 

cholesterol in men in intervention group 9 months after intervention. (p=.02) Specific changes, lipid levels, and data in women NR. 

Combining a serinteractive case conferences with reminders shows in increasing phyadherence-enhastrategies. 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-351

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Clark, 1998120

CC: NA Int: Live, Video 

CC: NA Int: Case-based learning, Clinical experiences, Lecture 

CC: NA Int: Multiple time or repetitive 

Impact of intervention on use of health care services by the patient 

Clinical outcome 

Mixed IG group patients were 

stat sig. less likely to have office visits. There was no impact on ED visits and hospitalizations (except in a subgroup analysis of low-income patients where the IG had fewer). 

The intervention positive impact ophysician and pareported behaviopatient outcomes also positively imby the interventioaddition, the impappeared to be mthan just that resfrom the increaseanti-inflammatory medication and tchange in practicbehavior and the management moseems to be impwell. 

Messina, 2002156

Int: Live, Audio, Print Int: Live, Print CC: Live, Audio CC: NA 

Int: Demonstration, Readings, Standardized patient, telephone counseling Int: Demonstration, Readings, Standardized patient CC: Readings, telephone counseling CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: Multiple time or repetitive CC: NA 

Patient mammography use. 

Clinical outcome 

Mixed Baseline ever users of 

mammography showed an increase in use among those who received BSTC compared to control pts (p=0.041). However, among never users, CME, and not BSTC, is more important in achieving regular use of mammography (but not to a statistically significant degree- lack of power) 

"...findings suggewomen who havprior mammogrado not screen remay require diffeapproaches to prscreening than wwho have never mammogram. Bstill potentially usmotivator to obtasubsequent mammograms bnot appear to be adequate for initiscreening in our of women who na mammogram." 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-352

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Unclear if Clinical Outcomes Objectives Met, Evaluation Duration Greater Than 30 DaysRodney, 198677

Int: Live, Video Int: Live, Video CC: NA 

Int: Demonstration, Lecture, Simulation (other than standardized patient or roleplay) Int: Demonstration, Lecture, Simulation (other than standardized patient or roleplay) CC: NA 

Int: One time Int: One time CC: NA 

Procedure outcomes from flexible sigmoidoscopy, as reported by physicians by phone or written survey 

Clinical outcome 

Unclear 

2 complications were reported among large group learners, and none in the small groups, but no test of significance mentioned. 

Physicians who participate in couflexible sigmoidohave a higher prof office utilizatiothese skills than who do not take Minimal differencbetween large angroup CME form
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-353

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Clinical Outcomes Objectives With No Control Group, Evaluation Duration Greater Than 30 DaysBrown, 2004135

Int: Live, Video, Print CC: NA 

Int: Case-based learning, Demonstration, Lecture, Problem-based learning or teambased learning, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Admission rates, written asthma action plan, and days absent from school 

Clinical outcome 

No control group 

The families of 36 children (13%) had an income of < $20,000, andthey were treated by 23 physicians. Low-income children in the treatment group tended to have higher levels of use of controller medications, to receive a written asthma action plan, and to miss fewer days of school, although these differences were not statistically significant compared to low-income children in the control group. However, low-income treatment group children were significantly less likely to be admitted to an emergency department (annual rate, 0.208 vs 1.441, respectively) or to a hospital (annual rate, 0 vs 0.029, respectively) for asthma care compared tochildren in the control group. 

"The physician’s interactive seminbeen shown to easthma care and outcomes. The imthe program is nreserved merely patients with moresources. The gdecline in emergdepartment use wchildren from lowfamilies." 
Evidence table 12. Effectiveness of continuing medical education on short-term and long-term clinical outcomes 

F-354

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Kutcher, 200265

Int: Live, Print Int: Live, Print 

Int: Academic detailing, Discussion group, Lecture, Readings, Int: Discussion group, Lecture, Readings 

Int: One time Int: One time 

Patient satisfaction, compliance, treatment outcome and office visits. 

Clinical outcome 

No control group

Patients were satisfied in both groups, but no between group comparison was done, no differences were found in compliance or withdrawals. Patients treated by physicians in the enhanced group did make fewer office visits. Both patient groups improved with regard to self reported signs of depression (p<0.001) but the between group comparison showed no differences between groups. Patients treated by physicians in the enhanced group were less likely to be sent for adjunctive psychotherapy, (p <0.0001). 

"A well-designeddirectional, brief, and low-cost eduprogram can incrfamily physiciansknowledge of deimprove their diaskills, and optimitreatment of dep

*A complex intervention, which involved both physician and patient education, was used. 

AMI = acute myocardial infarction; BMI = body mass index; BSTC = barrier-specific telephone counseling; CES-D = Center for Epidemiologic Studi

continuing medical education; CI = confidence interval; DBP = diastolic blood pressure; ED = emergency department; EH = emotion handling; HRQOlife; IDS = Inventory of Depressive Symptomatology; IG = intervention group; LOS = length of stay; NA = not applicable; NR = not reported; NSAID 

inflammatory drug; PCP = primary care provider; PD = problem defining; QI = quality improvement; SBP = systolic blood pressure; SCL-90 = Sympstandard deviation; SET = standard educational tools; SF-12 = Short Form-12; FQ = fluoroquinolone; TED = teleconferenced educational detailing; WAntibiotic Resistance Network; 
Evidence table 13. Characteristics of systematic reviews evaluating the effectiveness of simulation in medical education 

F-355 

Author, year 

Simulation methods 

Health care professionals Study designs Comparisons 

End date for search 

Psychomotor Skills Haque, 200626 Virtual reality Medical students; 

Physicians-in-training (residents or fellows) 

Randomized controlled trials; Prospective studies (Note: In order for studies to be included, they should have reported results with statistical data). 

Simulation vs other medical education; Experienced operators vs novice operators in a simulated environment 

2004 

Aucar, 200529 Partial task 

simulation; Computer simulation 

Medical students; Physicians-in-training (residents or fellows); Fully trained physicians; Nonsurgical personnel 

Randomized controlled trials; Nonrandomized controlled trials; Prospective studies; review articles 

Simulation vs other simulation; Simulation vs other medical education; Simulation vs no education 

2004 (not clearly reported, authors wrote that search was conducted until 'present') 

Gerson, 200430

Partial task simulation; Computer simulation; Virtual reality 

Physicians-in-training (residents or fellows); Fully trained physicians; novice; clerical staff 

Randomized controlled trials; Nonrandomized controlled trials; Prospective studies; Case-control studies 

Simulation vs other simulation; Simulation vs other medical education; Simulation vs no education; Expert vs novice 

January 2004 

Sutherland, 200631

Full simulation; Partial task simulation; Computer simulation 

Medical students; Physicians-in-training (residents or fellows); Fully trained physicians; other groups not specified 

Randomized controlled trials 

Simulation vs other simulation; Simulation vs other medical education; Simulation vs no education 

April 2005 

Gaffan, 2006158

Partial task simulation; Computer simulation; Standardized patient 

Medical students Randomized controlled 

trials; Nonrandomized controlled trials; descriptive; cohort 

Simulation vs other medical education; Simulation vs no education 

August 2004 
Evidence table 13. Characteristics of systematic reviews evaluating the effectiveness of simulation in medical education 

F-356 

Author, year 

Simulation methods 

Health care professionals Study designs Comparisons 

End date for search 

Ravert P, 2002159

Full simulation; Partial task simulation; Computer simulation 

Medical students; Physicians-in-training (residents or fellows); Nurses; Nursing students 

Randomized controlled trials; Nonrandomized controlled trials; Prospective studies 

Simulation vs other medical education 

NR 

Communication SkillsGaffan, 2006158

Partial task simulation; Computer simulation; Standardized patient 

Medical students Randomized controlled 

trials; Nonrandomized controlled trials; descriptive; cohort 

Simulation vs other medical education; Simulation vs no education 

August 2004 

Spangler, 2002160

Standardized patient role play 

Medical students Randomized controlled 

trials; Prospective studies 

Simulation vs other medical education 

June 2002 

Cognitive Skills Hmelo, 199028 Computer 

simulation, not clear 

Medical students; Physicians-in-training (residents or fellows); other health professional students (e.g., dietary, allied health, pharmacy, nursing, occupational health, dental, respiratory health); medical/nursing staff 

Not specified, but 78% had controls 

Simulation vs other medical education; Simulation vs no education 

NR 
Evidence table 13. Characteristics of systematic reviews evaluating the effectiveness of simulation in medical education 

F-357 

Author, year 

Simulation methods 

Health care professionals Study designs Comparisons 

End date for search 

Ravert P, 2002159

Full simulation; Partial task simulation; Computer simulation 

Medical students; Physicians-in-training (residents or fellows); Nurses; Nursing students 

Randomized controlled trials; Nonrandomized controlled trials; Prospective studies 

Simulation vs other medical education 

NR 

Other Issenberg, 200527

None, discussed features of highfidelity simulation 

Medical students; Physicians-in-training (residents or fellows); Fully trained physicians 

Randomized controlled trials; Nonrandomized controlled trials; Prospective studies 

None June 2003 

NR = not reported 
Evidence table 14. Results of systematic reviews evaluating the effectiveness of simulation on medical education 

F-358 

Psychomotor Skills 

Aucar, 200529

Outcome Effectiveness of simulation in improving performance of a single cohort of subjects. 

Comparison was made before and after simulation training. 

Objective Psychomotor skills 

Results Out of nine studies, eight found that simulation improved performance. Ninth 

study, which enrolled experienced surgeon only, found marked variability in the measurable skills. 

Outcome To evaluate the role of simulation in assessing the effectiveness of simulators in 

differentiating experienced from novice users. 

Objective Psychomotor skills 

Results Simulation is effective in differentiating between novice and expert users. 

Outcome To evaluate the performance on a simulator with performance on live animals or 

human surgical subjects. 

Objective Psychomotor skills 

Results Performance on simulation correlates well with performance on live animals or 

human subjects. Trainees trained on simulators may perform better on live animalsor human subjects as compared to untrained trainees (one study). 

Overall Conclusion: oveducational intervention Author’s Main Conclusserve as an objective meamong surgeons. The prseveral inconsistencies rthe acceptance of currenvalid for certification of sare needed that correlate performance of capable using defined criteria, wiperformance on simulato

Gaffan, 2006158

Outcome Student satisfaction with learning. 

Objective Attitudes 

Results Student satisfaction with structured clinical instruction models was positive. 

Outcome Ability to detect breast lumps using standardized patients. 

Objective Psychomotor skills 

Results In two studies, teaching of breast examination with standardized patients had 

increased detection in intervention groups compared with normal teaching. p<.05 and p<.001. 

Outcome Ability to detect breast lumps using models 

Objective Psychomotor skills 

Results Lump detection higher for group trained with dynamic model, p<0.001. In two more 

studies, post-test improvement in lump detection, p<.05 and p<.01, in intervention groups trained with models. Third study showed intervention group had higher sensitivity in detecting breast lumps, p=.001, but lower specificity, p=.001. 

Outcome Knowledge of breast cancer detection. 

Objective Psychomotor skills 

Results One study showed equivalent knowledge test results in group taught by 

standardized patients and group taught by normal teaching. 

Outcome Giving bad news 

Objective Psychomotor skills 

Results Four studies looked at giving bad news education, three with SPs, one with role 

play and SPs, and one with role play alone. All showed improvement in competence in giving bad news. 

Overall Conclusion: Oveducational intervention Author’s Main Conclussimulation: Use of standbreast examination imprin clinical assessment; thteach breast examinatiosensitivity for detecting baided learning modules superior to other types o
Evidence table 14. Results of systematic reviews evaluating the effectiveness of simulation on medical education 

F-359 

Outcome Student satisfaction with learning detection of testicular lumps. 

Objective Attitudes 

Results One study showed positive feedback from students in learning with mannikin Zack 

Sutherland, 200631

Outcome Surgical trainees trained with physical or model training vs other forms of training, 

including no training. 

Objective Psychomotor skills 

Results Four studies found that model training may be better than no training and standard 

training, but overall there were mixed results. 

Outcome Surgical trainees trained on computer simulation versus video simulation. 

Objective Psychomotor skills 

Results Mixed results in seven studies; may have depended on types of tasks, with 

computer simulating producing better results for tasks such as incisions, but not for knot tying times. 

Outcome Surgical trainees trained on computer simulating vs no training. 

Objective Psychomotor skills 

Results Those trained on computer simulation performed better than those with no training 

in eight of nine studies. 

Outcome Surgical trainees trained on computer simulation vs physical trainer. 

Objective Psychomotor skills 

Results One study showed computer simulation training to be superior to live pig (p=.0005)

Outcome Surgical trainees with two or more types of computer simulating; MIST-VR 

outcome. 

Objective Psychomotor skills 

Results One study showed more intensive training is more effective than easy level 

training; 2nd study showed inconsistent results. 

Outcome Surgical trainees trained with video simulation vs no training or other forms of 

training. 

Objective Psychomotor skills 

Results Six studies comparing video simulation with no training showed inconsistent 

results; five studies comparing video with other forms of training showed no differences. 

Outcome Surgical trainees trained on computer simulation vs standard training. 

Objective Psychomotor skills 

Results Four out of five studies showed statistical differences with improved results in 

computer simulation group. Authors concluded that computer simulation comparisons varied, potentially confounded by the different components of standard training, as well as by the different intensities of time allowed on the simulator in computer simulation groups. 

Overall Conclusion: Oveducational intervention. Author’s Main Conclusgenerally showed better but was not convincingly training (such as surgica(particularly when assesperformance). Video simconsistently better resulttraining at all, and there determine if video simulastandard training or the usimulation may have beetraining, and cadaver trathan model training. 

Ravert P, 2002159

Outcome Effect of simulation on skill or knowledge acquisition. 

Objective Knowledge or cognitive skills (Psychomotor skills) 

Overall Conclusion: Paresults 
Evidence table 14. Results of systematic reviews evaluating the effectiveness of simulation on medical education 

F-360 

Results Five out of nine studies included in this review included medical students or 

physicians-in-training. Two out of the five studies had a control or comparison group, and one study had a cross-control design. Overall, 75% of the nine studies showed positive effects of simulation on skill or knowledge acquisition as determined by effect size which was measured as d-index. The effect sizes ranged from 0.34 to 5.06. 

Author’s Main Conclusof computer-based simuknowledge and skill acqucomputer-based simulatis enormous, but more reeffectiveness is needed. 

Gerson, 200430

Outcome To evaluate the role of upper GI endoscopy simulator in differentiating between a 

novice and an expert in performing upper GI endoscopy.(Validation). 

Objective Psychomotor skills 

Results Authors found two studies and they showed that a simulator was able to 

distinguish a novice from an expert. One study also found that novice trained on the simulator reached the level of the experts after three weeks of simulator training. 

Outcome To evaluate the role of simulation in training colonoscopy by comparing novice GI 

fellows trained on simulators with traditionally trained novice GI fellows. 

Objective Psychomotor skills 

Results Simulator-trained novice fellows performed better than the novice fellows trained 

traditionally in all performance aspects except for time of insertion (p<.05) and that this advantage continued up to 30 colonoscopies. 

Outcome To determine optimal time for using simulation in the training of GI fellows. 

Objective Psychomotor skills 

Results Authors found one study which concluded that colonoscopy simulators are most 

effective early in the training of GI fellows. 

Outcome To evaluate the role of simulation in training ERCP by comparison of live pig 

model, Erlangen Endo-trainer model, and a computer simulation model. Each model was rated by novices and expert faculty for realism, utility in training, and ease of use. 

Objective Realism, usefulness in teaching basic and advanced ERCP skills. 

Results Authors found one study that reported Erlangen Endo-trainer model to be superior 

to other models. 

Outcome To evaluate the role of simulation in training flexible sigmoidoscopy by comparing 

simulator-trained trainees with trainees trained with standard methods or no training. 

Objective Psychomotor skills 

Overall Conclusion: Oveducational intervention Author’s Main Conclussigmoidoscopy simulatoclinical training of residepatient comfort is the pribased upon the current straining has been demon2. Use of the colonoscopfellows early in the trainiless than 30 proceduresbeneficial, based upon dFurther prospective studthese preliminary finding3. Indications for ERCP cannot be made until vaconducted. 4. Because current statesimulators have been deexpert endoscopists from endoscopy, sigmoidoscotechnology may somedamethod to judge compet
Evidence table 14. Results of systematic reviews evaluating the effectiveness of simulation on medical education 

F-361 

Results Authors found three studies. One study reported that simulator training is better 

than no training in teaching flexible sigmoidoscopy to family medicine residents as measured by insertion time (p=.03), directional errors (p<.01), and examination time (p=.01). Second study reported that bedside teaching was superior to simulator training in teaching flexible sigmoidoscopy to medical residents as measured by insertion time, negotiating rectosigmoid junction, reaching spleenic flexure, and performing reteroflexion. The third study found no procedural skills differences between the groups of 2nd-year residents trained on a simulator and those trained using standard methods. 

Outcome To evaluate the role of flexible sigmoidoscopy simulator in differentiating between 

a novice and an expert in performing sigmoidoscopy. 

Objective Psychomotor skills 

Results Authors found two studies and they showed that a flexible sigmoidoscopy 

simulator was able to distinguish a novice from an expert. 

Outcome To evaluate the role of simulation in assessing the effectiveness of colonoscopy 

simulators in differentiating novice from experts 

Objective Psychomotor skills 

Results Colonoscopy simulators can differentiate a novice from an expert as measured by 

procedure time, insertion time, and time in and out. 

Haque, 200626

Outcome Effectiveness of virtual simulation in differentiating between experienced (those 

who have performed more than 50 such procedures) and novice (those who have performed less than 10 procedures) trainees based on the "error score" (the number of "wall-strikes" experienced by the trainee or the ratings provided by an external expert of the trainee’s performance). 

Objective Psychomotor skills 

Meta-analysis Yes 

Significant Results Yes 

Results Virtual reality simulators help to differentiate between a novice and an experienced 

trainee as evidenced by low error scores of the experienced trainees to complete a given surgical task in a virtual reality simulation environment. Standardized effect size = -1.325 (95%CI; -2.125, -0.525)[Random-effects model, p <0.0001 for homogeneity]. 

Outcome Effectiveness of virtual simulation in differentiating between experienced (those 

who have performed more than 50 such procedures) and novice (those who have performed less than 10 procedures) trainees based on the "task completion time" (amount of time, in seconds or minutes, taken by the trainee to complete task) 

Objective Psychomotor skills 

Meta-analysis Yes 

Significant Results Yes 

Overall Conclusion: No after educational interveAuthor’s Main Conclusreality simulators can lescomplete a given surgicadiscriminate between the inexperienced trainees. 
Evidence table 14. Results of systematic reviews evaluating the effectiveness of simulation on medical education 

F-362 

Results Virtual reality simulators help to differentiate between a novice and an experienced 

trainee as evidenced by the decreased amount of time taken by experienced trainees to complete a given surgical task in a virtual reality simulation environment. Standardized effect size = -1.059 (95% CI; -1.331, -0.786)[fixedeffects model, p = 0.13 for homogeneity; authors report performing a random effects model with quite similar results.] 

Outcome Effectiveness of transference of skills from the simulation environment to the 

operating room with experienced surgeons or physicians, trained on simulations, operating on real patients or animal subjects. The control group was experienced surgeons or physicians trained using traditional methods. Both, simulation-trained and traditional method-trained, groups were compared using the "task completion time" (amount of time, in seconds or minutes, taken by the trainee to complete task). 

Objective Psychomotor skills 

Meta-analysis Yes 

Significant Results Yes 

Results The simulation-trained group took much less time than the traditionally trained 

group. Using the fixed effect model, a significant standardized effect size of -1.178 (95% CI; -1.706, -0.651) was obtained, but the inadequacy of analysis was indicated by lack of homogeneity (P <0.0001). Reanalysis using the random effects model yielded a statistically significant standardized effect size of -2.175(95% CI; -3.865, -0.485). 

Outcome Effectiveness of transference of skills from the simulation environment to the 

operating room with experienced surgeons or physicians, trained on simulations, operating on real patients or animal subjects. The control group was experienced surgeons or physicians trained using traditional methods. Both, simulation-trained and traditional method-trained, groups were compared using the "error score" (the number of "wall-strikes" experienced by the trainee or the ratings provided by an external expert of the trainee’s performance). 

Objective Psychomotor skills 

Meta-analysis Yes 

Significant Results No 

Results The simulation-trained group had a smaller error score than the traditionally 

trained group. Using the fixed effect model, a significant standardized effect size of -0.974(95% CI; -1.457, -0.491) was obtained, but the inadequacy of analysis was indicated by lack of homogeneity (P <0.0001). Reanalysis using the random effects model yielded a statistically insignificant standardized effect size of -1.565(95% CI; -3.445, 0.314). 
Evidence table 14. Results of systematic reviews evaluating the effectiveness of simulation on medical education 

F-363 

Communication Skills 

Spangler, 2002160

Outcome Student demonstration of effective tobacco cessation counseling. 

Objective Psychomotor skills 

Results Nine studies with active (role play or standardized patient) component to 

education: all but one showed significant improvement at posttest observation of skills; one did not show an improvement when SP group was compared with role playing group in follow-up SP assessment. 

Outcome Student confidence in tobacco cessation counseling skills 

Objective Attitudes 

Results Three studies demonstrated that students working with standardized patients 

expressed increased confidence in smoking cessation counseling. One RCT showed increased confidence in intervention students compared with controls. 

Overall Conclusion: Oveducational intervention Author’s Main Conclusmethods (e.g., the use ocounseling, standardized playing, or a combination effective for teaching tobtraditional didactic metho

Gaffan, 2006158

Outcome Student satisfaction with learning. 

Objective Attitudes 

Results Student satisfaction with structured clinical instruction models was positive. 

Outcome Ability to detect breast lumps using standardized patients. 

Objective Psychomotor skills 

Results In two studies, teaching of breast examination with standardized patients had 

increased detection in intervention groups compared with normal teaching. p<.05 and p<.001. 

Outcome Ability to detect breast lumps using models 

Objective Psychomotor skills 

Results Lump detection higher for group trained with dynamic model, p<0.001. In two more 

studies, post-test improvement in lump detection, p<.05 and p<.01, in intervention groups trained with models. Third study showed intervention group had higher sensitivity in detecting breast lumps, p=.001, but lower specificity, p=.001. 

Outcome Knowledge of breast cancer detection. 

Objective Psychomotor skills 

Results One study showed equivalent knowledge test results in group taught by 

standardized patients and group taught by normal teaching. 

Outcome Giving bad news 

Objective Psychomotor skills 

Results Four studies looked at giving bad news education, three with SPs, one with role 

play and SPs, and one with role play alone. All showed improvement in competence in giving bad news. 

Outcome Student satisfaction with learning detection of testicular lumps. 

Objective Attitudes 

Results I One study showed positive feedback from students in learning with mannikin 

Zack 

Overall Conclusion: Oveducational intervention Author’s Main Conclussimulation: Use of standbreast examination imprin clinical assessment; thteach breast examinatiosensitivity for detecting baided learning modules superior to other types o
Evidence table 14. Results of systematic reviews evaluating the effectiveness of simulation on medical education 

F-364 

Cognitive Skills 

Hmelo, 199028

Outcome Achievement 

Objective Unclear 

Meta-analysis Yes 

Significant Results Unclear 

Results The average effect size for achievement was 0.63. 

Overall Conclusion: neAuthor’s Main Concluscomputer-assisted instrumodality, how it may bespromote learning and procare environment require CAI may have a role in pof health practitioners, asituations that their profe

Ravert P, 2002159

Outcome Effect of simulation on skill or knowledge acquisition. 

Objective Knowledge or cognitive skills (Psychomotor skills) 

Results Five out of nine studies included in this review included medical students or 

physicians-in-training. Two out of the five studies had a control or comparison group, and one study had a cross-control design. Overall, 75% of the nine studies showed positive effects of simulation on skill or knowledge acquisition as determined by effect size which was measured as d-index. The effect sizes ranged from 0.34 to 5.06. 

Overall Conclusion: Paresults Author’s Main Conclusof computer-based simuknowledge and skill acqucomputer-based simulatis enormous, but more reeffectiveness is needed. 

Other 

Issenberg, 200527

Outcome To evaluate the features and uses of high-fidelity simulators that lead to effective 

learning. 

Objective Effective learning 

Results The important features of high-fidelity simulator that lead to effective learning are 

following: 1. feedback is provided during learning experience 2. Learners engage in repetitive practice 3. Simulator is integrated in overall curriculum 4. Learners practice with increasing levels of difficulty 5. Adaptable to multiple learning strategies 6. Clinical variation in a simulated environment can increase the number and variety of patients a learner encounters 7. Controlled Environment in which learners make and detect mistakes without consequences 8. Individualized and standardized learning 9. Outcomes are clearly defined 10. Face validity-realism of simulator 

Overall Conclusion: UnAuthor’s Main Conclusfield needs improvement quality, high-fidelity mededucationally effective aeducation complements care settings. 

SP = standardized patient; RCT = randomized controlled trial; MIST-VR = Minimally Invasive Surgical Trainer Virtual Reality; ERCD = endoscopic 

EUS = endoscopic ultrasound; CAI = computer assisted instruction 
Evidence table 15. Effects of audience characteristics on the effectiveness of continuing medical education 

F-365 

Author, year 

Audience characteristic 

Educational technique Outcome Results 

Other characanalyzed 

Juzych, 200592 Years in practice Live case studies and printed material Antibiotic prescribing rates No effect None Kiang, 200533 Years in practice Multiple interventions 

but specifics for groups not identified 

Clinician knowledge and beliefs regarding antibiotic prescribing for upper respiratory infections 

Clinicians in practice for >10 years improved in their beliefs about antibiotic prescribing but not in their knowledge compared to physicians with < 10 

years practice experience 

None 

Leopold, 200580 Years in practice Printed guide or hands-on instruction or 

video instruction 

Clinician confidence in performing knee joint injection and objective evaluation of joint injection technique 

No effect Age, number o

injections perflast year, gendphysician vs. nphysician 

Beaulieu, 2002140 Years in practice 90-minute workshop using case discussion, 

role play, and educational handouts 

Performance of physicians in including recommended screening items and excluding nonrecommended screening items during periodic health exam 

Physicians with less than 11 years of experience ordered fewer unnecessary tests order and fewer other tests ordered, but results not stratified by intervention 

None 

Gerstein, 199953 Years in practice Seven-hour small group workshop for 

family physicians 

Sub-scores on participant questionnaire about diabetes knowledge, attitude, practice behavior 

Greater number of years in practice predicted an improvement in selfreported practice behavior but no improvement in attitude or diabetes knowledge 

Gender, family practice certifisolo or group full or part-timpractice size 
Evidence table 15. Effects of audience characteristics on the effectiveness of continuing medical education 

F-366 

Author, year 

Audience characteristic 

Educational technique Outcome Results 

Other characanalyzed 

Grady, 199779 Years in practice Educational presentation 

with/without cue enhancement with/without feedback rewards 

Cueing had a positive impact on mammography rates above education alone, but feedback and rewards had no added benefit 

No effect Age, gender, p

setting, board certification, sforeign medicagraduate vs. Ugraduate, grouAMA memberresidency vs nresidency 

Leopold, 200580 Age Printed guide or hands-on instruction or 

video instruction 

Clinician confidence in performing knee joint injection and objective evaluation of joint injection technique 

No effect Years in pract

number of injeperformed in lgender, physicnon-physician 

Davis, 2004101 Age Live interactive case- based teleconferences Change in prescribing patterns of asthma 

controller medications 

Non-significant trend towards older physicians being less likely to prescribe controller medications 

Unclear (analyfactors mentionot described) 

Harris, 200239 Age Interactive, online, case-based program 

on domestic violence 

Confidence in managing domestic violence patients 

No effect, although data not shown 

Gender, previodomestic violetraining 

Mazmanian, 2001161 Age Commitment-to- change statement with 

or without a signature by the CME course participant 

Whether or not physicians changed their behaviors, depending on if they had a commitment to change, and also depending on whether or not they signed a commitment to change form 

No effect Gender 
Evidence table 15. Effects of audience characteristics on the effectiveness of continuing medical education 

F-367 

Author, year 

Audience characteristic 

Educational technique Outcome Results 

Other characanalyzed 

Grady, 199779 Age Educational presentation 

with/without cue enhancement with/without feedback rewards 

Cueing had a positive impact on mammography rates above education alone, but feedback and rewards had no added benefit 

Intervention more effective in older physicians in bivariate but did not hold in multivariate analysis 

Age, gender, psetting, board certification, sforeign medicagraduate vs. Ugraduate, grouAMA memberresidency vs nresidency 

Soumerai, 1987105 Age Printed educational materials and visits by 

clinical pharmacists 

Intensity of prescribing target drugs. 

No effect Practice settin

specialty, profboards, intenstarget drug usMedicaid pracnursing home size 

Leopold, 200580 Gender Printed guide or hands-on instruction or 

video instruction 

Clinician confidence in performing knee joint injection and objective evaluation of joint injection technique 

Females confidence and objective performance improved more than males 

Years in practnumber of injeperformed in lage, physician non-physician 

Harris, 200239

Gender Interactive, online, 

case-based program on domestic violence 

Confidence in managing domestic violence patients 

No effect, although data not shown 

Age, previous domestic violetraining 

Mazmanian, 2001161

Gender Commitment-tochange statement with 

or without a signature by the CME course participant 

Whether or not physicians changed their behaviors, depending on if they had a commitment to change, and also depending on whether or not they signed a commitment to change form. 

No effect Age 

Gerstein, 199953

Gender Seven-hour small 

group workshop for family physicians 

Sub-scores on participant questionnaire about diabetes knowledge, attitude, practice behavior 

No effect Years in pract

family practice certification, sgroup practicepart-time, prac
Evidence table 15. Effects of audience characteristics on the effectiveness of continuing medical education 

F-368 

Author, year 

Audience characteristic 

Educational technique Outcome Results 

Other characanalyzed 

Grady, 199779

Gender Educational 

presentation with/without cue enhancement with/without feedback rewards 

Cueing had a positive impact on mammography rates above education alone, but feedback and rewards had no added benefit 

No effect Age, gender, p

setting, board certification, sforeign medicagraduate vs. Ugraduate, grouAMA memberresidency vs nresidency 

Gerstein, 199953

Board certification (family practice) 

Seven-hour small group workshop for family physicians 

Sub-scores on participant questionnaire about diabetes knowledge, attitude, practice behavior 

No effect Years in pract

gender, solo opractice, full otime, practice 

Grady, 199779

Board certification (family practice or IM) 

Educational presentation with/without cue enhancement with/without feedback rewards 

Cueing had a positive impact on mammography rates above education alone, but feedback and rewards had no added benefit 

Greater effect on non board certified in bivariate model but not significant in multivariate 

Age, gender, psetting, board certification, sforeign medicagraduate vs. Ugraduate, grouAMA memberresidency vs nresidency 

Soumerai, 1987105

Board certification Printed educational 

materials and visits by clinical pharmacists 

Intensity of prescribing target drugs 

No effect Age, practice 

specialty, profboards, intenstarget drug usMedicaid pracnursing home size 

Grady, 199779

Race Educational 

presentation with/without cue enhancement with/without feedback rewards 

Cueing had a positive impact on mammography rates above education alone, but feedback and rewards had no added benefit 

Intervention more effective in nonwhites in multivariate analysis 

Age, gender, psetting, board certification, sforeign medicagraduate vs. Ugraduate, grouAMA memberresidency vs nresidency 
Evidence table 15. Effects of audience characteristics on the effectiveness of continuing medical education 

F-369 

Author, year 

Audience characteristic 

Educational technique Outcome Results 

Other characanalyzed 

Grady, 199779 Practice setting: solo vs. other Educational presentation 

with/without cue enhancement with/without feedback rewards 

Cueing had a positive impact on mammography rates above education alone, but feedback and rewards had no added benefit 

Intervention more effective in solo practitioners in multivariate analysis 

Age, gender, psetting, board certification, sforeign medicagraduate vs. Ugraduate, grouAMA memberresidency vs nresidency 

Soumerai, 1987105 Practice setting: rural vs. non-rural Printed educational materials and visits by 

clinical pharmacists 

Intensity of prescribing target drugs. 

No effect Age, practice 

board certificaprofessional bintensity of taruse, Medicaid size, nursing hpractice size 

Grady, 199779 Specialty (FP vs. IM) Educational presentation 

with/without cue enhancement with/without feedback rewards 

Cueing had a positive impact on mammography rates above education alone, but feedback and rewards had no added benefit 

No effect Age, gender, p

setting, board certification, sforeign medicagraduate vs. Ugraduate, grouAMA memberresidency vs nresidency 

Lewis, 199370 Specialty Day-long program on 

HIV using videotape, group discussion, small group workshops, and lecture 

Increased frequency of sexual history questions asked 

Internists, but not FPs or Ob/Gyns, were affected by the intervention. 

None 

Lane, 199169 Specialty Multiple CME 

interventions AND lowcost mammography 

Compliance with mammography screening guidelines 

Ob/Gyns had the highest base-line screening levels, but still showed an increase in screening recommendations after the intervention. FP's improved significantly more than OB/Gyns. 

Not specified 
Evidence table 15. Effects of audience characteristics on the effectiveness of continuing medical education 

F-370 

Author, year 

Audience characteristic 

Educational technique Outcome Results 

Other characanalyzed 

Des Marchais, 199061

Specialty 2 training sessions on 

interpersonal skills 

Pre- and post- average deviation score: participant's scoring of interpersonal interactions in videotape, compared with median group score 

Psychiatrists had less deviation from median of group both pre- and post-, compared with family physicians 

None 

Soumerai, 1987105 Specialty (IM vs. FP) Printed educational materials and visits by 

clinical pharmacists 

Intensity of prescribing target drugs. 

No effect Age, practice 

board certificaprofessional bintensity of taruse, Medicaid size, nursing hpractice size 

Grady, 199779 U.S. vs. foreign medical graduation Educational presentation 

with/without cue enhancement with/without feedback rewards 

Cueing had a positive impact on mammography rates above education alone, but feedback and rewards had no added benefit 

No effect Age, gender, p

setting, board certification, sforeign medicagraduate vs. Ugraduate, grouAMA memberresidency vs nresidency 

Grady, 199779 Residency trained vs. no residency Educational presentation 

with/without cue enhancement with/without feedback rewards 

Cueing had a positive impact on mammography rates above education alone, but feedback and rewards had no added benefit 

Intervention more effective in non residency trained in bivariate analysis but not in multivariate 

Age, gender, psetting, board certification, sforeign medicagraduate vs. Ugraduate, grouAMA memberresidency vs nresidency 

AMA = American Medical Association; FP = family practitioner; Ob/Gyn = obstetrician/gynecologist; IM = internal medicine; US = United States; C

education 
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-371 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Knowledge / Cognitive SkillsFordis, 200542 Knowledge test about cholesterol / CAD risk and 

management, using multiple-choice questions and case vignettes with fixed-choice responses. 

Yes No Face / content 

(Not reported) 

NA Current 

study 

Internal consistency (interitem) reliability 

Caaaat

Macrae, 200485 Locally developed test of critical appraisal skills 

pilot tested on group of residents with internal consistency (in pilot and current study) and interrater reliability (pilot) as described below. Small modification of evaluation- decreasing number of articles used from 3 (in pilot) to 2 (in current), but claimed that this change has "minimal impact" on alpha reliability (dropped to 0.74) or "the evidence of validity" which wasn't otherwise quantified. 

Yes 

Yes 

(questions changed) 

Face, content, and construct validity (experts and comparisons not reported) 

 Internal 

consistency (interitem) reliability 

Ca
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-372 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Face / content (Team of experts not specified) 

KR 20 = 0.89 Prior 

study 

Internal consistency 

SitK0

Andersen, 199036

Diagnostic Knowledge Inventory, a measure of psychiatric knowledge based on DSM-III / DSMIIIR criteria using 18 paragraph-length case vignettes about affective and anxiety disorders (discussed in intervention) and other disorders not discussed. Item-pool of 72 vignettes reviewed by team of experts and reduced through item-analysis in trained and untrained sample. Response options yes/no, 24-choice diagnoses, 7-point Likert for treatment options. Pre- and post-test versions with randomized vignette order. 

No Yes (no 

changes)

External construct (known group) 

55% difference in score between experts & non-experts; p<0.0001 for "diagnosis" cases 

Prior study 

Equivalence reliability 

Rp

Doucet, 199840 The Key Features Problem (KFP) was a 28-

item mailed exam developed from guidelines by investigators to test on topics related to headache. 38-item pilot test was given to intervention and control groups for item-analysis. 

Yes No Face / content 

(Experts/ sources: Two neurologists with interest in headache treatment ) 

NA Current 

study 

Internal consistency (interitem) reliability 

KRm(o
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-373 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

White, 198548 Knowledge test for in- hospital treatment of 

acute myocardial infarction, combining multiple-choice action items and true/false concept items corresponding to course objectives. Pilot tested on 15 physicians. Different versions used same items with change in order. 

Yes No Face / content 

(Experts/ sources: 4 cardiovascular faculty and 2 family practice faculty ) 

NA Current 

study 

Internal consistency (interitem) reliability 

Ca0

Chung, 200454 34-item multiple-choice questionnaire to assess 

bioterrorism diagnosis and management, using case scenarios. After review by expert panel, test was given to pilot group of 3 physicians in one exam period. Final test delivered by web. 

Yes No Face / content 

(Experts/ sources: Physicians in EM, Ped EM, Peds, ID, medical toxicology, and medical informatics 

NA Equivalence 

reliability 

Aad8
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-374 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Face / content (Experts/ sources: AAN advisory panel) 

NA Current 

study 

Gifford, 1999142 Questionnaire / test using three clinical scenarios 

about dementia diagnosis and management, with open-ended responses about diagnosis, closedended choice of 12 diagnostic tests, additional write-in diagnostic tests, and additional write-in options. Test was developed by study neurologists, reviewed by advisory panel, and piloted in 8 neurologists. Mailed 8-page test was scored by 3 physician investigators 

Yes No 

Concurrent Criterion (Chart review on subset of 22 physicians' charts) 

Percent agreement: 95-99% for EEG, Apo E test, and referral to Safe Return Program; 47-77% for referral to Alzheimer's Association and diagnosis of and prescription for depression; 27-49% for neuroimaging use 

Current study 

Chan, 199921 Multiple choice test assessing physician 

knowledge of geriatric psychiatry, administered by e-mail. 

No Yes (no 

changes)

Face / content (Experts/ sources: geriatrician, psychiatrist, and family physician) 

NA Current 

study 
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-375 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Gerrity, 199932 54-item knowledge test about depression 

diagnosis and management. Pilot testing by facilitators and investigators, with retention of items if all 7 agreed on answer. 

Yes Yes 

(changed, but unclear how; test based on objecttives and derived from other test) 

Face / content (Experts/ sources: literature review and workshop contents) 

NA Current 

study 

Maxwell, 198457

30-item multiple-choice test for evaluating knowledge gains at medical evaluation committee meetings. Senior resident and experts in relevant fields reviewed test, then pretested in resident physicians. 

Yes No Face / content 

(Experts/ sources: NR) 

NA Current 

study 
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-376 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Premi, 199338 Knowledge test about the diagnosis and 

management of chest pain. 94 questions covering 5 specified domains. Reviewed by 8 residents for interpretability, reading level, ambiguity, doublebarreled questions, jargon, value-laden terms, positive-negative wording, and length. Two versions created with software for randomized sampling and balancing. 

Yes No Face / content 

(Experts/ sources: Consultants and practicing physicians) 

NA Current 

study 
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-377 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Face / content (Experts/ sources: seven experts designed course & reviewed content of the scenarios) 

NA Current 

study 

Gifford, 199646 Survey instrument to assess subjects' decision 

making and adherence to 16 practice recommendations on disease detection, diagnostic test use, and treatments for neurology topics. Developed by 2 investigators from guidelines and course content, using 7 clinical scenarios. Response options included write-in diagnoses, 12-choice test options, write-in management, and other open-ended questions. Pilot tested in 27 neurologists. 

Yes No 

External (known group) validity (Comparison: Readers within the intervention group were more likely to report adherence to practice recommendations) 

Readers were 1.7-6.3 x more likely to report adherence 

Current study 

Stewart, 200543 Physician knowledge test, measured by 

questionnaire, of preventive / screening practices related to perimenopausal patient (21 items) and type 2 diabetes (22 items). 

No Yes 

(unclear if 

changed)

External construct (known group) 

Statistic not reported 

Current study 
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-378 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Slotnick, 199317 24-item knowledge test about drug prescribing 

information for two drugs, Bumex (older drug) and Mazicon (newer drug). Response options included multiple-choice and true/false. 

No No Equivalence 

reliability 

ScaSBpccbr

Hergenroeder, 200260

Knowledge test about ankle and knee examination 

No Yes 

(unclear if 

changes)

 Reliability: not 

specified 

Sr

Attitudes (and/or Knowledge)Meredith, 200051

Self-administered, mailed physician questionnaire on depression treatment. Questionnaires included batteries of questions that have been previously evaluated for reliability & validity & batteries that were specifically developed for their study. Included items of depression knowledge (12-item scale) measuring endorsement of AHRQ guidelines. Response options: 5-point Likert scale from very false to very true, 3 reversed items. 

No Yes (No 

changes 

made) 

Face / content (Experts/ sources: Panel of clinicians determined "nonambiguous" indicators of knowledge) 

NA Current 

study 
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-379 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Face / content (3 experts not specified) 

NA Current 

study 

Test-retest reliability 

Cck0apcr

Mann, 199752 60-item physician questionnaire about 

knowledge and attitudes towards cholesterollowering practices. Response options included true/false and Likert. Experts agreed on >90% of questions and rest deleted. Piloted by FPs for internal consistency. 

No No 

 Internal 

consistency 

K

Mann, 199752 Physicians' perceptions of self-efficacy in counseling 

on cholesterol-lowering, hypertension, and smoking management practices, using 14-items and Likert scale. 

No No Test-retest 

reliability 

Ccpprprpr

Chodosh, 200662

Short scale for providers' perceptions of quality of care for dementia patients, comprising Likert scales and administered through mailed survey 9 months after intervention initiated. 

Yes No Internal 

consistency (interitem) reliability 

Ca
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-380 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Lockyer, 200264

Written survey for knowledge and comfort in dementia management. 5-point scale for involvement-level in patient management; true/false and multiplechoice knowledge items for total score of 14; 3-point comfort score from 13 items. 

No No Internal 

consistency (interitem) reliability 

Caf(p& ac

Elliott, 199755 Survey of physician and nurse knowledge and 

attitudes towards cancer pain management: 15 knowledge items on 5-point scale and 9 attitude items on 11-point scale 

No Yes 

(modified: 

unclear 

how) 

 Internal 

consistency (interitem) reliability 

Cak0panafpan2 o
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-381 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

White, 200463

Case-based 25 item, multiple-choice questionnaire of scenarios faced by community physicians treating asthma patients, to measure knowledge, skills, attitudes (confidence) towards asthma management. Visual analog scale used to assess confidence about answers. 

Unclear Yes 

(unclear if 

changed)

 Internal 

consistency (interitem) reliability 

Ca0iwan 

Skills (communication, psychomotor, or procedural)Hergenroeder, 200260

Clinical Skills Assessment Examination (CSAE) using standardized patient to evaluate performance of ankle / knee exam by checklist: Interrater reliability of baseline rater evaluated in previous study of residents' performance using the same CSAE as that used in current study. SP (certified athletic trainer) used published checklist to rate as observer. 

No Yes 

(unclear if 

changes)

 Inter-rater 

reliability 

AkCaaakr
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-382 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Skills (communication, psychomotor, or procedural) Assessed in the Practice Setting (Practice Behaviors)Greenberg, 198525

Standardized patients' assessment of the appropriateness of medical plan given by the learner for managing common pediatric problems. 6 sets of SP mothers/children with case histories on 4 topics; SP mother completed checklist after visit; senior author rated records after visits for validation. 

No No Concurrent 

Criterion (Senior author's evaluation of medical records) 

Correlation coefficient: 0.60 p<0.001 

Current study 

Gerrity, 199932 Kaplan's 3-item Participatory DecisionMaking scale, as 

completed by standardized patient after visit. 

No Yes 

(adapted, 

unclear 

how) 

 Internal 

consistency (interitem) reliability 

Ca

Roter, 199584 Simulated patient visits about globus hystericus 

associated with unresolved grief were audiotaped and then scored by blinded coders for clinical proficiency. 97 clinical and psychologic items; 3 male physicians at Hopkins trained as blinded SPs; 10 categories and overall summed score as well as subscales. 

No No Internal 

consistency (interitem) reliability 

Cao0sfito0soRts0
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-383 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Roter, 199584 Proficiency in the management of emotional 

distress by physicians audiotaped in a few real patient visits. The audiotapes were analyzed to determine how often the physicians used each skill that was mentioned in the CME activities. This evaluates the reliability of blinded coders based on 20 tapes. 

No No Inter-rater 

reliability 

Cc0ehsfd

Inter-rater reliability for coding 

KKf(Ks(

Beaulieu, 2002140

Standardized patient visits for the periodic health exam (male and female scenario), 2 scenarios (man / woman). 

No Yes 

(questions,

response 

options 

changed)

Degree of conformity of SP with scenario 

9f8
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-384 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Carney, 199588 Physicians' cancer-control clinical skills as measured 

through standardized patients after encounters.. Reliability was tested for standardized-patient accuracy in case replication and in recall of study variables - monitored by audiotaping of encounters through hidden recorders. Pilot tested in 3 practice sessions, videotaped; only SPs with >90% recall accuracy and maintenance of scenarios retained; 9 SPs used 

No No Inter-rater 

reliability 

Ki(1

Levinson, 1993138

RIAS content analysis of audiotaped patient encounters (measurement of communication skills behaviors); also 6-point scale for Global Affect Ratings for physicians and patients. Interrater reliability tested for 34 tapes, with random 2nd coder to check for coder decay and drift; interrater reliability for 17 components (others not frequent enough to do analysis). 

Unclear Yes (no 

changes)

 Inter-rater 

reliability 

PCca00PA0grs
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-385 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Gerrity, 199932 Standardized patients to assess physician 

communication on depression, completed form after visit. 

No Yes (no 

changes)

 Inter-rater 

reliability 

Sr

Gerrity, 199932

ABIM Patient Satisfaction Scale, 9-item, 5-point Likert scales, completed by standardized patients after visit 

No Yes 

(shorter version compared with citation) 

Validity: not specified 

 Previous 

study 

Reliability: not specified 

 

Stewart, 200543 Physician behavior in prevention and diabetes 

management, as assessed by standardized patient and checklists 

No Yes 

(unclear if 

changes 

made) 

Validity: not specified 

Self-reported Practice Behaviors with Knowledge and/or Attitudes

Face / content (Experts/ sources: literature review and workshop contents) 

NA Current 

study 

Test-retest reliability 

CcIcc0

 Equivalence 

reliability 

N

Gerstein, 199953

Survey about diabetes care knowledge, attitudes, and practice behaviors, as selfreported by primary care clinicians. 120 true/false items drafted / modified by endocrinologists, administered in two 58-question halves. 18 FPs tested in crossover design to test for testretest and equivalence. Translated to and from French translation. Attitude scales used a 5-point Likert scale. 

No Yes 

(unclear if 

changes 

made) 

External construct (known group) (Comparison: Administration to medical students, residents, family physicians, and diabetes specialists (endocrinologists)) 

ANOVA p=0.02 for attitude, p=0.0001 knowledge, p=0.012 practice 

Prior study 

Internal consistency (interitem) reliability 

Caak0
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-386 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

McBride, 2000144

Physician and staff questionnaires about use of prevention care documentation tools and services 

No No Concurrent 

Criterion (Comparison: physician and staff phone interviews for 22%) 

Statistic not reported 

Current study 

Schroy, 199982 Pre- and post-provider survey (mail or phone) to 

assess provider attitudes / practices in colorectal cancer screening. Based on ACS 1989 survey: Piloted to full-time providers within site. Yes/no and Likert responses. Actual utilization through appt logs / referrals. 

Yes Yes 

(unclear)

Concurrent validity Kappa: 0.67 

between selfreport and utilization (referrals) for sigmoidoscopy 

Current study 

Clark, 200078 Physician self-report of clinical behaviors related 

to asthma management 

No Yes (no 

changes)

Predictive Criterion (Comparison: parent description of physician behavior ) 

Statistic not reported 

Prior study 
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-387 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Terry, 198149

Self-assessment questionnaires about knowledge, attitudes, and clinical practice behaviors related to COPD. 44- multiple-choice questions constructed using "standard test development procedures"; 25 items on subsequent tests, chosen based on reliability coefficients 

No No Internal 

consistency (interitem) reliability 

Caot

Lane, 200145 Physician barriers to screening for breast 

cancer (CME need) based on skills, knowledge, and resources on a preintervention survey, with physicians categorized as high or low need for CME. Cronbach's alphas were calculated to determine inter-item reliability of the subscales of the test of CME intervention need. 

No Yes (no 

changes) 

 Internal 

consistency (interitem) reliability 

Ca(0aisrfefcn

Brown, 199972 Learner (clinician) questionnaire about 

dealing with patients and routine communication style (skills, attitudes, and behaviors in communication) 

No Yes 

(unclear if 

changes 

made) 

Validity: not specified 

 Previous 

study 
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-388 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Practice Behaviors

Face / content (Experts/ sources: Peer group of experienced clinicians and consultants) 

NA Current 

study 

Intra-rater reliability 

K

Sibley, 1982136

Abstraction of medical chart data to assess quality of documentation of clinical care processes for common medical conditions. Nurse abstractors trained and blinded; outcome was proportion of episodes of care classified as superior or adequate. 

No No 

 Inter-rater 

reliability 

KCa

Brown, 199972 Art of Medicine Survey, a patient questionnaire 

about clinician's communication behavior and patient's reaction to it. Being administered by contractor to HMO, but no published reference to development or use of questionnaire. Mailed survey, 8 items; survey shown to be correlated with observed and experimentally manipulated differences in communication performance 

No Yes (no 

changes)

Concurrent Criterion (Comparison: Satisfaction) 

Spearman Correlation coefficient: 0.92 

Current study 

Internal consistency (interitem) reliability C

c07 coq
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-389 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Ozer, 2005104 Adolescent Report of the Visit: an independent 

survey of provider screening behavior involving interviews by staff with patients / families after well visits. Distributed by research/clinic staff after MD visit, completed anonymously, yes/no responses. Included 45 items physician behavior; described as "valid indicator of delivery of services" with "construct validity." 

Unclear Yes 

(length 

changed) 

Construct validity Statistic not 

reported 

Prior study 

Schectman, 199616

Claims based prescribing data as a measure of practice behavior 

No No Concurrent 

Criterion (Comparison: Chart audit data) 

100% reliability for prescribing drug info, 92% for prescribing MD - % agreement? 

Current study 
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-390 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

 Inter-rater 

reliability 

K(oocsp

Socolar, 1998134 Chart audit of quality of physician documentation 

in evaluation for sexual abuse: rater scores of individual history and physical exam items documented. Up to 5 records from each MD assessed by two blinded reviewers; 3 reviewers total; 16 items assesed in each; if disagreed on 2 items or more by >=2 Likert scale, 3rd reviewer 

No Unclear 

 Test-retest 

reliability 

K(odt

Margolis, 2004118

Blinded chart review data abstraction to assess preventive services for children - interrater reliability assessed on random 20% of charts 

No Yes 

(unclear if 

changes 

made) 

 Inter-rater 

reliability 

Kafps

Fordis, 200542

Chart review to assess appropriate lipid screening and treatment in high-risk patients. Generalized Kappa averaged across dichotomously scored outcomes of primary interest (lipid screening and drug treatment); other outcomes for percent agreement. 

No Unclear Inter-rater 

reliability 

Pa9K<rs
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-391 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Schectman, 2003108

Chart audit assessment of guideline-consistent behavior for low back pain management: independent patient record review (100 records, oversampled for utilization events) by clinician investigators blinded to prior audit findings 

No No Inter-rater 

reliability 

KDtdcwrmioemcao

Jennett, 198876

Chart abstraction for practice behaviors related to 3 cardiovascular and 3 cancer topics. 2 part-time data abstractors reviewed records selected by appointment / billing records; intraconsistency checks for both abstractors 1 year following training period with still high level of consistency 

Yes Unclear Inter-rater 

reliability 

Su>
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-392 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

McBride, 2000144

Chart review for use of cardiovascular risk tools and documentation of cardiovascular risk factor assessment and management; blinded abstracters at baseline, but not at 12 or 18 months; 100% of charts reviewed at baseline; at 12 / 18 months, random 2nd reviews on 10% 

No No Inter-rater 

reliability 

Hsrsddaro1

White, 198548

Medical record audit for appropriate in hospital management of acute myocardial infarction. 1 trained abstractor on project staff; not aware of desired medical practices for most objectives; Iowa Foundation for Medical Care helped develop audit; 1 audit per objective; exclusion criteria specified for cases; rated on 3-point scale; field tested; reaudit time not specified 

Yes No Intra-rater 

reliability 

Pa8
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-393 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Rost, 200122

Services Assessment of Children and Adolescents: Patient selfreport telephone interview about pharmacotherapy; concordance of physician depression treatment behavior with AHCPR guidelines was measured. 

No Yes 

(added questions) 

 Test-retest 

reliability 

Sr

Moran, 1996121 College of Family Physicians of Canada 

Practice Assessment Program (PASS). Clinical patient care, charting, prevention, and drug use as measured by PASS scores in database; physician questionnaire, patient questionnaire, chart review based on consensus criteria. 

Yes Yes 

(unclear if 

changes 

made) 

Validity: not specified 

 Prior 

study 

Reliability: not specified 

 
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-394 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Clinical OutcomesMaiman, 198847

Assessment of medication compliance by mothers' self-reports, as compared with home visit liquid/pill assessment: number of missed doses, measured during home visit 8 days after visit with MD. 20-minute structured interview with mothers 

No Unclear Concurrent 

Criterion: (Comparison: mother self-report of missed doses in home interview vs. liquid / pill check vs. physician report 

Correlation coefficient: Liquid/pill vs. self-report of missed doses 0.241; liquid/pill vs. self-report 0.367; compared with pediatricians report p<=0.05 

Current study 

Kim, 1999137 Patient survey about physician provision of 

preventive services, validated by medical record review. Surveyed by mail or f/u telephone; Usually medical records did not support pt recall of offered services 

No No Concurrent 

Criterion: (Comparison: medical record review ) 

Percentage agreement: Disagreements 20-40% of time 

Current study 

Roter, 199584

General Health Questionnaire, 28-item, to detect psychiatric distress in patients. "Selfadministered screening test to detect psychiatric distress in community and nonpsychiatric medical settings." Cut-off for scores not specified. 

No Yes 

(unclear if 

changes 

made) 

 Internal 

consistency (interitem) reliability 

Ca
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-395 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Elliott, 199755 Brief Pain Inventory: measure of cancer 

patients pain over last 7 days using 4-items and 11-point Likert scale 

No Yes 

(unclear) 

 Internal 

consistency (interitem) reliability 

Ca

Elliott, 199755 Survey of patient and family knowledge (5-point 

scale) and attitudes (11-point scale) towards cancer pain management, modified from provider questionnaires - items for knowledge did not have adequate internal consistency (statistic not reported). 

Yes Yes 

(modified: 

unclear 

how) 

 Internal 

consistency (interitem) reliability 

C0af(ooskswaic

Gullion, 1988132

Medical record abstraction to assess provider behavior and clinical outcomes regarding appropriate medication and behavioral management of hypertension 

Yes No Inter-rater 

reliability 

Pa8aabrar

Kim, 1999137

Chart review for preventive care services provided 

No No Inter-rater 

reliability 

Pa1
Evidence table 16. Validity and/or reliability of tools to measure the effectiveness of continuing medical education 

F-396 

Author, year Evaluation method 

Pilot or cognitive testing 

Method previously used (If "yes," did method change)

Type of validity measured (comparison method) 

Statistic for validity measure 

Statistic measured for current or previous study 

Type of reliability measured 

Srm

Norris, 200081 Patient questionnaires regarding physical activity 

and health status: SF-36, Washburn's Physical Activity Scale for the Elderly (PASE), Paffenbarger's physical activity index; administered in-person at baseline and 6 months. 

No Yes 

(modified 

PASE 

subscales, 

unclear 

how) 

Validity: not specified 

 Previous 

study 

Reliability: not specified 

 

NA = not applicable; NR = not reported; EM = emergency medicine; Ped = pediatrics; ID = infectious disease; AAN = American Academy of Neurol

questionnaire; AHRQ = Agency for Healthcare Research and Policy; AHCPR = Agency for Health Care Policy and Research (now AHRQ); SP = stan

continuing medical education; RIAS = Roter Interactional Analysis System; KR = Kuder-Richardson; ANOVA = analysis of variance; COPD = chron

diseases; HMO = health maintenance organization; ABIM = American Board of Internal Medicine; EEG = electroencephalography 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-26

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Adams, 1998125

Overall: 21 

Physiciansin-training (residents or fellows), nurse practitioners 

M overall: 12 F overall: 17 

Overall: 37.8 

Internal medicine 

Learners’ graduation from medical school, overall: 9.3 

Equivalent University/ 

medical school faculty 

NR 

Allison, 2005147 NR None F study group 

range: (35.4-41.1) 

Study group range: 44.5-44.5 

Family medicine, general practice (NOS), internal medicine, pediatrics 

NR Equivalent Private 

practice, health plan 

NR 

Andersen, 199036

Overall: 41 (100) 

None NR NR Family 

medicine, general practice (NOS), internal medicine 

NR NR Private 

practice, health plan, county public health system 

NR 

Anderson, 1996103

Study group range: 15-17 (100%) 

None M study 

group range: 15-17 (100%) 

Study group range: 50-52 

General practice (NOS) 

Learners’ graduation from medical school, study group range: 24-26 

Equivalent NR External 

audits, participation was voluntarybut those invited were high prescribers of regulated drugs 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-27

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Beaulieu, 2002140

Overall: 108 (100) 

None M study 

group range: 19-38 (61-68) F study group range: 12-18 (32-39) 

Study group range: 41.6 

Family medicine 

Years of experience, study group range: (<11->20) 

Minor differences 

Private practice, community health center 

NR 

Beaulieu, 200444

Overall: 52 

Pharmacists NR NR General 

practice (NOS) 

NR NR NR Educational 

credit 

Bjornson, 199014

Overall: 576 (100%) 

None NR NR Family 

medicine, general practice (NOS), internal medicine 

NR Equivalent NR NR 

Block, 198837

Overall: 64 (100) 

None NR NR Family 

medicine, internal medicine 

NR NR Private 

practice 

NR 

Bloomfield, 200568

Overall: 92 (100) 

None NR NR Primary care 

physician (NOS) 

NR NR Hospital 

staff, military/ government 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-28

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Brown, 199972 Study group 

range: 19-25 (66-78) 

Physician assistants, nurse practitioners, optometrists 

M study group range: 19-20 (62-65) F study group range: 10-12 (35-38) 

Overall: (30 - 60) 

Obstetrics and gynecology, pediatrics, primary care physician (NOS), surgery, medical specialists 

NR Equivalent Health 

plan 

NR 

Brown, 2004135

Overall: 23 (100) 

None NR NR Pediatrics NR NR Private 

practice 

NR 

Browner, 1994122

Overall: (100) 

None M overall: 

(92) F overall: (8) 

NR Family 

medicine, general practice (NOS), internal medicine 

NR NR Private 

practice 

NR 

Bunting, 2004123

Overall: 200 (100) 

None M study 

group range: 65-66 (69-72) F study group range: 26-29 (28-31) 

NR Family 

medicine, general practice (NOS), internal medicine, obstetrics and gynecology, pediatrics, surgery, rheumatology, urology, nephrology, medical biochemistry 

Learners’ graduation from medical school, study group range: (less than 7 years before intervention->37 years before intervention) 

Equivalent NR NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-29

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Carney, 199588

98 (100%) 

None M overall: 

(92-97) F overall: (3-8) 

Study group range: 43.7-44.5 

Family medicine, internal medicine 

NR Minor 

differences 

Private practice 

NR 

Casebeer, 1999131

Overall: 28 (100) 

None NR NR Family 

medicine, general practice (NOS), internal medicine 

NR Equivalent Private 

practice 

NR 

Chan, 199921

Overall: 23 (100) Study group range: 11-12 

None M overall: 

(72) M study group range: (66.7-76.9) 

Overall: 42.2 Study group range: 40.4-43.8 

Family medicine 

Mean years in practice, overall: 13.5 study group range: 11.6-15.3 

Minor differences 

NR NR 

Chassin, 1986106

Overall: 1483 physicians (120 hospitals) (100) 

None NR NR Obstetrics 

and gynecology, all physicians with delivery privileges 

NR Large 

differences 

University/ medical school faculty, hospital staff 

Quality improvement 

Cherkin, 199183 NR Physician assistants M overall: 22 

F overall: 7 

NR Family 

medicine, internal medicine 

Median years of practice, study group range: 8-12 

NA Private 

practice, health plan 

Personal improvement, CME credit 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-30

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Chodosh, 200662

Study group range: 63-85 (88.5-90) 

Physician assistants, nurse practitioners 

M study group range: 40-55 (57.1-57.3) F study group range: 30-41 (42.7-42.9) 

NR Family 

medicine, internal medicine 

NR Equivalent Health 

plan 

NR 

Chung, 200454 Study group 

range: 15-21 (48-66) 

Physiciansin-training (residents or fellows) 

F overall: 29 

NR Emergency 

medicine 

NR NR Hospital 

staff 

NR 

Clark, 1998120

NR NR M overall: 

60 F overall: 40 

Study group range: (30-39: 22%, 40-49: 37%, 50-59: 27%, over 60: 14%) 

Pediatrics NR NR NR NR 

Clark, 200078

Overall: (100) 

NR M overall: 

(60) F overall: (40) 

Overall: (30-39: 22%, 40-49: 37%, 50-59: 27% over 60: 14%) 

Pediatrics NR Equivalent NR Personal 

improvement 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-31

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Cohn, 200219

Overall: 13-21 (77-86) 

NR M study 

group range: (48-54) F study group range: (46-52) 

NR Obstetrics 

and gynecology 

NR Minor 

differences 

NR NR 

Costanza, 199235

Overall: 116 (100) 

Support staff, radiologists, community women 

M overall: (80.4-82.8) F overall: (19.6-17.2) 

NR Family 

medicine, general practice (NOS), internal medicine, obstetrics and gynecology 

Learners’ graduation from medical school, study group: 34.9-36% up to 1965, 30-33.3% in 1966-76, 34-31.8% in 1977 or later 

Large differences 

Private practice, health plan 

NR 

Cummings, 1989112

Overall: 44 (100) 

NR M study 

group range: 16-22 (80-92) F study group range: 2-4 (8-20) 

NR Family 

medicine, internal medicine 

NR Minor 

differences 

Private practice 

San Francisco Medical Society endorsement, personal improvement, CME credit 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-32

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Cummings, 1989113

Overall: 81 (100) 

None M overall: 

58 (100) F overall: 23 (100) 

NR Internal 

medicine 

NR Equivalent Health 

plan 

San Francisco Medical Society endorsement, personal improvement, CME credit 

Cummings, 1989117

Overall: 59 (100) 

None M study 

group range: 16-22 (80-92) F study group range: 2-4 (8-20) 

NR Family 

medicine, internal medicine 

NR Minor 

differences 

Private practice 

NR 

Curran, 200034

Overall: 52 (100) 

None M overall: 

42 (81) F overall: 10 (19) 

NR Dermatology Learners’ 

completion of residency overall: (74.5% had 11+ years) 

Equivalent Private 

practice, health plan 

NR 

Davis, 2004101

Overall: 54 (100) 

None NR NR Pediatrics NR NR Health 

plan 

Quality improvement, CME credit, monetary gain to charity of choice 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-33

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Derebery, 2002153

Overall: 258 (100) 

None NR NR Occupational 

medicine physician 

NR NR Large 

occupational health group 

CME credit 

Des Marchais, 199061

Overall: 25 (100) 

None NR NR Family 

medicine, psychiatry 

NR NA University/ 

medical school faculty 

NR 

Dietrich, 2000110

NR Nurse 

practitioners 

M study group range: (50-75) F study group range: (25-50) 

Overall: 44 

Family medicine, internal medicine, pediatrics 

Overall: 10 years in current practice 

Minor differences 

NR NR 

Dormuth, 200415 Study group 

range: 241-258 (100) 

None M study 

group range: (83-89) 

Study group range: 45.6-46.2 

General practice (NOS) 

NR Equivalent Health 

plan 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-34

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Doucet, 199840

Overall: 87 (100) 

None M overall: 

35 M study group range: (26 - 51) F overall: 52 F study group range: (49 - 74) 

NR Primary care 

physician (NOS) 

NR Large 

differences 

NR NR 

Elliott, 199755

NR Nurses M overall: 

(90) F overall: (10) 

NR Family 

medicine, general practice (NOS), internal medicine, surgery 

Years in practice, overall: 15.3 

NR NR NR 

Evans CE, 198656

Overall: 76 (100) 

None NR NR Internal 

medicine, primary care physician (NOS) 

NR Equivalent Private 

practice 

NR 

Fordis, 200542 Study group 

range: 44-49 (100) 

None M study 

group range: 22-29 (50-59) F study group range: 20-22 (41-50) 

NR Family 

medicine, internal medicine, obstetrics and gynecology 

Learners’ graduation from medical school, median by study group: 17 years (1985), 13 years (1989) 

Equivalent Hospital 

staff 

Monetaryfinancial gain 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-35

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Frush, 200687

Overall: (70.5-74.4) 

Nurses, paramedics 

NR NR Emergency 

medicine, pediatrics 

Learners’ completion of residency, study group range: (5.7-5.8) 

Equivalent University/ 

medical school faculty, hospital staff 

NR 

Gerbert, 200286

Overall: 71 (100) 

None M overall: 

(56%) F overall: (44%) 

Overall: 40 

General practice (NOS), internal medicine, general internists 

NR Minor 

differences 

University/ medical school faculty, private practice, hospital staff, military/ government 

NR 

Gerrity, 199932

Overall: 49 (100) 

 M study 

group range: 14-15 (58-61) F study group range: 9-11 (39-42) 

Study group range: 45.4-47.7 

Family medicine, internal medicine, primary care physician (NOS) 

Learners’ graduation from medical school, study group range: 16.8-20 

Equivalent Private 

practice, solo, specialty group, hospitalaffiliated clinic 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-36

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Gerstein, 199953

Overall: 290 (100) Study group range: 113-177 (100) 

None M study 

group range: 82 - 107 (60.5 - 72.6) F study group range: 31 - 70 (27.4 - 39.5) 

NR Family 

medicine 

Learners’ graduation from medical school, overall: 16-21 

Large differences 

Private practice 

NR 

Gifford, 199646

Overall: 492 (100) 

None M study 

group range: (85-89) F study group range: (11-15) 

Study group range: 45.5-46.2 

Neurology Number of 

years in practice, overall: 13 

Equivalent University/ 

medical school faculty, private practice 

NR 

Gifford, 1999142

Overall: 417 (100) 

None M study 

group range: (78-85) F study group range: (15-22) 

Study group range: 46-47 

Neurology Years in 

practice, study group range: 11 - 14 

Equivalent University/ 

medical school faculty, private practice, hospital staff, health plan 

NR 

Goldberg, 200193

NR Administrative 

personnel 

NR NR Neurology, 

surgery, orthopedic surgeons 

NR Equivalent Private 

practice 

Quality improvement 

Goldstein, 2005146

Overall: 36 

Nurse practitioners 

NR NR NR NR Equivalent Military/ 

government 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-37

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Goldwater, 2001157

NR Nurses, 

pharmacists, hospital personnel 

NR NR NR NR NR Hospital 

staff 

NR 

Gonzales, 1999111

NR Physician 

assistants, nurse practitioners, nurses 

NR NR Family 

medicine, internal medicine 

NR Equivalent Health 

plan 

NR 

Grady, 199779

NR NR M overall: 

(88.4) F overall: (11.6) 

Overall: 46.76 

Family medicine, general practice (NOS), internal medicine 

Learners’ completion of residency overall: 15.22 (1-45) 

Large differences 

Private practice 

NR 

Greenberg, 198525

Overall: 23 (100) 

None NR NR Pediatrics NR NR Private 

practice 

NR 

Gullion, 1988132

Overall: 111 (100) 

None M overall: 

(96) F overall: (4) 

Overall: 46.3 

Family medicine, general practice (NOS), internal medicine 

NR NR NR NR 

Hagen, 2005126

Overall: 33 

Nurses, pharmacists, family members 

NR NR NR NR NR Long 

term care facility 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-38

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Harris, 200239

Overall: 65 (100) 

None M overall: 

(70) M study group range: (64.3-73) F overall: (30) F study group range: (27-35.4) 

Study group range: 42.6-43.7 

Emergency medicine, family medicine, internal medicine, pediatrics, orthopedics 

Learners’ graduation from medical school, study group range: 16.7-17.7 

Equivalent NR NR 

Harris, 200575 Study group 

range: 43-47 

None M study 

group range: 27-30 (93.1-93.8) 

NR Family 

medicine 

NR Equivalent NR NR 

Heale, 198866

Overall: 53 (100) 

None NR NR Family 

medicine 

Years in practice, overall: 10.2 

NR NR NR 

Herbert, 2004102

Overall: 200 (100) 

None NR NR Family 

medicine 

NR Equivalent Private 

practice 

Already belonged to PBSG learning program, personal improvement 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-39

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Hergenroeder, 200260

Overall: 75 (100) 

None M study 

group range: 20-23 (55-61) F study group range: 13-22 (39-45) 

NR Pediatrics Years after 

residency, study group range: 15.5-18 (1-48) 

Equivalent Health 

plan 

NR 

Howe, 1997145

NR NR NR NR NR NR NR NR NR 

Jennett, 198876

Overall: 31 (100) 

None M overall: 

25 F overall: 6 

Overall: (35-60) 

Family medicine 

Learners’ graduation from medical school, number in each decade: 5 -1950s, 6-1960s, 19- 1970s; all at least 2 years in active practice 

Equivalent Private 

practice 

NR 

Juzych, 200592 Study group 

range: 9-19 

Nurses, Pharmacists 

NR NR Internal 

medicine, pediatrics 

NR NR Health 

plan 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-40

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Kemper, 200658

Overall: 374 (29.5) 

Physician assistants, nurses, medical students, pharmacists, nutritionists, dietitians 

M overall: (25.3) 

Overall: 40.3 (27.4-53.2) 

NR NR Equivalent University/ 

medical school faculty 

NR 

Kiang, 200533 Study group 

range: 400-600 

Physician assistants, nurse practitioners 

M study group range: (51-61) 

Mean years in practice, study group range: 12.6-13.8 

Emergency medicine, family medicine, general practice (NOS), internal medicine, pediatrics 

NR Equivalent All types 

of practices 

Public demand/ patient expectations 

Kim, 1999137

Overall: 48 (100) 

None M study 

group range: (85-86%) 

Study group range: 42-45 

Family medicine, internal medicine, subspecialists with large proportion of primary care 

Overall: 10 years employed in health plan (mean) study group: range 4-32 years employed in health plan 

Equivalent Health 

plan 

NR 

Kottke, 1989114

Overall: NR 

None M overall: 

66 (86) F overall: 10 (14) 

Study group range: 37.9 - 44.3 

Family medicine 

NR Equivalent NR NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-41

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Kronick, 200371

Overall: 81 (100) 

None M study 

group range: 30-31 (75-75.6) F study group range: 10 (24.4-25) 

NR Family 

medicine 

Year starting first practice, study group range: 1980 - 1984 

Minor differences 

Private practice 

NR 

Kutcher, 200265

Overall: 68 

None M study 

group range: 16-19 (46-63) F study group range: 11-19 (37-54) 

Overall: (31 and up) 

Family medicine 

Duration of practice, study group range: (20-25) 

Minor differences 

Private practice 

Personal improvement, voluntary 

Labelle, 200467

Overall: 40 (100) 

None NR NR General 

practice (NOS) 

NR NR NR NR 

Lane, 199169

NR Medical 

students, other health professional students 

NR NR Family 

medicine, general practice (NOS), obstetrics and gynecology 

NR Equivalent Private 

practice, 14% not in private practice, but practice description not provided 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-42

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Lane, 200145

Overall: 282 (100) 

None M study 

group range: 106-135 (82.8-88.3) F study group range: 19-22 (11.7-17.2) 

Study group range by age category: </=44: 44-49 (33.3-36.7), >=65: 9-18 (7.5-12.2) 

Family medicine, internal medicine, obstetrics and gynecology 

Learners’ graduation from medical school, study group range: (21-30) 

Minor differences 

Private practice, hospital staff, public health center 

Quality improvement 

Leopold, 200580

Overall: 48 

Physician assistants, nurse practitioners, osteopathic physicians 

NR NR NR NR NR NR NR 

Levinson, 1993138

Overall: (100) 

NR M overall: 

44 F overall: 9 

NR Family 

medicine, internal medicine 

Learners’ graduation from medical school, overall: 15 (3-47) 

NR NR Reduced 

malpractice premium 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-43

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Lewis, 199370

Overall: 253 (100) 

None F overall: 

(37.1) 

Study group range: 37-47 

Family medicine, internal medicine, obstetrics and gynecology 

NR Large 

differences 

Health plan 

HMO mandate 

Lin, 199773 NR NR NR NR Family medicine, 

internal medicine 

NR NR Health 

plan 

NR 

Lin, 2001133

Overall: 109 (100) 

None M study 

group range: 40-44 (75.5-78.6) F study group range: 12-13 (21.4-24.5) 

NR Primary care 

physician (NOS) 

NR Minor 

differences 

Health plan 

NR 

LindsayMcIntyre, 1987115

Overall: 82 (100) 

None NR NR Family 

medicine 

Learners’ graduation from medical school, overall: (15-20) 

NR Private 

practice 

NR 

Lockyer, 200264

Overall: 637 (100) 

None NR NR Family 

medicine 

NR NR Private 

practice 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-44

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Maclure, 199896

Overall: (100) 

NR M overall: 

(76) F overall: (24) 

NR General 

practice (NOS) 

NR NR NR NR 

Macrae, 200485

Overall: 86 (100) 

None NR NR Surgery Learners’ 

graduation from medical school, overall: 23 

Equivalent University/ 

medical school faculty, private practice, hospital staff 

NR 

Maiman, 198847

Overall: 83 (100) 

None M overall: 

(80) F overall: (20) 

Overall: 46 

Pediatrics Learners’ 

completion of residency, overall: 14 years 

NR Private 

practice 

NR 

Mann, 199752

Overall: 59 (100) 

None M overall: 

49 F overall: 10 

Study group range: 42.4-45 

Family medicine 

NR Minor 

differences 

NR NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-45

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Margolis, 2004118

The mean (range) number of clinicians in the 22 intervention practices was 5.6 (1-12) and in the 22 control practices 4.4 (1-12) 

Nurses, clerical 

NR NR Family 

medicine, pediatrics 

NR Minor 

differences 

Private practice, hospital staff 

NR 

Maxwell, 198457

NR NR NR NR NR NR NR Hospital 

staff 

NR 

Mazmanian, 199824

Overall: (100) 

NR NR NR Primary care NR NR NR NR 

Mazmanian, 2001161

Overall: 88 (100) 

None M overall: 

76 (100) F overall: 16 (100) 

Overall: 42 

NR NR NR NR Personal 

improvement 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-46

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

McBride, 2000144

Overall: 160 (100) 

Physician assistants, nurse practitioners, nurses, pharmacists, medical technicians, nursing assistants 

M study group range: (81) F study group range: (19) 

NR Family 

medicine, general practice (NOS), internal medicine, geriatrician 

Years in practice, study group range: 12-14 

Equivalent Private 

practice, health plan 

NR 

McClellan, 2003119

Overall: 477 (100) 

None NR NR Family 

medicine, general practice (NOS), internal medicine, obstetrics and gynecology, endocrinology 

NR NR NR NR 

McMahon, 1988155

Overall: 34 (100) 

None NR NR Internal 

medicine 

NR NR University/ 

medical school faculty, private practice 

Hospital credentialing 

Mehler, 200599 NR NR NR NR Family medicine, 

internal medicine, academic 

NR NR Hospital 

staff 

Quality improvement 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-47

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Meredith, 200051

Overall: (86.2) study group range: (81.0-93.9) 

Physician assistants, nurse practitioners 

F overall: (35.6) F study group range: (30.6-41.4) 

Overall: 43.7 study group range: 43.1-44.0 

Family medicine, general practice (NOS), internal medicine 

Mean years from training, overall: 11.7 study group range: 11.1-12.0 

Equivalent Health 

plan 

NR 

Messina, 2002156

Overall: (100) 

None NR NR Primary care 

physician (NOS) 

NR NR NR NR 

Moran, 1996121

Overall: 15 (100) study group range: 5-10 (100) 

None M study 

group range: 5-6 (60-100) F study group range: 0-4 (0-40) 

NR Family 

medicine, general practice (NOS), the specialty of the experimental group was not specified although none were residency trained suggesting that they were GPs 

Learners’ graduation from medical school, all five of the experimental groups graduated between 1950-1960. Six of the ten control physicians graduated in the 1980s. 

Large differences 

Private practice 

Participation was voluntarybut intervention physiccians were invited because they were identified as having serious deficiencies in their practices 

Mukohara, 200513

Overall: 107 

None F study 

group range: (38.9-47.2) 

Study group range: 40.8-41.7 

General practice (NOS), internal medicine 

NR Equivalent University/ 

medical school faculty 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-48

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Myers, 2004148

Overall: 838 

NR NR NR Primary care 

physician (NOS) 

NR Not 

applicable 

Private practice 

NR 

Norris, 200081

Overall: 32 (100) 

None NR NR Primary care 

physician (NOS) 

NR Equivalent Health 

plan 

NR 

Ockene, 1996124

Overall: 45 (100) 

None M overall: 

30 (66.7) M study group range: 9-11 (59-79) F overall: 15 (33.3) F study group range: 3-7 (21-41) 

Overall: 38.1 study group range: 36.8-39.3 

Internal medicine 

Learners’ graduation from medical school, overall: 11.1 study group range: 9.9-12.9 

Minor differences 

Health plan 

NR 

Ozer, 2005104

Overall: 86 

Nurse practitioners 

F study group range: (62.2-64.1) 

Study group range: 41.4-44.1 

Pediatrics Learners’ 

graduation from medical school, study group range: 14-15 

Minor differences 

Hospital staff, health plan 

NR 

Pazirandeh, 2002128

Overall: 134 (100) 

None NR NR Family 

medicine, internal medicine, obstetrics and gynecology 

NR NR NR NR 

Pereles, 1996163

NR None NR NR NR NR NR NR NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-49

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Perera, 1983107

Overall: 26 (100) 

None NR Overall: 

(37.2-40.5) 

Primary care physician (NOS) 

NR Minor 

differences 

Health plan 

NR 

Pimlott, 2003143

Overall: 274 (100) 

None M study 

group range: 145-170 (82.5-86.3) F study group range: 23-36 (13.7-17.5) 

Study group range: 50.6-50.7 (32-81) 

Primary care physician (NOS) 

Learners’ graduation from medical school, study group range: 24.5 

Minor differences 

NR NR 

Pinto, 199874

Overall: (100) 

None M overall: 

26 F overall: 13 

Overall: 44.1 study group range: 43.7-44.6 

Family medicine, general practice (NOS), internal medicine 

Learners’ completion of residency, overall: 9.2 

Equivalent Private 

practice 

Monetary/ financial gain 

Premi J, 199338

Overall: 40 (100) 

None NR NR Military 

physicians (specialty not reported) 

Years in clinical practice, overall: 4.8 

NR Military/ 

government 

NR 

Premi, 199441

Overall: 152 (100) 

None NR NR Family 

medicine, general practice (NOS) 

NR NR NR NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-50

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Rabin, 199897

NR NR NR NR General 

practice (NOS), internal medicine, obstetrics and gynecology, primary care physician (NOS) 

NR NR NR NR 

Rahme, 2005100 Study group 

range: 29-84 

None NR NR General 

practice (NOS) 

NR NR Health 

plan 

Educational credit 

Ray, 198520

Overall: 372 (100) 

None NR NR NR NR NR Private 

practice 

Tennessee Medicaid program prescription behavior 

Ray, 2001109

Overall: 209 (100) 

None NR NR Primary care 

physician (NOS) 

NR Equivalent NR NR 

Rodney, 198677

Overall: 196 (100) 

None NR NR Family 

medicine, internal medicine, other sigmoidoscopists 

NR NR Private 

practice 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-51

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Rosenthal, 200559

Study group range: 1-12 

Nurses, clerical 

NR NR Family 

medicine, pediatrics 

NR Minor 

differences 

Private practice, hospital staff 

CME credit 

Rost, 200122

Overall: (66.6) 

Nurses NR NR Primary care 

physician (NOS) 

NR Equivalent Private 

practice 

NR 

Roter, 199584

Overall: 63 

Physiciansin-training (residents or fellows) 

M overall: 63 M study group range: 18-23 (82-96) F overall: 6 F study group range: 1-4 (4-18) 

Overall: 40.3 (27-67) study group range: 38.5-42.7 

Family medicine, internal medicine 

Years in practice, overall: 10.8 (1-39 years) study group range: 9.3 – 12.5 

Equivalent Private 

practice, hospital staff, health plan 

NR 

Schectman, 1991139

NR Physician 

assistants, nurse practitioners 

NR NR Internal 

medicine 

Experience, study group range: (7 -13) 

NR Health 

plan 

HMO mandate 

Schectman, 1995130

NR NR NR NR Family 

medicine, internal medicine 

NR Equivalent University/ 

medical school faculty, private practice, HMO practitioners 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-52

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Schectman, 199616

Study group range: 13-190 

Physician assistants, nurse practitioners 

NR NR Family 

medicine, internal medicine 

Learners’ completion of residency, study group range: (7 -10 years) 

Large differences 

Health plan 

Quality improvement 

Schectman, 2003108

Overall: 120 

Physician assistants, nurse practitioners 

M study group range: 7-14 (36-58) F study group range: 9-13 (42-64) 

NR Family 

medicine, internal medicine 

Years of practice, study group range: 8.7- 11.0 

Minor differences 

Health plan, group practices affiliated with notfor-profit group model HMOs 

NR 

Schroy, 199982 NR Physician assistants, 

nurse practitioners 

M study group range: 48 F study group range: 52 

Study group range: 36 - 38 

Primary care physician (NOS) 

NR Equivalent Primary 

care physicians at neighborhood health centers 

NR 

Schwartzberg, 199798

NR NR M overall: 

(78) F overall: (22) 

NR Family 

medicine, internal medicine, geriatrics 

NR NR NR NR 

Sharif, 200294 Study group 

range: 6-9 (100) 

None NR NR Pediatrics NR NR University/ 

medical school faculty 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-53

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Short, 200650 Study group 

range: 23-29 

None M study 

group range: (52-56) 

Overall: 47 

Family medicine, internal medicine, obstetrics and gynecology, pediatrics, psychiatry 

Learners’ completion of residency, overall: 17 years 

Equivalent Private 

practice 

CME credit, monetary/ financial gain 

Sibley, 1982136

Overall: 16 (100) 

None NR NR Family 

medicine 

NR Equivalent NR NR 

Slotnick, 199317

Overall: 33 (100) 

None M study 

group range: (66-100) F study group range: (0-34) 

Study group range: 43-47 

Family medicine, general practice (NOS), internal medicine 

NR NR NR NR 

Socolar, 1998134 NR None M study group 

range: (64-67) F study group range: (33-36) 

Overall: (<35: 12%, 35-50: 57%, >50: 31%) 

Family medicine, pediatrics, 5% specialty not specified 

NR Equivalent NR NR 

Solomon, 2004127

Overall: 32 (100) 

None F study 

group range: (30-45) 

NR Rheumatology Learners’ 

completion of fellowship, overall: 13 

Minor differences 

Private practice 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-54

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Soumerai, 1987105

Overall: 435 (100) 

None NR Overall: 

51 

Family medicine, general practice (NOS), internal medicine, 18% not specified 

NR Equivalent NR External 

audits 

Stein, 200195

Overall: 63 

Nurses, nursing home staff 

NR NR Primary care 

physician (NOS) 

NR NR NR Quality 

improvement 

Stewart, 200543

Overall: 58 

None M study 

group range: (63-71) 

NR Family 

medicine 

Learners’ graduation from medical school, study group range: 17.9-18.6 

Minor differences 

Private practice military/ government 

NR 

Stross, 1985116

Overall: (100) 

None NR NR Primary care 

physician (NOS) 

NR NR Private 

practice, hospital staff 

NR 

Terry, 198149

Overall: 44 (100) 

None NR NR Family 

medicine, general practice (NOS), internal medicine 

NR NR Rural 

western Pennsylvania 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-55

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Thom, 2000129

Overall: (100) 

 F study 

group range: 1-2 (10-20) 

Overall: 47 

Family medicine 

Years in practice, overall: (15-17) 

Equivalent NR NR 

Tziraki, 2000141

Overall: 616 (100) 

None M overall: 

510 (83) F overall: 106 (17) 

NR Family 

medicine, general practice, (NOS), internal medicine 

NR Minor 

differences 

Private practice 

NR 

Wells, 2000151

Overall: 114 (87) 

Nurse practitioners 

NR NR Family 

medicine, internal medicine 

NR NR Private 

practice, health plan, public health clinics 

NR 

White, 198548

Overall: 103 (100) study group range: 40-63 (100) 

None NR NR Family 

medicine, internal medicine 

NR NR Hospital 

staff 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-56

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

White, 200463

Overall: 52 

NR NR NR Family 

medicine 

NR NR Private 

practice, hospital staff, health plan, military/ government 

NR 

Wilson, 1988154

Overall: 83 

None M overall: 

74 M study group range: (81.5-93.1) F overall: 9 

Study group range: 40.5-41.77 

Family medicine 

% Graduated from medical school in past 15 years: 45-83 

Minor differences 

Private practice 

NR 

Winickoff, 198423

Overall: 16 (100) 

None NR NR Internal 

medicine 

NR NR Health 

plan 

Quality improvement 

Worrall, 1999152

Overall: 42 

None NR NR Family 

medicine 

NR NR Fee-forservice 

physiccians 

NR 
Evidence Table 2. Characteristics of study population and study setting in studies assessing the effectiveness of continuing m

F-57

Author, year 

N (%) of fully trained physicians 

Other health care professionals attending CME 

Gender, n (%) Males (M) Females (F) 

Mean age (age range) 

Specialty of learners 

Years from training, mean (range) 

Comparable groups 

Type of practice 

Motivating factors 

Zuckerman, 200418

Overall: (100) 

None NR NR Family 

medicine, general practice (NOS), internal medicine, primary care physician (NOS) 

NR NR NR NR 

CME = continuing medical education; GP = general practitioner; HMO = health maintenance organization; NCI = National Cancer Institute; NHLBI = 

Blood Institute; NOS = not otherwise specified; NR = not reported; PBSG = practice-based small-group; RWJ = Robert Wood Johnson Foundation 
Evidence table 3. Description of CME activity 

F-58 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Received formal intensive training 

sessions on counseling techniques and the approach to alcohol counseling as well as practice supports to promote screening and counseling, n assigned: 17 

Live Lecture, 

mentor/ preceptor, office cuing system 

Individuals T: 2.5 W: NR 

D: NR 

Multiple time or repetitive 

Adams, 32 months NR N1998125

Concurrent control 

Received usual alcohol counseling intervention with weekly conferences at which alcohol counseling was encouraged in one session over two years, n assigned: 12 

Live Lecture Individuals T: NR W: Once 

only D: NR 

NR 32 months NR N

Intervention Intervention group that received 

multicomponent Internet CME modules, n assigned: NR 

Computerbased offline Feedback, (chlamydia screening 

rates for the physician's entire office), readings 

Practice settings/ teams 

T: 0.2 per module W: NR D: 1 year 

Multiple time or repetitive 

Allison, NR NR Y2005147

Concurrent control 

Comparison group that received flat-text, Internet-based CME modules on women's health, n assigned: NR 

Computerbased offline Readings Practice setting/ teams T: NR W: NR D: 1 year Multiple time or repetitive NR NR Y
Evidence table 3. Description of CME activity 

F-59 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

 Primary care physicians receiving no training on affective and anxiety disorders in primary care, n assigned: NR 

NA NA NA T: NA 

W: NA D: NA 

Andersen, NA NA NR N199036

Intervention Primary care physicians receiving 

3.5 hour seminar on diagnosis, treatment, and referral of affective and anxiety disorders in primary care, n assigned: NR 

Live, video Case-based 

learning, lecture 

Individuals T: 3.5 W: Once 

only D: 3.5 hours 

One time 1-8 weeks Yes N

Concurrent control 

Control, n assigned: 18, n analysis: 15 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Intervention Education and notification, 

n assigned: 18, n analysis: 17 

Live, video Case-based 

learning, discussion group, lecture 

Individuals T: NR W: Once 

only D: 1 day 

One time 6 months NR Y

Anderson, 1996103

Concurrent control 

Notification only, n assigned: 18, n analysis: 17 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA Y
Evidence table 3. Description of CME activity 

F-60 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention "Exposed" physicians attended workshop 

incorporating features of effective educational interventions, n assigned: 59, n analysis: 31 

Live Case-based 

learning, problembased learning or team-based learning, readings, role play 

Individuals, practice setting/ teams 

T: 1.5 W: Once only D: NR 

Beaulieu, One time 4-6 months Yes N2002140

Concurrent control 

"Unexposed" physicians attended no such workshops until after the SP visit, n assigned: 49, n analysis: 56 

NA NA NA T: NA 

W: NA D: NA 

NA 4-6 months NA N
Evidence table 3. Description of CME activity 

F-61 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Workshop only, n assigned: NR Live Case-based learning, 

discussion group, problembased learning or team-based learning, readings 

Individuals, practice settings/ teams 

T: NR W: Once only D: NR 

One time 6 months Yes Y

Concurrent control 

Control group, n assigned: NR 

NA NA NA W: NA 

D: NA 

NA 6 months NA N

Intervention Workshop and a decision tree, 

n assigned: NR 

Live, print Case-based 

learning, discussion group, problembased learning or team-based learning, programmed learning, readings 

Individuals, practice setting/ teams 

T: NR W: Once only D: NR 

One time 6 months Yes Y

Beaulieu, 200444

Intervention Decision tree only, n assigned: NR Print Discussion group, programmed 

learning, readings 

Individuals T: NR W: Once 

only D: NR 

One time 6 months Yes Y
Evidence table 3. Description of CME activity 

F-62 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Received information packet including letter, 

copy of study, and questionnaire, n assigned: 288, n analysis: 141 

Print Readings Individuals T: NR W: NA 

D: NA 

Bjornson, One time 4 months NR N199014

Concurrent control 

Control, n assigned: 288, n analysis: 288 

NA NA NA T: NA 

W: NA D: NA 

NA 4 months NR N
Evidence table 3. Description of CME activity 

F-63 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Control hospital in comparison city (no provider or public education), n assigned: 38, n analysis: 23 

NA NA NA T: NA 

W: NA D: NA 

Block, NA NA NA N198837

Intervention Demonstration hospital (targeted with 

provider and public education), n assigned: 50, n analysis: 41 

Live, print Lecture, point 

of care 

Individuals T: NR W: NR 

D: NR 

Multiple time or repetitive 

NR NR Y
Evidence table 3. Description of CME activity 

F-64 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Multicomponent intervention plus 

patient letters, n assigned: NR 

Live, print, patient letter prompting patient to discuss with provider about letter contents (treatment) 

Discussion group, lecture, patient informs provider 

Individuals, practice setting/ teams 

T: 1 W: Once only D: 1 year 

One time NR Yes Y

Concurrent control 

Control, n assigned: NR 

NA NA NA T: NA 

W: NA 

NA NA NA N

Intervention Multicomponent intervention plus 

progress notes, n assigned: NR 

Live, print, progress notes reminding PCP about appropriate approach 

Discussion group, lecture, point of care 

Individuals, practice settings/ teams 

T: 1 W: Once only D: 1 year 

One time NR Yes Y

Bloomfield, 200568

Intervention Multicomponent intervention plus 

computer chart reminders, n assigned: NR 

Live, print, chart reminders appearing on cover page of patient's computerized medical record 

Discussion group, lecture, point of care 

Individuals, practice settings/ teams 

T: 1 W: Once only D: 1 year 

One time NR Yes Y
Evidence table 3. Description of CME activity 

F-65 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Clinicians who attended training 

program in communication skills immediately, n assigned: 37, n analysis: 30 

Live Clinical 

experiences, discussion group, lecture, role play, clinicians audio taped interaction with patients and listened between workshops 

Individuals T: 4 W: NR 

D: 10 hours 

Multiple time or repetitive 

Brown, >=6 months NR N199972

Concurrent control 

Clinicians who would receive training program on communication skills at a later time, n assigned: 32, n analysis: 29 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Concurrent control 

Control, n analysis: 11 

NA NA Individuals T: NA W: 1 

day/week D: NA 

Brown, NA 22 months NA N2004135

Intervention Interactive seminar, n analysis: 12 Live, video, print Case-based learning, 

demonstration, lecture, problembased learning or team-based learning, readings 

Individuals T: 4-6 W: 1 

day/week D: 2-3 weeks 

Multiple time or repetitive 

22 months NR U

a
Evidence table 3. Description of CME activity 

F-66 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Physician practices randomized to receive 

a standard 3 hour CME seminar and syllabus on cholesterol management, n assigned: 55, n analysis: 45 

Live, print Lecture, 

readings 

Individuals T: 3 hours W: Once 

only D: NR 

One time 18 months Yes N

Intervention Physician practices randomized to receive 

intensive CME on cholesterol management, n assigned: 57, n analysis: 47 

Live, print Academic 

detailing, case-based learning, lecture, point of care, readings 

Individuals, practice settings/ teams 

T: NR W: NR D: 3 months 

Multiple time or repetitive 

18 months Yes N

Browner, 1994122

Concurrent control 

Physician practices who received no CME, n assigned: 62, n analysis: 48 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Concurrent control 

Control group, n assigned: 100, n analysis: 98 

NA NA Individuals T: NA W: NA 

D: 2 years 

NA 2 years No U

a

Bunting, 2004123

Intervention Multifaceted education group (meeting + 

feedback), n assigned: 100, n analysis: 95 

Live, print Feedback, 

readings 

Individuals T: NR W: NA 

D: 2 years 

Multiple time or repetitive 

2 years No U

a
Evidence table 3. Description of CME activity 

F-67 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Did not receive training, n assigned: 32, n analysis: 32 

NA NA Individuals T: NA W: NA 

D: NA 

NA NA NA U

a

Carney, 199588

Intervention Received CME on cancer detection, 

n assigned: 25, n analysis: 25 

Live, video Demonstration, 

discussion group, feedback, lecture, programmed learning, role play 

Individuals T: 8 W: Once 

only D: NR 

One time 1 year NR U

a

Concurrent control 

No training on hypercholesterolemic management, n assigned: 14, n analysis: 12 

NA NA NA T: NA 

W: NA D: NA 

Casebeer, NA NA N1999131

Intervention Three interactive audio conferences 

plus chart reminders on managing hypercholesterolemic patients, n assigned: 14, n analysis: 14 

Audio, telephone conferences 

Case-based learning, discussion group, point of care 

Individuals T: NR W: NR 

D: NR 

Multiple time or repetitive 

9 month NR N
Evidence table 3. Description of CME activity 

F-68 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

No small group learning, but was provided with similar resources, n assigned: 12, n analysis: 11 

NA NA NA T: NA 

W: NA D: NA 

Chan, NA NA NA N199921

Intervention Internet based problem-based small 

group learning, n assigned: 11, n analysis: 8 

Internet, not real time 

Problembased learning or team-based learning 

Individuals T: NR W: NR 

D: 2 months 

Multiple time or repetitive 

2 months NR N

Intervention Educational interventions to 

reduce pelvimetry rates, n assigned: 64 hospitals, n analysis: 64 hospitals 

Live, print Feedback, 

lecture, 16 of 64 hospitals received only print mailings to physicians with privileges, without lecture 

Practice setting/ teams 

T: NR W: NR 

Chassin, One time 10 months Yes Y1986106

Concurrent control 

No interventions regarding pelvimetry rates, n assigned: 56 hospitals, n analysis: 56 hospitals 

NA NA NA T: NA 

W: NA 

NA NA NA N
Evidence table 3. Description of CME activity 

F-69 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention University physicians: received 2 lectures 

and 1 video, n assigned: 23, n analysis: 15 

Live, video Demonstration, 

discussion group, lecture

Individuals T: NR W: NR 

D: NR 

Multiple time or repetitive 

4 months Yes U

a

Cherkin, 199183

Intervention Community physicians--received 1 

workshop, n assigned: 17, n analysis: 14 

Live, video Demonstration, 

discussion group, lecture 

Individuals T: 1.5 W: Once 

only D: NR 

One time 1-2 months Yes U

c

Intervention Primary care providers at clinics 

randomized to a comprehensive management program for patients with dementia (ACCESS) and a provider education program, n analysis: 96 

Live, internet, not real time, print 

Discussion group, lecture 

Practice setting/ teams 

T: NR W: NR D: NR 

Multiple time or repetitive 

Chodosh, 9 months No Y200662

Concurrent control 

Primary care providers at clinics randomized to no management program or educational program, n analysis: 70 

NA NA Individuals, NA T: NA W: NA 

D: NA 

NA NA NA N
Evidence table 3. Description of CME activity 

F-70 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Web intervention group, 

n analysis: 31 

Live, internet, not real time 

Case-based learning, lecture, readings 

Individuals T: 1 lecture + website 

access W: NA D: 1 month 

Multiple time or repetitive 

Chung, 6 months No N200454

Concurrent control 

Control group, n analysis: 32 

Live Lecture Individuals T: 1 W: Once 

only D: 1 month 

One time 6 months NA N
Evidence table 3. Description of CME activity 

F-71 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

No educational intervention provided, n assigned: NR 

NA NA Individuals T: NA W: NA 

D: NA 

Clark, NA 22 months NA N1998120

Intervention Attended seminars on asthma management, 

n assigned: NR 

Live, video Case-based 

learning, clinical experiences, lecture 

Individuals T: 5 W: NR 

D: 2-3 weeks 

Multiple time or repetitive 

22 months NR N

Concurrent control 

Pediatricians NOT receiving CME intervention, n assigned: 36, n analysis: 33 

NA NA Individuals T: NA W: NA 

D: NA 

Clark, NA 2 years NR N200078

Intervention Pediatricians receiving CME intervention, 

n assigned: 38, n analysis: 34 

Live, video Case-based 

learning, demonstration, lecture 

Individuals T: 5 W: NR 

D: 2-3 weeks 

Multiple time or repetitive 

2 years NR N
Evidence table 3. Description of CME activity 

F-72 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Single intervention community, 

n assigned: 30, n analysis: 13 

Live, print Academic 

detailing 

Individuals T: NR W: NA 

D: NR 

Multiple time or repetitive 

3-6 months NR N

Intervention Double intervention community, 

n assigned: 29, n analysis: 21 

Live, print Academic 

detailing 

Individuals T: NR W: NA 

D: NR 

Multiple time or repetitive 

3-6 months NR N

Cohn, 200219

Concurrent control 

Control community, n assigned: 24, n analysis: 22 

NA NA NA T: NA 

W: NA D: NR 

NA 3-6 months NR N

Concurrent control 

Comparison, no intervention done, n assigned: 64, n analysis: 64 (pre) and 55 (post) 

NA NA NA T: NA 

W: NA D: NA 

Costanza, NA NA NA N199235

Intervention Intervention: received CME on breast cancer 

screening via both direct lectures, CBE practice sessions, and office reminder systems, n assigned: 52, n analysis: 52 (pre) and 45 (post) 

Live, print Discussion 

group, lecture, point of care, simulation (other than standardized patient or role-play) 

Individuals T: NR W: NR 

D: NR 

Multiple time or repetitive 

NR Yes N
Evidence table 3. Description of CME activity 

F-73 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Physicians participating in a 3-

part seminar, n assigned: 31, n analysis: 24 

Live, video, print 

Demonstration, discussion group, lecture, role play 

Individuals T: W: NR 

D: 5-14 weeks 

Multiple time or repetitive 

Cum- 1 year Yes Nmings, 1989112

Concurrent control 

Did not participate in seminar--control group, n assigned: 28, n analysis: 20 

NA NA Individuals, NA T: NA W: NR 

D: NA 

NA 1 year Yes N

Concurrent control 

Control group did not participate in the seminar, n assigned: 41, n analysis: 41 

NA NA Individuals, NA T: NA W: NR 

D: NA 

NA 1 year Yes U

a

Cummings, 1989113

Intervention Intervention group participated in a 3-part 

seminar, n assigned: 40, n analysis: 40 

Live, video, print 

Case-based learning, demonstration, discussion group, lecture, role play 

Individuals T: 3 W: NR 

D: 5-14 weeks 

Multiple time or repetitive 

1 Year Yes U

c
Evidence table 3. Description of CME activity 

F-74 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Physicians participating in 

seminars about smoking cessation counseling, n assigned: 31, n analysis: 24 

Live, video, patient education materials 

Case-based learning, demonstration, role play, selfreflection 

Individuals T: 3 W: 1 

day/week D: 5-14 weeks 

Multiple time or repetitive 

Cum- 1 year NR Nmings, 1989117

Concurrent control 

Physicians receiving no training, n assigned: 28, n analysis: 20 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Concurrent control 

No CME instruction/course, n assigned: 22 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Intervention Computer and web- based instructional 

course on dermatological office practices, n assigned: 16 

Computerbased offline, Internet, not real time 

Case-based learning, programmed learning, computerbased courseware 

Individuals T: NR W: NR 

D: NR 

Multiple time or repetitive 

NR Yes U

c

Curran, 200034

Intervention Computer and web- based instructional 

course on dermatological office practices, n assigned: 14 

Computerbased offline, internet, not real time 

Case-based learning, programmed learning, computerbased courseware 

Individuals T: NR W: NR 

D: NR 

One time NR Yes U

c
Evidence table 3. Description of CME activity 

F-75 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Primary care physicians who were 

part of the Dean Health System (Madison, WI), n assigned: 30, n analysis: 20 

Live, video, audio, audio teleconferencing 

Case-based learning, discussion group, problembased learning or team-based learning 

Individuals T: 2.25-3.00 W: 1 

day/week D: 3 weeks 

Multiple time or repetitive 

6 months Yes U

c

Davis, 2004101

Concurrent control 

Those primary care physicians who chose not to participate in the study- from the same clinics (Dean Health System) as the intervention group, n assigned: 34 

NA NA NA T: NA 

W: NA D: NA 

NA 6 months U

a
Evidence table 3. Description of CME activity 

F-76 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Physicians who did not participate in educational workshop and were given a copy of the low back pain manual with no specific instructions, n analysis: 151 

Print Readings Individuals T: NA W: NA 

D: NR 

Derebery, NA 1 year No N2002153

Intervention Physicians who participated in 

education intervention during CMC Professional Development Conference, n analysis: 107 

Live, print Demonstration, 

discussion group, lecture, problembased learning or team-based learning, readings, simulation (other than standardized patient or role-play) 

Individuals T: 10 (2 workshop+8 

reading) W: NA D: 

In addition to workshop, physicians received updated low back pain manual to read on their own within 2-3 months for additional CME credit 

1 year No N
Evidence table 3. Description of CME activity 

F-77 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Practical training in assessing 

interpersonal skills of their learners, followed by theorybased training, n assigned: 12 

Live, video Case-based 

learning, demonstration, discussion group, reflection about personal experiences 

Individuals T: NR W: NR 

D: NR 

Des NR NR NR NMarchais, 199061

Intervention Theory-based training in assessing 

interpersonal skills, followed by practical training, n assigned: 13 

Live, video Case-based 

learning, demonstration, discussion group, reflection about personal experiences 

Individuals T: NR W: NR 

D: NR 

NR NR NR U

c
Evidence table 3. Description of CME activity 

F-78 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Completed statewide survey about sun protection counseling and did not practice in intervention communities, n analysis: 12 

NA NA NA T: NA 

W: NA D: NA 

Dietrich, NA NA NA N2000110

Intervention Community-based and practice-based 

interventions to increase sun protection counseling,n analysis: 29 

Live, print, telephone call 

Academic detailing, lecture, problembased learning or team-based learning, readings 

Practice setting/ teams 

T: 30-45 minutes for first office meeting, otherwise NR W: NR D: NR 

Multiple time or repetitive 

about 1 year 

NR Y

Concurrent control 

Control group that received intervention 3-8 months later after the intervention group,n assigned: 241 

Print Readings Individuals, 

practice settings/ teams 

T: NR W: NR D: NR 

Multiple time or repetitive 

Dormuth, NR NR Y200415

Intervention Intervention group that received evidencebased drug therapy 

letters, n assigned: 258 

Print Readings Individuals, 

practice setting/ teams 

T: NR W: NR D: 3.17 years 

Multiple time or repetitive 

NR NR Y
Evidence table 3. Description of CME activity 

F-79 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Lecture group, n assigned: 49, n analysis: 29 for the pre- and post-tests; 26 for the KFP assessments; and 26 for the CME evaluation 

Live Discussion 

group, lecture 

Individuals T: 2 W: Once 

only D: NA 

One time 3 months Yes U

a

Doucet, 199840

Intervention PBL group, n assigned: 38, 

n analysis: 34 for the pre and post-tests; 21 for the KFP test; and 29 for the CME evaluation 

Live Problembased 

learning or team-based learning 

Individuals T: 6 W: 1 

day/week D: 3 weeks 

Multiple time or repetitive 

3 months Yes U

a
Evidence table 3. Description of CME activity 

F-80 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Extensive intervention including training 

opinion leaders on cancer pain management and then dissemination of information to their communities, n assigned: 3 communities 

Live, print Case-based 

learning, clinical experiences, discussion group, lecture, readings 

Practice settings/ teams 

T: NR W: NR D: 1 year 

Multiple time or repetitive 

Elliott, 15 months NR N199755

Concurrent control 

No intervention, n assigned: 3 communities 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Concurrent control 

No education or office materials about hypertension, n assigned: 35 physicians (29 practices) 

NA NA NA T: NA 

W: NA D: NA 

Evans NA 21 months NA NCE, 198656

Intervention 14 weekly information packets about 

hypertension diagnosis and management, as well as office materials, n assigned: 41 physicians (33 practices) 

Print Readings, 

chart cue materials offered, but not necessarily implemented 

Individuals, practice setting/ teams 

T: NR W: 1 day/week D: NR 

Multiple time or repetitive 

21 months NR U

a
Evidence table 3. Description of CME activity 

F-81 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Online CME about cholesterol 

management; 20 randomly selected for chart review in phase II, n assigned: 52, n analysis: 44 test (17 chart review) 

Internet, real time (e.g. streaming) internet, not real time, print, risk calculator 

Case-based learning, lecture, programmed learning, readings 

Individuals T: 3.8 W: NR 

D: 2 weeks 

Multiple time or repetitive 

5 months Yes Y

Intervention Live CME about cholesterol 

management (exists in phase I and phase II); 20 randomly selected for chart review in phase II, n assigned: 51, n analysis: 49 test (19 chart review) 

Live, print, risk calculator 

Case-based learning, lecture, programmed learning, readings 

Individuals T: 1.75 W: Once 

only D: 10 days 

One time 5 months Yes Y

Fordis, 200542

Concurrent control 

No training on cholesterol management, n assigned: 20, n analysis: 18 chart review 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Intervention Web-based education program, 

n analysis: 43 

Internet, not real time, video 

Lecture, simulated scenario 

Individuals T: NR W: NR 

D: NR 

education was available for 3 months 

Frush, NR No Y200687

Concurrent control 

Control, n analysis: 44 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N
Evidence table 3. Description of CME activity 

F-82 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Control group, n analysis: 32 

NA NA Individuals T: NA W: NA 

D: NA 

Gerbert, NA 8 weeks No N200286

Intervention Intervention group, n analysis: 39 Internet, not real 

time 

Case-based learning, feedback, programmed learning, readings 

Individuals T: NR W: NA 

D: NR 

Multiple time or repetitive 

8 weeks No N

Intervention Depression Education Program, two 4-hour 

sessions about recognition and management of depression and syllabus, n assigned: 27, n analysis: 23 

Live, video, audio, print 

Clinical experiences, discussion group, feedback, lecture, readings, role play 

Individuals T: 8 W: 1 

day/week D: 2 weeks 

Multiple time or repetitive 

Gerrity, 2-6 weeks NR N199932

Concurrent control 

No training on depression recognition and management, n assigned: 29, n analysis: 26 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Intervention Workshop on diabetes-specific 

clinical practice guidelines, n assigned: 1807, n analysis: 177 

Live Case-based 

learning, discussion group 

Individuals T: 7 W: Once 

only D: 1 day 

Gerstein, One time 24 months NR N199953

Concurrent control 

Control group that did not attend workshop, n analysis: 113 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N
Evidence table 3. Description of CME activity 

F-83 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Neurologists subscribing to AAN 

CME course, who received movement disorders module, n assigned: 248, n analysis: 176 

Video, print Programmed 

learning, readings 

Individuals T: 6.0 hours median 

W: NR D: NR 

Multiple time or repetitive 

Gifford, 4-5 months Yes N199646

Concurrent control 

Neurologists subscribing to AAN CME course, who did not receive the movement disorders module, n assigned: 244, n analysis: 183 

NA NA NA T: NA 

W: NA D: NA 

NA 4-5 months NA N
Evidence table 3. Description of CME activity 

F-84 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Neurologists who received mailed 

CONTINUUM Dementia Care Course, resource manual, seminar, and mailings: postintervention evaluation, n assigned: 139, n analysis: 95 

Live, print Case-based 

learning, discussion group, lecture, readings, mailings 

Individuals, practice settings/ teams 

T: 3 for seminar, otherwise NR W: Once only D: 4 months 

One time 6 months NR N

Historical control 

Baseline neurologists who do not receive teaching about dementia care: preintervention evaluation, n assigned: 139, n analysis: 108 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Gifford, 1999142

Concurrent control 

Control neurologists who received no training about dementia care: postintervention evaluation, n assigned: 139, n analysis: 102 

NA NA NA T: NA 

W: NA D: NA 

NA NA N
Evidence table 3. Description of CME activity 

F-85 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention 5 communities in W. Washington State with 

higher than average annual number of spine surgeries performed, n assigned: 5 

Live, video, print 

Academic detailing, discussion group, lecture 

Practice setting/ teams 

T: NR W: NR D: 30 months 

Multiple time or repetitive 

Goldberg, NA NR Y200193

Concurrent control 

5 communities in W. Washington State with higher than average annual number of spine surgeries performed, n assigned: 5 

NA NA NA T: NA 

W: NA D: 30 months 

NA NA NR N
Evidence table 3. Description of CME activity 

F-86 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Individualized- intervention group, 

n analysis: 20 

Live, print Discussion 

group, readings, packet which had patientspecific information form and advisory about guideline concordance of patient's antihypertensive drug regimen 

Individuals T: NR W: NA 

D: NR 

Received multiple items for 

Goldstein, NR NR Y2005146

Intervention General-intervention group, 

n analysis: 16 

Live, print Discussion 

group, readings, packet which had patientspecific information form 

Individuals T: NR W: NA 

D: NR 

Received multiple items for 

NR NR Y

Intervention Standard educational tools, 

n assigned: NR 

Print Academic 

detailing, point of care, readings 

Practice settings/ teams 

T: NA W: NR D: 14 months 

Multiple time or repetitive 

Goldwa- NR No Nter, 2001157

Intervention Therapeutic interchange, 

n assigned: NR 

Print Academic 

detailing, point of care 

Practice setting/ teams 

T: NA W: NR D: 14 months 

Multiple time or repetitive 

NR No N
Evidence table 3. Description of CME activity 

F-87 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Limited intervention, received household 

and office-based materials only, n assigned: 31 

Print Academic 

detailing, point of care 

Individuals, practice settings/ teams 

T: NA W: Once only D: NR 

NA 1 year NR U

a

Intervention Full intervention, received household 

and office-based materials and an educational intervention, n assigned: 28 

Live, print Academic 

detailing, demonstration, lecture, point of care 

Individuals, Practice setting/ teams 

T: 0.5 W: Once only D: 30 minutes 

One time 1 year NR U

a

Gonzales, 1999111

Concurrent control 

Control, n assigned: 34 

NA NA Individuals, 

practice settings/ teams 

T: NA W: NA D: NA 

NA NA NR U

a

Intervention Cue enhancement, education, and peer 

comparison feedback and rewards (money),n analysis: 20 

Live, print Feedback, 

lecture, point of care, readings, financial reward for compliance 

Individuals, practice settings/ teams 

T: NR W: NR D: 1 year 

Multiple time or repetitive 

NR NR N

Concurrent control 

Education only, n analysis: 23 

Live, print Lecture, 

readings 

Individuals, practice setting teams 

T: NR W: Once only D: NR 

One time NR NR N

Grady, 199779

Intervention Cue enhancement (posters, chart 

stickers) plus education, n analysis: 18 

Live, print Lecture, point 

of care, readings 

Individuals, practice settings/ teams 

T: NR W: NR D: 1 year 

Multiple time or repetitive 

NR NR N
Evidence table 3. Description of CME activity 

F-88 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Lecture-based education on 

headaches or behavior problems, n assigned: 26 

Live Lecture Individuals T: 1 W: Once 

only D: 1 hour 

Green- One time 6-9 months NR Nberg, 198525

Intervention Case-based presentations on 

enuresis or sleep disorders, n assigned: 22 

Live Case-based 

learning 

Individuals T: 1 W: Once 

only D: 1 hour 

One time 6-9 months NR N

Concurrent control 

No education about hypertension management, n assigned: 27, n analysis: 26 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Intervention Education about behavioral 

management for hypertension, n assigned: 28, n analysis: 27 

Print, telephone conference 

Discussion group, feedback, readings 

Individuals T: 1 hour call + reading 

time W: NR D: NR 

Multiple time or repetitive 

11 months NR N

Intervention Education about medication and 

behavioral management for hypertension, n assigned: 30, n analysis: 29 

Print, telephone conference 

Discussion group, feedback, readings 

Individuals T: 2 hours on call + reading 

time W: NR D: NR 

Multiple time or repetitive 

11 months NR N

Gullion, 1988132

Intervention Education about medication 

management for hypertension, n assigned: 27, n analysis: 25 

Print, telephone conference 

Discussion group, feedback, readings 

Individuals T: 1 hour call + reading 

time W: NR D: NR 

Multiple time or repetitive 

11 months NR N
Evidence table 3. Description of CME activity 

F-89 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Education for physicians, nursing 

staff, pharmacists, and family members, n assigned: NR 

Live, print, laminated copies to be posted at stations and in charts 

Lecture, readings 

Practice setting/ teams 

T: 1.75 W: NR D: NR 

Multiple time or repetitive 

Hagen, 6 months NR Y2005126

Concurrent control 

No education, n assigned: NR 

NA NA NA T: NA 

W: NA D: NA 

NA NA NR N

Intervention Intervention group participating in the 

online domestic violence education program, n assigned: 50, n analysis: 28 

Internet, not real time 

Case-based learning, feedback, readings 

Individuals T: NR W: NR 

D: 2 weeks 

Multiple time or repetitive 

Harris, 3 weeks Yes N200239

Concurrent control 

Control group not receiving education intervention about domestic violence. n assigned: 49, n analysis: 37 

NA NA NA T: NA 

W: NA D: NA 

NA 3 weeks NA N

Intervention Intervention group that received TED 

(teleconferenced education detailing), n assigned: 43, n analysis: 32 

Live, audio, print, teleconference 

Case-based learning, lecture, readings 

Individuals, practice setting/ teams 

T: 8 W: NR D: 8, one hour sessions 

Multiple time or repetitive 

Harris, 12 months Yes Y200575

Concurrent control 

Control group, n assigned: 47, n analysis: 29 

NA NA Individuals, 

practice settings/ teams 

T: NA W: NA D: NA 

NA NA NA N
Evidence table 3. Description of CME activity 

F-90 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Small group, problem based sessions for a 

one day CME, n analysis: 22 

Live Problembased 

learning or team-based learning 

Individuals T: NR W: Once 

only D: 1 day 

One time 7 months NR N

Intervention Large group, case problem discussion 

educational method for one day CME, n analysis: 12 

Live Case-based 

learning, discussion group 

Individuals T: NR W: Once 

only D: 1 day 

One time 7 months NR N

Heale, 198866

Intervention Traditional lecture educational method 

for 1 day CME, n analysis: 27 

Live Lecture Individuals T: NR W: Once 

only D: 1 day 

One time 7 months NR N
Evidence table 3. Description of CME activity 

F-91 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Control module and no portrait, n assigned: 56 

Live NA Individuals T: <2 W: Once 

only D: NR 

One time 6 months NA N

Intervention Control module and portrait, 

n assigned: 48 

Live Clinical 

experiences, feedback, readings 

Individuals T: <2 W: Once 

only D: NR 

One time 6 months NR U

a

Intervention Experimental module and portrait, 

n assigned: 49 

Live Case-based 

learning, clinical experiences, demonstration, discussion group, problem- or team-based learning, readings 

Individuals T: <2 W: Once 

only D: NR 

One time 6 months NR U

a

Herbert, 2004102

Intervention Experimental module and no portrait, 

n assigned: 47 

Live Case-based 

learning, clinical experiences, demonstration, discussion group, problem- or team-based learning 

Individuals T: <2 W: Once 

only D: NR 

One time 6 months NR U

a
Evidence table 3. Description of CME activity 

F-92 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Video + skills instruction intervention 

group, n assigned: 52, n analysis: 42 

Live, video, print 

Demonstration, feedback, simulation (other than standardized patient or role-play) 

Individuals T: NR W: Once 

only D: NR 

Hergen- One time NR No Nroeder, 200260

Intervention Video only intervention group, 

n assigned: 49, n analysis: 33 

Video, Print Demonstration Individu- als T: NR W: Once 

only D: NR 

One time NR No N
Evidence table 3. Description of CME activity 

F-93 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Patients diagnosed at 5 rural hospitals with 

intensive intervention: visits by urban oncologists, audit with feedback, physician reminder systems, n assigned: 201 patients (pre and post combined)/5 facilities 

NR Feedback, 

lecture, point of care, readings (the major difference between the groups was the intensity of the intervention) 

NR T: NR 

W: NR D: 2 years 

Multiple time or repetitive 

2 years NR N

Intervention Urban patients, received less 

intensive interventions, n assigned: 947 patients/4 facilities 

NR Feedback, 

point of care, readings 

NR T: NR 

W: NR D: 2 years 

NR 2 years NR N

Intervention Patients from rural areas but received 

care at urban centers which received only audit and feedback (less intensive) intervention, n assigned: 265 patients/4 facilities 

NR Feedback, 

point of care, readings 

NR T: NR 

W: NR D: 2 years 

NR 2 years NR N

Howe, 1997145

Intervention Patients diagnosed and treated in 

hospitals receiving only audit and feedback (less intensive) intervention,n assigned: 114 patients/4 facilities 

NR Feedback, 

point of care, readings 

NR T: NR 

W: NR D: 2 years 

Multiple time or repetitive 

2 years NR N
Evidence table 3. Description of CME activity 

F-94 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Cancer medicine CME, 

n assigned: 9, n analysis: 9 

Live, print, teleconference 

Discussion group, feedback, mentor/ preceptor, readings 

Individuals T: 6.8 W: NR 

D: 6-8 weeks 

Multiple time or repetitive 

12 months NR U

a

Concurrent control 

No CME, n assigned: 11, n analysis: 11 

NA NA NA T: NA 

W: NA D: NA 

NA 12 months NA N

Jennett, 198876

Intervention Cardiovascular CME, n assigned: 11, 

n analysis: 10 

Live, print, teleconference 

Discussion group, feedback, mentor/ preceptor, readings 

Individuals T: 7.7 hours W: NR 

D: 6-8 weeks 

Multiple time or repetitive 

12 months NR U

a

Concurrent control 

Control group, n assigned: 9, n analysis: 9 

NA NA NA T: NA 

W: NA D: NA 

Juzych, NA NA NA N200592

Intervention Educational intervention group, 

n assigned: 21, n analysis: 14 

Live, print Case-based 

learning, lecture, readings 

Individuals, practice setting/ teams 

T: NR W: NR D: NR 

Not perfectly clear, but seems to be a single half day session 

NR NR Y
Evidence table 3. Description of CME activity 

F-95 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Pull + drip, n assigned: 318, 

n analysis: 202 

Internet, not real time 

Discussion group, problembased learning or team-based learning 

Individuals T: NR W: NR 

D: NR 

Online discussions; casebased, selfinstructional modules 

NR Yes U

a

Intervention Push + drip, n assigned: 318, 

n analysis: 206 

Internet, not real time 

Discussion group, problembased learning or team-based learning 

Individuals T: NR W: NR 

D: NR 

Online discussions; casebased, selfinstructional modules 

NR Yes U

a

Intervention Pull + bolus, n assigned: 313, 

n analysis: 177 

Internet, not real time 

Discussion group, problembased learning or team-based learning 

Individuals T: NR W: NR 

D: NR 

Online discussions; casebased, selfinstructional modules 

NR Yes U

a

Kemper, 200658

Intervention Push + bolus, n assigned: 318, 

n analysis: 195 

Internet, not real time 

Discussion group, problembased learning or team-based learning 

Individuals T: NR W: NR 

D: NR 

Online discussions; casebased, selfinstructional modules 

NR Yes U

a
Evidence table 3. Description of CME activity 

F-96 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Minnesota group, 2002, n assigned: 400 

NA NA NA T: NA 

W: NA D: NA 

not clear NA NA N

Concurrent control 

Minnesota group, 1999, n assigned: 400 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Intervention Wisconsin group, 1999, 

n assigned: 400 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Kiang, 200533

Intervention Wisconsin group, 2002, 

n assigned: 600 

Many things were made available but it is not reported as to which groups used what methods. Options included live, regional meetings, CD-ROMs, mailings, grand rounds 

Not clear NR T: NR 

W: NR D: NR 

not clear NR NR Y
Evidence table 3. Description of CME activity 

F-97 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Educational materials about preventative care practices only, n assigned: 24 MDs, 905 patients, n analysis: 20 MDs 

Print Readings NA T: NA 

W: NR D: 1 year 

Multiple time or repetitive 

Kim, 2-2.5 years NR N1999137

Intervention Educational materials about preventative 

care practices, peercomparison feedback, and academic detailing from a pharmacist three times over a year, n assigned: 24 MDs, 905 patients, n analysis: 21 MDs 

Live, print Academic 

detailing feedback, readings 

Individuals T: NR W: NR 

D: 1 year 

Multiple time or repetitive 

2-2.5 years NR N

Intervention Only educational materials for patient 

distribution, n assigned: 22 

Print Readings Individuals T: NA W: NA 

D: NA 

NR 1 year NR U

c

Intervention Workshop training and educational materials 

for patient distribution,n assigned: 27 

Live, print Demonstration, 

discussion group, lecture, readings 

Individuals T: 6 W: Once 

only D: 6 hours 

One time 1 year NR U

c

Kottke, 1989114

Concurrent control 

Control, n assigned: 17 

NA NA NA T: NA 

W: NA D: NA 

NA 1 year NR U

c
Evidence table 3. Description of CME activity 

F-98 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Control physicians receiving no training, n assigned: 40, n analysis: 40 

NA NA NA T: NA 

W: NA D: NR 

Kronick, NA 3 months NA N200371

Intervention Individualized training sessions on evidencebased information 

retrieval by trained hospital librarians, n assigned: 41, n analysis: 30 

Live Case-based 

learning, demonstration 

Individuals T: 3 W: Once 

only D: NR 

One time 3 months NA N
Evidence table 3. Description of CME activity 

F-99 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention General educational intervention program 

(small group info format), n analysis: 30 

Live, print Discussion 

group, lecture, readings 

Individuals T: 1 W: Once 

only D: NR 

Kutcher, One time 6 months No N200265

Intervention Enhanced educational intervention program 

(small group info format), n analysis: 35 

Live, print Academic 

detailing, discussion group, lecture, readings 

Individuals T: 1.25 W: Once 

only D: NR 

One time 6 months No N

Intervention GPs attending first OSCE and workshop 

and participating in OSCE 12 months later, n assigned: 24, n analysis: 12 

Live, print Case-based 

learning, demonstration, discussion group, lecture, problembased learning or team-based learning, role play, standardized patient 

Individuals, practice setting/ teams 

T: NR W: Once only D: 12 months 

Multiple time or repetitive 

Labelle, 12 months NR N200467

Intervention GPs attending a 6 and 12 month OSCE, 

n assigned: 16, n analysis: 13 

Live Case-based 

learning, role play, standardized patient 

Individuals T: NR W: Once 

only D: 12 months 

Multiple time or repetitive 

12 months NR N
Evidence table 3. Description of CME activity 

F-100 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Multiple CME interventions and lowcost 

mammography, n analysis: NR 

Live, print ARS, clinical 

experiences, demonstration, discussion group, feedback, lecture, mentor/ preceptor, readings 

NR T: NR 

W: NR D: NR 

Multiple time or repetitive 

NR Yes N

Concurrent control 

No CME intervention and no mammography cost intervention, n analysis: NR 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Intervention Multiple CME interventions AND no 

cost intervention with mammography, n analysis: NR 

Live, print ARS, clinical 

experiences, demonstration, discussion group, feedback, lecture, mentor/ preceptor, readings 

NR T: NR 

W: NR D: NR 

Multiple time or repetitive 

NR Yes N

Lane, 199169

Intervention No CME intervention but free 

mammography, n analysis: NR 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N
Evidence table 3. Description of CME activity 

F-101 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention High-need physicians, as judged by their 

responses to a questionnaire on the need for breast cancer screening, received either an in-office or a self-learning intervention, n assigned: 201, n analysis: 128 

Live Lecture, 

standardized patient 

Individuals T: 1-3 W: Once 

only D: 1 day 

Lane, One time 6 months Yes Y200145

Concurrent control 

Control physicians were also designated as high-need and lowneed, but neither group received any CME intervention materials, n assigned: 223, n analysis: 154 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Intervention CD-ROM video demonstrating 

technique, n assigned: NR 

Computerbased offline Demonstra- tion, readings Individu- als T: NR W: NA D: NR NA NR NR U

a

Intervention Hands-on instruction by a trained tutor, 

n assigned: NR 

Live Demonstration, feedback,

mentor/ preceptor, simulation (other than SPor role-play) 

Individuals T: 0.08-0.17 W: Once 

only D: NR 

One time NR NR U

a

Leopold, 200580

Intervention Printed guide, n assigned: NR Print Readings Individu- als T: NR W: NA 

D: NR 

NA NR NR U

a
Evidence table 3. Description of CME activity 

F-102 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Control physicians not receiving CME, n assigned: 15, n analysis: 15 

NA NA NA T: NA 

W: NA D: NA 

Levinson, NA 1 month NA N1993138

Intervention Physicians attending a one time short CME 

program on communication skills, n assigned: 16, n analysis: 15 

Live Case-based 

learning, lecture 

Individuals T: 4.5 hours W: Once 

only D: NR 

One time 1 month Yes N

Intervention Comparison, received only written 

information, n assigned: 106 

Print Readings NR T: NR 

W: NR D: NR 

One time 3 months NR N

Concurrent control 

Control, n assigned: 93 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Lewis, 199370

Intervention Experimental, received live CME 

program on HIV and taking a sexual history, n assigned: 54 

Live, video Discussion 

group, lecture 

NR T: NR 

W: Once only D: 1 day 

One time 3 months NR N
Evidence table 3. Description of CME activity 

F-103 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Multifaceted educational 

intervention and collaborative treatment model, n assigned: 134 preintervention/109 postintervention, n analysis: 83 preintervention/65 postintervention 

Live, video, print 

Academic detailing, feedback, lecture, readings, role play 

Practice settings/ teams 

T: NR W: NR D: 1 year 

Multiple time or repetitive 

6 months NR U

a

Lin, 199773

Concurrent control 

Control practices (usual care), n assigned: NR, n analysis: NR 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Intervention Intervention group, n analysis: 56 Live Academic detailing, 

case-based learning, demonstration, discussion group, role play 

Individuals, practice setting/ teams 

T: NR W: NR D: 3 months 

Multiple time or repetitive 

1 year NR U

a

Lin, 2001133

Concurrent control 

Usual care group, n analysis: 53 

Live, NA NA NA T: NR 

W: NA D: NR 

Multiple time or repetitive 

1 year NR U

a
Evidence table 3. Description of CME activity 

F-104 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Family physicians providing usual care, n assigned: NR 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Intervention Family physicians with counseling training, 

nicotine gum, chart cue, intervention flowsheet, and patient self-help materials, n assigned: NR 

Live, video, print 

Demonstration, lecture, point of care, standardized patient 

Individuals T: 3-4 hour training 

session W: NR D: 2 months 

Training 1 time, cueing with each patient 

1 year NR N

LindsayMcIntyre, 1987115

Intervention Family physicians with nicotine gum and 

chart cue, n assigned: NR 

Print Point of care Individuals T: NR W: NR 

D: 2 months 

Multiple time or repetitive 

1 year NR N

Intervention Introductory course (track 1) on 

Alzheimer's disease, n analysis: NR 

Live Case-based 

learning, discussion group, lecture, role play 

Individuals T: 6.5 W: Once 

only D: 1 day 

Lockyer, One time 3 months Yes N200264

Intervention Advanced course (track 2) on 

Alzheimer's disease, n analysis: NR 

Live Case-based 

learning, discussion group, lecture 

Individuals T: 6.5 W: Once 

only D: 1 day 

One time 3 months Yes N
Evidence table 3. Description of CME activity 

F-105 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention 1 hour teleconference participants, 

n assigned: 94 

Live, video Lecture, 

teleconference 

Individuals T: NR W: NR 

D: NR 

One time 3 months NR N

Concurrent control 

Matched control for the teleconference group, n assigned: 188 

NA NA NA T: NA 

W: NA D: NA 

NA 3 months NA N

Intervention Small group workshops, 

n assigned: 70 

Live Discussion 

group, lecture 

Individuals T: NR W: NR 

D: NR 

NR 3 months NR N

Concurrent control 

Matched control for the small group workshop group, n assigned: 140 

NA NA NA T: NA 

W: NA D: NA 

NA 3 months NA N

Intervention Physicians receiving 2 newsletters, 

n assigned: 258 

Print Readings Individuals T: NR W: NR 

D: NR 

Multiple time or repetitive 

3 months NR N

Maclure, 199896

Concurrent control 

Control group for newsletter, n assigned: 241 

NA NA NA T: NA 

W: NA D: NA 

NA 3 months NA N
Evidence table 3. Description of CME activity 

F-106 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Intervention group participating in 

electronic "journal club" & receiving 8 packages, each consisting of 1 clinical and 1 methodological article along with questions designed to guide critical appraisal, n assigned: 44, n analysis: 26 

Print Readings, 

listserv discussion group with moderator 

Individuals T: NR W: NR 

D: 8 months 

Multiple time or repetitive 

Macrae, 6 weeks Yes N200485

Concurrent control 

Control group received only the 8 clinical articles, n assigned: 37, n analysis: 29 

Print Readings Individuals T: NR W: NR 

D: 8 months 

Multiple time or repetitive 

6 weeks NR N

Intervention 2 session didactics and printed materials, 

n assigned: 33, n analysis: 32 

Live, print Discussion 

group, lecture, readings 

Individuals T: 5 hours W: NR 

D: NR 

Multiple time or repetitive 

6 months NR N

Concurrent control 

No additional education, n assigned: 27, n analysis: 24 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Maiman, 198847

Intervention Mailed printed materials, 

n assigned: 30, n analysis: 27 

Print Readings Individuals T: NR W: NR 

D: NR 

One time 6 months NR N
Evidence table 3. Description of CME activity 

F-107 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Training workshop on cholesterol-lowering 

practices with FP, dietician, educational specialist, and internist, n analysis: 17 

Live, video, print 

Case-based learning, demonstration, discussion group, readings, simulation (other than standardized patient or role-play) 

Individuals T: NR W: NR 

D: NR 

Multiple time or repetitive 

15 months NR N

Concurrent control 

No workshop or chart cues, n analysis: 15 

NA NA NA T: NA 

W: NA D: NA 

NR 15 months NR N

Mann, 199752

Intervention Training workshop plus chart cues 

cholesterol-lowering interventions for patients in their clinical practice, n analysis: 19 

Live, video, print, cue stickers on medical chart 

Case-based learning, demonstration, discussion group, point of care, readings, simulation (other than standardized patient or role-play) 

Individuals T: NR W: NR 

D: NR 

Multiple time or repetitive 

15 months NR N
Evidence table 3. Description of CME activity 

F-108 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Control, n analysis: 22 

NA NA NA T: NA 

W: NA D: NA 

Margolis, NA NA No N2004118

Intervention Plan-do-study-act process improvement,

n analysis: 22 

Live, organized set of tools otherwise unspecified 

Academic detailing, feedback, lecture, unspecified organized set of tools 

Practice setting/ teams 

T: NR W: NR D: 12 months 

Multiple time or repetitive 

18 months Yes Y

Concurrent control 

Physicians matched by department chairman to be similar in type and length of practice to committee members, n assigned: NR 

NA NA NA T: NA 

W: NR D: NA 

Maxwell, NA NR NA N198457

Intervention Physician members of three evaluation 

committees at a hospital, n assigned: NR 

Live Case-based 

learning, discussion group 

Practice setting/ teams 

T: NR W: NR D: 1 year 

Multiple time or repetitive 

NR NR Y
Evidence table 3. Description of CME activity 

F-109 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Received lecture only,n assigned: 13 medical schools (340 physicians), n analysis: 13 schools (153 physicians) 

Live Lecture Individuals T: 1 W: Once 

only D: 1 hour 

Mazmani One time 45 days NR Nan, 199824

Intervention Received lecture as well as information 

regarding barriers to changing practice, n assigned: 12 medical schools (398 physicians), n analysis: 12 schools (146 physicians) 

Live Lecture Individuals T: 1 W: Once 

only D: 1 hour 

One time 45 days NR N

Concurrent control 

This group's questionnaires included a line on which they could sign indicating their commitment to change practice behaviors, n assigned: 55, n analysis: 43 

Live Discussion 

group, lecture 

Individuals T: NR W: Once 

only D: 1 day 

Mazmani One time 3 months Yes Nan, 2001161

Intervention This group's questionnaires did not 

include a signature line, n assigned: 55, n analysis: 45 

Live Discussion 

group, lecture 

Individuals T: NR W: Once 

only D: 1 day 

One time 3 months Yes N
Evidence table 3. Description of CME activity 

F-110 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Consultation group: primary care practices 

receiving 1-day conference and HEART Kits to improve prevention care systems PLUS 3 consultation meetings and 2 reinforcement visits during 1 year, n assigned: 11 practices 

Live, print Academic 

detailing, discussion group, feedback, lecture, point of care, problembased learning or team-based learning 

Practice setting/ teams 

T: NR W: NR D: 1 year 

Multiple time or repetitive 

6 months NR Y

Intervention Prevention coordinator group: 

primary care practices receiving 1-day conference and HEART Kits PLUS prevention coordinator working 4.5 hours per week to promote practice systems improvements for prevention care, n assigned: 11 practices 

Live, print Lecture, point 

of care, problembased learning or team-based learning, patient education 

Practice settings/ teams 

T: NR W: NR D: 1 year 

Multiple time or repetitive 

6 months NR Y

McBride, 2000144

Concurrent control 

Conference-only group: primary care practices receiving 1-day conference and HEART Kits to improve prevention care systems, n assigned: 12 practices 

Live, print Lecture, point 

of care 

Practice settings/ teams 

T: NR W: NR D: 1 day 

One time 18 months NR Y
Evidence table 3. Description of CME activity 

F-111 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Combined intervention: primary 

care practices receiving 1-day conference, HEART Kits, 3 consultation meetings, 2 reinforcement visits, and a prevention coordinator, n assigned: 11 practices 

Live, print Academic 

detailing, discussion group, feedback, lecture, problembased learning or team-based learning, patient education 

Practice settings/ teams 

T: NR W: NR D: 1 year 

Multiple time or repetitive 

6 months NR Y

Intervention Intervention physicians received 

material at baseline, 2, 4 and 6 months, n assigned: 247, n analysis: 236 

Video, print Feedback, 

readings 

Individuals T: NR W: NR 

D: 6 months 

Multiple time or repetitive 

McClellan, 1.5-2 years NR Y2003119

Concurrent control 

Comparison physicians received no materials. n assigned: 230, n analysis: 223 

NA NA Individuals T: NA W: NA 

D: NA 

NA 1.5-2yrs NR Y
Evidence table 3. Description of CME activity 

F-112 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Internists on another IM service in the hospital with a high LOS, n assigned: NR 

Live Discussion 

group, lecture 

Individuals T: NR W: Once 

only D: NR 

One time 16 months NR Y

Intervention Internists admitting to 3 hospital services 

with high LOS, n assigned: NR 

Live Discussion 

group, feedback, lecture 

Individuals T: 1 hour W: Once 

only D: NA 

Multiple time or repetitive 

16 months NA Y

Concurrent control 

Physicians on other units; received no intervention, n assigned: NR 

NA NA NA T: NA 

W: NA D: NA 

NA 16 months NA N

McMahon, 1988155

Intervention Internists admitting to 4 services with high 

LOS, n assigned: NR 

Live Discussion 

group, lecture 

Individuals T: NR W: Once 

only D: NR 

One time 16 months NR Y

Intervention Electronic detailing intervention (using 

facsimiles and emails), n assigned: 415, n analysis: 415 

Internet, not real time, print 

Academic detailing, readings 

Practice setting/ teams 

T: NR W: NR D: NR 

Multiple time or repetitive 

3 months NR Y

Intervention Direct detailing intervention (using 

face-to-face education sessions), n assigned: 146 

Live, print Academic 

detailing, lecture 

Practice settings/ teams 

T: NR W: NR D: NR 

Multiple time or repetitive 

3 months NR Y

Mehler, 200599

Concurrent control 

Control, n assigned: 323 

NA NA NA T: NA 

W: NA D: NA 

NA 3 months NA N
Evidence table 3. Description of CME activity 

F-113 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Physician education & QI therapy 

intervention (reduced copayments for use of practice psychotherapists), n assigned: 58 

Live, print Lecture, 

readings 

Practice settings/ teams 

T: NR W: NA D: 6 months 

NA 12 months 

(18 months after implementation) 

NR Y

Intervention Physician education & QI meds intervention 

(improved resources for antidepressant medication management), n assigned: 49 

Live, print Lecture, 

readings 

Practice setting/ teams 

T: NR W: NA D: 6 months 

Multiple time or repetitive 

12 months (18 months after implementa-tion) 

NR Y

Meredith, 200051

Concurrent control 

Usual care, n assigned: 53 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N
Evidence table 3. Description of CME activity 

F-114 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Physician CME & no patient training, 

n assigned: 342, n analysis: 154 

Live, print Demonstration, 

readings, standardized patient 

Individuals T: NR W: NA 

D: NR 

Multiple time or repetitive 

3 years NR U

a

Concurrent control 

No training: control physicians, control patients, n assigned: 489, n analysis: 115 

NA NA NA T: NA 

W: NA D: NA 

NA 3 years NR U

a

Intervention Physician CME (nurse educator training + SP 

visit) & patients with BSTC, n assigned: 335, n analysis: 149 

Live, audio, print 

Demonstration, readings, standardized patient, telephone counseling 

Individuals T: NR W: NA 

D: NR 

Multiple time or repetitive 

3 years NR U

a

Messina, 2002156

Concurrent control 

No physician CME, & patient BSTC only, n assigned: 435, n analysis: 92 

Live, audio Readings, 

telephone counseling 

Individuals T: NR W: NA 

D: NR 

Multiple time or repetitive 

3 years NR U

a

Intervention Cases - in need of remedial CME as 

identified by the Manitoba College of Physicians, n assigned: 5 

Live Case-based 

learning, discussion group 

Individuals T: 30 W: NR 

D: 10 days 

Multiple time or repetitive 

Moran, 18 months NR N1996121

Concurrent control 

Controls, n assigned: 10 

Live Case-based 

learning, discussion group 

Individuals T: 30 W: NR 

D: 10 days 

Multiple time or repetitive 

NR NR N
Evidence table 3. Description of CME activity 

F-115 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Received healthrelated news from email from Yahoo! Health, n assigned: 53, n analysis: 45 

Computerbased offline Readings Individu- als T: NR W: Once only 

D: NR 

Multiple time or repetitive 

Mukohara, NR NR N200513

Intervention Received WBJC readings which had 

summaries of 1 or 2 articles from medical/health journals, n assigned: 54, n analysis: 51 

Computerbased offline Readings Individu- als T: NR W: Once only 

D: NR 

Multiple time or repetitive 

NR NR N

Concurrent control 

Control group, n analysis: 198 practices 

NA Readings Individuals T: NR W: NA 

D: NA 

Myers, NA 6 years NR Y2004148

Intervention Physician-oriented reminder-feedback 

and educational outreach intervention, n analysis: 120 practices 

Live, audio, print 

Academic detailing, feedback, lecture, readings 

Individuals T: NR W: NA 

D: 5 years 

Multiple time or repetitive 

6 years NR Y
Evidence table 3. Description of CME activity 

F-116 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

No training on physical activity counseling, n assigned: 17 MDs, 463 patients, n analysis: 460 patients 

NA NA NA T: NA 

W: NA D: NA 

Norris, NA NA NA N200081

Intervention 1-hour workshop on Physician-Based 

Assessment and Counseling for Exercise, a behaviorbased tool for PCPs counseling healthy adults, n assigned: 15 MDs, 384 patients, n analysis: 362 patients 

Live, print, follow-up phone calls about protocol 

Lecture, point of care, opinionleader from clinic teaching 

Practice setting/ teams 

T: 1 W: NR D: NR 

Multiple time or repetitive 

6 months NR N

Intervention Lipid intervention physician training, 

n assigned: 17, n analysis: 115 

Live, video Feedback, 

lecture, role play, standardized patient 

Individuals T: 3 W: 1 

day/week D: 2 weeks 

2 types of training sessions, each provided once 

Evaluation was conducted over a 2 year time span 

NR N

Concurrent control 

Usual care, n assigned: 14, n analysis: 92 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Ockene, 1996124

Intervention Lipid intervention physician training plus 

practice management,n assigned: 14, n analysis: 118 

Live, video Feedback, 

lecture, role play, standardized patient 

Individuals, practice settings/ teams 

T: 3 W: 1 day/week D: 2 weeks 

2 types of training sessions, each provided once 

Evaluation was conducted over a 2 year time span 

NR Y
Evidence table 3. Description of CME activity 

F-117 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Comparison group, n assigned: 44, n analysis: 37 

NA NA NA T: NA 

W: NA D: NA 

Ozer, NA NA NA N2005104

Intervention Intervention group, n assigned: 42, 

n analysis: 39 

Live, print Discussion 

group, lecture, role play, screening and charting tools 

Practice setting/ teams 

T: 8 W: Once only D: NR 

One time 8 months NR Y

Intervention Intervention group receiving didactic 

lectures on the results of screening and the effect of prudent management on osteoporosis outcome, n analysis: 53 

Live, print Lecture, 

readings, Q & A period 

Individuals T: NR W: Once 

only D: NR 

One time 6 months No U

a

Pazirandeh, 2002128

Concurrent control 

Control group receiving no intervention, n analysis: 81 

NA NA Individuals T: NA W: NA 

D: NR 

NA 6 months No U

c
Evidence table 3. Description of CME activity 

F-118 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Wrote commitment to change after CME 

course, n assigned: 7 

Live NR Individuals T: NR W: NR 

D: NR 

NR 3 months NR U

c

Pereles, 1996163

Concurrent control 

CME course only, n assigned: 9 

Live NR Individuals T: NR W: NR 

D: NR 

NR 3 months NR U

c

Intervention Primary care physicians enrolled in 

a preceptorship for sigmoidoscopy training, 2nd training, n assigned: 13 

Live Clinical 

experiences, demonstration 

Individuals T: 8 W: 2 

days/week D: 2 days 

One time 3 months Yes N

Concurrent control 

Primary care physicians who did not sign up for sigmoidoscopy training, n assigned: 74 

NA NA NA W: NA NA NA NA N

Perera, 1983107

Intervention Primary care physicians enrolled in 

a preceptorship for sigmoidoscopy training, 1st training, n assigned: 13 

Live Clinical 

experiences, demonstration 

Individuals T: 8 W: 2 

days/week D: 2 days 

One time 7-10 

months 

Yes N
Evidence table 3. Description of CME activity 

F-119 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Control group receiving mailed packages of feedback education materials on antihypertensives, n analysis: 206 

Print Feedback, 

readings 

Individuals T: NR W: NA 

D: 6 months 

Multiple time or repetitive 

6 months NR U

c

Pimlott, 2003143

Intervention Intervention group receiving mailed 

packages of feedback about participants' prescribing (of benzodiazepines) and evidence-based education materials, n analysis: 168 

Print Feedback, 

readings 

Individuals T: NR W: NA 

D: 6 months 

Multiple time or repetitive 

6 months NR U

c

Concurrent control 

Physicians and patients were given a manual about exercise counseling but no formal training or instruction, n assigned: 17, n analysis: 15 

Print Readings Individuals T: NR W: NR 

D: NR 

Given materials to readexposure up to the physicians 

Pinto, 8 months NR N199874

Intervention Physicians were given the manual as well as 

an hour training session and follow-up reminders. Patients were also given reminders about exercise, n assigned: 17, n analysis: 12 

Live, print Discussion 

group, readings, role play 

Individuals T: 1 W: Once 

only D: 1 hour 

One time 8 months NR N
Evidence table 3. Description of CME activity 

F-120 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Review articles only on the management of chest pain, n assigned: 10, n analysis: 10 

Print Readings Individuals T: NR W: NR 

D: NR 

NR NR NR N

Concurrent control 

No education on the management of chest pain, n assigned: 10, n analysis: 7 

NA NA NA T: NA 

W: NA D: NA 

NA NA NR N

Intervention educational workbook (video + questions/ 

answers) and review articles on the management of chest pain, n assigned: 10, n analysis: 4 

Video, print Case-based 

learning, readings 

Individuals T: NR W: NR 

D: NR 

NR NR NR N

Premi J, 199338

Intervention Educational workbook (video + questions/ 

answers) on the management of chest pain, n assigned: 10, n analysis: 9 

Video, print Case-based 

learning, readings 

Individuals T: NR W: NR 

D: NR 

NR NR NR N
Evidence table 3. Description of CME activity 

F-121 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Family and general practice MDs waitlisted for CME and receiving no training, n assigned: 52, n analysis: 46 

NA NA NA T: NA 

W: NA D: NA 

Premi, NA NA NA N199441

Intervention Family and general practice MDs who 

participated in twice monthly small group community CME about hormone replacement, asthma, congestive heart failure, otitis media, and other topics, n assigned: 100, n analysis: 76 

Live, print Case-based 

learning, discussion group, problembased learning or team-based learning, readings 

Individuals T: 24 D: 1 year (for 

this evaluation) 

Multiple time or repetitive 

3 months Yes N

Intervention Educational materials alone, 

n assigned: NR 

Print Readings Individuals T: NR W: NR 

D: NR 

NR 3 months NR N

Concurrent control 

No intervention, n assigned: NR 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Rabin, 199897

Intervention Educational materials and simulated patient 

instructor, n assigned: NR 

Live, print Readings, 

standardized patient 

Individuals T: NR W: NR 

D: NR 

Multiple time or repetitive 

3 months NR N
Evidence table 3. Description of CME activity 

F-122 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Workshop and decision tree group, 

n analysis: 84 

Live, print Discussion 

group, lecture, readings 

Individuals T: 1.5 W: Once 

only D: NA 

One time 5 months Yes Y

Concurrent control 

Control group, n analysis: 82 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Intervention Decision tree group only, 

n analysis: 54 

Print Readings Individuals T: NA W: NA 

D: NA 

NA 5 months Yes Y

Rahme, 2005100

Intervention Workshop group only, n analysis: 29 Live Discussion group, lecture Individu- als T: 1.5 W: Once 

only D: NA 

One time 5 months Yes Y

Concurrent control 

No intervention about prescribing behavior, n assigned: 98+150, n analysis: 248 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Intervention Drug-educator visit about contraindicated 

antibiotics or oral cephalosporins, n assigned: 71+76, n analysis: 147 

Live Academic 

detailing 

Individuals T: NR W: Once 

only D: 1 visit 

One time 2 years NR N

Ray, 198520

Intervention Physicians receiving a physician-counselor 

visit about contraindicated antibiotics or oral cephalosporins, n assigned: 44+45, n analysis: 89 

Live Academic 

detailing 

Individuals T: 0.25 W: Once 

only D: 1 visit 

One time 2 years NR N
Evidence table 3. Description of CME activity 

F-123 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention visit by a physician- educator on the 

reasons to switch to drugs other than NSAIDS, as well as chart reminder stickers, n assigned: 110, n analysis: 103 

Live, print Academic 

detailing, readings, chart reminders 

Individuals T: NR W: NR 

D: NR 

Multiple time or repetitive 

Ray, 1 year NR Y2001109

Concurrent control 

No educational intervention on slowing NSAID prescribing, n assigned: 110, n analysis: 106 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N
Evidence table 3. Description of CME activity 

F-124 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention One of 3 small-group courses (12-45 per 

group), n assigned: 82, n analysis: 61 

Live, video Demonstration, lecture, 

simulation (other than standardized patient or role-play) 

Individuals T: 4 W: Once 

only D: 1 day 

One time 12-18 

months 

Yes N

Intervention Large group course, n assigned: 114, 

n analysis: 94 

Live, video Demonstration, lecture, 

simulation (other than standardized patient or role-play) 

Individuals T: 15 W: 4 

days/week D: 4 days 

One time 12-18 

months 

Yes N

Rodney, 198677

Concurrent control 

Randomly contacted physicians, of whom 22% had received other CME on flexible sigmoidoscopy, n assigned: 97, n analysis: 97 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Concurrent control 

Practices that receive only 1 of the 4-step intervention, n analysis: 22 

Review of data 

Review of practice's data 

Practice settings/ teams 

T: NR W: NR D: NR 

Rosen- One time NR NR Ythal, 200559

Intervention Practices that receive the full 4-step 

intervention, n analysis: 22 

Live, handheld, review of data 

Lecture, readings, review of practice's data 

Practice setting/ teams 

T: NR W: NR D: NR 

Multiple time or repetitive 

NR NR Y
Evidence table 3. Description of CME activity 

F-125 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Primary care practices without a mental health professional on faculty, n assigned: 6 practices (12 doctors),n analysis: 6 practices (12 doctors) 

NA NA NA T: NA 

W: NA D: NA 

Rost, NA NA NA N200122

Intervention Primary care practices without a mental 

health professional on faculty, n assigned: 6 practices (12 doctors),n analysis: 6 practices (12 doctors) 

By phone Conference 

calls 

Practice setting/ teams 

T: 6 W: NR D: 2 months 

Multiple time or repetitive 

6 months NR Y

Intervention Emotion-handling skills education, 

n assigned: 22 

Live Discussion 

group, lecture, role play, standardized patient 

Individuals T: 8 W: 1 

day/week D: 2 weeks 

Multiple time or repetitive 

Immediate – 6 months 

NR N

Concurrent control 

No education control, n assigned: 24 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Roter, 199584

Intervention Problem defining skills education, 

n assigned: 23 

Live Discussion 

group, lecture, role play, standardized patient 

Individuals T: 8 W: 1 

day/week D: 2 weeks 

Multiple time or repetitive 

Immediate – 6 months 

NR N
Evidence table 3. Description of CME activity 

F-126 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Received memo on the use of allergy trial 

packs, n assigned: 20 

Print Readings, 

directed physicians to use allergy trial packs 

Practice setting/ teams 

T: NR W: NR D: NR 

NR NR NR Y

Concurrent control 

Did not receive memo,n assigned: 190 

NA NA Practice 

settings/ teams 

T: NA W: NA D: NA 

NA NA NA N

Schectman, 1991139

Concurrent control 

Did not receive memo,n assigned: 13 

NA NA Practice 

settings/ teams 

T: NA W: NA D: NA 

NA NA NA N
Evidence table 3. Description of CME activity 

F-127 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Private FFS, received only memo, n assigned: 33 (the study only provides the total in the private practice group) 

Print Readings NR T: NR 

W: NR D: several weeks 

Multiple time or repetitive 

6 months NR Y

Intervention Private FFS, received memo and feedback 

on H2 blocker use, n assigned: 33 (the study only provides the total in the private practice group) 

Print Feedback, 

readings 

NR T: NR 

W: NR D: several weeks 

Multiple time or repetitive 

6 months NR Y

Concurrent control 

Group modelacademic, received only memo, n assigned: 21 (the study only provides the total in the academic group) 

Print Readings NR T: NR 

W: NR D: several weeks 

Multiple time or repetitive 

6 months NR Y

Schectman, 1995130

Intervention Group model- academic, received 

memo and feedback on H2 blocker use, n assigned: 21 (the study only provides the total in the academic group) 

Print Feedback, 

readings 

NR T: NR 

W: NR D: several weeks 

Multiple time or repetitive 

6 months NR Y
Evidence table 3. Description of CME activity 

F-128 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Group model- nonacademic, 

received memo and feedback on H2 blocker use, n assigned: 9 (the study only provides the total in the nonacademic group) 

Print Feedback, 

readings 

NR T: NR 

W: NR D: several weeks 

Multiple time or repetitive 

6 months NR Y

Concurrent control 

Group model, nonacademic and received only memo, n assigned: 9 (the study only provides the total in the nonacademic group) 

Print Readings NR T: NR 

W: NR D: several weeks 

Multiple time or repetitive 

6 months NR Y

Intervention All providers at the largest HMO site 

receiving memo about anti-histamine use, n assigned: 27 

Print, samples of first generation anithistamines to use with patients 

Readings Practice 

setting/ teams 

T: NR W: NR D: 1 month 

Schect- One time 6 months No Yman, 199616

Concurrent control 

Providers in 4 control HMO sites, n assigned: 14 

NA NA NA T: NA 

W: NA D: NA 

NA 6 months NA N
Evidence table 3. Description of CME activity 

F-129 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

No clinician intervention, n analysis: 56 

Video, print NA Individuals, 

practice settings/ teams 

T: NR W: NA D: 1 year 

NA 1 year NR U

a

Schectman, 2003108

Intervention Clinical practice guideline, 

n analysis: 50 

Live, video, print 

Lecture, readings 

Individuals, practice setting/ teams 

T: NR W: NA D: 1 year 

Multiple time or repetitive 

1 year NR U

a

Concurrent control 

Comparison sites, n assigned: 5 comparison sites 

NA NA NA T: 1 

W: NA D: NA 

Schroy, NA 1 year NR Y199982

Intervention Academic detailing intervention sites, 

n assigned: 4 intervention sites 

Live Academic 

detailing, discussion group, lecture 

Individuals, practice setting/ teams 

T: 1 W: Once only D: NR 

One time 1 year NR Y

Intervention Attended seminar on homecare and given 

written materials on the topic, n assigned: 355, n analysis: 131 

Live, print Case-based 

learning, lecture, readings 

Individuals T: NR D: half day Schwartz One time 3 months NR N

berg, 199798

Concurrent control 

No seminar attended, n assigned: 249, n analysis: 204 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N
Evidence table 3. Description of CME activity 

F-130 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Pediatricians from 2 inner-city pediatric clinics in the same academic medical center who did not attend workshop, n assigned: 9 

NA NA NA T: NA 

W: NA D: NA 

Sharif, NA NA NA N200294

Intervention Pediatricians from 2 inner-city pediatric 

clinics in the same academic medical center who attended workshop, n assigned: 6 

Live Lecture, role 

play 

Individuals T: NR W: Once 

only D: 1 day 

One time 3 weeks NR N

Concurrent control 

No CME, n assigned: 37, n analysis: 29 

NA NA T: NA 

W: NA D: NA 

Short, NA NA NA N200650

Intervention Online CME program, n assigned: 44, 

n analysis: 23 

Internet, not real time 

Lecture, problembased learning or team-based learning, readings, multiple media, interactivity 

Individuals T: NR W: NR 

D: 4 hours minimum and up to 16 

Multiple time or repetitive 

12 months Yes U

c
Evidence table 3. Description of CME activity 

F-131 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

No education, n assigned: 8 

NA NA NA T: NA 

W: NA D: NA 

Sibley, NA NA NA N1982136

Intervention 18 continuing- education packages, 

n assigned: 8 

Audio, print Case-based 

learning, lecture, readings 

Individuals T: 3-4 (median) 

W: NR D: not clear but seems to be available for an 18 month window 

Multiple time or repetitive 

18 months NR N

Slotnick, 199317

Intervention Education about a newer drug (Mazicon) 

through enhanced advertising with a "clinical challenge" scenario, n assigned: 10 

Print Readings Individuals T: NR W: Once 

only D: 1 session 

One time Immediate NR N

Intervention Education about a newer drug (Mazicon) 

through enhanced advertising with a "clinical challenge" scenario and an introduction on how to use the "clinical challenge", n assigned: 11 

Print Readings Individuals T: NR W: Once 

only D: 1 session 

One time Immediate NR N
Evidence table 3. Description of CME activity 

F-132 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Education about an older drug (Bumex) 

through enhanced advertising with a "clinical challenge" scenario and an introduction on how to use the "clinical challenge", n assigned: 9 

Print Readings Individuals W: Once only 

D: 1 session 

One time Immediate NR N

Intervention Education about a newer drug (Mazicon) 

through conventional advertising, n assigned: 11 

Print Readings Individuals T: NR W: Once 

only D: 1 session 

One time Immediate NR N

Intervention Education about an older drug (Bumex) 

through enhanced advertising with a "clinical challenge" scenario, n assigned: 8 

Print Readings Individuals T: NR W: Once 

only D: 1 session 

One time Immediate NR N

Intervention Education about an older drug (Bumex) 

through conventional advertising, n assigned: 12 

Print Readings Individuals T: NR W: Once 

only D: 1 session 

One time Immediate NR N
Evidence table 3. Description of CME activity 

F-133 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

No formal feedback on charting, n assigned: 75, n analysis: 45 

NA NA Individuals T: NA W: NA 

D: NA 

Socolar, NA 1 year NR N1998134

Received feedback on documentation of child abuse cases, n assigned: 72, n analysis: 42 

Print Feedback, 

readings 

Individuals T: NR W: NR 

D: 1 year 

Multiple time or repetitive 

1 year NR N

Intervention Intervention group: 3 part intervention for 

GIOP, n analysis: 10 

Live, print Discussion 

group, lecture, readings 

Individuals T: NR W: NR 

D: 6 weeks 

Multiple time or repetitive 

Solomon, 6 months NR N2004127

Concurrent control 

Control arm, n analysis: 11 

NA NA NA T: NA 

W: NA D: NA 

NA 6 months NA N

Concurrent control 

No intervention, n assigned: 162, n analysis: 54-70 

NA NA NA T: NA 

W: NA D: NA 

NA 9 months NA U

c

Intervention Academic detailing and printed materials, 

n assigned: 141, n analysis: 69-75 

Live, print Academic 

detailing, readings, brochures 

Individuals T: 33 minutes 

W: 1 day/week D: 6 months 

Multiple time or repetitive 

9 months NR U

c

Soumerai, 1987105

Intervention Print only educational intervention, 

n assigned: 132 

Print Readings, 

brochures 

Individuals T: NR W: NR 

D: NR 

Multiple time or repetitive 

9 months NR U

c
Evidence table 3. Description of CME activity 

F-134 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Not counseled to reduce NSAID use, n assigned: 10 homes (77 patients), n analysis: 10 homes (71 patients) 

NA NA NA T: NA 

W: NA D: NA 

Stein, NA NA NA N200195

Intervention Education and instructions on how to 

reduce the use of NSAIDS among residents over 65, n assigned: 10 homes (81 patients), n analysis: 10 homes (76 patients) 

Live, handheld, by phone 

Readings, study physician visit, algorithm 

Practice setting/ teams 

T: NR W: NR D: NR 

Multiple time or repetitive 

3 months NR Y

Intervention Intervention group receiving case-based 

on-line learning modules, n assigned: 27 

Internet, not real time 

Case-based learning, discussion group, readings 

Individuals T: NR W: NA 

D: 4 weeks 

Multiple time or repetitive 

Stewart, 6 months NR Y200543

Concurrent control 

Control group, n assigned: 31 

NA NA T: NA 

W: NA D: NA 

NA NA NA N
Evidence table 3. Description of CME activity 

F-135 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Received no intervention, n assigned: 3 communities (>15 primary care physicians each) 

NA NA NA T: NA 

W: NA D: NA 

Stross, NA NA NA N1985116

Intervention Educationally influential physicians 

received training about osteoarthritis management, n assigned: 3 communities (>15 primary care physicians each) 

Live, print, "audiovisual materials" 

Feedback, readings, use of "educationally influential" physician 

Practice setting/ teams 

T: NR W: NR D: NR 

Multiple time or repetitive 

1 year NR N

Concurrent control 

Education on other pulmonary topics, n assigned: 6 

Audio, print Case-based 

learning, lecture, readings 

Individuals T: 4 W: NR 

D: 9 months 

Multiple time or repetitive 

6 months Yes N

Intervention Group meetings about educational needs 

and 2 audiovisual programs about management and patient counseling, n assigned: 10 

Live, audio, print 

Case-based learning, discussion group, lecture, readings 

Individuals T: NR W: NR 

D: 18 months 

Multiple time or repetitive 

6 months Yes N

Terry, 198149

Intervention Group meetings about educational needs, 2 

audiovisual programs about management and patient counseling and feedback on questionnaire performance, n assigned: 7 

Live, audio, print 

Case-based learning, discussion group, feedback, lecture, readings 

Individuals T: NR W: NR 

D: 18 months 

Multiple time or repetitive 

6 months Yes N
Evidence table 3. Description of CME activity 

F-136 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Two audiovisual programs about 

management and patient counseling, n assigned: 12 

Live, audio, print 

Case-based learning, lecture, readings 

Individuals T: 4 W: NR 

D: 9 months 

Multiple time or repetitive 

6 months Yes N

Intervention Two audiovisual programs about 

management and patient counseling and feedback on questionnaire performance, n assigned: 9 

Audio, print Case-based 

learning, feedback, lecture, readings 

Individuals T: NR W: NR 

D: 9 months 

Multiple time or repetitive 

6 months Yes N

Intervention CME on patient trust, n assigned: 10 Live, video Demonstra- tion, 

discussion group, lecture, role play 

Individuals T: 7 W: Once 

only D: NR 

One time 6 months NR U

a

Thom, 2000129

Concurrent control 

No CME, n assigned: 10 

NA NA Individuals T: NA W: NA 

D: NA 

NA 6 months NR U

a

Intervention Received the National Cancer Institute 

training manual on nutrition counseling and systems change, n assigned: 256, n analysis: 205 

Print Readings Practice 

settings/ teams 

T: NR W: NA D: NA 

Tziraki, One time 4-6 months NR Y2000141

Concurrent control 

Received no training on nutrition counseling and systems change, n assigned: 255, n analysis: 222 

NA NA NA T: NA 

W: NA 

NA NA NA N
Evidence table 3. Description of CME activity 

F-137 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Received a 3-hour training on nutrition 

counseling and systems change to support nutrition counseling, n assigned: 244, n analysis: 191 

Live, print Discussion 

group, lecture, problembased learning or team-based learning, role play 

Practice setting/ teams 

T: 3 W: Once only D: NA 

One time 4-6 months NR Y

Concurrent control 

Usual care, including AHRQ depression practice guidelines, n assigned: 16 clinics, 443 patients, n analysis: 16 clinics, 374 patients 

NA NA NA T: NA 

W: NA D: NA 

Wells, NA NA NA N2000151

Intervention Quality improvement interventions with 

either medication adherence support or cognitive behavioral therapy; QI included institutional support, training of local leaders, 2-day MD workshop, manuals, monthly lectures, academic detailing, audit with feedback, 1-day staff workshop, and support materials,n assigned: 30 clinics, 913 patients, n analysis: 30 clinics, 752 patients 

Live, print Academic 

detailing, discussion group, feedback, lecture, point of care, readings 

Practice setting/ teams 

T: NR W: NR D: NR 

Multiple time or repetitive 

NR NR Y
Evidence table 3. Description of CME activity 

F-138 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Intervention Generalists receiving unitopic (cardiology) 

university-based CME program, n analysis: 31 

Live NR Individuals T: NR W: 4 

days/week D: 4 days 

One time 6 months NR N

Intervention Generalists receiving a multitopic universitybased CME, 

n analysis: 73 

Live NR Individuals T: NR W: 4 

days/week D: 4 days 

One time 6 months NR N

Concurrent control 

Generalists in communities receiving no training on myocardial infarction, n assigned: 40 MDs (4 communities), n analysis: 40 MDs (4 communities) 

Live, NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

White, 198548

Intervention Generalists in communities receiving 

training on management of myocardial infarction, n assigned: 63 MDs (8 communities), n analysis: 63 MDs (8 communities) 

Live Case-based 

learning, discussion group, lecture 

Individuals T: 3.5 W: Once 

only D: 3.5 hours 

One time 6 months N

Intervention Small-group problem- based learning (PBL) 

sessions, n assigned: 23, n analysis: 23 

Live Case-based 

learning, discussion group 

Individuals T: 1 W: Once 

only D: NA 

White, One time 3 months NR N200463

Concurrent control 

Didactic lecture sessions, n assigned: 29, n analysis: 29 

Live Case-based 

learning, discussion group, lecture 

Individuals T: 1 W: Once 

only D: NA 

One time 3 months NR N
Evidence table 3. Description of CME activity 

F-139 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Family physicians who received nicotine gum, but no training, n assigned: 27, n analysis: 27 

NR NR Individuals T: NA W: NA 

D: NA 

MDs were told to offer nicotine gum 

NA NA N

Concurrent control 

Family physicians who received usual care, n assigned: 25 

NA NA NA T: NA 

W: NA D: NA 

NA NA NA N

Wilson, 1988154

Intervention Family physicians who received nicotine gum 

plus training, n assigned: 22 

Live NR Individuals T: 4 hours W: Once 

only D: NA 

One time 1 YEAR NR U

c

Intervention Internists in HMO who received 6 months of 

monthly feedback on colorectal cancer screening practices compared with colleagues, 1st group to receive intervention, n assigned: 8 

Print Feedback Practice 

setting/ teams 

T: NR W: NR D: 6 months 

Multiple time or repetitive 

Winickoff, 12 months NR Y198423

Intervention Internists in HMO who received 6 months of 

monthly feedback on colorectal cancer screening practices compared with colleagues, 2nd group to receive intervention, n assigned: 8 

Print Feedback Practice 

settings/ teams 

T: NR W: NR D: 6 months 

Multiple time or repetitive 

6 months NR Y
Evidence table 3. Description of CME activity 

F-140 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Control participants received a copy of the clinical practice guideline in the mail, n assigned: 20 

Print Readings Individuals T: NA W: NA 

D: NA 

One time 6 months NR U

c

Worrall, 1999152

Intervention Participants who received 3 hour 

educational intervention and access to a psychiatrist, n assigned: 22 

Live Case-based 

learning, discussion group, lecture 

Individuals T: 3 W: Once 

only D: 3 hours (it was a workshop) 

One time 6 months NR U

c
Evidence table 3. Description of CME activity 

F-141 

Author, year Group Group description, N

Media used 

Educational methods 

Intervention design 

Hours in CME activity (T), days/week exposed (W), intervention duration (D) 

Amount of exposure 

Evaluation duration 

Accredited 

Poopje

Concurrent control 

Received only a newsletter, no educational materials,n assigned: 10972 

NA NA NA T: NA 

W: NA D: NA 

NA NA NR N

Intervention Received "underutilization 

packages" of educational materials (doctors whose postAMI patients had not been prescribed a beta-blocker), n assigned: 328 

Print Readings Individuals T: NR W: NA 

D: 2 weeks 

One time 30 days NR U

c

Zuckerman, 200418

Intervention Received "noncompliant 

packages" of educational materials (doctors whose postAMI patients were noncompliant with beta-blocker therapy),n assigned: 157 

Print Readings Individuals T: NR W: NA 

D: 2 weeks 

One time 30 days NR U

c

AAN = American Academy of Neurology; ACCESS = Alzheimer's Disease Coordinated Care for San Diego Seniors; ACE = angiotensin-converting eHealthcare Research and Quality; AMI = acute myocardial infarction; ARS = audience response system; BSTC = barrier specific telephone counseling

examination; CDE = complete diagnostic evaluation; CDH = chronic daily headache; CHD = coronary heart disease; CME = continuing medical educdiethylstilbestrol; DSM = Diagnostic and Statistic Manual of Mental Disorders; FPs = family practitioner; GIOP = glucocorticoid-induced osteoporosi

HMO = health maintenance organization; IHD = ischemic heart disease; IM = internal medicine; KFP = Key Features Program; LOS = length of stay; = medication induced headache; NA = not applicable; NCEP = National Cholesterol Education Program; NR = not reported; NSAID = non-steroidal a= osteoarthritis; OSCE = objective structured clinical examination; PBL = problem-based learning; PCP = primary care provider; PHE = physical heal

improvement; QI = quality improvement; SP = standardized patient; TED = teleconferenced education detailing; WAP = written action plan; WBJC = Club 
Evidence table 4. Quality of studies assessing the effectiveness of continuing medical education 

F-142 

Author, year Randomized 

Randomization scheme described 

Blinded evaluation 

Blinding described 

Withdrawals described 

Power analysis described 

Sufficienpower to statisticsignifica

Adams, 1998125

Yes No No NR NR No No 

Allison, 2005147

Yes Yes No Not described No Yes No 

Andersen, 199036

Yes No NR NR No No NR 

Anderson, 1996103

Yes No NR NR Yes No NR 

Beaulieu, 2002140

Yes Inappropriate Yes Yes Yes Yes No 

Beaulieu, 200444

No NR No NR No No NR 

Bjornson, 199014

Yes No NR Not described Yes Yes No 

Block, 198837 No NR NR NR No No NR Bloomfield, 200568

Yes Yes NR Not described Yes Yes Yes 

Brown, 199972 Yes Inappropriate NR NR NR Yes Yes Brown, 2004135 Yes Yes No NR No No NR Browner, 1994122

Yes Yes No NR No Yes Yes 

Bunting, 2004123

No NR NR NR No No NR 

Carney, 199588 Yes No Yes Yes Yes Yes Yes Casebeer, 1999131

Yes No NR NR No No 

Chan, 199921 Yes No NR Not described Yes Yes Yes Chassin, 1986106

Yes Inappropriate NR NR No No NR 

Cherkin, 199183

No NR No NR No No NR 

Chodosh, 200662

Yes No No NR No No NR 
Evidence table 4. Quality of studies assessing the effectiveness of continuing medical education 

F-143 

Author, year Randomized 

Randomization scheme described 

Blinded evaluation 

Blinding described 

Withdrawals described 

Power analysis described 

Sufficienpower to statisticsignifica

Chung, 200454 Yes Yes No NR No Yes No Clark, 1998120 Yes No NR NR No No NR Clark, 200078 Yes No NR NR Yes Yes Yes Cohn, 200219 No NR NR NR No No No Costanza, 199235

No NR No NR No Yes Yes 

Cummings, 1989112

Yes Yes Yes Yes Yes No NR 

Cummings, 1989113

Yes Yes Yes Yes No No NR 

Cummings, 1989117

Yes Yes Yes Yes No No No 

Curran, 200034 Yes Yes NR NR No No NR Davis, 2004101 No NR No NR No No NR Derebery, 2002153

No NR NR NR No No No 

Des Marchais, 199061

Yes No NR NR No No No 

Dietrich, 2000110

Yes Inappropriate No NR No No NR 

Dormuth, 200415

Yes Yes Yes Not described No No NR 

Doucet, 199840 No NR NR NR No No NR Elliott, 199755 Yes No NR NR No No Evans CE, 198656

Yes Yes Yes Inappropriate Yes Yes Yes 

Fordis, 200542 Yes Yes NR NR Yes Yes Yes Frush, 200687 Yes No NR NR No No NR Gerbert, 200286

Yes No NR NR No No NR 

Gerrity, 199932 Yes Yes Yes Yes Yes No NR 
Evidence table 4. Quality of studies assessing the effectiveness of continuing medical education 

F-144 

Author, year Randomized 

Randomization scheme described 

Blinded evaluation 

Blinding described 

Withdrawals described 

Power analysis described 

Sufficienpower to statisticsignifica

Gerstein, 199953

No NR NR NR No Yes No 

Gifford, 199646 Yes Yes NR NR Yes Yes Yes Gifford, 1999142

Yes Yes Yes Not described Yes Yes Yes 

Goldberg, 200193

Yes No NR NR No Yes Yes 

Goldstein, 2005146

Yes Yes NR Not described Yes No NR 

Goldwater, 2001157

No NR NR NR No No NR 

Gonzales, 1999111

No No NR Not described No No NR 

Grady, 199779 Yes Yes NR NR Yes No NR Greenberg, 198525

No Yes Yes No No NR 

Gullion, 1988132

Yes No Yes Not described Yes No NR 

Hagen, 2005126

NR No NR Not described No No NR 

Harris, 200239 Yes No NR NR No No NR Harris, 200575 Yes Yes NR Not described No No NR Heale, 198866 Yes Yes NR NR No No NR Herbert, 2004102

Yes Yes Yes Yes Yes No NR 

Hergenroeder, 200260

Yes Yes Yes Inappropriate No No NR 

Howe, 1997145 No NR NR NR No No NR Jennett, 198876 Yes Yes Yes Yes Yes No NR Juzych, 200592 No No No Not described No No NR Kemper, 200658

Yes Yes No NR No No NR 
Evidence table 4. Quality of studies assessing the effectiveness of continuing medical education 

F-145 

Author, year Randomized 

Randomization scheme described 

Blinded evaluation 

Blinding described 

Withdrawals described 

Power analysis described 

Sufficienpower to statisticsignifica

Kiang, 200533 No No NR Not described No Yes Yes Kim, 1999137 Yes No NR NR Yes No NR Kottke, 1989114 Yes Yes NR NR No Yes No Kronick, 200371

Yes No NR NR No Yes Yes 

Kutcher, 200265

No NR NR Not described No No NR 

Labelle, 200467 No NR No NR No No NR Lane, 199169 No NR NR NR No Yes Yes Lane, 200145 No NR NR NR Yes No NR Leopold, 200580

Yes No No Not described No No NR 

Levinson, 1993138

Yes No Yes Yes Yes Yes Yes 

Lewis, 199370 No NR No NR No No NR Lin, 199773 No NR No NR No No NR Lin, 2001133 Yes Yes NR NR No No NR LindsayMcIntyre, 1987115

Yes No No NR Yes No NR 

Lockyer, 200264

No NR NR NR No No NR 

Maclure, 199896

Yes Inappropriate NR NR No No NR 

Macrae, 200485

Yes No Yes Yes Yes Yes Yes 

Maiman, 198847

Yes Yes Yes Yes Yes No NR 

Mann, 199752 Yes No NR NR No No NR Margolis, 2004118

Yes Yes NR Not described No Yes Yes 
Evidence table 4. Quality of studies assessing the effectiveness of continuing medical education 

F-146 

Author, year Randomized 

Randomization scheme described 

Blinded evaluation 

Blinding described 

Withdrawals described 

Power analysis described 

Sufficienpower to statisticsignifica

Maxwell, 198457

No NR Yes Yes No No NR 

Mazmanian, 199824

Yes No NR NR No No NR 

Mazmanian, 2001161

Yes Yes Yes Yes No No NR 

McBride, 2000144

Yes No No NR No Yes No 

McClellan, 2003119

Yes Yes NR NR No Yes Yes 

McMahon, 1988155

No NR NR NR No No NR 

Mehler, 200599 Yes Yes No NR No No NR Meredith, 200051

Yes Yes No NR No No NR 

Messina, 2002156

No NR NR NR No No NR 

Moran, 1996121 No NR No NR No No NR Mukohara, 200513

Yes Yes NR Not described No Yes Yes 

Myers, 2004148 Yes No No NR No Yes No Norris, 200081 Yes Yes NR NR No Yes No Ockene, 1996124

Yes Inappropriate NR NR No No NR 

Ozer, 2005104 No No NR Not described Yes No NR Pazirandeh, 2002128

No NR No NR No No NR 

Pereles, 1996163

Yes No NR NR No No NR 

Perera, 1983107

Yes No NR NR Yes No NR 

Pimlott, 2003143

Yes Yes NR NR No No NR 
Evidence table 4. Quality of studies assessing the effectiveness of continuing medical education 

F-147 

Author, year Randomized 

Randomization scheme described 

Blinded evaluation 

Blinding described 

Withdrawals described 

Power analysis described 

Sufficienpower to statisticsignifica

Pinto, 199874 Yes Yes No NR No No No Premi J, 199338

Yes No NR NR No No No 

Premi, 199441 No NR NR NR No No NR Rabin, 199897 Yes No Yes Not described No No NR Rahme, 2005100

Yes Yes NR NR No No NR 

Ray, 198520 No NR NR NR No No NR Ray, 2001109 Yes Yes NR NR Yes No NR Rodney, 198677

No NR No NR No No NR 

Rosenthal, 200559

Yes Yes No Not described No Yes Yes 

Rost, 200122 Yes Yes Yes Yes Yes Yes Yes Roter, 199584 Yes No NR NR Yes Yes Yes Schectman, 1991139

No NR No NR No No NR 

Schectman, 1995130

Yes No Yes Yes No Yes Yes 

Schectman, 199616

No NR No NR No No NR 

Schectman, 2003108

Yes Yes NR NR No Yes Yes 

Schroy, 199982 No NR NR NR Yes Yes Yes Schwartzberg, 199798

No NR NR NR Yes No NR 

Sharif, 200294 No NR NR NR No No NR Short, 200650 Yes Yes No NR Yes Yes No Sibley, 1982136 Yes Yes Yes Yes No Yes Yes Slotnick, 199317

No NR NR NR No Yes Yes 
Evidence table 4. Quality of studies assessing the effectiveness of continuing medical education 

F-148 

Author, year Randomized 

Randomization scheme described 

Blinded evaluation 

Blinding described 

Withdrawals described 

Power analysis described 

Sufficienpower to statisticsignifica

Socolar, 1998134

Yes Inappropriate NR NR Yes Yes Yes 

Solomon, 2004127

Yes No NR NR No Yes No 

Soumerai, 1987105

Yes Yes NR Not described No No NR 

Stein, 200195 Yes Yes Yes Yes No No NR Stewart, 200543

Yes Yes NR NR NR Yes Yes 

Stross, 1985116 Yes No NR Not described No No NR Terry, 198149 Yes Yes Yes Yes No No NR Thom, 2000129 No NR Yes Not described No No NR Tziraki, 2000141

Yes Yes Yes Yes Yes Yes Yes 

Wells, 2000151 Yes Yes NR NR Yes Yes Yes White, 198548 Yes Yes Yes Yes No No NR White, 200463 Yes Yes Yes NR No No NR Wilson, 1988154

Yes Yes Yes Yes Yes Yes No 

Winickoff, 198423

Yes No NR NR NR Yes Yes 

Worrall, 1999152

Yes Yes NR Not described No Yes Yes 

Zuckerman, 200418

No NR NR Not described No No NR 

*Total quality score was calculated using the Jadad8 criteria: 1) appropriateness of the randomization scheme, 2) appropriateness of the blinding, and 3

and drop-outs. NR = not reported
Evidence table 5. Quality of systematic reviews evaluating the effectiveness of simulation in medical education 

F-149

Author, year 

Question clearly stated 

Search methods described/ comprehensive 

Inclusion criteria reported/ appropriate 

Study quality Assessed/ appropriate 

Methodology reproducible 

Discuss differences in study design or population 

Results combined appropriately 

Concsuppdata 

Psychomotor skillsHaque, 200626 Yes Partially/ yes Yes/ yes No/ can't tell Can't tell Yes Yes Yes Aucar, 200529 Partially Yes/ partially No/ can't tell No/ can't tell Can't tell Partially Can't tell Yes Gerson, 200430 Partially Yes/ partially No/ can't tell Yes/ yes Can't tell Yes Can't tell Yes Sutherland, 200631 Yes Yes/ partially Yes/ yes Yes/ partially Can't tell Yes Can't tell Yes Ravert, 2002159 Yes Partially/ 

partially 

Yes/ yes No/ can't tell Can't tell No Can't tell Yes 

Gaffan, 2006158 Partially Yes/ partially Yes/ yes No/ can't tell Can't tell No Can't tell Yes Communication skills Gaffan, 2006158 Partially Yes/ partially Yes/ yes No/ can't tell Can't tell No Can't tell Yes Spangler, 2002160 Yes Yes/ partially Yes/ yes No/ can't tell Can't tell No Can't tell Yes Cognitive skills Hmelo, 199028 Partially Yes/ yes Partially/ 

partially 

No/ can't tell Can't tell No Yes Partia

Gaffan, 2006158 Partially Yes/ partially Yes/ yes No/ can't tell Can't tell No Can't tell Yes Other Issenberg, 200527 Yes Yes/ yes Yes/ yes Yes/ yes Yes Yes Yes Yes 
Evidence table 6. Reporting of adult learning principles in studies assessing the effectiveness of continuing medical education 

F-150 

Author, year 

Enables learners to be active contributors to their learning* 

Relate to learners' current work or life experience†

Tailored to learners' current or past experiences‡

Allow learners to identify their own learning goals and direct their education§

Allow learners to practice what they learn║

Provide support to self-directed learners¶

Receive feedback from teachers and/or peers during active learning#

Allow learnrefleclearn

Adams, 1998125

Poor Good Fair Poor Poor Poor Fair Good 

Allison, 2005147

Fair Good Fair Poor Poor Poor Poor Poor 

Andersen, 199036

Poor Good Poor Poor Poor Poor Poor Poor 

Anderson, 1996103

Fair Good Fair Poor Fair Poor Poor Fair 

Beaulieu, 2002140

Fair Good Poor Poor Fair Poor Poor Poor 

Beaulieu, 200444

Fair Good Good Poor Poor Fair Fair Poor 

Bjornson, 199014

Poor Good Poor Poor Poor Poor Poor Fair 

Block, 198837 Poor Good Poor Poor Poor Poor Poor Bloomfield, 200568

Fair Good Fair Poor Poor Fair Poor Poor 

Brown, 199972 Fair Good Poor Poor Fair Good Poor Good Brown, 2004135 Fair Good Fair Fair Poor Fair Fair Poor Browner, 1994122

Poor Fair Poor Poor Poor Poor Poor Poor 

Bunting, 2004123

Poor Good Poor Poor Good Fair Good Poor 

Carney, 199588 Fair Fair Poor Poor Fair Poor Poor Poor Casebeer, 1999131

Good Good Good Good Good Fair Poor Poor 

Chan, 199921 Good Good Good Good Good Good Fair Fair Chassin, 1986106

Poor Good Poor Poor Poor Poor Fair Poor 

Cherkin, 199183

Fair Good Fair Poor Poor Poor Poor Fair 

Chodosh, 200662

Poor Good Poor Poor Poor Poor Poor Poor 

Chung, 200454 Poor Fair Poor Poor Poor Fair Poor Poor 
Evidence table 6. Reporting of adult learning principles in studies assessing the effectiveness of continuing medical education 

F-151 

Author, year 

Enables learners to be active contributors to their learning* 

Relate to learners' current work or life experience†

Tailored to learners' current or past experiences‡

Allow learners to identify their own learning goals and direct their education§

Allow learners to practice what they learn║

Provide support to self-directed learners¶

Receive feedback from teachers and/or peers during active learning#

Allow learnrefleclearn

Clark, 1998120 Fair Good Fair Poor Poor Poor Poor Fair Clark, 200078 Good Good Fair Fair Fair Fair Fair Poor Cohn, 200219 Poor Fair Poor Poor Good Fair Poor Poor Costanza, 199235

Poor Fair Poor Poor Fair Poor Poor Poor 

Cummings, 1989112

Fair Good Good Fair Good Poor Fair Fair 

Cummings, 1989113

Fair Good Fair Fair Good Poor Fair Fair 

Cummings, 1989117

Fair Good Poor Poor Fair Poor Poor Good 

Curran, 200034 Good Good Good Fair Poor Good Good Good Davis, 2004101 Fair Good Fair Poor Poor Poor Fair Poor Derebery, 2002153

Good Good Fair Poor Fair Poor Fair Fair 

Des Marchais, 199061

Fair Good Poor Poor Fair Poor Fair Good 

Dietrich, 2000110

Poor Good Poor Poor Fair Fair Poor Poor 

Dormuth, 200415

Poor Good Poor Poor Poor Fair Poor Poor 

Doucet, 199840 Fair Good Good Poor Fair Poor Poor Fair Elliott, 199755 Poor Poor Fair Poor Poor Poor Poor Poor Evans CE, 198656

Poor Good Poor Poor Poor Fair Poor Poor 

Fordis, 200542 Fair Good Poor Poor Poor Fair Fair Poor Frush, 200687 Poor Good Fair Poor Fair Poor Poor Poor Gerbert, 200286

Fair Fair Poor Poor Poor Poor Fair Poor 

Gerrity, 199932 Good Good Good Good Good Good Good Good Gerstein, 199953

Poor Good Poor Poor Poor Poor Poor Poor 
Evidence table 6. Reporting of adult learning principles in studies assessing the effectiveness of continuing medical education 

F-152 

Author, year 

Enables learners to be active contributors to their learning* 

Relate to learners' current work or life experience†

Tailored to learners' current or past experiences‡

Allow learners to identify their own learning goals and direct their education§

Allow learners to practice what they learn║

Provide support to self-directed learners¶

Receive feedback from teachers and/or peers during active learning#

Allow learnrefleclearn

Gifford, 199646 Poor Good Poor Poor Poor Good Poor Poor Gifford, 1999142

Poor Good Poor Poor Poor Poor Poor Poor 

Goldberg, 200193

Good Good Poor Fair Fair Fair Poor Good 

Goldstein, 2005146

Fair Good Good Poor Poor Poor Poor Poor 

Goldwater, 2001157

Poor Good Poor Poor Good Poor Poor Poor 

Gonzales, 1999111

Poor Good Fair Poor Poor Poor Poor Poor 

Grady, 199779 Poor Fair Fair Poor Poor Poor Good Poor Greenberg, 198525

Fair Good Fair Poor Poor Poor Poor Poor 

Gullion, 1988132

Fair Good Good Poor Poor Poor Fair Poor 

Hagen, 2005126

Poor Good Fair Poor Poor Fair Poor Poor 

Harris, 200239 Poor Good Poor Poor Poor Fair Poor Poor Harris, 200575 Fair Good Fair Poor Fair Poor Poor Heale, 198866 Good Good Fair Poor Poor Poor Poor Poor Herbert, 2004102

Good Good Fair Poor Poor Fair Fair Poor 

Hergenroeder, 200260

Poor Fair Poor Poor Good Good Poor Poor 

Howe, 1997145 Poor Fair Poor Poor Poor Poor Good Poor Jennett, 198876

Fair Good Good Fair Good Poor Good Fair 

Juzych, 200592 Poor Good Fair Poor Poor Fair Poor Poor Kemper, 200658

Fair Fair Fair Poor Poor Fair Poor Poor 

Kiang, 200533 Poor Good Fair Poor Poor Poor Poor Poor Kim, 1999137 Poor Good Poor Poor Good Poor Good Fair 
Evidence table 6. Reporting of adult learning principles in studies assessing the effectiveness of continuing medical education 

F-153 

Author, year 

Enables learners to be active contributors to their learning* 

Relate to learners' current work or life experience†

Tailored to learners' current or past experiences‡

Allow learners to identify their own learning goals and direct their education§

Allow learners to practice what they learn║

Provide support to self-directed learners¶

Receive feedback from teachers and/or peers during active learning#

Allow learnrefleclearn

Kottke, 1989114 Fair Good Poor Poor Fair Poor Poor Fair Kronick, 200371

Good Good Fair Good Poor Good Poor Poor 

Kutcher, 200265

Fair Good Poor Poor Poor Poor Fair Fair 

Labelle, 200467 Fair Good Poor Poor Fair Poor Good Poor Lane, 199169 Fair Fair Poor Poor Fair Poor Poor Poor Lane, 200145 Poor Good Fair Poor Good Good Good Good Leopold, 200580

Fair Good Fair Poor Fair Fair Fair Poor 

Levinson, 1993138

Poor Poor Poor Poor Poor Poor Poor Poor 

Lewis, 199370 Poor Poor Poor Poor Poor Poor Poor Poor Lin, 199773 Poor Good Fair Poor Fair Poor Good Poor Lin, 2001133 Good Fair Fair Poor Fair Good Fair Fair LindsayMcIntyre, 1987115

Poor Good Poor Poor Fair Poor Poor Poor 

Lockyer, 200264

Fair Good Fair Poor Fair Poor Poor Poor 

Maclure, 199896

Poor Poor Poor Poor Poor Poor Poor Poor 

Macrae, 200485

Fair Good Poor Poor Poor Poor Poor Poor 

Maiman, 198847

Fair Good Poor Poor Poor Poor Poor Poor 

Mann, 199752 Fair Good Poor Poor Poor Fair Poor Poor Margolis, 2004118

Fair Good Fair Poor Poor Poor Fair Poor 

Maxwell, 198457

Fair Poor Poor Fair Poor Poor Fair Poor 

Mazmanian, 199824

Poor Poor Good Poor Poor Poor Poor Poor 
Evidence table 6. Reporting of adult learning principles in studies assessing the effectiveness of continuing medical education 

F-154 

Author, year 

Enables learners to be active contributors to their learning* 

Relate to learners' current work or life experience†

Tailored to learners' current or past experiences‡

Allow learners to identify their own learning goals and direct their education§

Allow learners to practice what they learn║

Provide support to self-directed learners¶

Receive feedback from teachers and/or peers during active learning#

Allow learnrefleclearn

Mazmanian, 2001161

Fair Poor Fair Poor Poor Poor Fair Poor 

McBride, 2000144

Good Good Poor Good Fair Good Poor Poor 

McClellan, 2003119

Poor Good Fair Poor Poor Poor Fair Poor 

McMahon, 1988155

Poor Fair Poor Poor Poor Poor Good Fair 

Mehler, 200599 Fair Good Fair Poor Poor Poor Poor Poor Meredith, 200051

Poor Good Poor Fair Poor Poor Poor Poor 

Messina, 2002156

Fair Good Fair Poor Poor Fair Poor Fair 

Moran, 1996121 Good Good Good Good Poor Good Fair Good Mukohara, 200513

Fair Fair Fair Poor Poor Fair Poor Poor 

Myers, 2004148 Poor Good Fair Poor Poor Good Fair Fair Norris, 200081 Poor Good Poor Poor Poor Fair Poor Poor Ockene, 1996124

Poor Good Fair Poor Fair Poor Fair Poor 

Ozer, 2005104 Fair Good Fair Poor Poor Poor Poor Poor Pazirandeh, 2002128

Poor Good Poor Fair Poor Poor Poor Poor 

Pereles, 1996163

Poor Poor Poor Poor Poor Poor Poor Poor 

Perera, 1983107

Poor Good Good Poor Fair Poor Fair Poor 

Pimlott, 2003143

Poor Good Fair Poor Poor Poor Good Poor 

Pinto, 199874 Good Good Poor Poor Fair Fair Fair Poor Premi J, 199338

Poor Fair Poor Poor Poor Poor Poor Poor 

Premi, 199441 Good Good Fair Good Fair Good Fair Fair 
Evidence table 6. Reporting of adult learning principles in studies assessing the effectiveness of continuing medical education 

F-155 

Author, year 

Enables learners to be active contributors to their learning* 

Relate to learners' current work or life experience†

Tailored to learners' current or past experiences‡

Allow learners to identify their own learning goals and direct their education§

Allow learners to practice what they learn║

Provide support to self-directed learners¶

Receive feedback from teachers and/or peers during active learning#

Allow learnrefleclearn

Rabin, 199897 Poor Good Fair Poor Poor Poor Fair Poor Rahme, 2005100

Fair Good Fair Poor Poor Poor Poor Poor 

Ray, 198520 Fair Good Good Poor Poor Poor Poor Poor Ray, 2001109 Poor Good Fair Poor Good Fair Fair Fair Rodney, 198677

Poor Good Poor Poor Fair Fair Fair Poor 

Rosenthal, 200559

Poor Good Fair Poor Poor Poor Poor Poor 

Rost, 200122 Fair Good Poor Poor Poor Poor Fair Poor Roter, 199584 Poor Fair Poor Poor Good Poor Fair Poor Schectman, 1991139

Poor Poor Poor Poor Poor Poor Poor Poor 

Schectman, 1995130

Poor Poor Poor Poor Poor Poor Fair Poor 

Schectman, 199616

Poor Good Poor Poor Poor Poor Poor Poor 

Schectman, 2003108

Fair Fair Poor Poor Poor Poor Good Poor 

Schroy, 199982 Fair Good Fair Poor Fair Poor Poor Fair Schwartzberg, 199798

Poor Fair Fair Poor Poor Fair Poor Poor 

Sharif, 200294 Fair Good Poor Poor Fair Poor Poor Poor Short, 200650 Good Good Fair Fair Poor Fair Poor Poor Sibley, 1982136 Fair Good Fair Fair Poor Fair Fair Poor Slotnick, 199317

Poor Fair Fair Poor Poor Poor Poor Poor 

Socolar, 1998134

Poor Poor Fair Poor Poor Poor Poor Poor 

Solomon, 2004127

Fair Good Fair Poor Good Fair Poor Fair 

Soumerai, 1987105

Fair Good Fair Poor Poor Poor Poor Fair 
Evidence table 6. Reporting of adult learning principles in studies assessing the effectiveness of continuing medical education 

F-156 

Author, year 

Enables learners to be active contributors to their learning* 

Relate to learners' current work or life experience†

Tailored to learners' current or past experiences‡

Allow learners to identify their own learning goals and direct their education§

Allow learners to practice what they learn║

Provide support to self-directed learners¶

Receive feedback from teachers and/or peers during active learning#

Allow learnrefleclearn

Stein, 200195 Poor Good Poor Poor Poor Fair Poor Poor Stewart, 200543

Fair Good Fair Poor Poor Poor Poor Poor 

Stross, 1985116 Fair Good Good Poor Poor Good Good Fair Terry, 198149 Good Good Fair Poor Poor Fair Fair Poor Thom, 2000129 Poor Fair Poor Poor Poor Poor Poor Poor Tziraki, 2000141

Good Good Good Good Fair Fair Poor Poor 

Wells, 2000151 Poor Fair Poor Poor Poor Poor Poor Poor White, 198548 Fair Good Fair Poor Poor Poor Poor Poor White, 200463 Fair Good Fair Poor Poor Poor Poor Poor Wilson, 1988154

Poor Good Poor Poor Poor Poor Poor Poor 

Winickoff, 198423

Poor Good Fair Poor Poor Poor Fair Poor 

Worrall, 1999152

Fair Good Fair Fair Fair Good Poor Poor 

Zuckerman, 200418

Poor Good Fair Poor Poor Poor Poor Poor 

* Ratings of "good" indicate that two or more of the following were reported: learners identify/choose a question OR actively contribute to finding the 

of their learning to others. Ratings of "fair" indicate that only one of the above were reported OR none of the above were reported but the curriculum 

learning methods such a interactive lectures or group discussions. Ratings of "poor" indicate that none of the above are described. †Ratings of "good" indicate that learners would recognize the curriculum as having practical or immediate value to their work or lives. Ratings of "fai

would recognize the curriculum as having theoretical or future value to their work or lives. Ratings of "poor" indicate that learning addresses an issue 

recognize as having value to their work or lives OR the curriculum's relevance to the learners is not clear. ‡Ratings of "good" indicate that the authors describe a needs assessment AND describe how the curriculum is tailored towards the needs of the learner

that the authors describe only one of the above. Ratings of "poor" indicate that the authors neither describe a needs assessment nor do they describe hotowards the needs of learners. 

§Ratings of "good" indicate that learners received complete freedom to pursue independent studies or projects during part or all of the curriculum. Rat

learners may choose from a range of learning methods or projects, but the range is limited. Ratings of "poor" indicate that the learners are limited to a 

is not described. 
Evidence table 6. Reporting of adult learning principles in studies assessing the effectiveness of continuing medical education 

F-157 

║Ratings of "good" indicate that the learners engage in applied or simulated activities during at least 50% of curriculum time. Ratings of "fair" indicat

or simulated activities <50% of curriculum time. Ratings of "poor" indicate that the curriculum does not provide opportunities for practicing knowleddescribed. 

¶Ratings of "good" indicate that the curriculum specifically allots faculty time/resources for supporting learners during independent learning projects. 

the curriculum provides only self-learning materials (e.g., online library or bulletin boards) or faculty are available for but not dedicated to supporting 

Ratings of "poor" indicate that the curriculum provides none of the above or it is not described. 

#Ratings of "good" indicate that the curriculum includes mechanisms for providing formative (feedback that is intended to help learners adjust their lea

completion of the curriculum) AND summative (feedback intended to inform learners of their progress upon completion of the curriculum) feedback t

indicate that the curriculum only includes mechanisms for providing one of the above. Ratings of "poor" indicate that feedback to learners is not prov**Ratings of "good" indicate that the curriculum describes mechanisms that are specifically intended to help learners reflect on their learning. These mreflection time and debriefing meetings or presentations that are intended for self-reflection. Ratings of "fair" indicate that the curriculum describes ledebriefing meetings or summary presentations, which are not specifically intended for reflection on learning but are likely to involve some reflection b"poor" indicate opportunities for self-reflection are not included in the curriculum or are not described. 

††Ratings of "good" indicate that the learners observe role models actually practicing goal behavior in clinical settings. Ratings of "fair" indicate learn

simulated settings. Ratings of "poor" indicate role modeling is not used or is not described. 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-158 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Knowledge Objectives Met, Evaluation Duration Not ReportedKiang, 200533

Int: NA CC: NA Int: many things were made available but it is not reported as to which groups used what methods; options included live, regional meetings, CDROMs, mailings, grandrounds CC: NA 

Int: NA CC: NA Int: not clear CC: NA 

Int: NA CC: NA Int: not clear CC: not clear 

Responses to questions about nonpredictive clinical factors and social factors

Knowledge Cognitive skills 

Yes Yes 

In Wisconsin, significant improvement occurred in the responses to the 2 questions about nonpredictive clinical factors and the social factor that may increase the likelihood of prescribing antimicrobial agents. Overall, Wisconsin clinicians demonstrated significant improvement regarding the influence of purulent nasal discharge (p=0.044) and productive cough (p=0.010) after accounting for temporal changes in Minnesota. 

In conclusion, thsuggests that the campaign had at modest positive the knowledge adecision-making primary care clinWisconsin. 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-159 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Curran, 200034 Int: Computer- based off-line, 

Internet, not real time CC: NA Int: Computerbased off-line, Internet, not real time 

Int: Case-based learning, Programmed learning, computer-based courseware CC: NA Int: Case-based learning, Programmed learning, computer-based courseware 

Int: Multiple time or repetitive CC: NA Int: One time 

Physician learning achievement

Knowledge Yes These results indicate that those physicians 

who participated in a computer-mediated instructional courseware program performed significantly better on a cognitive test of dermatologic office procedures than physicians who received no CME. A significant difference was also found between the mean ranked scores of experimental study group I and the control group (p=.000). A MannWhitney test of the mean ranked scores of experimental study group II and the control group also revealed a significant difference (p=.000). 

This evaluation rthat a hybrid commediated coursesystem was an emeans for increaknowledge and iself-reported comin dermatologic oprocedures, and participants were satisfied with the paced instruction of asynchronous computer conferfor collaborative information shariamong colleague

Costanza, 199235

Int: Live, Print CC: NA 

Int: Discussion group, Lecture, Point of care, Simulation (other than standardized patient or roleplay) CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician knowledge and attitudes toward breast cancer screening 

Knowledge Attitudes 

Yes Two barriers to use of 

mammography, concern that mammography is not cost-effective and concern about interpreting ambiguous reports, affected physicians less in the intervention group than in the control. 

This study demothat primary care physicians will chtheir screening presponse to interaimed at altering regarding mammbenefits or to bartheir sense of codevelopment. 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-160 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Andersen, 199036

Int: Live, Video CC: NA 

Int: Case-based learning, Lecture CC: NA 

Int: One time CC: NA 

Physicians' diagnostic accuracy for psychiatric conditions, as measured by pre and post-tests incorporating multiple clinical vignettes 

Knowledge Cognitive skills 

Yes Yes 

Compared with control, intervention physicians had significantly better post-test composite scores for affective and anxiety disorders; differences significant in 2 of 4 affective disorders (major depression and dysthymic disorder but not depression with psychotic features or bipolar), no specific anxiety disorders (0 of 4), and no somatic disorders. Pre-test scores negatively correlated with improvement in accuracy (more improvement on incorrect pre-test cases). 

A brief, single-seintervention can impact on physicpsychiatric diagnabilities. Additionphysicians’ particin the interventiomore likely to refpsychiatric patien

Block, 198837

Int: Live, Print CC: NA 

Int: Lecture, Point of care CC: NA 

Int: Multiple time or repetitive CC: NA 

Thresholds for treatment of cholesterol identified by physicians on test 

Knowledge Yes Physicians in the demonstration project 

reported lower values for normal, for initiation of diet therapy and lipid lowering therapy (p<0.05). 

Physicians in a community with bphysician and pueducation reportchanging their prmore significantly physicians in a comparison com
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-161 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Premi J, 199338

Int: Video, Print Int: Video, Print CC: Print CC: NA 

Int: Case-based learning, Readings Int: Case-based learning, Readings CC: Readings CC: NA 

Int: NR Int: NR CC: NR CC: NA 

Knowledge of chest pain diagnosis and management, as measured by test 

Knowledge Yes Using the test questions based on the workbook 

content, there was a significant increase in posttest scores of intervention (67->83) vs. control group (stable at 70) (p=0.0001), with no significant effect of review articles. Using general question about chest pain (not covered in the workbook), there was no difference comparing posttest to pretest scores in the groups (control 63.1-> 64.6%, intervention 59.9 -> 61.6%). 

"The program prsuccessful in impthe participating physicians' knowabout the diagnomanagement of pain." 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-162 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Knowledge Objectives Met, Evaluation Duration Less Than or Equal to 30 DaysHarris, 200239

Int: Internet, not real time CC: NA 

Int: Case-based learning, Feedback, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician knowledge, attitudes, and practice behaviors in managing domestic violence cases, as measured by survey 

Knowledge Attitudes 

Yes Yes 

Compared with control group, intervention group had a statistically significant better improvement in scores regarding provider fear of offending patients (p=0.008) and victimblaming (p=0.022). Compared with control group, intervention group had greater improvements in knowledge scores (p<0.001), but no statistically significant improvements in perceived system support or perceived frequency of asking about domestic violence. 

"An on-line DV eprogram can impphysician confidemeasured by selefficacy), attitudeself-reported knoin managing DV patients." 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-163 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Knowledge Objectives Met, Evaluation Duration Greater Than 30 DaysDoucet, 199840

Int: Live CC: Live 

Int: Problembased or teambased learning CC: Discussion group, Lecture 

Int: Multiple time or repetitive CC: One time 

The Key Feature Problems examination (evaluation of clinical reasoning skills) 

Knowledge Cognitive skills 

Yes Yes 

Whereas those enrolled in the lecture group had a mean score of 28 (SD = 5.23), participants of the PBL group had a mean examination score of 34.76 (SD = 5.96). This represents a 25% difference, deemed educationally significant. The difference in scores between the intervention and control groups was highly statistically significant (p=0.001). 

"Tests of knowleacquisition and ttests of clinical reskills indicated thPBL group benefmore than the legroup. In additionphysicians particthe PBL sessionenjoyed the interapproach and raprogram more hiParticipants in thgroup rated the psessions more fathan did their counterparts in thlecture group acseven of the nine program dimensHowever, physicthe lecture group report that the prheld their interescontributed to theknowledge and sprovided content to their practice. Physicians in the group also agreethe facilitators prthe concepts effebut were less incagree that the dicomponent of the enhanced their le
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-164 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Doucet, 199840

Int: Live CC: Live 

Int: Problembased or teambased learning CC: Discussion group, Lecture 

Int: Multiple time or repetitive CC: One time 

Pre-Post test assessing "knowledge acquisition in the area of headache diagnosis and management" (40 items, developed by a neurologist) 

Knowledge Yes There was no statistically significant difference 

observed in knowledge regarding headache diagnosis and management between the intervention and control groups at baseline (p=0.69). On the post-test, the intervention group performed at a statistically significantly higher level than the control group (p=0.05). Mean score for PBL group=33.30 (with SD = 3.67). Mean score for lecture group=31.38 (SD = 4.42). The authors, however, note that the observed post-test difference, at 6%, was not educationally significant. 

"Tests of knowleacquisition and ttests of clinical reskills indicated thPBL group benefmore than the legroup. In additionphysicians particthe PBL sessionenjoyed the interapproach and raprogram more hiParticipants in thgroup rated the psessions more fathan did their counterparts in thlecture group acseven of the nine program dimensHowever, physicthe lecture group report that the prheld their interescontributed to theknowledge and sprovided content to their practice. Physicians in the group also agreethe facilitators prthe concepts effebut were less incagree that the dicomponent of the enhanced their le
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-165 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Premi, 199441

Int: Live, Print CC: NA 

Int: Case-based learning, Discussion group, Problembased or teambased learning, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Knowledge about asthma, CHF, hormone replacement, and otitis media, as assessed on true-false test >3 months after topic discussed 

Knowledge Yes Pretest scores were similar in intervention and 

control (64.7 vs. 64.4) but improved to 76.8 (SD 11.7) in intervention vs. 65.7 in control (12.0), for a gain of 12.1% in intervention (p=0.0001). 

Small-group comCME may lead to knowledge gains discussed; unclepractice behaviorigorous evaluati
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-166 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Fordis, 200542

Int: Live, Print, Risk calculator Int: Internet, real time (e.g., streaming), Internet, not real time, Print, Risk calculator CC: NA 

Int: Case-based learning, Lecture, Programmed learning, Readings Int: Case-based learning, Lecture, Programmed learning, Readings CC: NA 

Int: One time Int: Multiple time or repetitive CC: NA 

Knowledge of cholesterol management, as measured by pretest to posttest 1 (immediate) and posttest 2 (12 week) scores 

Knowledge Yes Regarding the group main effect, the online 

CME group scored slightly higher than the live CME group when averaged across all 3 testing occasions (4.8% additional items correct, 95% CI, 0.6%-9.0%; partial w2= 0.01; p=.03). For both groups combined, posttest 1 scores surpassed pretest levels, and posttest 2 scores surpassed both pretest and posttest 1 levels (p<.001). The sizes of the differences from pretest to posttest 1 and pretest to posttest 2 were large, representing increases in percentage of items correct of 31.0% (95% CI: 27.0%-35.0%) and 36.4% (95% CI: 32.2%-40.6%), respectively. Although the increase from posttest 1 to posttest 2 was statistically significant (p<.001), the percentage increase was only 5.4% (95% CI: 2.6%-8.2%). 

Appropriately deevidence-based CME can producobjectively measchanges in behawell as sustained knowledge that acomparable or suthose realized froeffective live acti
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-167 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Stewart, 200543

Int: Internet, not real time CC: NA 

Int: Case-based learning, Discussion group, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician knowledge 

Knowledge Yes The intervention was associated with 

increased knowledge. Knowledge scores among the intervention group family physicians were higher than among control group family physicians on the prevention topic, at both 2 months and 6 months. The difference for the diabetes topic was in the expected direction but was not significant. 

The case-based discussion demoa mixed effect, wsignificant differeonly one of two cand for only two three outcomes (physicians’ knowand quality of praThe study identifpromising contineducation format based, on-line leas well as questifuture research rthe content and ocases presented line education. 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-168 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Beaulieu, 200444

Int: Live, Print Int: Live Int: Print CC: NA 

Int: Case-based learning, Discussion group, Problembased or teambased learning, Programmed learning, Readings Int: Case-based learning, Discussion group, Problembased or teambased learning, Readings Int: Discussion group, Programmed learning, Readings CC: NA 

Int: One time Int: One time Int: One time CC: NA 

Knowledge Yes Median pre and posttest scores (evaluating 

knowledge retention) were compared, showing an improvement in score for both questions describing patients with (p<0.1), and patients without GI risk factors (p<0.0001). Improvement was also said to be sustained at the 6 month evaluation. 

"The results of thevaluations have demonstrated thevidence-based interventions wesuccessful not onimproving physicknowledge regardiagnosis and management of also—more impoin changing their behavior to make appropriate therachoices for their The observed modification of thprescription pattereflects an improin their medical pwhich may lead tpatient outcomes generate greater efficiencies for thcare system." 

Cohn, 200219

Int: Live, Print Int: Live, Print CC: NA 

Int: Academic detailing Int: Academic detailing CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Provider knowledge 

Knowledge Yes Data strongly suggest that intervention 

increased DES knowledge (statistically significant among double, but not single intervention community). There were also significant increases in the proportion of healthcare providers in the intervention group who read national DES guidelines. 

"Academic detaiincrease DES knand history takinprimary care pro
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-169 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Lane, 200145

Int: Live CC: NA 

Int: Lecture, Standardized patient CC: NA 

Int: One time CC: NA 

One main outcome measure was whether or not physicians referred 90-100% of women over age 50 to get a mammogram every 1-2 years. The other main outcome was a reduction in the score of needing the CME activity (improved knowledge and understanding) 

Knowledge Practice behavior 

Yes Yes 

There was improvement in the need for CME scores of more physicians in the intervention group than the control group. The intervention significantly improved knowledge and behaviors about breast cancer screening practices. 

Participation in thactivity improved physicians' awarknowledge, and behaviors regardclinical breast exbreast cancer scpractices, as comto a control grou

Gifford, 199646

Int: Video, Print CC: NA 

Int: Programmed learning, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Factual knowledge about movement disorders, as measured by test 

Knowledge Yes For 6/7 factual knowledge questions, 

neurologists' knowledge about movement disorders was higher in the intervention group than in the control group (p<.01). Range for intervention was 44-86% correct and control 32-74% correct. 

The educational improved neuroloreported decisionmaking. 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-170 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Maiman, 198847

Int: Live, Print Int: Print CC: NA 

Int: Discussion group, Lecture, Readings Int: Readings CC: NA 

Int: Multiple time or repetitive Int: One time CC: NA 

Physician compliance knowledge 

Knowledge Yes Tutorial and printed materials increased 

pediatricians' knowledge about adherenceenhancing techniques; tutorials more than printed materials alone. 

CME increased pknowledge and compliance-enhapractices and resimprovement in madherence to the

White, 198548

Int: Live CC: NA Int: Live Int: Live 

Int: Case-based learning, Discussion group, Lecture CC: NA Int: NR Int: NR 

Int: One time CC: NA Int: One time Int: One time 

Knowledge gains on management of myocardial infarction after CME in multitopic, unitopic, or community CME, as measured by knowledge test pre-, post-, and 6 months postKnowl- edge Yes Generalizability study: Physicians in all groups had significant knowledge gains with retention at 6 months (baseline of 68.8-81.5% improving to 80.2-91.7% on post-test and 75.6- 82.8% at six months post-test; p<0.01). "A carefully conc

and executed traCME program canot only in signifiincreases in phyknowledge but arelated changes patient care prac"both knowledge behavioral changpersist for at leasmonths" 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-171 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Terry, 198149

Int: Live, Audio, Print Int: Live, Audio, Print Int: Audio, Print Int: Live, Audio, Print CC: Audio, Print 

Int: Case-based learning, Discussion group, Feedback, Lecture, Readings Int: Case-based learning, Discussion group, Lecture, Readings Int: Case-based learning, Feedback, Lecture, Readings Int: Case-based learning, Lecture, Readings CC: Case-based learning, Lecture, Readings 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive Int: Multiple time or repetitive CC: Multiple time or repetitive 

Knowledge and judgment in COPD management, as measured by selfassessment questionnaires at baseline, 8 months, and 18 months 

Knowledge Cognitive skills 

Yes Yes 

Intervention groups showed a 23% significant increase in scores on posttest 1 to match those of pulmonologists, while there was no improvement for control groups (p>0.05 for group differences). Intervention groups receiving feedback had similar scores on posttest 2 to intervention groups not receiving feedback. Among intervention physicians who did not agree to participate in standardized patients, posttest scores dropped to baseline. 

Physicians comphome study AV pincreased knowleabout diagnosis treatment of COPtheir behavior in simulated exercinot different from controls. Experimgroup physicians more patient-eduand smoking cesinformation durinvisits. Group mefor needs assessand feedback (giweeks after testsapparent effect bthe audiovisual m
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-172 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Short, 200650

Int: Internet, not real time CC: NA 

Int: Lecture, Problem-based or team-based learning, Readings, multiple media, interactivity CC: NA 

Int: Multiple time or repetitive CC: NA 

Educational outcome measure 

Knowledge Attitudes 

Yes Yes 

The MANOVA results showed a significant time by group interaction for the overall physician PREMIS scores (Wilks’s lambda = 0.274, p=0.001), indicating a change over 12 months for the study group that was significantly greater than for the control group. There were significant positive changes for the two background PREMIS scales (perceived preparation, p=0.000, and perceived knowledge, p=0.000), five of the six opinion scales (preparation, p=0.000; legal requirements, p=0.011; workplace issues, p=0.002; selfefficacy, p=0.013; and victim understanding, p=0.044); and the practice issues scale (p=0.000). Actual knowledge also improved, but the change was only significant at p <= 0.10 (p=0.06). The only scale that clearly showed no improvement was the opinions scale related to alcohol/drugs and IPV (p=0.445). 

This study shows asynchronous, interactive, onlinprogram developcadre of national in accordance wcurrent online edbest practices, csuccessful in chanumber of physicIPV knowledge, beliefs, and self-behaviors and prand that these chcan persist over months. 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-173 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Meredith, 200051

Int: Live, Print Int: Live, Print CC: NA 

Int: Lecture, Readings Int: Lecture, Readings CC: NA 

Int: Multiple time or repetitive Int: NA CC: NA 

Clinician knowledge about depression treatment 

Knowledge Yes Compared to physicians in usual care, physicians 

in both the intervention groups showed greater increases in overall depression knowledge (p=0.06). The change in overall depression knowledge scores was most pronounced for the QI therapy group (p=0.01 compared to usual care and p=0.09 compared to QI-meds). Knowledge scores did not change significantly for general treatment & antidepressant medication but there were significant increases in psychotherapy knowledge scores for the QI meds and QI therapy groups compared to usual care (p=0.04 & 0.04 respectively). 

Clinicians exposmultifaceted QI pfor depression in managed primarpractices gained knowledge abouassessing and trdepression over months following implementation. main activity accompanying thin knowledge waparticipation in thspecific types of and informal eduactivities that wethe study protoco
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-174 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Mann, 199752

Int: Live, Video, Print Int: Live, Video, Print Other: cue stickers on medical chart CC: NA 

Int: Case-based learning, Demonstration, Discussion group, Readings, Simulation (other than standardized patient or roleplay) Int: Case-based learning, Demonstration, Discussion group, Point of care, Readings, Simulation (other than standardized patient or roleplay) CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NR 

Physician knowledge, as measured by posttests 

Knowledge Yes Significant (p=0.002) group x time effect was 

seen in the overall knowledge score; statistically significantly higher scores were seen in the intervention group (compared to control, p=0.02) at posttest. Statistically significant increases in knowledge of screening, selecting and managing patients and on dietary modification principles, and a nonsignificant, but increasing trend (p=0.09) in dietary assessment techniques, was also seen when comparing the control group to both intervention groups. 

Educational trainworkshops appeeffective in chanphysicians' behashould thus, be calong with additioresearch on the mechanisms of wbehavior change 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-175 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Gerstein, 199953

Int: Live CC: NA 

Int: Case-based learning, Discussion group CC: NA 

Int: One time CC: NA 

Knowledge, attitudes, and practice behavior regarding diabetes care, based on participant questionnaire 

Knowledge Attitudes Practice behavior 

Yes Yes Yes 

After 40 days, participants' overall scores improved significantly while there was no change in the controls' scores (F =24.14; p<0.0001). Significant improvement was also noted in domains of attitude (F=31.75; p<0.0001), knowledge (F=4.23; P =0.041), and practice behavior (F=10.43; p=0.0014). However, improvement was not apparent after a year. Participants who completed 425-day assessment scored lower in attitude subscale compared to controls, despite having initially scored higher at the 40-day assessment. 

An interactive, smgroup, diabetes continuing educaprogram effectivedisseminates praguidelines to famphysicians. The such a program after 1 year. 

Knowledge Objectives Not Met, Evaluation Duration Not ReportedGerrity, 199932

Int: Live, Video, Audio, Print CC: NA 

Int: Clinical experiences, Discussion group, Feedback, Lecture, Readings, Role play CC: NA 

Int: Multiple time or repetitive CC: NA 

Knowledge about depression diagnosis and management, as measured on a 54-item written test 

Knowledge No Intervention and control group did not differ 

significantly in mean scores (41.5 vs. 39.3, p=0.136) or in their assessment of their knowledge in depression (64% vs. 44% rating selves as very knowledgeable about antidepressants, p=0.175). 

The Depression Education Prograchanged physiciabehavior and maimportant compothe efforts to impcare of depressepatients. 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-176 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Knowledge Objectives Not Met, Evaluation Duration Greater Than 30 DaysChung, 200454

Int: Live, Internet, not real time CC: Live 

Int: Case-based learning, Lecture, Readings CC: Lecture 

Int: Multiple time or repetitive CC: One time 

Knowledge of diagnosis and management of victims of bioterrorism.

Knowledge No There was no significant change from baseline to 

1 and 6 month follow-up for the intervention or control group. 

"Even shortly aftbioterrorist attackphysicians were to demonstrate aincrease in knowafter voluntarily participating in a based educationintervention. ...palearning through creation of medicsites on the concbioterrorism may the most effectivmethod to educaclinicians and, thprepare for future attacks." 

Elliott, 199755

CC: NA Int: Live, Print 

CC: NA Int: Case-based learning, Clinical experiences, Discussion group, Lecture, Readings 

CC: NA Int: Multiple time or repetitive 

Physician and nurse knowledge of CPM 

Knowledge No There was a trend toward improved knowledge in 

the intervention group, but it was not statistically significant. 

With the exceptioattitudes of patiecaregivers, there trend toward improvement in pratings as well as provider attitudes knowledge but thwas minor and thresults were not overwhelmingly convincing 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-177 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Evans CE, 198656

Int: Print CC: NA 

Int: Readings, Chart cue materials offered, but not necessarily implemented CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician knowledge of hypertension, as measured by multiplechoice posttest 

Knowledge No Mean knowledge scores did not differ between 

groups (50 and 52%, p>0.05). There were no significant correlations between scores and patients' blood pressures, but a weak correlation between scores and patient compliance (r=0.25, p<0.05). 

"Our study demono influence of a continuing mediceducation prograpractices of physon the control of pressure of hypepatients referred community survethese physicians program was beg

Knowledge Objectives With Mixed Results, Evaluation Duration Not ReportedMaxwell, 198457

Int: Live CC: NA 

Int: Case-based learning, Discussion group CC: NA 

Int: Multiple time or repetitive CC: NA 

Knowledge gains on topics under discussion in medical care evaluation committees, as measured by 30-item multiplechoice tests 

Knowledge Mixed Statistically significant improvement pre- to 

post- in only 1 of 3 departments. Scores improved in "successful committees" on specific topics compared with those of matched controls or their own scores on control topics (p<0.01). 

Medical care evameetings have educational valu
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-178 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Kemper, 200658

Int: Internet, not real time Int: Internet, not real time Int: Internet, not real time Int: Internet, not real time 

Int: Discussion group, Problembased or teambased learning Int: Discussion group, Problembased or teambased learning Int: Discussion group, Problembased or teambased learning Int: Discussion group, Problembased or teambased learning 

Int: online discussions; case-based, selfinstructional modules Int: online discussions; case-based, selfinstructional modules Int: online discussions; case-based, selfinstructional modules Int: online discussions; case-based, selfinstructional modules 

Differences of the three types of scores between the four groups 

Knowledge Attitudes 

No control group

No significant difference in any of the three outcome scores by delivery strategy. 

Results from this have important implications for professional edufuture research. Educators wishinthe Internet can confident that improvements arheavily dependecurriculum delivestrategy. The demethod that is eainstructors (bolusmay present the barriers for comp(such as full maiinstitutional firewparticularly for shintroductory courdelivery strategy easiest does not to result in substworse outcomesAdditional interveare needed to imclinicians' commbehavior. This stdemonstrated thInternet educatiosubstantially impclinicians' knowleconfidence regarthe delivery strat
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-179 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Kemper, 200658

Int: Internet, not real time Int: Internet, not real time Int: Internet, not real time Int: Internet, not real time 

Int: Discussion group, Problembased or teambased learning Int: Discussion group, Problembased or teambased learning Int: Discussion group, Problembased or teambased learning Int: Discussion group, Problembased or teambased learning 

Int: online discussions; case-based, selfinstructional modules Int: online discussions; case-based, selfinstructional modules Int: online discussions; case-based, selfinstructional modules Int: online discussions; case-based, selfinstructional modules 

Changes in knowledge, confidence and communication scores

Knowledge Attitudes 

No control group

Statistically significant improvement in scores after they were taken. 

Results from this have important implications for professional edufuture research. Educators wishinthe Internet can confident that improvements arheavily dependecurriculum delivestrategy. The demethod that is eainstructors (bolusmay present the barriers for comp(such as full maiinstitutional firewparticularly for shintroductory courdelivery strategy easiest does not to result in substworse outcomesAdditional interveare needed to imclinicians' commbehavior. This stdemonstrated thInternet educatiosubstantially impclinicians' knowleconfidence regarthe delivery strat
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-180 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Rosenthal, 200559

Int: Live, Handheld, Review of data CC: Review of data 

Int: Lecture, Readings, Review of practice data CC: Review of practice's data 

Int: Multiple time or repetitive CC: One time 

Parent reports of knowledge 

Knowledge 

No control group

For parents of both 1-month-olds and 6-montholds, the control and intervention parents did not differ in the adjusted change in the proportion of parents reporting parent knowledge in agespecific topics. 

An office system intervention imprparent reports of of anticipatory gubut did not changknowledge or pabehavior. 

Hergen-roeder, 200260

Int: Video, Print Int: Live, Video, Print 

Int: Demonstration Int: Demonstration, Feedback, Simulation (other than standardized patient or roleplay) 

Int: One time Int: One time 

Physician knowledge of performing a physical exam 

Knowledge 

No control group

There was a statistically significant increase between baseline and followup in knowledge test scores for both intervention groups, and there was a significant difference between the groups at followup. 

"This study demothat improvemenphysicians' knowand skills in perfoankle and knee pexaminations weassociated with tphysicians' particin either intervenimprovements in physicians' knowand skills in the aand knee examinwere greater in tvideotape plus sintervention grouthe videotape-alogroup." 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-181 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Des Marchais, 199061

Int: Live, Video Int: Live, Video 

Int: Case-based learning, Demonstration, Discussion group, Reflection about personal experiences Int: Case-based learning, Demonstration, Discussion group, Reflection about personal experiences 

Int: NR Int: NR 

Appropriate assessment of interpersonal skills of videotaped physiciansin-training, as measured by participants' scoring of the video compared with median score in group 

Knowledge No control 

group 

Both groups showed increased interrater agreement after the first portion of training, but no substantial or significant increase beyond that after the second portion of the training. Order of training (theoretical vs. practical first) had no significant effect. Psychiatrists had a higher reliability score before and after training compared with family physicians. 

A short, specific can increase inteagreement in the assessment of thdoctor-patient relationship. 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-182 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Knowledge Objectives With No Control Group, Evaluation Duration Less Than or Equal to 30 DaysSlotnick, 199317

Int: Print Int: Print Int: Print Int: Print Int: Print Int: Print 

Int: Readings Int: Readings Int: Readings Int: Readings Int: Readings Int: Readings 

Int: One time Int: One time Int: One time Int: One time Int: One time Int: One time 

Knowledge of prescribing information as applied to clinicallyoriented test questions on a post-intervention test 

Knowledge No control 

group

Higher mean test scores were seen for clinical challenge readers vs. conventional ad readers, with a reported effect size of 0.45 SD. There was no difference with an introductory ad explaining the use of the clinical challenge. Knowledge scores were higher for older drug than newer drug. Compared with those reading conventional ads, physicians using enhanced ads had RR 1.48 (1.07-2.05) for correctly answering questions drawn from the clinical challenge scenario, vs. RR 1.12 (NS) for those not covered in the scenario. 

Pharmaceutical aenhanced clinicachallenges increphysician knowletesting, specificaquestions that wdrawn from the cchallenges them
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-183 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Knowledge Objectives With Mixed Results, Evaluation Duration Greater Than 30 DaysChodosh, 200662

Int: Live, Internet, not real time, Print CC: NA 

Int: Discussion group, Lecture CC: NA 

Int: Multiple time or repetitive CC: NA 

Providers' knowledge regarding dementia care, as measured by test 

Knowledge Mixed Higher percentage of intervention than usualcare clinic providers 

answered both of knowledge-related questions on capacity determination correctly (adjusted OR 2.4, 95% CI: 1.2-4.8). No difference existed for questions on knowledge of areas of delirium evaluation, patient safety, and depression treatment. 

Despite a succesintervention demonstrating siimprovements in of care for patiendementia, providknowledge and awere minimally a
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-184 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Knowledge Objectives With No Control Group, Evaluation Duration Greater Than 30 DaysWhite, 200463

Int: Live CC: Live 

Int: Case-based learning, Discussion group CC: Case-based learning, Discussion group, Lecture 

Int: One time CC: One time 

Performance of questionnaire (knowledge) 

Knowledge No control 

group 

Groups did not differ on their performance. Their performance, however, varied across time, as indicated by the significant main effect for time, F (2,33)=18.10, p<.01, which was consistent for both groups. Both groups significantly improved their performance at the second administration and retained this level at the third, delayed testing.For the knowledge measure, there was a significant drop in performance comparing the delayed test 3 months later to the immediate post-test, but it was still significantly better than performance on the pretest. 

This study found evidence that PBbetter than more learning sessionfacilitating knowlgain, knowledge retention, or chaattitude about asmanagement. However, this stusimilar to other ssuggests that theweak evidence thphysicians attribuadditional value tmore interactive approach to CME
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-185 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Lockyer, 200264

Int: Live Int: Live 

Int: Case-based learning, Discussion group, Lecture, Role play Int: Case-based learning, Discussion group, Lecture 

Int: One time Int: One time 

Physician practice behaviors: knowledge scores 

Knowledge No control 

group

Both track 1 (introductory course) and track 2 (advanced course) participants improved statistically significantly between pre and posttest scores; track 1 had a moderate effect size difference (0.5), and track 2 a minimal effect size (0.2). Between tracks comparisons showed statistically significant differences between tracks for precourse assessment of knowledge but not for post-course assessment of knowledge. 

Track 1 (introduccourse) physiciaimproved moderwhile track 2 (adcourse) physiciashowed a small onegligible changknowledge, comfinvolvement in pcare for dementipatients. TrackinCME - assigning physicians to coubased on pre-coability, interest, oneeds further stu

Kutcher, 200265

Int: Live, Print Int: Live, Print 

Int: Academic detailing, Discussion group, Lecture, Readings Int: Discussion group, Lecture, Readings 

Int: One time Int: One time 

Knowledge test 

Knowledge No control 

group 

Both enhanced and general intervention groups improved from pre to post test (p<0.0001), but no comparison between groups was made. 

"A well-designeddirectional, brief, and low-cost eduprogram can incrfamily physiciansknowledge of deimprove their diaskills, and optimitreatment of dep
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-186 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Heale, 198866

Int: Live Int: Live Int: Live 

Int: Lecture Int: Case-based learning, Discussion group Int: Problembased or teambased learning 

Int: One time Int: One time Int: One time 

Physician knowledge of six topic areas addressed by CME 

Knowledge No control 

group (No for technique) 

Performance assessments on the knowledge MCQ were not provided. The only statement was that there was no difference in acquired or retained knowledge (7 months) among the three learning formats. 

Within a one day course in family medicine, the leaformat had no efacquired or retaiknowledge or on physician performthree patient proPhysicians rated small group probbased format hig

Greenberg, 198525

Int: Live Int: Live 

Int: Lecture Int: Case-based learning 

Int: One time Int: One time 

Knowledge regarding common pediatric problems, as assessed by a multiplechoice test on one of 4 topics 

Knowledge No control 

group (No for technique) 

Case-based learners showed improvement on 64% of the post-tests vs. 29% of lecture-based learners, but this was not statistically significant. 6-9 months later, 55% of the lecture-based learners and 43% of the case-based learners showed a decline in test scores; no tests of significance were reported. 

Case-based learcompared with lebased learning, wassociated with ssignificant advanteaching skills anbehavior in manaof common pediaproblems. 
Evidence table 7. Effectiveness of continuing medical education on short-term and long-term knowledge outcomes 

F-187 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objective 

Objectives 

met Summary of results Overall concl

Labelle, 200467

Int: Live, Print Int: Live 

Int: Case-based learning, Demonstration, Discussion group, Lecture, Problem-based or team-based learning, Role play, Standardized patient Int: Case-based learning, Role play, Standardized patient 

Int: Multiple time or repetitive Int: Multiple time or repetitive 

Long-term knowledge of written action plans and related asthma management criteria, as assessed by OSCE 

Knowledge No control 

group

Written action plan (WAP) knowledge increased between baseline and 12 months for those GPs in the second group (participating in the OSCE (objective structured clinical examination) at 6 months) p=0.01. WAP knowledge did increase, but not to a statistically significant degree for those not participating in the OSCE at 6 months (group 1), p=0.28. 

"This study demoa positive impact combination of a based, interactivworkshop featuripreformatted toodrafting of WAPsreinforcing OSCE months post-woron GP knowledgself-reported use WAPs. These resupport the concreviews of CME programs that intand sequential educational activproviding opportupractice appear promising in chaphysician practic

CC = concurrent control; CI = confidence interval; CME = continuing medical education; COPD = chronic obstructive pulmonary disease; DES = die

domestic violence; GI = gastrointestinal; GP = general practitioner; Int = intervention group; KFP = Key Features Program; MANOVA = multiple anamultiple choice questionnaire; NA = not applicable; NR = not reported; NS = not significant; OR = odds ratio; OSCE = objective structured clinical ex

based learning; PREMIS = Physician Readiness to Manage Intimate Partner Violence Survey; QI = quality improvement; SD = standard deviation 
Evidence table 8. Grading of the body of evidence for the effectiveness of continuing medical education and the effectiveness 

education  F-188 

Key Question 1/2 – Effectiveness of CME on Acq

Knowledge Attitudes Skills P

Be

Quantity of Evidence:

Number of studies 

40 36 22 

Quality and Consistency of Evidence:

Were study designs mostly randomized trials (high quality), nonrandomized controlled trials (medium quality), observational studies (low quality)? 

High High High 

Did the studies have serious (-1) or very serious (-2) limitations in quality? (Enter 0 if none) 

-1 -1 -2 

Did the studies have important inconsistency? (-1) -1 -1 -1 Was there some (-1) or major (-2) uncertainty about the directness (i.e. extent to which the people, interventions and outcomes are similar to those of interest? 

-1 -1 -1 

Were data imprecise or sparse? (-1) (i.e. lack of data or very wide confidence intervals that may change conclusions) 

-1 -1 -1 

Did the studies have high probability of reporting bias? (-1) -1 -1 0 Did the studies show strong evidence of association between intervention and recruitment outcome? (“strong” if significant relative risk or odds ratio > 2 based on consistent evidence from 2 or more studies with no plausible confounders or some other measure that suggests a large difference between study groups (+1); “very strong” if significant relative risk or odds ratio > 5 based on direct evidence with no major threats to validity or some other measure that suggests a very large difference between study groups (+2))- use your clinical judgment for absolute differences.

0 0 +1 

Did the studies have unmeasured plausible confounders that most likely reduced the magnitude of the observed association? (+1) 

0 0 +1 

Overall grade of evidence (high, moderate, low, very low) Very low Very low Low Ve

High = further research is very unlikely to change our confidence in the estimates; moderate = further research is likely to have an important impact on estimate of effect and may change the estimate; low = further research is likely to have an important impact on our confidence in the estimate of effect 

estimate; very low = any estimate of effect is very uncertain. 
Evidence table 8. Grading of the body of evidence for the effectiveness of continuing medical education and the effectiveness 

education  F-189 

Key Question3 

Psychomotor Skills Communication 

Skills 

Cognitive Sk

Quantity of Evidence:

Number of studies 

7 

172 studies 

2 

14 

2 

37 

Quality and Consistency of Evidence:

Were study designs mostly randomized trials (high quality), nonrandomized controlled trials (medium quality), observational studies (low quality)? 

Yes No No 

Did the studies have serious (-1) or very serious (-2) limitations in quality? (Enter 0 if none) 

Only 1/6 had reproducible methodology Quality of studies assessed in only 3 

None had reproducible methods Only 1 of 3 assessed study quality 

Not reproducibNo assessmenstudy quality 

Did the studies have important inconsistency? (-1) Yes No Yes Was there some (-1) or major (-2) uncertainty about the directness (i.e. extent to which the people, interventions and outcomes are similar to those of interest? 

No No No 

Were data imprecise or sparse? (-1) (i.e. lack of data or very wide confidence intervals that may change conclusions) 

Sparse, only 2 had 

quantitative data 

Sparse Sparse 

Did the studies have high probability of reporting bias? (-1) Yes Yes Yes Did the studies show strong evidence of association between intervention and recruitment outcome? (“strong” if significant relative risk or odds ratio > 2 based on consistent evidence from 2 or more studies with no plausible confounders or some other measure that suggests a large difference between study groups (+1); “very strong” if significant relative risk or odds ratio > 5 based on direct evidence with no major threats to validity or some other measure that suggests a very large difference between study groups (+2))- use your clinical judgment for absolute differences.

Of the quantitative 

results, one 

showed strong 

associations 

between 

transference of 

skills to OR (task 

completion rate and 

error rate) 

Not quantitated, but 

consistently better 

with role play and 

or simulated 

patients 

Average effect 

0.63 across 

studies tha

collected dat

one review 

Effect sizes ra

-.04 for comp

assisted trau

manageme

teaching to 5.0

anesthesia 

simulator 
Evidence table 8. Grading of the body of evidence for the effectiveness of continuing medical education and the effectiveness 

education  F-190 

Key Question3 

Psychomotor Skills Communication 

Skills 

Cognitive Sk

Did the studies have unmeasured plausible confounders that most likely reduced the magnitude of the observed association? (+1) 

Yes, large 

differences in types 

of learners, skill 

taught, intensity 

and frequency of 

simulation method, 

and outcome 

measures. 

Yes differences in 

learners, intensity 

and frequency, and 

measures 

Yes, differenc

learners, an

content bein

taught 

Overall grade of evidence (high, moderate, low, very low) Moderate Low Low 

High = further research is very unlikely to change our confidence in the estimates; moderate = further research is likely to have an important impact on estimate of effect and may change the estimate; low = further research is likely to have an important impact on our confidence in the estimate of effect 

estimate; very low = any estimate of effect is very uncertain. 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-191 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Attitudinal Objectives Met, Evaluation Duration Not ReportedBloomfield, 200568

Int: Live, Print, Patient letter prompting patient to discuss with provider about treatment Int: Live, Print, Chart reminders appearing on cover page of patient's medical record Int: Live, Print, Progress notes reminding PCP about appropriate approach CC: NA 

Int: Discussion group, Lecture: Patient informs provider Int: Discussion group, Lecture Point of care Int: Discussion group, Lecture Point of care CC: NA 

Int: One time Int: One time Int: One time CC: NA 

Provider view of live education, opinion leader, and prompts 

Attitudes Yes 99% of respondents 

agreed somewhat or completely that the workshop increased their knowledge and 86% stated that they were more likely to treat as a result of the workshop. 60% felt that the opinion leader had influenced their prescribing decisions and about 50% stated that they agreed or strongly agreed that the prompts positively influenced their prescribing, while 40% found the prompts annoying and 14% felt the prompts did more harm than good. 

In conclusion, thshows that a relasimple interventieducational workopinion leader inand prompts) batheoretical modeprovider behaviois designed to adempirically identibarriers, can ressubstantial improin provider prescbehavior. 

Mukohara, 200513

Int: Computerbased off-line CC: Computerbased off-line 

Int: Readings CC: Readings 

Int: Multiple time or repetitive CC: Multiple time or repetitive 

Time spent reading 

Attitudes Yes The control group 

reported a significant increase in the time they spent reading medical journals, amounting to 20% (26 minutes per week), while the WBJC group’s reading time decreased by 7% (10 minutes per week). The difference in the change in reading time between the groups was 35.3 minutes (95% CI: 5.1, 65.6). 

While doctors appreciated thessummaries, whicimproved their reefficiency, the intervention had impact on their uresearch evidencpractice. 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-192 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Curran, 200034 Int: Computer- based off-line, 

Internet, not real time CC: NA Int: Computerbased off-line, Internet, not real time 

Int: Case-based learning, Programmed learning, computer-based courseware CC: NA Int: Case-based learning, Programmed learning, computer-based courseware 

Int: Multiple time or repetitive CC: NA Int: One time 

Self-reported performance change 

Attitudes Yes Wilcoxon tests of the 

retrospective responses to the performance statements revealed significant differences between self-reported pre- and post-learning performance at p<.05. This suggests that participants experienced improved competencies in all performance areas related to dermatologic office procedures as a result of participating in the computer-mediated instructional program. 

This evaluation rthat a hybrid commediated coursesystem was an emeans for increaknowledge and iself-reported comin dermatologic oprocedures, and participants were satisfied with the paced instruction of asynchronous computer conferfor collaborative information shariamong colleague

Curran, 200034 Int: Computer- based off-line, 

Internet, not real time CC: NA Int: Computerbased off-line, Internet, not real time 

Int: Case-based learning, Programmed learning, computer-based courseware CC: NA Int: Case-based learning, Programmed learning, computer-based courseware 

Int: Multiple time or repetitive CC: NA Int: One time 

To determine participant (physician) reaction to the computerbased CME activity 

Attitudes Yes Participants were very 

satisfied with the selfpaced instruction and use of asynchronous computer conferencing for collaborative information sharing among colleagues. 

This evaluation rthat a hybrid commediated coursesystem was an emeans for increaknowledge and iself-reported comin dermatologic oprocedures, and participants were satisfied with the paced instruction of asynchronous computer conferfor collaborative information shariamong colleague
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-193 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Costanza, 199235

Int: Live Print CC: NA 

Int: Discussion group, Lecture, Point of care, Simulation (other than standardized patient or roleplay) CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician knowledge and attitudes toward breast cancer screening 

Knowledge Attitudes 

Yes Two barriers to use of 

mammography, concern that mammography is not cost-effective and concern about interpreting ambiguous reports, affected physicians less in the intervention group than in the control. 

This study demothat primary care physicians will chtheir screening presponse to interaimed at altering regarding mammbenefits or to bartheir sense of codevelopment. 

Lane, 199169

Int: Live Print Int: Live Print Int: NA CC: NA 

Int: ARS, Clinical experiences, Demonstration, Discussion group, Feedback, Lecture Mentor/Preceptor, Readings Int: ARS, Clinical experiences, Demonstration, Discussion group, Feedback, Lecture, Mentor/Preceptor, Readings Int: NA CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive Int: NO CME intervention, just free mammography CC: NA 

Change in physicians' attitudes about making mammography referrals 

Attitudes Yes Physicians remained 

concerned about cost, radiation exposure, unnecessary biopsies, cost effectiveness, patient discomfort and sufficiency of physician exams, as well as insufficient time with patients. However, there was a statistically significant reduction in concern about cost, radiation exposure and cost-effectiveness in the intervention group whereas there was no significant change in any attitudes in the control sites. 

Physicians in the interventions rep(self) an increase number of mammography r
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-194 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Andersen, 199036

Int: Live, Video CC: NA 

Int: Case-based learning, LectureCC: NA 

Int: One time CC: NA 

Physicians' treatment recommendations for psychiatric conditions, as measured by pre- and post-tests incorporating multiple case vignettes 

Attitudes Practice behavior 

Yes Yes 

Compared with control, intervention physicians were significantly more inclined to refer patients to mental health professional and less inclined to treat them in primary care. Referral was high on pretest for both groups, but experimental group showed increase for 6 of 7 (1 tie) disorders while control did for 2 of 5 (3 ties). For medication, there were no significant effects. For behavioral therapy, there were significant increases for intervention group. 

A brief, single-seintervention can impact on physicpsychiatric diagnabilities. Additionphysicians particthe intervention wmore likely to refpsychiatric patien
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-195 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Attitudinal Objectives Met, Evaluation Duration Less Than or Equal to 30 DaysHarris, 200239

Int: Internet, not real time CC: NA 

Int: Case-based learning, Feedback, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician knowledge, attitudes, and practice behaviors in managing domestic violence cases, as measured by survey 

Knowledge Attitudes 

Yes Yes 

Compared with control group, intervention group had a statistically significant better improvement in scores regarding provider fear of offending patients (p=0.008) and victimblaming (p=0.022). Compared with control group, intervention group had greater improvements in knowledge scores (p<0.001), but no statistically significant improvements in perceived system support or perceived frequency of asking about domestic violence. 

"An on-line DV eprogram can impphysician confidemeasured by selefficacy), attitudeself-reported knoin managing DV patients." 

Harris, 200239

Int: Internet, not real time CC: NA 

Int: Case-based learning, Feedback, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician self-efficacy in managing domestic violence cases, as measured by survey 

Attitudes Yes In the intervention group, 

self-efficacy improved by 17.8% versus -0.6% mean change in control group (p<0.001). 

"An on-line DV eprogram can impphysician confidemeasured by selefficacy), attitudeself-reported knoin managing DV patients." 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-196 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Attitudinal Objectives Met, Evaluation Duration Greater Than 30 DaysDoucet, 199840

Int: Live CC: Live 

Int: Problembased learning or team-based learning CC: Discussion group, Lecture 

Int: Multiple time or repetitive CC: One time 

Program evaluation assessing satisfaction 

Attitudes Yes 85% of the PBL 

respondents rated the program as excellent compared to 35% of the control group (p = 0.001). Except for "were well organized" and "facilitator presented concepts effectively," there were highly statistically significant differences between the two groups' self-reported satisfaction with the program domains. These were: 1. "Met its objectives" - p = 0.013 2. "Met my expectations" - p = 0.001 3. "Contributed to my knowledge and skills" - p = 0.004 4. "Program material was helpful" - p = 0.019 5. "Discussions enhanced my learning" - p = 0.000 

"Tests of knowleacquisition and ttests of clinical reskills indicated thPBL group benefmore than the legroup. In additionphysicians particthe PBL sessionenjoyed the interapproach and raprogram more hiParticipants in thgroup rated the psessions more fathan did their counterparts in thlecture group acseven of the nine program dimensHowever, physicthe lecture group report that the prheld their interescontributed to theknowledge and sprovided content to their practice. Physicians in the group also agreethe facilitators prthe concepts effebut were less incagree that the dicomponent of the enhanced their le
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-197 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Lewis, 199370

Int: Live, Video Int: Print CC: NA 

Int: Discussion group, Lecture Int: Readings CC: NA 

Int: One time Int: One time CC: NA 

Physician attitudes toward the course on HIV 

Attitudes Yes 58% approved of the 

activity, 20% were neutral, and 22% disapproved at the start. After the intervention, 68% approved, 18% were neutral, and 14% disapproved. This represented a positive impact of the intervention. 

These results suthat a medical edprogram that goebeyond standard and incorporates interactive formachange physician behaviors. 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-198 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Kronick, 200371

Int: Live CC: NA 

Int: Case-based learning, Demonstration CC: NA 

Int: One time CC: NA 

Physician comfort level with accessing medical information via internet. 

Attitudes Practice behavior 

Yes Mixed

Intervention group physicians increased their frequency of and comfort with accessing Internet medical information. Statistically significant differences in change from baseline between intervention and control group were seen with frequency of use of the Internet to address patient related questions (p=.009), in the comfort level using online databases (p=.032) and in the frequency of accessing online databases (p=.044). Non-statistically significant differences were seen in frequency of accessing email to answer patient-related questions and comfort in using email, the Internet, opinion of the value of the Internet and in accessing online full-text journals. 

"Rural physicianscomfort and comin use of computaddress patient pcan be improved individualized 3-htraining session. data suggest thaphysicians distanmedical libraries have excellent aevidence-based resources; as coto the Internet befaster, more unifreliable across communities, the will become easideliver." 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-199 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Fordis, 200542

Int: Live, Print, Risk calculator Int: Internet, real time (e.g., streaming), Internet, not real time, Print, Risk calculator CC: NA 

Int: Case-based learning, Lecture, Programmed learning, Readings Int: Case-based learning, Lecture, Programmed learning, Readings CC: NA 

Int: One time Int: Multiple time or repetitive CC: NA 

Participant satisfaction, as measured by questionnaire 

Attitudes Yes All live CME participants 

and 94% of online CME participants rated the learning experience as “good” or “excellent.” Nonparametric correlations revealed no significant associations between course satisfaction and test performance. Nearly all of the 40 online CME participants (95%) who attended the live Web conference rated it as “useful” or “very useful” and indicated that it provided an opportunity to solidify guideline knowledge and obtain answers to questions. 

Appropriately deevidence-based CME can producobjectively measchanges in behawell as sustained knowledge that acomparable or suthose realized froeffective live acti
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-200 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Brown, 199972

Int: Live CC: NA 

Int: Clinical experiences, Discussion group, Lecture, Role play, Clinicians audiotaped interaction with patients and listened between workshops CC: NA 

Int: Multiple time or repetitive CC: NA 

Selfassessment of clinicians' communication skills, attitudes, and behavior, as measured by participant questionnaire 

Attitudes Skills (psycho,motor, or procedural skills) Practice behavior 

Yes Yes Yes 

Intervention group noted substantial improvements compared with control in 8 of 24 skills / attitudes / behaviors: awareness of and confidence in dealing with patients whom they found difficult; abilities to compliment patients’ efforts, ask open-ended questions, address psychosocial factors, express empathy and reassurance, and clarify expectations. 33% of clinicians reported that fewer than 5% of visits were frustrating after the program (compared with 21% of clinicians at baseline). Three months after the program, clinicians in the intervention group reported that it had improved communication with patients. On average, however, the scores for clinicians’ rating of improvement in patient satisfaction and improvement in clinicians’ personal satisfaction in their work decreased below the midpoint on the five-point scale (mean rating, 2.85 for both items). 

”Thriving in a BuPractice: PhysiciPatient Communa typical continuimedical educatioprogram geared developing cliniccommunication snot effective in imgeneral patient satisfaction. To iglobal visit satisfcommunication straining programneed to be longemore intensive, tbroader range of and provide ongoperformance fee
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-201 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Lin, 199773

CC: NA Int: Live, Video, Print 

CC: NA Int: Academic detailing, Feedback, Lecture, Readings, Role play 

CC: NA Int: Multiple time or repetitive 

Physician attitudes toward the intervention 

Attitudes Yes This outcome was 

alluded to in the study but was not a primary endpoint in this paper (it was reported elsewhere). However, immediately post-intervention, physicians did have significant improvements in how they felt about the depression care they were delivering. 

The results do nosupport the concthis complex and aggressive interveffected a sustaiin practice behavmonths after the intervention. In fsome positive chwere noted immeafter the intervenprescribing pattewere lost 6 montthe intervention w

Maiman, 198847

Int: Live, Print, Int: Print CC: NA 

Int: Discussion group, Lecture, Readings Int: Readings CC: NA 

Int: Multiple time or repetitive Int: One time CC: NA 

Physician self-report of compliance enhancing behavior 

Attitudes Yes 78.2% of physicians in 

tutorial group, 62.9% of pediatricians in print only group, and 45.9% of controls reported undertaking 8 or more specific behaviors in their practice related to enhancing compliance. 

CME increased pknowledge and compliance-enhapractices and resimprovement in madherence to the

Pinto, 199874

CC: Print Int: Live, Print 

CC: Readings Int: Discussion group, Readings, Role play 

CC: Given materials to readexposure up to the physicians Int: One time 

Physician confidence in counseling on exercise 

Attitudes Yes The intervention group 

had a statistically significant higher rating of confidence than the control group (p<.05). 

The program impphysician confidecounseling and psatisfaction, but increase the phyreports of exerciscounseling provipatients. 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-202 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Short, 200650

Int: Internet, not real time CC: NA 

Int: Lecture, Problem-based learning or teambased learning, Readings, multiple media, interactivity CC: NA 

Int: Multiple time or repetitive CC: NA 

Educational outcome measure 

Knowledge Attitudes 

Yes Yes 

The MANOVA results showed a significant time by group interaction for the overall physician PREMIS scores (Wilks’s lambda = 0.274, p = 0.001), indicating a change over 12 months for the study group that was significantly greater than for the control group. There were significant positive changes for the two background PREMIS scales (perceived preparation, p = 0.000, and perceived knowledge, p = 0.000), five of the six opinion scales (preparation, p = 0.000; legal requirements, p = 0.011; workplace issues, p = 0.002; self-efficacy, p = 0.013; and victim understanding, p = 0.044); and the practice issues scale (p = 0.000). Actual knowledge also improved, but the change was only significant at p <= 0.10 (p = 0.06). The only scale that clearly showed no improvement was the opinions scale related to alcohol/drugs and IPV (p = 0.445). 

This study shows asynchronous, interactive, onlinprogram developcadre of national in accordance wcurrent online edbest practices, csuccessful in chanumber of physicIPV knowledge, beliefs, and self-behaviors and prand that these chcan persist over months. 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-203 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Harris, 200575

Int: Live, Audio, Print, Teleconference CC: NA 

Int: Case-based learning, Lecture, Readings CC: NA 

Int: Multiple time or repetitive CC: NA 

Evaluation of quality of teleconferenced educational detailing CME 

Attitudes Yes 65% of FPs felt their 

knowledge of the Canadian Diabetes Association guidelines was considerably increased, 76% (22/29) were stimulated to implement new office strategies, and 100% felt the specialist moderators were excellent. Half of the FPs felt they learned as effectively as at a typical CME event. 

CME delivered bteleconference wfeasible, well attewell received by participants, and improved some kdiabetes managepractices and oualthough primary improving HbA1C not achieved. 

Jennett, 198876

Int: Live, Print, Teleconference Int: Live, Print, Teleconference CC: NA 

Int: Discussion group, Feedback, Mentor/Preceptor, Readings Int: Discussion group, Feedback, Mentor/Preceptor, Readings CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Value of the educational experience in cardiovascular medicine 

Attitudes Yes FPs rated the program as 

4.3 for value and 4.4 for relevance on a scale of 1-5, with 5 at high end of scale. 

A carefully plannprogram, adherinessential learninprinciples, was ein changing officpractice of volundoctors as long amonths after the intervention. 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-204 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Jennett, 198876

Int: Live, Print, Teleconference Int: Live, Print, Teleconference CC: NA 

Int: Discussion group, Feedback, Mentor/Preceptor, Readings, Int: Discussion group, Feedback Mentor/Preceptor, Readings CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NA 

Value of the educational experience in cancer medicine 

Attitudes Yes FPs rated the value and 

relevance of the program as 3.4 and 3.5 on scale of 1-5, with 5 representing the high end of the scale. 

A carefully plannprogram, adherinessential learninprinciples, was ein changing officpractice of volundoctors as long amonths after the intervention. 

Mann, 199752

Int: Live, Video, Print Int: Live, Video, Print, Cue stickers on medical chart CC: NA 

Int: Case-based learning, Demonstration, Discussion group, Readings, Simulation (other than standardized patient or roleplay) Int: Case-based learning, Demonstration, Discussion group, Point of care, Readings, Simulation (other than standardized patient or roleplay) CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NR 

Physician attitude, as measured by posttests 

Attitudes Yes Significant (p=0.03) 

group x time effect was seen in the overall attitude score. Pretest scores showed statistically significantly more positive attitudes (p=0.02) of some among the control group. Among other attitudes, the intervention groups had more positive attitudes (p=0.02) at posttest. 

Educational trainworkshops appeeffective in chanphysicians’ behashould, thus, be continued along additional researthe mechanisms behavior change 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-205 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Rodney,198677

Int: Live, Video Int: Live, Video CC: NA 

Int: Demonstration, Lecture, Simulation (other than standardized patient or roleplay) Int: Demonstration, Lecture, Simulation (other than standardized patient or roleplay) CC: NA 

Int: One time Int: One time CC: NA 

Attitudes towards sigmoidoscopy, as measured by phone or written survey 

Attitudes Yes Both small and large 

group learners reported that sigmoidoscopy was a positive contribution to practice. 

Physicians who participate in couflexible sigmoidohave a higher prof office utilizatiothese skills than who do not take Minimal differencbetween large angroup CME form
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-206 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Rodney,198677

Int: Live, Video Int: Live, Video CC: NA 

Int: Demonstration, Lecture, Simulation (other than standardized patient or roleplay) Int: Demonstration, Lecture, Simulation (other than standardized patient or roleplay) CC: NA 

Int: One time Int: One time CC: NA 

Behavior related to flexible sigmoidoscopy use, as measured by phone or written survey 

Attitudes Cognitive skills Practice behavior 

Yes No Yes 

Small group learners were more likely to acquire additional training and teaching attachments for their sigmoidoscopes, and less likely to use small (35 cm) scopes; there was no difference in biopsy utilization. 90% of large group learners acquired scopes after training vs. 40-56% of small groups. Small groups were associated with shorter procedure times (p<0.05) for first 10 procedures, but otherwise no differences in times, insertion depths, or number of exams performed. Compared to a randomly surveyed group of physicians, those with CME were significantly more likely to perform flexible sigmoidoscopy. 

Physicians who participate in couflexible sigmoidohave a higher prof office utilizatiothese skills than who do not take Minimal differencbetween large angroup CME form
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-207 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Clark, 200078

Int: Live, Video CC: NA 

Int: Case-based learning, Demonstration, Lecture CC: NA 

Int: Multiple time or repetitive CC: NA 

Patient parent perception of physician performance.

Attitudes Yes Parents of patients of 

physicians in the intervention group were more likely to report that the doctor had communicated and educated effectively, and that the physician paid close attention to the family. 

Participating phyreported that thecommunicated ataught patients in sophisticated waParents of intervpatients reported physicians used of communicatioeducation strategenhance patient and satisfaction. Intervention patieshowed a decreahospitalization. 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-208 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Gerstein, 199953

Int: Live CC: NA 

Int: Case-based learning, Discussion group CC: NA 

Int: One time CC: NA 

Knowledge, attitudes, and practice behavior regarding diabetes care, based on participant questionnaire 

Knowledge Attitudes Practice behavior 

Yes Yes Yes 

After 40 days, participants' overall scores improved significantly while there was no change in the controls' scores (F =24.14; p<0.0001). Significant improvement was also noted in domains of attitude (F=31.75; p<0.0001), knowledge (F=4.23; P =0.041), and practice behavior (F=10.43; p=0.0014). However, improvement was not apparent after a year. Participants who completed 425-day assessment scored lower in attitude subscale compared to controls, despite having initially scored higher at the 40-day assessment. 

An interactive, smgroup, diabetes continuing educaprogram effectivedisseminates praguidelines to famphysicians. The such a program after 1 year. 

Attitudinal Objectives Not Met, Evaluation Duration Not ReportedBlock, 198837

Int: Live, Print CC: NA 

Int: Lecture, Point of care CC: NA 

Int: Multiple time or repetitive CC: NA 

Impact of CME on physician attitudes and behaviors as measured by questionnaire 

Attitudes No Physicians in 

demonstration project felt equally strong influence of CME courses and consensus statements on their practice behaviors compared with physicians in control group. 

MDs in a commuboth physician aeducation reportchanging their prmore significantly MDs in a compacommunity. 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-209 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Attitudinal Objectives Not Met, Evaluation Duration Greater Than 30 DaysChodosh, 200662

Int: Live, Internet, not real time, Print CC: NA 

Int: Discussion group, Lecture CC: NA 

Int: Multiple time or repetitive CC: NA 

Providers' perceptions of care quality, as measured by questionnaire 

Attitudes No No difference in 

providers' perceptions existed about quality of care. 

Despite a succesintervention demonstrating siimprovements in of care for patiendementia, providknowledge and awere minimally a

Elliott, 199755

CC: NA Int: Live, Print 

CC: NA Int: Case-based learning, Clinical experiences, Discussion group, Lecture, Readings 

CC: NA Int: Multiple time or repetitive 

Physician and nurse attitudes toward CPM 

Attitudes No There was a trend toward 

improved attitudes in the intervention group, but it was not statistically significant. 

With the exceptioattitudes of patiecaregivers, there trend toward improvement in pratings as well as provider attitudes knowledge but thwas minor and thresults were not overwhelmingly convincing 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-210 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Mann, 199752

Int: Live, Video, Print Int: Live, Video, Print, Cue stickers on medical chart CC: NA 

Int: Case-based learning, Demonstration, Discussion group, Readings, Simulation (other than standardized patient or roleplay) Int: Case-based learning, Demonstration, Discussion group, Point of care, Readings, Simulation (other than standardized patient or roleplay) CC: NA 

Int: Multiple time or repetitive Int: Multiple time or repetitive CC: NR 

Physicians' perceptions of selfefficacy in cholesterollowering practices, as measured on Likert scale survey (14 items) 

Attitudes No Overall posttest scores 

did not differ between intervention and control groups. Intervention groups (workshop and workshop + chart cue) had statistically significantly higher confidence on 1 of 14 practices at posttest 1 (dietary counseling) and 2 of 14 practices at posttest 2 (identifying patients to be screened and interpreting test results to patient). Results possibly due to multiple comparisons. 

Educational trainworkshops appeeffective in chanphysician behavishould, thus, be continued along additional researthe mechanisms behavior change 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-211 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Attitudinal Objectives With No Control Group, Evaluation Duration Not ReportedGrady, 199779

CC: Live, Print Int: Live, Print Int: Live, Print 

CC: Lecture, Readings Int: Lecture, Point of care, Readings Int: Feedback, Lecture, Point of care, Readings, Financial reward for compliance 

CC: One time Int: Multiple time or repetitive Int: Multiple time or repetitive 

Physician acceptance of the intervention 

Attitudes No 

control group 

Overall 38.7% of physicians used the educational materials and the Likert scale ratings of the cueing procedures, feedback, etc ranged from 2.9-3.8 (scale 1-5). 

Cueing (posters chart stickers) hapositive impact omammography rcompletion, and compliance abovbeyond educatioHowever, there wadded benefit frofeedback and finrewards for compIn addition, physacceptance of thinterventions wamarginal. There appears to be a impact of the phycharacteristics oresults of the inte
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-212 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Kemper, 200658

Int: Internet, not real time Int: Internet, not real time Int: Internet, not real time Int: Internet, not real time 

Int: Discussion group, Problembased learning or team-based learning Int: Discussion group, Problembased learning or team-based learning Int: Discussion group, Problembased learning or team-based learning Int: Discussion group, Problembased learning or team-based learning 

Int: Online discussions; case-based, selfinstructional modules Int: Online discussions; case-based, selfinstructional modules Int: Online discussions; case-based, selfinstructional modules Int: Online discussions; case-based, selfinstructional modules 

Differences of the three types of scores between the four groups 

Knowledge Attitudes 

No control group 

No significant difference in any of the three outcome scores by delivery strategy. 

Results from this have important implications for professional edufuture research. Educators wishinthe Internet can confident that improvements arheavily dependecurriculum delivestrategy. The demethod that is eainstructors (bolusmay present the barriers for comp(such as full maiinstitutional firewparticularly for shintroductory courdelivery strategy easiest does not to result in substworse outcomesAdditional interveare needed to imclinicians' commbehavior. This stdemonstrated thInternet educatiosubstantially impclinicians' knowleconfidence regarthe delivery strat
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-213 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Kemper, 200658

Int: Internet, not real time Int: Internet, not real time Int: Internet, not real time Int: Internet, not real time 

Int: Discussion group, Problembased learning or team-based learning Int: Discussion group, Problembased learning or team-based learning Int: Discussion group, Problembased learning or team-based learning Int: Discussion group, Problembased learning or team-based learning 

Int: Online discussions; case-based, selfinstructional modules Int: Online discussions; case-based, selfinstructional modules Int: Online discussions; case-based, selfinstructional modules Int: Online discussions; case-based, selfinstructional modules 

Changes in knowledge, confidence and communication scores

Knowledge Attitudes 

No control group 

Statistically significant improvement in scores after they were taken. 

Results from this have important implications for professional edufuture research. Educators wishinthe Internet can confident that improvements arheavily dependecurriculum delivestrategy. The demethod that is eainstructors (bolusmay present the barriers for comp(such as full maiinstitutional firewparticularly for shintroductory courdelivery strategy easiest does not to result in substworse outcomesAdditional interveare needed to imclinicians' commbehavior. This stdemonstrated thInternet educatiosubstantially impclinicians' knowleconfidence regarthe delivery strat
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-214 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Rosenthal, 200559

Int: Live, Handheld, Review of data CC: Review of data 

Int: Lecture, Readings, Review of practice's data CC: Review of practice data 

Int: Multiple time or repetitive CC: One time 

Parent reports of preventive health behaviors 

Attitudes No 

control group

For parents of both 1-month-olds and 6-montholds, the control and intervention parents did not differ in the adjusted change in the proportion of parents reporting parent behaviors in agespecific topics. 

An office system intervention imprparent reports of of anticipatory gubut did not changknowledge or pabehavior. 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-215 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Leopold, 200580

Int: Print, Int: Computerbased off-line Int: Live 

Int: Readings Int: Demonstration, Readings Int: Demonstration, Feedback, Mentor/Preceptor, Simulation (other than standardized patient or roleplay) 

Int: NA Int: NA Int: One time 

Confidence in task 

Attitudes No 

control group 

Before the instruction, confidence skills did not differ between the three randomized groups. When stratified by demographic variables, gender and type of practitioner were considered to be significantly different with confidence of skill prior to instruction. After instruction, confidence levels increased for all groups. 

Even low-intensiof instruction impindividuals’ conficompetence, and assessment of thin performing the straightforward psychomotor tassimulated knee iHowever, men aphysicians disproportionateloverestimated thboth before and training, a finding worsened as conincreased. The inrelationship betwconfidence and competence that observed before educational interas well as the demographic diffthat we noted shraise questions ahow complex new procedures shouintroduced and wself-trained practshould begin to pthem. 
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-216 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Attitudinal Objectives With Mixed Results, Evaluation Duration Greater Than 30 DaysNorris, 200081

Int: Live, Print, Followup phone calls about protocol CC: NA 

Int: Lecture, Point of care, Opinion-leader from clinic teaching CC: NA 

Int: Multiple time or repetitive CC: NA 

Physician self-reported knowledge, attitudes, skills, and behavior in counseling patients about physical activity, as measured by questionnaire 

Attitudes Mixed Significant differences in 

3 of 10 reported items: Intervention providers perceived themselves to be more knowledgeable about physical activity counseling (p=0.002), perceived time and effectiveness to be less of a barrier to counseling (p=0.003), and were 22% more likely to counsel patients about physical activity (p=0.03) than control physicians. 

"A one-time PACcounseling sessiminimal reinforcea setting with higbaseline levels odoes not further activity. Contemadvanced in stagbehavior change

Chodosh, 200662

Int: Live, Internet, not real time, Print CC: NA 

Int: Discussion group, Lecture CC: NA 

Int: Multiple time or repetitive CC: NA 

Providers' attitudes regarding dementia care, as measured by questionnaire 

Attitudes Mixed More intervention 

physicians strongly endorsed statement "Older patients with dementia are difficult to manage in primary care" (p=0.03). No other differences in attitude between intervention and usual-care providers were observed. 

Despite a succesintervention demonstrating siimprovements in of care for patiendementia, providknowledge and awere minimally a
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-217 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Unclear if Attitudinal Objectives Met, Evaluation Duration Greater Than 30 DaysSchroy, 199982

Int: Live CC: NA 

Int: Academic detailing, Discussion group, Lecture CC: NA 

Int: One time CC: NA 

Provider attitudes or practices related to colorectal cancer screening as determined from a preeducational vs. posteducational survey instrument, based on an instrument that was similar to one used by the American Cancer Society in 1989 

Attitudes Practice behavior 

Unclear Yes 

At year 1, there were no significant differences in concern about patient fear and discomfort, time, procedural skills, or cost. However, the mean scores for equipment availability, efficacy, and yield were all significantly higher in the noncompliant group than the compliant group (p = 0.001). Most of the compliant providers were at the intervention sites, which indicate that availability of on-site screening may trump perceived barriers for sigmoidoscopy adherence. At baseline, 24% of providers at intervention sites and 19% at comparison sites reported recommending screening sigmoidoscopy. Overall, self-reported compliance rates increased by 36% at the intervention site, vs. 7% at the comparison site (p = 0.001). 

"In summary, this clearly shows thaacademic detailinform of an outreadidactic educatioseminar followed implementation osigmoidoscopy sis an effective stenhancing providcompliance with screening guidel
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-218 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Schroy, 199982

Int: Live CC: NA 

Int: Academic detailing, Discussion group, Lecture CC: NA 

Int: One time CC: NA 

Physician concerns about patient fear, discomfort, time, procedural skill, cost, equipment availability, evidence and yield 

Attitudes Unclear Providers were asked to rate their concerns on a 

Likert scale survey pre and one year post intervention. Results were NR by study groups but by compliant vs. noncompliant providers, with the statement that more compliant providers were in the intervention group. 

"In summary, this clearly shows thaacademic detailinform of an outreadidactic educatioseminar followed implementation osigmoidoscopy sis an effective stenhancing providcompliance with screening guidel

Attitudinal Objectives With No Control Group, Evaluation Duration Greater Than 30 DaysWhite, 200463

Int: Live CC: Live 

Int: Case-based learning, Discussion group CC: Case-based learning, Discussion group, Lecture 

Int: One time CC: One time 

Program evaluation 

Attitudes No 

control group

There was no significant difference between the two groups with respect to the overall rating of the CME program or to the knowledge gained at each test administration. The only statistically significant difference was found in the subjective assessment of the educational value of the program (4.36 vs. 3.93; p=.04) in the PBL session. The attrition rates for the third test administration for the lecture-based participants were higher than the attrition rate of PBL participants. 

This study found evidence that PBbetter than more learning sessionfacilitating knowlgain, knowledge retention, or chaattitude about asmanagement. However, this stusimilar to other ssuggests that theweak evidence thphysicians attribuadditional value tmore interactive approach to CME
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-219 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Lockyer, 200264

Int: Live Int: Live 

Int: Case-based learning, Discussion group, Lecture, Role play Int: Case-based learning, Discussion group, Lecture 

Int: One time Int: One time 

Physician practice behaviors: comfort level

Attitudes No 

control group

Both track 1 (introductory course) and track 2 (advanced course) participants improved statistically significantly between pre and posttest scores; track 1 had a moderate effect size difference (0.5), and track 2 a minimal effect size (0.2). Between tracks comparisons showed statistically significant differences between tracks for both precourse and postcourse assessment of comfort. 

Track 1 (introduccourse) physiciaimproved moderwhile track 2 (adcourse) physiciashowed a small onegligible changknowledge, comfinvolvement in pcare for dementipatients. TrackinCME - assigning physicians to coubased on pre-coability, interest, oneeds further stu

Cherkin, 199183

Int: Live, Video Int: Live, Video 

Int: Demonstration, Discussion group, Lecture Int: Demonstration, Discussion group, Lecture 

Int: Multiple time or repetitive Int: One time 

Selfassessment of provider knowledge of how to manage back pain 

Attitudes No 

control group

The percentage of providers who believed they knew exactly what they needed to do to effectively manage patients with low back pain increased from 32% to 71% (p<0.01). 

Both HMO and ppractice achievegoal. As a result intervention abouthe physicians feconfident in their manage back pa

Cherkin, 199183

Int: Live, Video Int: Live, Video 

Int: Demonstration, Discussion group, Lecture Int: Demonstration, Discussion group, Lecture 

Int: Multiple time or repetitive Int: One time 

Satisfaction with the intervention 

Attitudes No 

control group 

85% of the providers felt the 3 didactic components to have been "somewhat" useful, almost half (46-48%) found the first two components to be "very" useful. 

Both HMO and ppractice achievegoal. As a result intervention abouthe physicians feconfident in their manage back pa
Evidence table 9. Effectiveness of continuing medical education on short-term and long-term attitude outcomes 

F-220 

Author, 

year Media used 

Educational 

techniques 

Amount of 

exposure 

Main 

outcome 

measure 

Type of 

objecttive 

Objectives 

met Summary of results Overall concl

Cherkin, 199183

Int: Live, Video Int: Live, Video 

Int: Demonstration, Discussion group, Lecture Int: Demonstration, Discussion group, Lecture 

Int: Multiple time or repetitive Int: One time 

Level of physician comfort in the management of back pain 

Attitudes No 

control group

62% of participants felt "somewhat" more confident with back-pain management. About 50% felt more knowledgeable about the psychosocial and scientific aspects of low back pain. 

Both HMO and ppractice achievegoal. As a result intervention abouthe physicians feconfident in their manage back pa

Heale, 198866

Int: Live Int: Live Int: Live 

Int: Lecture Int: Case-based learning, Discussion group Int: Problembased learning or team-based learning 

Int: One time Int: One time Int: One time 

Perception of the program by participants 

Attitudes No 

control group (yes for technique)

The small group problembased learning participants differed significantly, and gave the CME day the highest rating (p value not provided). 

Within a one day course in family medicine, the leaformat had no efacquired or retaiknowledge or on physician performthree patient proPhysicians rated small group probbased format hig

ARS = audience response system; ATP = allergy trial pack; CC = concurrent control; CME = continuing medical education; DV = domestic violence; 

= intervention group; IPV = intimate partner violence; KFP = Key Features Program; MANOVA = multiple analysis of variance; NA = not applicableproblem-based learning; PCP = primary care provider; SD = standard deviation; WBJC = Weekly Browsing Journal Club;