Self-Study from Web-Based and Printed Guideline Materials

Many primary care physicians have not learned about key advances in caring for patients after acute myocardial infarction, including the use of -blockers (1) and angiotensin-converting enzyme inhibitors (2). Although professional organizations have disseminated guidelines for the care of acute myocardial infarction (3, 4), important therapies continue to be underused (5). Physicians read guidelines to stay up-to-date (6), but reading without additional interaction is relatively ineffective for learning (7) and has little effect on practice (8). Physicians therefore need better methods for learning from guidelines. Self-assessment tests are often used to enhance self-study of printed materials (9, 10), but physicians can now choose from a growing number of World Wide Web sites that offer on-line self-study (11). Since the 1960s, computer-assisted instruction has been explored as a method for medical education (12-15). In theory, computer-assisted instruction can enhance self-study by tailoring presentations to learners' individual needs and by including interactive simulations (16). Two previous studies comparing computer-assisted instruction systems with carefully constructed self-study controls found mixed results (17, 18). In both of these studies, the systems were developed for and tested among medical students before the advent of the Web. Therefore, physicians have little direct evidence to inform their choices among currently available self-study methods. We conducted a randomized trial to compare the educational outcomes produced by a Web-based tutorial on myocardial infarction guidelines with those produced by self-assessmentbased and study from print versions of the same guidelines. We hypothesized that print-based self-study would result in substantial learning but Web-based self-study would produce incrementally greater knowledge gains, retention, learning efficiency, and satisfaction. We conducted this trial among residents rather than community physicians because residents frequently use self-study materials and because we anticipated that they would be more feasible to recruit. Methods Design of Self-Study Alternatives Guidelines and Learning Objectives The American College of Cardiology (ACC), the American Heart Association (AHA), and the American College of PhysiciansAmerican Society of Internal Medicine (ACPASIM) allowed us to use their guidelines for this trial. We wrote 20 cognitive learning objectives (19), covering knowledge from one or both guidelines that would be important for physicians who provide primary care to patients after myocardial infarction. One of the authors, an expert in evidence-based cardiology, edited the objectives for content validity and importance. The final set of learning objectives is shown in the Appendix Table. Knowledge Tests We wrote multiple-choice questions that had one correct response demonstrating the knowledge in a single learning objective. Thirty attending physicians and fellows in internal medicine, family medicine, and cardiology critiqued the clarity and difficulty of prospective questions by using an electronic mail quiz system that tracked each question's average score (20). These data were used to assemble a 20-question pretest and a different 20-question post-test. The tests included one question per learning objective and were approximately equivalent in overall difficulty. Expected scores on the final pretest and post-test differed by less than 6%. For each question on the final test forms, participants could place a check mark next to the statement, I would like to review available evidence or recommendations on this topic. We referred to this feature as the review flag. For both of the self-study alternatives in this trial, learners initially completed the pretest on paper. Web-Based Self-Study Materials We constructed a Web-based learning system called SAGE (Self-Study Acceleration with Graphic Evidence). Participants who used SAGE copied their pretest answers into the system and were then taken to the main tutorial page, which presents an overview of the user's pretest results for each learning objective. By clicking on hyperlinks, users could view a sequence of learning resources for each objective. The first resource for each objective was the relevant pretest answer. From there, users could click on hyperlinks to view the relevant guideline passages (highlighted segments of text in an on-line version of the complete guideline document). When guidelines referred to landmark clinical trials, users could click on a link to open a graphic evidence browser (Figure 1). Complete instructions for using SAGE were provided in the system. The program was intended to combine two strategies of computer-assisted instruction: a tutorial strategy, which focuses the learner on deficits in previous knowledge, and a simulation strategy, which reinforces knowledge through interaction with graphic models of randomized trial evidence. We wrote software for SAGE in Server-side JavaScript, using the Netscape Enterprise Server (Mountain View, California) running on a Sun Ultrasparc2 workstation (Mountain View, California). For animated statistical graphics, we wrote a Java applet. Figure 1. The SAGE (Self-Study Acceleration with Graphic Evidence) randomized trial viewer. Printed Self-Study Materials Printed materials for the control group included the same self-assessment test and guidelines as the Web-based materials. The printed materials included 1) a booklet showing the correct answer for each pretest question and the learning objective each question was intended to evaluate and 2) complete reprints of the ACPASIM and ACC/AHA guidelines. Learners were instructed to compare their pretest answers with the correct answers and look up the evidence or guideline recommendations for each learning objective that they may not have met by using the outline structure of the ACPASIM guidelines and the index of the ACC/AHA guidelines. Materials in the control group were intended to provide learning through self-assessment and study from the actual guideline documents as distributed by the specialty societies. The control materials contained the same pretest answers and guideline content as SAGE but lacked the hypermedia navigation features and the detailed, interactive, graphic presentations of landmark trial evidence. Of note, however, the guideline materials for both self-study groups contained all of the information that learners would need to meet each learning objective, including narrative descriptions of the key trial results that were available graphically in SAGE. Study Sample We recruited residency programs in family and internal medicine at four universities. Approval for research involving humans was obtained from each university's institutional review board. A total of 5 family medicine and 7 internal medicine programs agreed to participate. We recorded the name, sex, and training year of every resident across all 12 programs in a confidential recruitment database. Letters were placed in residents' mailboxes inviting them to attend a 1.25-hour learning session in which they would be randomly assigned to study from computer or printed materials on the care of myocardial infarction. An honorarium of $30 was offered for completing the trial. The trial was also announced at resident conferences. All residents who attended a session and gave consent became eligible participants. Self-Study Sessions Participants used the self-study materials in a single session that began with the pretest and ended with an immediate post-test. After completing the pretest, participants kept carbon copies of their answers, returned their originals, and received their randomly generated study group assignment. Those assigned to the control group were given the printed materials, and those assigned to SAGE went to an adjacent computer classroom, where they logged on and entered their pretest answers. Participants in both groups were asked to study until they felt that they had met the learning objectives, at which point the self-study materials were removed and they immediately took the post-test on paper. Each session was proctored to ensure that all learning was attributable to self-study from the assigned material without use of extraneous materials or discussion with colleagues. No irregularities occurred. Participants could not attend more than one session and were not allowed to keep the printed self-study materials or to access SAGE after completing their session. Study Group Assignment Individual participants were randomly assigned to study groups by a computer program that generated blocked randomization stratified by the participants' sex and residency year. As each resident arrived at a session, he or she was registered on a laptop computer that contained coded records from the recruitment database. As the first participants finished the pretest, a randomization program run on the laptop generated a balanced pool of assignments for each sexresidency year category (using a block size of 2 [21]). For each registered participant, the program chose an assignment at random from the appropriate pool. As participants turned in their pretests, they and the investigators looked up their assignments on the laptop screen. Study group assignment was therefore simultaneously revealed to the participants and to the investigators. Outcome Measures Knowledge scores from multiple-choice tests were calculated by assigning 1 point per correct answer; unanswered items were considered incorrect. This created scores that could range from 0 to 20 points. The primary outcome of the trial was knowledge on the immediate post-test. The time spent studying was measured by using a time-stamp machine for participants in the control group and the server log for participants in the SAGE group. Learning efficiency was calculated as the point gain from pretest to immediate post-test divided by the time