Outcomes-Based Courses: Using Pre and Post Course Measurements to Examine and Enhance Course Success

Questionnaire-based pre/post course testing can be used to extend the cognitive processes provided in short (2to 6-hour) courses. When participants participate in preand post-course assessments and implement the changes they commit to at the end of a course, course designers can have an impact on knowledge and workplace practices. A carefully designed pre-course assessment offered prior to an educational activity can help professionals draw upon their workplace experiences prior to a course and during that course. Jocelyn Lockyer, PhD, Director, Continuing Medical Education and Professional Development, Associate Professor, Department of Community Health Sciences, Faculty of Medicine, University of Calgary. Richard Ward, MD FCFP, is Senior Consultant, MAINPROCourses, Continuing Medical Education and Professional Development, and Clinical Associate Professor, Department of Family Medicine, Faculty of Medicine, University of Calgary Herta Fidler, MSc, Research Associate, Continuing Medical Education and Professional Development, Faculty of Medicine, University of Calgary John Toews, MD FRCPC, Associate Dean, Continuing Medical Education and Professional Development and Professor, Department of Psychiatry, Faculty of Medicine, University of Calgary Margaret Churcher, MD CCFP, Consultant, MAINPRO-C Courses, Continuing Medical Education and Professional Development, Faculty of Medicine, University of Calgary Background It is assumed that professionals learn and gain information from continuing professional development programs (Daley, 2002). However, studies involving randomized control trials of continuing medical education (CME) programs show the results of short courses can be variable (Davis, Thomson, Oxman, & Haynes, 1995; Davis, Thomson, Freemantle, Wolf, Mazmanian, & Taylor-Vaisey, 1999). These systematic reviews of the literature suggest that short courses, particularly those that are didactic (lecture based), often fail to show change in physician behavior or patient outcomes. The ones that do show improvement use multiple learning methods; involve enabling and reinforcing strategies as well as facilitating strategies; and have a needs assessment that was used to determine the gap between current and desired clinical practice. Outside of medicine, there are even fewer studies and little attempt to measure changes in outcomes from short course curriculum (Greiner and Knebel, 2003). Daley (2002) has theorized that, while continuing professional development (CPD) programs play a vital role in providing professionals with up to date information, the context in which the professional practices frames what they learn and how they use the information in practice. Proponents of a situated view of learning argue that one cannot separate the learning process from the situation in which the learning is presented. Knowledge and the process of Journal of Executive Education Fall 2003 31 learning within this framework are viewed as a product of the activity, context, and culture in which it is developed and used (Brown, Collins, & Duguid, 1989). One of the most powerful methods by which this is attained is through cognitive apprenticeships. For example, Brandt, Farmer and Buckmaster (1993) suggest that students learn through five sequential phases: Modeling, in which they observe more experienced members of the community or hear about the ‘tricks of the trade’; Approximating, in which they try out an activity while articulating and reflecting on what they have seen and done; Scaffolding, in which coaching and teaching aids assist in developing competency, while support gradually drops away; Self-directed learning, in which students assume responsibility for the learning; and Generalizing, in which students discuss and relate what they have learned to subsequent practice situations. In medicine, this is done through problembased or case-based learning in which a patient’s case is presented and physicians discuss how they would manage that case. In this model, physicians might be explicitly asked to state the words they would use to query why the patient had come for the visit or the patient’s understanding of medications prescribed. Roleplaying may be used to ensure the student understands how to use a specific tool. For example, we explicitly taught physicians how to do the mini mental state examination (MMSE), by a role-playing exercise. Such enabling experiences try to provide authentic and not decontextualized contexts for the learning. While we accept that situated cognition strategies are useful when professionals are attending an educational event, very often the events are short (i.e., two to four hours). This time may be insufficient to ensure the types of reflection that are so necessary to learn new skills and approaches. It may not allow the learner to generalize to other patient cases that are similar but have unique nuances. Practice and feedback are often limited. We posit that a carefully designed pre-course assessment offered prior to an educational activity can help professionals draw upon their workplace experiences prior to a course and during that course. A pre-test can situate the learning early by focusing on the salient aspects of the educational program. That same test, readministered one to three months after an event, can stimulate reflection on the activity and the learner’s ability to apply this activity in the workplace. It may help the learner uncover barriers to the application of newly acquired approaches. Further, the data that are generated from preand post-testing can be useful in assessing a course’s worth, altering the design, and creating evidence that the educational unit is performing appropriately. This paper will offer some general guidelines on the development of preand post-testing, describe the experiences at the University of Calgary in the area of preand post-course testing for courses offered to family physicians, and finally offer some suggestions related to research that might be done to further establish the value of preand post-testing as an integral part of a continuing professional development strategy in other professions. PreAnd Post-Testing There are no hard and fast rules related to the approaches one might take with preand post-testing. Some favor a strictly open-ended qualitative approach to the pre-course assessment in which learners will describe their 32 Fall 2003 Journal of Executive Education intended learning expectations, barriers to learning, the role(s) they play in this area, and even the complexities of their practice vis-à-vis the learning tasks. Others who work in environments in which quantitative data are preferred and understood by decision makers may prefer to take a more quantitative approach to the pre-course. Such an approach may use multiple-choice testing to assess the learner’s knowledge base or scales (Likert-type or agreement-disagreement) to examine learner comfort or approaches to practice. The approach to questioning should be tied to the goals of the educational activity to enable students to prepare for the program and teachers to use the data to guide their pre-course preparation. Similarly at the post-course phase, one to three months later, a variety of approaches may be taken depending on the desired strategy and use of the data. For example, it may be appropriate to continue with an open-ended approach to facilitate open reflection related to content and skills learned and used since the course, barriers encountered in applying the new knowledge in the workplace, and intended further learning. This line of questioning will help learners think about their progress since the course and set goals for the future. It may be equally appropriate to repeat a quantitative precourse assessment to examine change since the course. In some cases, a combination of quantitative and qualitative data may be helpful to learner, teacher and the educational unit. The timing of the preand post-course assessments needs to be considered carefully. Too much time in advance of a course may mean that the professional’s goals and expectations have shifted. Even environmental barriers may be reduced or heightened prior to a course. However, too short a time reduces the teacher’s ability to receive and use the data in a meaningful way. Post-course assessment has its challenges as well. Too little time may mean the professional has not had a chance to practice the new approach. Too much time may limit the learner’s ability to recall the salient aspects of the course and its application to practice. Data that are generated in such circumstances may not be valid. Certainly, the validity of self-report data has been questioned particularly when objective examinations of medical records fail to provide the evidence that the desired change has taken place despite self-report data to the contrary (Adams, Soumerai, Lomas, & Ross-Degnan, 1999). The University Of Calgary Approach And Findings The University of Calgary Office of Continuing Medical Education and Professional Development regularly uses a pre/post assessment approach in conjunction with its short courses. In part, following the systematic review of the randomized control trial literature provided by Davis, et al. (1995), we pioneered this approach believing that it would make it more likely that our courses would have greater impact. Our initial work in teaching physicians to better manage alcohol-related problems further reinforced our thinking (Lockyer, el-Guebaly, Simpson, Gromoff, & Toews, 1996). Later as the College of Family Physicians of Canada changed its accreditation criteria for programs, assigning extra value to those that included preand postcourse assessment, reflective exercises and small group (<10 professionals), we began to develop and offer outcomesbased courses in which these were standard components. Journal of Executive Education Fall 2003 33 Our outcomes-based courses take a fai