Psychometric Challenges of Using Simulations for High-Stakes Assessment

119 O ver the past two decades, the use of simulations in medical education has become widespread (1-4). In the United States, most medical schools have well-established, standardized patient (SP) programs for teaching clinical skills. Standardized patients are laypeople who are trained to realistically and consistently portray actual patients for purposes of assessment and/or training. In addition, many medical schools have incorporated computer-based training modules, part-task-trainers (e.g., breast models, pelvic models), and full-scale integrated simulators to aid in the instruction of undergraduate medical students. The reasons for the increasing popularity of simulation-based training are clear: They can provide students with realistic clinical experiences without risks to patients; the tasks/scenarios can be designed to meet important needs, with increasing complexity introduced in a controlled way; skills can be practiced repeatedly, with tailoring to individual needs; and the likelihood of transfer from instruction to real-world situations is enhanced (5,6). Similarly, simulations are increasingly used for assessment, both for low-stakes tests in medical schools and residency training programs and for high-stakes licensing and certification examinations. Dozens of medical schools in North America have implemented end-ofcourse, end-of-clerkship, and comprehensive examinations involving SPs (7). The Medical Council of Canada has used a multistation SP-based examination (Medical Council of Canada Qualifying Examination Part II) as part of the licensure process since 1993 (8,9); the Educational Commission for Foreign Medical Graduates (ECFMG) introduced the Clinical Skills Assessment in 1998 as part of the certification requirements for graduates of international medical schools (10); and the General Medical Council of the United Kingdom has conducted an SP-based assessment as one component of the Professional Linguistics and Assessment Board examination since roughly the same time (11). Beginning in the summer of 2004, a high-stakes, SP-based examination will be included in the United States Medical Licensing Examination (USMLE). For medical students graduating in 2005 or thereafter, passing this examination, known as USMLE Step 2 Clinical Skills (CS), will be required for licensure in all US jurisdictions. In addition, another simulation-based assessment method, computer-based case simulations, has been used in USMLE Step 3 since 1999. On the horizon are a variety of other simulation-based assessment methods, including part-task-trainers (e.g., pelvic replicas) for teaching and testing psychomotor skills, virtual reality and haptic systems for teaching and testing surgical and other skills, and integrated simulators combining sophisticated life-size mannequins with computer programs allowing the mannequins to produce realistic physiological responses to pharmacologic and other interventions. These methods hold promise for testing the proficiency of individual members of healthcare teams and, for high-fidelity operating-room theater-type simulators, for the teams as a whole (12). Although the fidelity, cost, and efficiency of simulation-based assessment methods vary extensively, their utility for high-stakes assessment depends crucially on their psychometric characteristics. In this chapter, we summarize a range of simulation-based assessObjectives