Evaluation: measuring performance, ensuring competence, achieving long-term excellence.

THE education of medical students, residents, and practicing physicians is necessary to achieve high-quality health care. For this education to be effective, it must begin with a clear delineation of the goals that learners need to achieve. Evaluation of their subsequent mastery of these goals is an essential element of the educational process. In 1999, the Accreditation Council for Graduate Medical Education changed the focus of residency and fellowship program reviews to emphasize educational outcomes as a key measure of the quality of training programs. An objective of this change in focus is to improve educational outcomes and thus enhance the quality of medical care and increase patient safety. These educational outcomes can only be judged through effective evaluation. Educational evaluation has two distinct aspects, generally classified as formative and summative evaluation. Formative evaluation, which can also be called feedback, is used to guide learners during the educational process, identifying those areas they have mastered and those areas that need additional work. With formative evaluation, a teacher’s relationship with a learner is similar to that of a coach’s relationship with an athlete. Learners, like athletes, need to invest the necessary time and effort to achieve their goals. Teachers are responsible for monitoring the learners’ progress and providing guidance to help them achieve peak performance. For formative evaluation to be effective, it must be provided close to the time of the observed evaluation and must contain specific observations that a learner can use to improve performance. This feedback must also be delivered in an appropriate setting and in a constructive manner. Unfortunately, most medical educators have not received formal training in providing evaluation and feedback. The result is often feedback that is deficient in both quality and quantity. To help learners achieve their maximum potential, medical educators must be skilled in providing feedback that helps learners change unsatisfactory performance into satisfactory performance and satisfactory performance into outstanding performance. Summative evaluation is used to assess the learners’ knowledge and/or performance at the completion of an educational activity, and it allows for judgment of the learners’ success in achieving their educational goals. Examples of summative evaluation include final examinations, end-ofrotation faculty evaluations, and patient surveys. Summative evaluation is not intended to change behavior during the educational activity; however, it can be used to develop future activities by identifying areas for improvement. Comparison of the results of summative evaluations with learners’ self-evaluations can be used by both teachers and learners to assess the learners’ skills in self-evaluation. In the cognitive domain of learning, summative evaluation has traditionally relied on the use of written and oral examinations. Standardized multiple choice examinations provide an efficient method to assess medical knowledge. Large amounts of material can be tested and large numbers of learners can be assessed and compared using statistically reliable and valid methods. The use of criterion-referenced standards to determine passing performance on an examination allows for objective evaluation. Passing the examination is determined by specific performance criteria and not by the performance of the cohort of learners taking the examination. Although they are more difficult to standardize and administer than written examinations, oral examinations may allow the assessment of more complex cognitive abilities, including analysis, synthesis, and evaluation. In constructing an oral examination, attention must be given to develop an examination format that provides a uniform and reproducible experience for multiple examinees. This requires careful construction of the questions and establishment of clearly defined criteria for grading. Education and assessment of the examiners are essential to make the oral examination reliable and to minimize examiner bias, thus ensuring an examinee would have the same result if examined by different examiners or given a different set of questions. Measuring the validity of the oral examination requires a metric against which examinee performance can be compared. Identification of such a metric may be difficult and can impair assessment of the examination’s validity. High-fidelity simulation provides a new method to evaluate a learner’s performance in all the educational domains—cognitive, affective, and psychomotor—in real time. Anesthesiology has been in the forefront of the development of simulation as an educational tool in medicine. The next frontier in the use of simulation may be the development of summative examinations that can be used in making high-stakes decisions regarding residency education, Accepted for publication October 13, 2009. The authors are not supported by, nor maintain any financial interest in, any commercial activity that may be associated with the topic of this article.