Evaluation of residents: challenges and opportunities.

In this issue of the Journal, two groups1,2 address differing but overlapping aspects of the problems facing medical educators as they strive to improve the quality of graduate medical education and the outcomes of that educational process. Though these two papers highlight yet again the shortcomings and challenges inherent in evaluating residents, they also offer some hope and a framework for developing improved systems for evaluation. In the first paper, Yao and Wright address “The Challenge of Problem Residents.”1 They base their paper on a review of the medical literature that sought articles focusing on the identification, underlying causes, management and prevention of problem residents. Notably, the literature review did not seek articles that addressed similar problems in medical students or other health science students. In addition, the authors scrutinized other information sources, such as the American Board of Internal Medicine (ABIM) publications and meeting proceedings from the Society of General Internal Medicine and the Association of Program Directors in Internal Medicine and conducted structured interviews with a convenience sample of program directors. For purposes of their paper, Yao and Wright defined a “problem resident” as one who demonstrates a performance deficiency, in at least one of the seven areas of clinical competencies that the ABIM evaluates, which was not corrected after a single intervention. Yao and Wright provide a thoughtful synthesis of the information obtained from these three sources and develop a useful framework for faculty and program directors to identify and manage this small (4–7% of all residents) but important cohort of residents whose problems persist despite initial attempts to resolve them. The authors correctly note that such residents pose special challenges, often consuming a disproportionate amount of the faculty's efforts during remediation efforts and adversely affecting the training experience of the other residents.3 Of concern to all should be their observation that “the amount of published literature addressing many topics related to problem residents is inadequate.” Notably, a prior review4 of this topic in 1993 led to a similar observation: “surprisingly little has been written about problem residents…” The majority of the literature extant focuses on the impaired (psychiatric illness and substance abuse) resident while relatively little focuses on such critical dimensions as clinical skills, medical judgement and professionalism. Given Yao's earlier work5 indicating a high prevalence of such problems, this gap in our knowledge and understanding is of real concern. The literature that does exist suggests that deficiencies in resident performance might be even more prevalent than currently recognized, in part due to variability in standards and observational abilities among faculty,6 and the failure to use observation methods specific to the skills being assessed.7 Holmboe et al.,2 as if answering Yao and Wright's call for more and better research, address an important problem in their study: namely, can we improve the quality of faculty's written evaluation of residents and increase the amount and quality of feedback that residents report they receive? The simple answer from their study was yes. The overall effect, however, was at best modest though nonetheless important, given the similarly modest nature of their intervention, which could be easily implemented at any other residency program. Appropriately, the authors raise the concern that much more intense interventions may be needed to produce larger and perhaps sustainable changes in faculty evaluation and feedback behaviors, an observation of special concern at a time when competition for faculty time and resources is growing steadily. This observation suggests that other strategies for increasing the quality and quantity of resident observations need to be considered. We need to better understand the barriers that limit these observations. In particular, we need to address both the development and the communication among faculty, of the standards that serve as the foundation for our collective judgements.8 Approaches that involve the stimulated recall of faculty observations of a learner's performances suggest additional methods that might increase the quality and quantity of evaluation information, without at the same time requiring complex evaluation systems or methods.9 Indeed, the greatest yield in evaluation information may not come from seeking the perfection of any one method, but from the application of an array of evaluation methodologies10 across a range of venues (e.g. inpatient, outpatient). As our authors correctly state, we need rigorous research to assess the best methods and systems for evaluating our residents (and students) and for assisting them in correcting any deficiencies identified during the evaluation process. We need also to look outside medicine in other fields and professions for best practices in evaluation. Our residents and the public they will ultimately serve deserve nothing less.