The Mini-CEX (Clinical Evaluation Exercise): A Preliminary Investigation

In 1972, the American Board of Internal Medicine abandoned the oral examination of residents for logistic and psychometric reasons. It then delegated to program directors the task of evaluating the essential components of residents' clinical competence, including clinical skills. Over the years, the American Board of Internal Medicine has worked with program directors to develop efficient, effective local evaluation systems, and it has recommended that the clinical evaluation exercise (CEX) be part of the process [1]. The traditional CEX is conducted by an experienced physician who observes a resident while that resident interviews a single patient (unfamiliar to the resident), does a complete physical examination, presents findings, and plans the patient's management. After the exercise, the evaluator gives the resident substantive feedback and documents the experience on a form provided by the Board. Later, the resident gives the evaluator a written record of the patient work-up for review. The traditional CEX takes about 2 hours. Approximately 82% of residents have one such evaluation during their first year of training, and a much smaller percentage (32%) have more than one [2]. As a measurement device, the traditional CEX is limited in three ways. First, the resident is observed by only one evaluator, and studies have shown that even experienced physicians differ from one another when observing exactly the same events [3]. Second, the resident is observed with only one patient; patient problems vary considerably, and this is probably one reason why physician performance in one case does not predict performance in others [4]. Third, because most physicianpatient encounters are short and focused, the traditional CEX, with its emphasis on completeness, is a somewhat less relevant measure of clinical skill. The unreliability of the observer, the variation of resident performance from patient to patient, and the artificiality of the task mean that the traditional single-interaction CEX is not a dependable measure of a resident's clinical competence. As a result of these shortcomings, the reproducibility or generalizability of the CEX ratings has been found to be less than 0.30, and the CEX has been appropriately criticized as a measurement instrument [3, 5, 6]. Our goal was to respond to those concerns by exploring the logistic and psychometric properties of a variation on the traditional CEX format, called the mini-CEX. The mini-CEX is designed around both the skills that residents most often need in actual patient encounters and the educational interactions that attending physicians routinely have with residents during teaching rounds. A single faculty member observes and evaluates a resident while that resident conducts a focused history and physical examination (expected to take about 20 minutes). Because the interaction should be relatively short and because it occurs as a natural part of the training environment, each resident can be evaluated on several occasions by different faculty members. In this study, we report initial data on the psychometric and logistic characteristics of the mini-CEX. Methods Instrumentation For each mini-CEX, a single faculty member observed and evaluated a resident while that resident conducted a focused history and physical examination in an inpatient, outpatient, or emergency department setting. After asking the resident for a diagnosis and treatment plan, the faculty member completed a short evaluation form and gave the resident feedback. For most residents, the mini-CEXs were collected within a 4-month period. All formal mini-CEX evaluation data were collected on a two-sided, 1-page form that was the same for all study sites. Information was gathered from the resident on level of training and satisfaction with the mini-CEX; the latter was rated on a 9-point scale anchored with the words lowest and highest. Ratings were collected from the evaluator for the resident's overall clinical competence and for four components of competence: history-taking skill, physical examination skill, clinical judgment and synthesis, and humanistic qualities. The ratings were made on a 9-point scale, on which 1, 2, and 3 were unsatisfactory; 4 was marginal; 5 and 6 were satisfactory; and 7, 8, and 9 were superior. A box was also provided so that the evaluator could select insufficient contact to judge. For the purposes of our study, we averaged the four component skill ratings, and the result is referred to as the calculated average score. In addition to providing data on the resident, the evaluator recorded information on the site of the evaluation (inpatient service, clinic, or emergency department) and the patient's major medical problems and diagnoses. Unlike the traditional CEX, which focuses on an exhaustive assessment, the mini-CEX concentrates on the resident's ability to solve patient problems regardless of the nature of the encounter. Therefore, new and return visits with a particular resident were included and identified. Finally, the evaluator was asked to rate his or her level of satisfaction with the mini-CEX format. Because each resident was evaluated on several occasions by different faculty members, the mini-CEX certainly differs from the traditional CEX in terms of quantity. However, it also differs qualitatively. The traditional CEX rewards the resident for being thorough and thoughtful in an environment unconstrained by the realities of clinical practice. In contrast, the exact nature of any mini-CEX encounter is more variable, because the challenge it poses is more dependent on the particular setting and patient. Consequently, this method of evaluation rewards the resident's ability to focus and prioritize diagnosis and management within the context of clinical practice. Participants Five residency programs in Pennsylvania volunteered to assess residents using the mini-CEX. In general, the mix of programs reflects the heterogeneity found in residency education. Of the 397 useable evaluation forms collected, 9 were for residents who had received only a single mini-CEX. These 9 evaluations were excluded from the study to permit calculation of the psychometric characteristics of the format, so 388 evaluations formed the basis for analysis. The evaluations reflected the performance of 88 residents; each resident had between 2 and 10 evaluations (mean, 4.4 2.2; median, 4). These evaluations were collected throughout the year of training; 78 of the residents (89%) were in their first year, 9 (10%) were in their second year, and 1 (1%) was in his or her third year. The process incorporated the efforts of 97 attending physicians, each of whom conducted between 1 and 26 evaluations (mean, 4.0; median, 2.0). Thirty-three attending physicians each conducted only 1 evaluation. Analysis Our results are reported in three parts. First, we describe the nature of the 388 encounters in terms of the setting of the mini-CEX, the types of patients seen, and the patients' diagnoses or problems. Second, we examine the mini-CEX from the perspective of the 97 evaluators, emphasizing their ratings of the residents and their satisfaction with the mini-CEX format. Finally, we analyze the performance of the 88 residents in terms of their competence ratings, the reproducibility of those ratings, and resident satisfaction with the mini-CEX format. Results are given SD. Results Nature of the Encounters Of the 388 encounters, 54% occurred in an inpatient setting, 38% in an outpatient setting, and 14% in the emergency department (information on setting was missing for 4% of encounters). The overall competence ratings were similar in all settings (P = 0.59, data not shown), but the duration of the encounter differed significantly according to setting (P = 0.003). The average inpatient encounter lasted 34.9 minutes, the average outpatient encounter lasted 27.7 minutes, and the average emergency department encounter lasted 28.2 minutes. Fifty-eight percent of the encounters represented the first visit of a patient to a particular resident, and 36% were return visits to that resident. There were no statistically significant differences in overall clinical competence scores or component scores for encounters with new and returning patients. The average first visit (35.3 minutes) was longer than the average follow-up visit (25.3 minutes; P < 0.0001). As a group, the mini-CEXs took between 5 and 100 minutes (mean, 31.5 19.3 minutes; median, 25 minutes). There were small but statistically significant Pearson product moment correlations of 0.10 to 0.16 between the duration of the evaluation and performance ratings (P < equals 0.05), indicating that longer encounters were generally associated with higher resident performance ratings. Duration also correlated with satisfaction with the mini-CEX format on the part of residents (0.19, P = 0.002) and faculty members (0.27, P < 0.001). For 239 of the 388 encounters (62%), the patient's problems or diagnoses were specified by the evaluator; they covered a broad range and included a representative array of common problems such as hypertension, diabetes, congestive heart failure, coronary artery disease, arthritis, chronic obstructive pulmonary disease, abdominal pain, and peptic ulcer. In addition, several patients had problems that crossed from traditional internal medicine into other disciplines, including neurology (seizures), substance abuse (alcohol and drugs), psychiatry (depression and schizophrenia), dermatology (rashes), gynecology (ovarian cysts), prevention (obesity or smoking), and ophthalmology (cataracts). Many patients, particularly elderly persons and persons with the acquired immunodeficiency syndrome, had multiple problems. Some patients had acute problems, such as myocardial infarction or respiratory failure. Evaluator Performance The mean ratings given by the 97 evaluators were 6.6 or 6.7 for history-taking skill, physical examination skill, clinical