The Mini-CEX: A Method for Assessing Clinical Skills

When the American Board of Internal Medicine (ABIM) stopped administering the oral examination in 1972, it asked program directors to evaluate the clinical competence of candidates for certification. The ABIM has since recommended that program directors use a traditional clinical evaluation exercise, or CEX, as one form of assessment for residents, particularly first-year residents. In this exercise, which is based on the bedside oral examination, a faculty member evaluates the resident as he or she performs a complete history and physical examination on an inpatient and then reaches diagnostic and therapeutic conclusions (1). The CEX takes about 2 hours, and residents are assessed during their first year of training (2). The CEX has been criticized as an evaluation instrument because the results are unlikely to be generalizable beyond the observed encounter (3-6). Physician performance is case specific, and the CEX assesses the performance of the resident with only one patient. Moreover, faculty members vary in stringency, and the CEX incorporates the ratings of only one examiner. In response to these problems, the ABIM proposed the mini-CEX. In this variation on the traditional CEX, one faculty member evaluates a resident with one patient in a 15- to 20-minute encounter; several of these assessments are conducted throughout the year. The encounters can occur in various settings (ambulatory, emergency department, and inpatient), so the patients present a broader range of challenges. This allows the residents to be evaluated by different faculty members as they interact with several patients who pose a wider range of problems. In addition to offering better evaluation, the multiple-patient mini-CEX differs from the traditional CEX in what it can evaluate. The traditional CEX focuses on the resident's thoroughness in an environment that is uninfluenced by the time constraints of medical practice. In contrast, multiple mini-CEX encounters are more variable because the challenges they pose depend on a broader range of settings, patients, and tasks. Furthermore, mini-CEXs assess the resident's ability to focus and prioritize diagnosis and management within the context of real clinical practice. Preliminary study of the mini-CEX verified that it 1) assessed residents in a broader range of clinical situations than the traditional CEX, 2) produced scores that were more reliable than those based on the traditional CEX, and 3) offered the residents more opportunity for observation and feedback (7, 8). However, these findings were based on only 88 residents from five programs in a limited geographic region. Moreover, the study did not address a series of important questions about changes in competence throughout the first year of training, the complexity of patient problems, the focus of the encounter, and the relative amount of time spent observing resident performance and offering feedback. The current study replicated and expanded on the previous work. Specifically, we collected the data for the study during a 1-year period from 21 training programs, representing a range of resident ability. Methods Procedure For each mini-CEX encounter, one examiner observed the resident conduct a focused interview or physical examination in an inpatient, outpatient, emergency department, or other setting. After asking the resident for diagnostic or therapeutic decisions, the examiner completed the rating form (Appendix Figure) and provided feedback. The rating form was a card that produced two copies, and the same form was used across all sites of the study. For each encounter, the examiner recorded the date, the complexity of the patient's problem on a 3-point scale (low, moderate, and high), the sex of the patient, the type of visit (new or return), the setting (ambulatory, inpatient, emergency department, or other), the number of minutes spent observing the encounter, and the number of minutes spent giving feedback. The examiner also noted whether the focus of the encounter was data gathering, diagnosis, treatment, or counseling. Using a 9-point scale (in which 1 to 3 were unsatisfactory, 4 was marginal, 5 and 6 were satisfactory, and 7 to 9 were superior), the examiner rated the resident on interviewing, physical examination, professionalism, clinical judgment, counseling, organization and efficiency, and overall competence. The examiner also rated his or her own satisfaction with the method as a valid and efficient assessment device on a 9-point scale in which 1 was dissatisfied and 9 was very satisfied. For all items, the examiner could select not applicable where appropriate and a small number of examiners assigned ratings in 0.5-point increments. In addition, a total score was calculated as the mean of the six component ratings. Participants Twenty-one residency programs nonsystematically volunteered to assess residents by using this format. They were primarily, but not exclusively, from the northeastern United States. The programs represented a broad range of resident abilities, as judged by their program pass rates on the ABIM certifying examination. A total of 1228 usable evaluation forms, representing data from 421 residents, were returned to the investigators. The residents were evaluated on 1 to 8 encounters, with a mean (SD) of 2.9 1.5 and a median of 3. The evaluations were conducted by 316 examiners who evaluated a mean of 3.9 4.2 residents and a median of 2 residents. Statistical Analysis Analysis was conducted at the level of the individual encounter, the resident, and the examiner. For the individual encounters, we describe the complexity, setting, and focus of the encounters, as well as the patients' sex and type of visit. Performance across the four quarters of the academic year are also described. For the residents, we average their ratings over all their encounters and analyze the relationships among the components of competence and the reliability of the total score. For the examiners, we also average their ratings and analyze their satisfaction with the method and its relationship with other aspects of the encounter. Nonparametric statistical tests, including the MannWhitney U, KruskalWallis (with a chi-square approximation), and Spearman , were applied to these data because some of the variables are categorical and others may not meet assumptions of normality. Analyses were conducted by using SAS software (SAS Institute, Inc., Cary, North Carolina) and SPSS software (SPSS, Inc., Chicago, Illinois). All educational tests are fallible, and generalizability theory offers a family of analysis-of-variancebased indices to quantify how well their scores are generalizable beyond the specific behaviors they elicit (9-11). Details of the application of generalizability theory in this study can be found in the Appendix, but briefly, the total scores were analyzed to estimate variance components for the residents and for measurement error. These reflect the variability in ratings that would occur if each resident were examined by a large number of evaluators while seeing a large number of patients. The variance components were used to estimate what the 95% CIs would be for total scores based on 1 to 14 encounters. Analogous to its counterparts in biostatistics, the CI has the advantage of simultaneously addressing issues of psychometric and practical significance. Results Table 1 presents data on the numbers of programs, evaluators, residents, and encounters involved in the study. The degree of participation varied greatly. Unless otherwise noted, data reported are the mean (SD). Table 1. Number of Examiners, Residents, and Encounters Included in the Study Encounters The mean age of the patients was 55 18 years; 45% were men and 49% were women (sex was not recorded for 6% of the encounters). The complexity of patient problems was rated as low in 20% of the encounters, moderate in 54% of encounters, and high in 17% of encounters (9% were missing complexity ratings). Thirty-eight percent of the encounters took place in the inpatient setting, 52% were in the outpatient setting, 7% were in the emergency department setting, and 2% were in other settings (mostly the critical care unit) (2% were missing settings). Thirty-seven percent of the encounters were based on new patients, and 42% were follow-up visits (and for 22% of the encounters, this information was not specified). Each encounter called for residents to perform one or more tasks. Of the 1228 encounters, 56% required data gathering, 40% required diagnosis, 34% required therapy, and 26% required counseling. The median (mean [SD]) time the examiner spent observing the resident interact with the patient was 15 (18 12.1) minutes, and the time spent providing feedback to the resident was 5 (7.6 5.3) minutes. The amount of time spent observing (P < 0.001) and giving feedback (P < 0.001) was greater for new visits than return visits. Likewise, time spent observing (P < 0.001) and giving feedback (P = 0.007) increased with the complexity of the patient's problem. The examiner filled in the patient's problems or diagnoses for 1131 encounters (92%). The problems covered a broad range of presenting symptoms, including abdominal pain, chest pain, cough, dizziness, fever, headache, low back pain, shortness of breath, and weight gain. The content represented common internal medicine problems, such as arthritis, asthma, chronic obstructive pulmonary disease, congestive heart failure, coronary artery disease, diabetes, and hypertension. Routine physical examinations, pelvic examinations, and breast examinations were included, as were problems such as seizure, substance abuse, depression, dementia, and rash. Many of the patients had several problems (such as congestive heart failure, hypertension, and diabetes) or acute problems (such as sepsis and myocardial infarction). Problem complexity varied significantly by the setting (P < 0.001) and the type of patient (P = 0.001) but n