Use of a Structured Assessment to Measure Nontechnical Aspects of Otolaryngology Residency Training.

An educational curriculum that produces competent clinicians is thecornerstoneof all residency trainingprograms.The development of this clinical expertise has many components. Some, such as knowledge acquisition, technical expertise, andexperience,are more tangible and measurable. Others, such as clinical judgment and critical thinking, aremore enigmatic andyet equally if notmore important. The challengeofhowto teach these less tangibleattributeshasconfoundedmedical educators since Hippocrates and continues todo so. Lackingbetterdefinition,weaseducators tend to rely ona“weknowitwhenwesee it”methodofassessingour trainees’ progress in their acquisition. In general, there tends to be agreement among the faculty about resident expertise at the extremes: the residents we knowwill to do the right thing regardlessof their level of trainingand the residentswhose clinical decision-makingwehavedifficulty trustingdespite an adequate knowledge base. Sorting out the residents in between anddefiningthespecific issuesare thatare impedingtheirprogress along the clinical continuum ismore difficult. To do better we need to be develop validmethods formeasuring these “less tangible” qualities. With this in mind, Shin and colleagues1 from the Harvard Medical School program have over the past 6 years developed a measurement instrument (Clinical Practice Instrument [CPI]) designed to be used for trainee assessment in the context of a highly structured oral board format examination. The CPI is similar in concept to what is used more extensively in quality-of-life surveys. It contains 21 questions completed by the examiner, which have either Likert scale or dichotomous responses depending on the issue being measured. The instrument is designed to assess the domains of information gathering, knowledge base, systematic organization, integration of aggregate patient findings, clinical judgment, and understanding the individual elements required to execute a treatment plan. This instrument has been validated, in their earlier published work,2 to be reliable between different raters, internally consistent, responsive to change, and valid from a criterion and discrimination standpoint. Building on this previous validation work, the authors describe their experience using this instrument for repeated testing of the same residents over the course of their residency training. They found that not only were total scores significantly different among postgraduate year (PGY) levels of training (PGY 2 level vs PGY 4 or PGY 5 level) but also that domain scores related to information gathering and organizational skills were acquired earlier in training, whereas knowledge base and clinical judgment improved later in residency. Clearly, the concept of a structured assessment technique, suchas theCPI, is an improvementover thecurrent typical oral board format in which the criteria for assessment are less specific and rely more heavily on the examiner’s “gestalt” of the candidate.What is less clear is whether the questions asked in theCPIaretrulyable toassess intangibles, suchas judgmentand critical thinking, as factors independentof knowledgeacquisition and overall experience. The findings of this article1 suggest this to be the case in the context of how the authors defined these domains relative to the CPI questions asked; however, do these domain scores really reflect these qualities in everyday practice? Acquiring (and teaching) the ability differentiate a sick patient who is getting better from one who seemswell but is getting sicker,when tooperate and,more important,whennot to,whento treatandwhentoobserveareextraordinarily complex issues. They are, however, at the core of the clinical expertise we strive to create. A continued focus on refining the questions in the assessment tool is needed. Thiswill come through a broader,multiinstitutional application of this tool to the assessment of different formsof real-time clinical interactionpairedwithmore long-term follow-up of trainees as they go out into practice. Only by doing this will we better understand whether we are measuring the right things. In the interim,Shinandcolleagues1 are to be congratulated for their sustained efforts,whichhave helped raise awareness and focus on these issues.