Quality Assurance Data for Residents' Global Performance Ratings

The Accreditation Council for Graduate Medical Education (ACGME) has defined six areas of competency that are expected of a new practitioner. Residency programs must require their residents to master competencies in the six areas, but little guidance exists about how to assess these competencies. To assist residency programs, the ACGME has developed the Toolbox of Assessment Methods. The Toolbox recommends global ratings as a potentially applicable assessment method for selected competencies in two areas (patient care and practice-based learning and improvement). Global performance ratings are the most widely used measure of residents’ performances. A literature review of global ratings in residency education concluded that they provide relatively reliable and consistent evaluations. When used in an OSCE, global ratings have been shown to be reliable and valid. Global ratings used over time in actual clinical settings, however, have been described as ‘‘. little more than a popularity contest . ’’ due to the unreliability of raters’ memories and the paucity of observation. Other problems with global performance ratings in actual clinics (GPRACs) are leniency error, which also contributes to the low numerical precision of individual ratings, and the lack of behaviorally specific written comments. Many criticisms of the GPRAC as a measurement method would be less serious if the GPRAC were viewed as a low-cost, noninvasive epidemiologic screening test for marginal performance, analogous to a blood pressure screening. Prescriptive screening is aimed at detecting disease that can be better managed when detected early in presumptively healthy individuals. Continuing this screening analogy, a substandard (below-average) GPRAC for John Q Resident (i.e., positive screening test) would indicate possible marginal performance. However, in order to diagnose and treat the problem, additional data must be obtained, such as assigning John Q to a rotation with intensive observation by several attending faculty. If GPRAC were viewed as a prescriptive screening test, then measurement problems such as low numerical precision of a single observation and paucity of observation would be less serious because additional diagnostic performance data would precede the initiation of definitive action. This paper proposes a method to generate quality assurance data for the GPRAC. The completed rating forms are only one of four parts (context surrounding the ratings, faculty ability and motivation to judge performance, completed rating forms, and the program’s willingness to take administrative action regarding a marginal resident) that comprise the GPRAC system. The four parts of a GPRAC system function interdependently as a unified whole. A GPRAC system can be improved by making changes to one or more of the system’s parts. Quality assurance addresses the question of how GPRAC quality can be ensured and managed. Internal quality monitoring provides quantitative data to assess how well a system is functioning. This study proposes an internal quality monitoring system for the GPRAC based on four prerequisite conditions that should exist in a residency program. First, GPRAC forms must be routinely completed and returned to the residency program director. Second, attending faculty must write candid appraisals when a resident’s performance is marginal. (A national survey of residency directors indicated that about 7% of residents have problems in one or more areas of competency.) Third, the program must be willing to act by diagnosing and by treating a performance problem that is identified by a small number of GPRACs. Fourth, the program should have a systematic GPRAC database to support more severe administrative actions when educational diagnosis and treatment fail. Internal data for quality monitoring can be collected for each prerequisite condition described above, specifically: