Commentary on Reliability of the AMA Guides to the Evaluation of Permanent Impairment

T hank you for inviting me to review the article “Reliability of the AMA Guides to the Evaluation of Permanent Impairment,” which appears in this issue of JOEM. Overall, I appreciate the authors’ willingness to tackle the important question of this study, and I am pleased to see it published primarily because of the timeliness and relevance of the general issues it tries to address. Nevertheless, a substantial number of significant inherent limitations of their study remain and deserve mention, some of which cannot be overcome with a pilot effort of this kind. Perhaps the usefulness of this article will be to initiate a broader scientific dialogue and hopefully to stimulate more thorough and meaningful comparisons of this kind. The objectives as set forth in the article include determining the reliability of the AMA Guides for back injury, and presenting a broader research agenda. The authors stray from their stated agenda to also compare differences of magnitude of ratings between the AMA Guides fifth and sixth editions. The latter comparison is perhaps more a question of “construct validity” than “reliability” per se1 and was never thoroughly investigated in this study, because an accepted “gold standard” for absolute rating values has yet to be established. Perhaps the key issue the authors should be addressing is the overall reliability of AMA impairment rating guides. Is reliability good or isn’t it? How do we define “good”? If it is good, how do we make it better? If it needs improvement, how can we achieve this? It has been well recognized that reliability problems exist for multiple iterations of the AMA Guides, as the authors’ references and citations, and other salient references2,3 already attest. This article tries to set the stage to move these issues forward with a clarion call for a new and broader research agenda, for which I wholeheartedly agree. Unfortunately, their approach has significant pitfalls, which can reasonably be expected to undermine some of their conclusions: To begin with, the analytical focus (albeit first step) taken by these authors, in comparing the fifth versus sixth editions of the AMA Guides, reduces these issues to a very narrow (indeed trivial) platform of comparison. We are left with a potentially misleading snapshot of only one anatomical region (cervical) in only one organ system (spine) for the entire musculoskeletal system. Furthermore, in approaching their comparison of the spine, they are only comparing the “diagnosis-related estimate” method of AMA Guides fifth edition to the “diagnosis-based impairment” method of AMA Guides sixth edition, respectively. They have not included the relative reliability estimates and comparisons also necessary and possible if one includes the problem-prone reliability estimates derived from the surface “range-of-motion” method,4 which is embedded in the AMA Guides fifth edition of the spine, and which has been deliberately excluded from the AMA Guides sixth edition methodology (in favor of the exclusive diagnosis-based impairment approach) because of the fundamental reliability concerns surrounding its use.5 Clearly, such broader comparisons must be made before drawing any meaningful conclusions concerning the relative merits of the analytical platforms of various editions of the Guides. The methods employed for this study appear demonstrably narrow and contrived. Given the “pilot” nature of their study, this limitation appears unavoidable and does not obviate the relevance of this article. The above consideration notwithstanding, my additional concerns are enumerated below as follows: Rather than simply stating that they chose low back injuries as their rating focus, the authors might have researched the literature available on the most frequent regional impairments or the most frequent ratable diagnoses for the fifth (or earlier) edition of the AMA Guides (see, for example, the Guides Newsletter6,7) to put their choice in better perspective. In fact, nonspecific backache, shoulder, carpal tunnel, and lumbar disc injuries are also among the five most frequently rated conditions and may warrant similar comparisons in a pilot study of this kind. The selection and training of participants of this study appears particularly troublesome. According to the description provided, the “training” afforded to the novice raters of this study seems perfunctory and probably grossly inadequate. Regardless of the edition, the AMA Guides requires a high degree of medical knowledge and experience to perform ratings correctly. A number of educational venues are available to afford opportunity for physicians to gain the necessary knowledge and expertise to perform ratings correctly (see, for example, courses8,9 and certifications8 offered regularly to physician raters); such courses typically require several days of intensive didactics and precourse preparation as well as some operating familiarity with the AMA Guides to complete the training course with optimal proficiency. A conservative estimate would suggest that there are at least 3000 active certified independent medical examiners with formal training and expertise in use of at least one edition of the AMA Guides. The AMA Guides10 mandates a level of “knowledge, skill, and ability generally accepted by the medical scientific community” when evaluating medical impairment, which is clearly not the case here. I believe that the use of naive and inexperienced raters is a significant probable methodological flaw implicating the relevancy of their findings, and that their results must be qualified accordingly. Time afforded to raters for performance of impairment ratings may also have been problematic. Each rater was allotted a 3-hour block of time (total) to perform impairment ratings according to 20 case descriptions provided, for an average 9-minute allocation per impairment rating. My own experience suggests that, in an actual case setting, a minimum of 20 minutes is required to adequately extrapolate and organize data from a