New Instruments for Assessing Disability: Not Quite Ready for Prime Time

Measurement of patient-reported outcomes (recently termed “PROs”) has been on the ascendancy in medicine, and especially in rheumatology, for nearly 25 years. These outcomes, which can be recorded by patients on questionnaires, such as the Health Assessment Questionnaire (HAQ) from our group (1) and the Medical Outcomes Study 36-question Short Form (SF36) (2), offer important advantages over traditional physician-recorded “process” measures. These advantages include the measurement of outcomes using the values of patients, the focus on longer-term outcomes, and the emphasis on the quality as much as the quantity of life. PROs can include the assessment of physical function or disability, treatment side effects, medical care costs, pain, or other content areas. Furthermore, they are easier to administer and less expensive than physician-observed health status measures (3). As a group, PROs have been proven to be reliable, valid, and sensitive to change, displaying these attributes in greater abundance than do physicianreported measures. Accordingly, PROs have become standard in the evaluation and approval of new drugs as well as in observational studies and randomized trials. They have been translated and culturally adapted to scores of languages and cultures and validated in literally thousands of studies (4). Our increasing reliance on PROs represents a major change in the evaluation of treatments and services to patients. Rheumatologists have been among the leaders of these efforts. After a quarter of a century, it seems reasonable to ask whether a new generation of outcome assessment instruments should supersede the old. Surely, it may be time to move our efforts to the next plateau, to take a set of good ideas and to improve them further. There is no shortage of suggestions for change or for innovations promising improvements. The article by Wolfe et al in this issue of Arthritis & Rheumatism, analyzing the disability index (DI) of the HAQ, is an innovative step toward the refinement of PROs as they relate to rheumatic disease (5). The conclusions to be discussed in this editorial are as follows: 1) the time for improvement of the HAQ and other instruments has come; 2) Wolfe and colleagues have made the best approach yet toward these issues in rheumatology; and 3) the newer techniques of item response theory (IRT) and computerized adaptive testing (CAT) have great promise, especially when used in combination. Nevertheless, I suggest that 1) a potential limitation of these approaches is loss of face validity, which differentiated PROs from physician-based observations in the first place; 2) human insights cannot easily be delegated to machines; and 3) what has been proposed to this point may represent change but not improvement. First, it is important to consider the background, which may seem complex but is essential for understanding both the promise and the pitfalls of these new approaches. IRT uses statistical modeling techniques to analyze multiple items that attempt to assess the same outcome dimension, such as questionnaire items on the HAQ DI. Using data about the questionnaire items themselves, one can assess quantitative aspects of any resulting index and of alternative indices. A problem arises, however, because IRT approaches require unidimensionality (i.e., that all of the items on a scale measure the same concept, as, for example, with walking a short distance and walking a longer distance). When the content area is not unidimensional (as with disability), face validity can be lost, with unfortunate consequences. The simplest IRT statistical model, used by Wolfe et al, originates from the work of Rasch and focuses on the relative difficulty of items (6). More recent modeling approaches include a parameter for item discrimination that allows some items to be more James F. Fries, MD: Stanford University School of Medicine, Palo Alto, CA. Address correspondence and reprint requests to James F. Fries, MD, Stanford University, Division of Rheumatology and Immunology, Stanford University School of Medicine, 1000 Welch Road, Suite 203, Palo Alto, CA 94304-5755. E-mail: jff@stanford.edu. Submitted for publication April 22, 2004; accepted in revised form June 22, 2004.