Use of administrative data for clinical quality measurement

T In this edition of the Journal, an article comparing outcomes in California and New York State demonstrates that the low-volume programs in California are the main reason for the disparity in cardiac surgical outcomes between the two states. The report presents results in a way that we are all used to seeing, risk adjusted with confidence limits, statistical significance, and the like, with primary and secondary conclusions that make sense. Although everything the authors assert may be right on the money, we need to use caution in interpreting this kind of report, because for the California outcomes it relies exclusively on administrative data, and there lies a slippery slope. The first problem with using claims data in clinical studies is that it is starting with an inexact tool. Administrative data are generated by professional coders who work in hospital medical records departments. Coders are trained to review charts and identify the diseases and procedures that define the most resourceintensive diagnosis-related group (DRG) assignment for each hospitalization. By definition, DRGs encompass similar but not identical patient categories, so coding does not create exact lists of clinical conditions. Going back to International Classification of Diseases, Ninth Revision (ICD-9) codes helps, but it does not solve this fundamental problem. More importantly, diagnosis codes are not dated in administrative data and do not reliably distinguish between comorbidities and procedural complications. For example, diagnoses such as stroke, atrial fibrillation, and renal failure could either increase the risk of surgery as comorbid variables or represent postoperative sequelae. This makes for a reliable but not very useful risk-adjustment system: bad outcomes obviously do predict bad outcomes. In addition, hospitals spend variable amounts of time, energy, and money to make coding better, but their view is toward optimizing reimbursement, not perfecting clinical accuracy. 3M, the company that brought us Post-It Notes, takes coding optimization a step further with a product called DRG Assurance, which helps institutions to improve documentation of revenuecritical diagnoses and secondarily improves the intensity of coding. Coding practices therefore vary significantly among institutions. In the final analysis, as seen through the distorted prism of administrative data, a program like DRG Assurance that was designed to yield higher institutional payment changes coding, increases the recorded comorbidity level of an institution’s patients, increases the complication rate, and reduces risk-adjusted mortality. In reality, of course, the outcomes have not changed, just the coding. If administrative data are inaccurate and labile, why does anyone use them for clinical research? The main reason is because these data are accessible to everyone and relatively inexpensive to use. In fairness, at times administrative data may be the only source of information available to look at a clinical question. In these circumstances, the accuracy caveats have to be prominently acknowledged and conclusions drawn with a conservative brush, lest we unwittingly validate a growing approach to quality measurement with which we are