Discordance of Databases Designed for Claims Payment versus Clinical Information Systems: Implications for Outcomes Research

Insurance claims data are being used increasingly to study clinical outcomes and quality of care [1-4]. Each year, hospital-specific mortality rates, adjusted by clinically modified International Classification of Diseases (ICD-9-CM) codes from Medicare bills, are released by the Health Care Financing Administration (HCFA) [1-3]. Using ICD-9-CM data to adjust for illness severity, threefold differences for surgeon-specific mortality in Philadelphia were found by Williams and colleagues [4]. Many of the Patient Outcomes Research Teams (PORTs), supported by the Agency for Health Care Policy Research, are using ICD-9-CM coded Medicare discharge abstracts to examine the process of medical care, including physician- and hospital-specific performance [5-7]. The potential advantages of using insurance claims data sets for clinical research have been described in many previous publications [8]. They include 1) large samples of geographically dispersed patients; 2) longitudinal records; 3) data already collected and available; and 4) defined sampling frames. The question remains: Are data collected to obtain insurance reimbursement a valid proxy for data collected for clinical care and research purposes? Such validity is essential to identify clinically relevant populations and to adjust for illness severity and differences in outcomes [9]. Six reabstracting studies have attempted to answer this question with respect to analysis of patients discharged after acute myocardial infarction [10-15]. These studies selected patients with the ICD-9-CM code 410, the code for acute myocardial infarction. By examining medical records, they found that clinical criteria for an acute myocardial infarction were met in 43% to 87% of records where the code was used at discharge. Errors resulted when the physician listed the acute myocardial infarction incorrectly, when a myocardial infarction occurred in a previous admission, or when myocardial infarction was ruled out (if it was the admitting diagnosis). A substantial limitation of five of these studies was that they selected patients based on claims data. Thus, the groups selected for review were only those patients with an ICD-9-CM code for myocardial infarction. Using this design, it was only possible to obtain estimates of disagreement in one direction; patients who had a condition coded in the clinical data set, but not in the claims data set, could not be examined. A second limitation of the previous studies is that their comparison gold standard was based on retrospective review of information recorded in the discharge summary or medical chart. Medical record data are limited by the unstructured way in which they are collected. Inaccuracies in these sources cannot be identified in such a study, and it is possible that in some disagreements with ICD-9-CM codes, the medical record is incorrect. Our study examined the suitability of billing data compared with clinical data (prospectively collected for cardiology research and patient care) for use in clinical outcomes research. The descriptors for coronary artery disease that we examined were those listed as important determinants of prognosis by an expert panel from the American College of Cardiology [16]. Methods Insurance Claims Data The administrative or insurance claims information comprised all discharge abstracts from Duke University Medical Center between July 1985 and May 1990 containing any procedure code for coronary arteriography. All discharged patients, regardless of insurance status or age, were routinely classified by ICD-9-CM codes recorded by trained medical record technicians based on the attending physician's listed discharge diagnoses, the discharge summary, and selected information from the progress notes and from the test result sections of the hospital chart [17]. These records contained up to 30 diagnostic codes and 9 procedure codes. After the technician had assembled the ICD-9-CM codes, the discharge abstract and the chart were returned to the attending physician for final approval by signature; ICD-9-CM codes were not generated for patients having outpatient cardiac catheterization unless they were subsequently admitted for further evaluation or treatment. The records for the subgroup of Medicare patients in this study were sent by Duke Hospital to the North Carolina Medicare intermediary and, thus, reflect the Duke Hospital data contained in the Health Care Financing Administration data sets. Clinical Database Data The clinical information consisted of important diagnostic and prognostic information about coronary artery disease routinely collected on standardized data forms by the cardiology fellow doing the cardiac catheterization for suspected ischemic heart disease. Information collected included details from the patient history, physical examination, laboratory studies, and cardiac catheterization, as previously described [18]. Each new fellow entering the catheterization laboratory was given a 3-hour training session on variable definitions and use of the data forms and was given an operations manual covering these details. In addition, all data were reviewed for accuracy by the attending angiographer associated with the case; additional consistency, range check, and other quality control measures were done during the data entry process by trained research technicians. This information was stored in the Duke Databank for Cardiovascular Disease, a completely separate and independent system from the hospital administrative records described above. Records Matching and Variable Definitions Records from the administrative and clinical files were matched by unique, patient hospital identification numbers and hospitalization dates. Only the first matching clinical record for each patient was included in the analysis. Twelve clinical variables were mapped to ICD-9-CM codes according to an algorithm developed by the Patient Outcomes Research Team for chronic ischemic heart disease (Table 1) (Romano PS, Roos LL. Unpublished observations). The variables studied were selected if they met two criteria: 1) They were considered to be determinants of prognosis for coronary artery disease according to an expert panel from the American College of Cardiology; 2) they could be mapped to diagnoses contained in the ICD-9-CM coding system [16, 17]. The definitions of the clinically identified conditions appear in the Appendix. Table 1. International Classification of Diseases-9-CM and Clinical Detail Map Appendix.Glossary of Terms Data Analysis Based on the clinical condition and the ICD-9-CM map described above, two-by-two tables were constructed to assess the agreement between the data sources. For the claims data, a condition was considered to be absent if it was not coded. For the clinical data, patients with missing data were excluded from the analysis for the specific missing condition. Kappa statistics were generated for each condition to measure agreement while controlling for chance agreement [19]. Confidence intervals and test statistics for proportions were calculated by the normal approximation. For the diagnoses of acute myocardial infarction, congestive heart failure, angina, and unstable angina, we reviewed a random sample of 15 clinical-positive and claims-negative charts as well as 15 claims-positive and clinical-negative charts for each diagnosis to illustrate the major reasons for disagreement. In addition to the comparisons made in the overall data sets, subsets defined by age, fiscal year, and sex were compared to determine if the coding accuracy varied according to these factors. Results The study group consisted of 12 937 consecutive patients having inpatient cardiac catheterization between July 1985 and May 1990. Although each record represented the first cardiac catheterization in the claims records, from the perspective of the clinical records, 89% involved the first catheterization, 8% involved the second catheterization, and the remaining 3% involved the third or subsequent catheterization. The patients had a mean age of 58.8 years, 34% were women, and the racial composition was 88% white, 10% black, and 2% other. At cardiac catheterization, the mean left ventricular ejection fraction was 52%. The distribution of the number of diseased major epicardial vessels (zero, one, two, or three) was 23%, 26%, 23%, and 28%, respectively. Overall, the study group characteristics were similar to those of other large angiographic registries except for the greater proportion of women and the higher mean age [20, 21]. Measures of Agreement Specific measures of agreement between clinical database and ICD-9-CM variables are listed in Table 2 in descending order of value (the agreement rate adjusted for chance agreement). Kappas ranged from 0.83 for diabetes mellitus to 0.09 for unstable angina. Of the 12 conditions, only 3 (diabetes, acute myocardial infarction, and hypertension) were identified by the claims data more than 50% of the time that they were identified by the clinical data. Table 2. Comparison of Agreement by Condition Ranked by Kappa Value In the clinical data set, two conditions were graded according to severity, congestive heart failure, and mitral regurgitation. With increasing severity levels, the claims data were more likely to identify the presence of these conditions. Claims data identified 31% of clinically identified congestive heart failure that was New York Heart Association class I and II and identified 45% of class III and IV heart failure (P < 0.0001) [22]. Similarly, claims data identified 40% of grades I and II mitral regurgitation and identified 69% of grades III and IV mitral regurgitation (P < 0.0001). When all diagnoses were considered together, the overall agreement of ICD-9-CM codes with clinical data was 0.75 (99% CI, 0.75 to 0.76). The proportion of conditions in the clinical data set identified by claims data was 0.39 (99% CI, 0.38 to 0.39) (Table 3). Stratified by fisc