Hospital Quality Data: CMS Needs More Rigorous Methods to Ensure Reliability of Publicly Released Data
Released: 2006-02-24T00:00:00

Letter

Background

Medicare spends over $136 billion annually on inpatient hospital care for its beneficiaries. To help ensure the quality of the care it purchases through Medicare, CMS launched the Hospital Quality Initiative in 2003. This initiative aims to refine and standardize hospital data, data transmission, and performance measures as part of an effort to stimulate and support significant improvement in the quality of hospital care.

One component of this broader initiative is CMS’s participation in the Hospital Quality Alliance (HQA), a public-private collaboration that seeks to make hospital performance information more accessible to the public, payers, and providers of care. Before the enactment of MMA, HQA had organized a voluntary program for hospitals to submit data on quality of care measures intended for public reporting. For its part as a participant in HQA, CMS set up a central database to receive the data submitted by hospitals and initiated plans for a Web site to post information on hospital quality of care measures. Thus, CMS had a data collection infrastructure in place when MMA established the financial incentive for hospitals to submit quality data.

Selection of Measures

The 10 measures chosen by the Secretary of Health and Human Services for the APU program are the original 10 measures that were adopted by HQA. HQA subsequently adopted additional measures that relate to the same three conditions—heart attacks, heart failure, and pneumonia—and others that relate to surgical infection prevention. (See table 1 for a listing of the APU-measure set and the expanded-measure set.) Hospitals participating in HQA were encouraged to submit data on the additional measures, but data submitted on the additional measures did not affect whether a hospital received its full payment update under the APU program. CMS and the QIOs have tested these measures for validity and reliability, and all measures have been endorsed by the National Quality Forum, which fosters agreement on national standards for measurement and public reporting of health care performance data.

To minimize the data collection burden on hospitals by the APU program, CMS and the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) have worked to align their procedures and protocols for collecting and reporting the specific clinical information that is used to score hospitals on the measures. JCAHO-accredited hospitals— approximately 82 percent of hospitals that participate in Medicare—have since 2002 submitted data to JCAHO on the same measures as those in the APU-measure set as well as many of those in the expanded-measure set. Beginning with the first calendar quarter of data submitted by hospitals for the APU program, hospitals had the option of submitting the same data to CMS that many of them were already collecting for JCAHO. In November 2004, CMS and JCAHO jointly issued a manual laying out the aligned procedures and protocols for discharges beginning January 1, 2005.

Collection, Submission, and Reporting of Quality Data

Hospitals use CMS’s definition of the eligible patient population to identify the patients for whom they should collect and submit quality data for each measure. The definition is based on the primary diagnosis and, for the two cardiac conditions, the age of the patient. Specifically, hospitals use diagnostic codes and demographic information from the patients’ medical and administrative records to determine eligibility based on protocols established by CMS.

Once the eligible patients have been identified, hospitals extract from their patients’ medical records the specific data items needed for the Iowa Foundation for Medical Care (IFMC) to calculate a hospital’s performance, following detailed data abstraction guidelines developed by CMS. Hospitals may submit data for all eligible patients for a given condition, or if they have more than a specified number of eligible patients, they may draw a random sample according to a formula, and submit data for those patients only. These data are put into a standardized data format and submitted quarterly through a secure Internet connection to the QIO “clinical warehouse” administered by IFMC. IFMC accepts into the clinical warehouse only the data that meet the formatting and other specifications established by CMS and that are submitted before the specified deadline for that quarter. About 80 percent of hospitals rely on data vendors— which typically are collecting the same data for JCAHO—to submit the data for them.

IFMC aggregates the information from the individual patient records to generate a rate for each hospital on each of the measures for which the hospital submitted relevant clinical data. These rates show how often a hospital provided the specific service or activity designated in the measures to patients for whom that service or activity was appropriate. Hospitals also collect information on each patient that identifies patients for whom the particular service or activity would not be called for, such as patients with a condition that would make prescribing aspirin or beta blockers medically inappropriate.

CMS posts on its Hospital Compare Web site each hospital’s rates for all the APU and expanded measures for which it submitted data. In November 2004, CMS first posted these rates, based on data from the first quarter of calendar year 2004. It subsequently posted new rates in March 2005, based on the first two quarters of calendar year 2004 data, and again in September and December 2005 with additional quarters of data. CMS continues to update these rates quarterly, using the four most recent quarters of data available. There can be up to a 14-month time lag between when patients are treated by the hospital and when the resulting rates are posted on the CMS Web site. (See fig. 1.)

CMS had to make its determination of hospital eligibility for the fiscal year 2005 annual payment update decision approximately 1 month after hospitals submitted their data for the first quarter.

Implementation of the APU Program

In implementing the APU program, CMS uses the same policies and procedures for collecting and submitting quality data as are used for HQA. For the first annual payment update determined by the APU program, which applied to fiscal year 2005, hospitals were required to begin submitting data by July 1, 2004, for the patients discharged during the first calendar quarter of 2004 (January through March 2004). Data were received from 3,839 hospitals, over 98 percent of those affected by the MMA provision. These figures include 150 hospitals that certified to CMS that they had no eligible patients with the three conditions during the first calendar quarter of 2004. Hospitals that have no eligible patients are not penalized and receive the full annual payment update. For the second annual payment update determined by the APU program, which applied to fiscal year 2006, participating hospitals were required to continue to submit data in accordance with the quarterly deadlines set by CMS. Failure to meet the requirements of the program and qualify for the full annual payment update in one year does not affect a hospital’s ability to participate in and qualify for the full update in the succeeding year.

CMS has assigned primary responsibility to the 53 QIOs to inform hospitals about the APU program’s requirements and to provide technical assistance to hospitals in meeting those requirements. This includes assistance to hospitals in submitting their data to the clinical warehouse provided by IFMC.

Other Reporting Systems

There are several organizations that administer reporting systems that collect clinical data, some of which also release their data to the public. Some of these organizations are in the public sector, such as state health departments, and some are in the private sector, such as accreditation bodies. Several of these systems have been in existence for a number of years, including one for as long as 16 years. Hospitals, health plans, nursing homes, and other external organizations submit data to these systems on a range of medical conditions, which for most of these systems includes at least one cardiac condition (e.g., percutaneous coronary intervention, coronary artery bypass grafting, heart attack, heart failure). Many of these systems make the results of the data they have collected available for public use. For example, one public organization has been collecting individual, patient-level data on cardiac surgeries from hospitals for the past 16 years and creates reports based on the data collected, which it subsequently posts on its Web site. Additionally, data collected by these reporting systems can also be used for quality improvement efforts and to track performance over time. (For more background information on other reporting systems, see app. II, table 3.)

CMS Has Processes for Checking Data Accuracy but Has No Ongoing Process to Check Completeness

CMS has processes for ensuring the accuracy of the quality data submitted by hospitals for the APU program, but has no ongoing process to assess whether hospitals are submitting complete data. To check accuracy, IFMC, a CMS contractor, electronically checks the data as they are submitted to the clinical warehouse. In addition, CDAC independently audits the data submitted by hospitals. Specifically, it reabstracts the quality data from medical records for a sample of five patients per quarter for each hospital and compares its results to the quality data submitted by hospitals. The data are deemed to be accurate if there is 80 percent or greater agreement between these two sets of results, a standard that hospitals had to meet for the APU-measure set to qualify for their full annual payment update for fiscal year 2006. To check completeness, CMS has twice compared the number of cases submitted by each hospital for the APU program for a given period to the number of claims the hospital submitted to Medicare, once for the fiscal year 2005 update and once for the fiscal year 2006 update. However, these analyses did not address non-Medicare patient records and the approach that CMS took in these analyses was not capable of detecting incomplete data for all hospitals. CMS has not put in place an ongoing process for checking the completeness of the data that hospitals submit for the APU program that would provide accurate and consistent information for all patients and all hospitals. Moreover, CMS has not required hospitals to certify that they submitted data for all eligible patients or a representative sample thereof.

CMS Checks Data Accuracy Electronically and Through an Independent Audit

CMS employs two processes to check and ensure the accuracy of the quality data submitted by hospitals for the APU program. First, at the time that data are submitted to the clinical warehouse, IFMC, a CMS contractor, electronically checks the data for inconsistencies and missing values. The results are shared with hospitals. After the allotted time for review and correction of the submissions, no more data or corrections may be submitted by hospitals for that quarter. These checks are done whether the hospital submits its data directly to the warehouse or through a data vendor.

Second, CDAC conducts quarterly independent audits to verify that the data submitted by hospitals to the clinical warehouse accurately reflect the information in their patients’ medical records. From among all the patient records submitted to the clinical warehouse each quarter, CMS randomly selects for CDAC’s reabstraction five patient records from each participating hospital. CDAC sends a request for these patients’ medical records to the hospitals, and they send photocopies of the records to CDAC for reabstraction. A CDAC abstractor reviews the medical record, determines if or when a specific action occurred—such as the time when a patient arrived at the hospital—and records that data field accordingly. Once the CDAC reabstraction is complete, the response previously entered into that field by the hospital is compared to that entered by the CDAC abstractor, and CDAC notes whether the two responses match. If they do not match, a second CDAC abstractor reviews the medical record to make a final determination. The results of the CDAC reabstraction are sent to the clinical warehouse, where the individual data matches and mismatches are summed to produce an accuracy score for each hospital. The accuracy score represents the overall percentage of agreement between data submitted by the hospital and data reabstracted by CDAC across all five cases. It is based on all the APU and expanded measures for which the hospital submitted data. The score, along with information from CDAC on where the mismatches occurred and why, is shared with the hospital and the hospital’s local QIO. CMS considers hospitals achieving an accuracy score of 80 percent or better to have provided accurate data. Hospitals with accuracy scores below 80 have the opportunity to appeal their reabstraction results.

In applying these processes for the fiscal year 2005 annual payment update, CMS did not require hospitals to meet the 80 percent accuracy threshold for the 10 APU measures to qualify for the full update. Rather, to receive their full payment update, hospitals only had to pass the electronic data checking performed when they submitted their data to the clinical warehouse for the first calendar quarter of the APU program—for discharges that occurred from January 2004 through March 2004. Although the accuracy scores were not considered for the payment update, CMS calculated an accuracy score for each quarter in which the hospital submitted at least six cases to the clinical warehouse. Each quarter the accuracy score was based on data for all the measures submitted by the hospital in that quarter and was derived from five randomly selected patient records. Along with the accuracy score, hospitals received information on where mismatches occurred and the reasons for the mismatches.

In contrast to the prior year, CMS applied the 80 percent threshold for accuracy as a requirement for hospitals to qualify for their full fiscal year 2006 annual payment update. IFMC continued to check electronically all of the data as they were submitted for each quarter and calculated accuracy scores quarterly for each hospital. CMS decided to base its payment update decision on the accuracy score that hospitals obtained for the third calendar quarter of 2004—for discharges that occurred from July 2004 through September 2004. This meant that the payment decision rested on the reabstraction results obtained from 5 randomly selected patient records. If a hospital met the 80 percent accuracy threshold based on all of the quality data it submitted, it received the full payment update. However, if a hospital failed to meet the 80 percent threshold, CMS recomputed the accuracy score using only the data elements required for the APU-measure set. For hospitals that failed again, CMS combined the CDAC reabstraction results from the third calendar quarter of 2004 with the CDAC results from the fourth calendar quarter of 2004 to produce an accuracy score derived from 10 patient medical records. CMS then computed accuracy scores first for all the quality data submitted by the hospital and finally for the APU-measure set, if needed to reach the 80 percent threshold. As a result, even though CMS assessed hospital accuracy primarily on the basis of data that exceeded those required for the APU-measure set, hospitals were not denied the full annual payment update except on the basis of the APU-measure set. A possibility does exist, however, that a hospital could have qualified for the full update based on its results for all the data it submitted, even if it would have failed using the APU-measure set. This could happen if the hospital submitted data that matched the CDAC abstractors’ entries more consistently for the data entries used exclusively in computing the expanded measures, such as those relating to smoking cessation counseling, than for the data required by the APU-measure set.

In the future, CMS intends to base its decisions on hospital eligibility for full annual payment updates on accuracy assessments from more than one quarter. Although its concerns about potential alignment issues affecting data for the first two quarters of the APU program led the agency to rely primarily on data from the third calendar quarter for the fiscal year 2006 update, CMS stated that its goal was to use accuracy assessments from four consecutive quarters when it determines hospital eligibility for the fiscal year 2007 full annual payment update.

CMS uses the accuracy scores in making decisions on payment updates, but the scores do not affect the information posted on the Hospital Compare Web site. The Web site transmits to the public the rates on the APU and expanded measures that derive from the data that the hospitals submitted to the clinical warehouse. CMS does not post the accuracy scores generated from the CDAC reabstraction process on the Web site or indicate if the hospital rates are based on data that met CMS’s 80 percent threshold for accuracy.

CMS Has No Ongoing Process to Ensure Completeness of Data Submitted for the APU Program

Although CMS has recognized the importance of obtaining quality data for the APU program on all eligible patients, or a representative sample if appropriate, it has not put in place an ongoing process to ensure that this occurs. For the fiscal year 2005 annual payment update, CMS checked that hospitals submitted data for at least a minimum number of patients by using Medicare claims data to estimate the number of “expected cases” that each hospital should have submitted to the clinical warehouse. To do this, it first calculated the average number of patients for each of the three conditions that each hospital had billed Medicare for over the previous eight calendar quarters (January 2002 through December 2003). Then, if the average number of Medicare claims for a condition was large enough to entitle the hospital to draw a sample instead of submitting data for all the eligible patients to the clinical warehouse, CMS reduced the number of “expected cases” based on the size of the sample. CMS told each hospital what its expected numbers of heart attack, heart failure, and pneumonia patients were. If the actual number of patients for whom hospitals submitted data for the APU program was lower, the hospitals were instructed to send a letter to their local QIO, signed by the hospital’s CEO or administrator, stating that the hospital had fewer discharged patients for that condition than CMS had estimated. If such a letter was filed, the hospital qualified for the full annual payment update. In the end, no hospital participating in the APU program was denied a full annual payment update for fiscal year 2005 for submitting data on an insufficient number of patients or any other reason.

For the fiscal year 2006 update decision, CMS took a different approach to using Medicare claims data to address the issue of completeness. CMS used Medicare claims data to check whether hospitals that billed Medicare for any cases with one of the three conditions submitted at least one case to the clinical warehouse. To do this, CMS compared each hospital’s Medicare claims for the three conditions for the four calendar quarters of 2004 to the hospital’s submissions to the clinical warehouse for those same quarters. CMS identified instances where hospitals had submitted one or more claims for payment to Medicare for any of the three conditions for a quarter when they had not submitted any cases with one of those conditions to the clinical warehouse. On this basis, CMS determined that 110 hospitals would not qualify for the full payment update for fiscal year 2006.

CMS conducted two additional analyses involving a comparison of the same Medicare claims data and quality data submissions to identify hospitals that may have submitted incomplete data for the APU program, but these analyses did not affect hospital eligibility for the full fiscal year 2006 payment update. The additional analyses identified (1) a set of hospitals that may have submitted samples of their eligible cases to the clinical warehouse when, according to the applicable sampling rules, they should have submitted data on all their cases; and (2) another set of hospitals that failed to submit cases to the clinical warehouse for all of the three conditions for which they filed Medicare claims in that quarter. However, in contrast to the hospitals that did not qualify for their full payment update, the hospitals in the second set submitted to the clinical warehouse at least one case for one of the three conditions. A CMS official stated that the agency plans to educate the hospitals identified by these additional analyses on the data submission and sampling requirements for the APU program.

The analysis that CMS conducted using Medicare claims data for its fiscal year 2005 update decision and the three analyses it conducted in conjunction with its fiscal year 2006 update decision shared two limitations: none addressed the completeness of data submissions for non- Medicare patients, and none could detect incomplete data for all hospitals. Given that non-Medicare patients represent a substantial proportion of the patients treated for heart attacks, heart failure, and pneumonia, any minimum number of “expected cases” based on Medicare claims inherently underestimates the total number of patients for which hospitals should have submitted quality data for the APU program. Moreover, the approaches taken in the analyses conducted for both fiscal year updates could not detect incomplete data for many hospitals. For example, in the fiscal year 2005 analysis, the difference between the number of cases expected under the CMS sampling rules and the higher number expected under the sampling rules that applied to JCAHO-accredited hospitals meant that JCAHO-accredited hospitals treating more patients than the minimum CMS sample of seven could have failed to submit data on most of the cases that exceeded the CMS minimum and still have met the number of expected cases set by CMS. The analysis that CMS conducted to determine hospital eligibility for the full fiscal year 2006 update also could identify only certain hospitals that submitted incomplete data, in this case limited to hospitals that submitted no patient data at all to the clinical warehouse in a given quarter.

CMS officials acknowledged that the lack of information on non-Medicare patients and the imprecise adjustments that CMS made to take account of the varying sampling procedures that hospitals could have followed limited the conclusions that CMS could draw from its Medicare claims data analysis for the fiscal year 2005 update. Because of these limitations, CMS officials described their effort as a rough check for inconsistencies between data submitted by hospitals to the clinical warehouse and the cases that the hospitals had billed to Medicare.

CMS has not combined these limited efforts to monitor the completeness of hospital quality data submissions with efforts to clearly inform hospital officials of their obligation to submit complete data. For example, CMS has not explicitly listed submission of complete data as a requirement for participating in the APU program on the “Notice of Participation” that the hospital CEO or administrator must sign when hospitals enroll. The notice states requirements for participating hospitals—including that they must register with the QualityNet Exchange Web site and that they must submit data for all measures specified in the APU-measure set by established deadlines. The notice indicates that the submitted data will undergo validation, a reference to the CDAC reabstraction process. However, the notice does not stipulate that hospitals must submit data for all eligible cases, or for a representative sample if appropriate.

We interviewed health professionals familiar with the APU program, several of whom raised concerns about data completeness. One expert in the area of outcomes research noted the potential for systematic underreporting by hospitals. He suggested that, as one approach to detect systematic underreporting, CMS could compare not only the number of patients for whom data were submitted and Medicare claims filed, but also the characteristics of patients for cases submitted to the APU program to the patient characteristics of comparable cases submitted to Medicare for payment. Another expert in the area of clinical quality improvement expressed his concern that the APU program did not verify the completeness of the data. He observed that hospitals have flexibility in determining which patients are included through their assignment of the patient’s primary diagnosis. A QIO official echoed this concern, noting the risk that hospitals could decide to not submit cases where patients had not received the services or activities assessed by the APU measures.

Data Accuracy Baseline Was High Overall, but Statistically Uncertain for Many Hospitals, and Data Completeness Baseline Cannot Be Determined

We could determine a baseline level of accuracy for the quality data submitted for the APU program but not a baseline level of completeness. We found a high overall baseline level of accuracy when we examined CMS’s assessment of the data submitted by hospitals for the first two calendar quarters of 2004. The median accuracy score exceeded 90 percent, which was well above the 80 percent accuracy threshold set by CMS, and about 90 percent of hospitals met or exceeded that threshold for both the first and the second calendar quarters of 2004. For most hospitals whose accuracy scores were well above the threshold, the results were statistically certain. However, for approximately one-fourth to one-third of all the hospitals that CMS assessed for accuracy, the statistical margin of error for their accuracy score included both passing and failing accuracy levels. Consequently, for these hospitals, the small number of cases that CMS examined was not sufficient to establish with statistical certainty whether the hospital met the threshold level of data accuracy. Accuracy did not vary between rural and urban hospitals, and small hospitals provided data as accurate as those from larger hospitals. The completeness baseline could not be determined because CMS did not assess the extent to which all hospitals submitted data on all eligible patients, or a representative sample thereof, for the first two calendar quarters of 2004, and consequently there were no data from which to derive such an assessment.

Baseline Level of Data Accuracy Was High Overall, and Large Majority of Hospitals Met Accuracy Threshold

Overall, the baseline level of data accuracy for the first two quarters of the APU program was high. The median accuracy score achieved by hospitals ranged between 90 and 94 percent, with slightly higher values in the second quarter and for the APU-measure set. (See fig. 2.) In addition, with at least half the hospitals receiving accuracy scores above 90, relatively few failed to reach the 80 percent threshold set by CMS.

In both quarters, 90 to 92 percent of hospitals obtained accuracy scores meeting the threshold using the APU-measure set, and 87 to 90 percent met the threshold using the expanded-measure set (see table 2). The 8 to 13 percent of hospitals that did not meet the accuracy threshold represented approximately 300 to 500 hospitals across the country.

There were minimal differences in baseline accuracy scores among hospitals characterized by urban or rural location and small or large capacity, but variation across hospitals served by different data vendors was more substantial. Rural hospitals and smaller hospitals generally received accuracy scores similar to those of urban hospitals and larger hospitals. Among the hospitals that used JCAHO-certified data vendors to submit their quality data to the clinical warehouse, a higher percentage of hospitals served by certain data vendors met the 80 percent threshold than did the hospitals served by other data vendors (see app. III, table 8).

Passing the 80 Percent Threshold Is Statistically Uncertain for One-Fourth to One-Third of Hospitals

While the baseline level of data accuracy achieved by hospitals in the aggregate was well above the 80 percent threshold, for approximately one- fourth to one-third of hospitals the determination that a particular hospital met the 80 percent threshold was statistically uncertain. This uncertainty stems primarily from the small number of cases examined for accuracy from each hospital. Because CDAC’s reabstraction of the data is limited to five patient records per quarter, the greater sampling variability found in small samples leads to relatively large confidence intervals, reflecting low statistical precision, for the accuracy score of any specific hospital. Across all hospitals, the median difference between the upper and lower limits of the confidence interval was 14.0 percentage points using the APU- measure set for first-quarter discharges, dropping to 11.8 percentage points in the second quarter. For the expanded-measure set, the median confidence interval was 14.6 percentage points in the first quarter and 13.0 percentage points in the second.

The wide confidence intervals meant that for a substantial number of hospitals it was statistically uncertain whether a different sample of cases would have altered their result from passing the 80 percent threshold to failing, or vice versa. For most hospitals there was statistical certainty that their baseline accuracy score met CMS’s 80 percent accuracy threshold. However, other hospitals had confidence intervals for their accuracy scores where the upper limit was 80 or above and the lower limit was less than 80. Because the confidence interval around the accuracy score computed for each of these hospitals bracketed the accuracy threshold set by CMS, their results were statistically uncertain. Consequently, for these hospitals, the small number of cases that CMS examined was not sufficient to establish whether the hospital met the threshold level for data accuracy. One-third of all the hospitals that CMS assessed for accuracy fell into this uncertain category for first-quarter 2004 discharges using the APU-measure set. (See fig. 3.) This proportion declined to about one-fourth of the hospitals for the second quarter. When the expanded-measure set was used—as CMS has done when calculating its quarterly accuracy scores—the proportion of hospitals whose accuracy scores were statistically uncertain increased compared to the APU- measure set for both the first and the second quarter.

These confidence intervals would narrow if CMS drew on multiple quarters of data to bring more cases into the computation of the accuracy scores. CMS has stated its intention to base this accuracy assessment on four quarters of hospital quality data, but so far every accuracy score it has generated and reported to hospitals has been based on a single quarter of data. Moreover, its implementation of the fiscal year 2006 payment update called for using only one quarter of data, with the possibility of adding one more quarter of data for hospitals that failed to meet the accuracy threshold based on the single quarter of data.

No Data Were Available to Provide Baseline Assessment of Completeness of Hospital Quality Data

There were no data available from which to estimate a baseline level of completeness for the first two calendar quarters of data submitted for the APU program. In contrast to the system of quarterly reabstractions performed by CDAC to check the accuracy of quality data submitted by hospitals, CMS did not conduct any corresponding assessment of the extent to which all hospitals submitted data on all the cases, or a representative sample of such cases, that met CMS’s eligibility criteria for the first two calendar quarters of 2004.

The information that CMS did collect was not suitable for estimating the baseline level of data completeness. The Medicare claims data analysis conducted by CMS on the first calendar quarter of data submitted for the APU program was not designed to provide valid information on the magnitude of data incompleteness for each hospital, which is what is needed to estimate a baseline level of data completeness. Although CMS could identify instances where certain hospitals failed to provide quality data on all eligible cases, CMS’s analysis did not produce comparable information on data completeness for every hospital. As noted above, it lacked information on non-Medicare patients and could not adjust properly for the sample sizes that JCAHO-accredited hospitals would have drawn if they followed JCAHO’s sampling rules rather than CMS’s. The limitations in the CMS analysis would affect some hospitals more than others, depending on how many non-Medicare patients a hospital treated and whether it applied the JCAHO sampling rules. Consequently, had we used information from this analysis to estimate baseline data completeness, our results would have been distorted by the uneven impact of those factors on the information produced for different hospitals.

In addition, we found no data for assessing the baseline completeness of the quality data provided by hospitals submitting samples of their eligible cases to the clinical warehouse. For hospitals that submitted a sample, their quality data could be incomplete, even if they submitted the expected number of cases, if their samples were not selected in a way that ensured they were representative of all a hospital’s patients. If a hospital did not follow appropriate procedures to provide for random selection, the sample might not be representative and therefore could be incomplete. Because the available information from CMS focused on the number of cases submitted, and not on how they were selected, we could not address this aspect of data completeness.

Other Reporting Systems Use Various Methods to Ensure Data Accuracy and Completeness, Notably an Independent Audit

Other reporting systems that collect clinical performance data have adopted various methods to ensure data accuracy and completeness, and officials from these systems stressed the importance of including an independent audit in these activities. Most other reporting systems that conduct independent audits incorporate three methods as part of their process that CMS does not use in its independent audit. Specifically, these systems include an on-site visit, focus their audit on a selected number of facilities or reporting entities, and review a minimum of 50 patient medical records per reporting entity.

Other Reporting Systems Use Various Methods to Check Data

Other reporting systems that collect clinical performance data have adopted various methods to ensure data accuracy and completeness. To check data accuracy, all the other reporting systems we examined assess the data when they are submitted, typically using computers to detect missing or out-of-range data. (See app. II, tables 4 and 5.) In addition, all the other systems have developed standardized data collection processes and measures. When checking data completeness, all the other systems compare submitted data with data from another source, whether inside the facility, such as pharmacy or laboratory records, or outside the facility, such as state hospital discharge data or Medicare claims data. Officials reported that these analyses were done annually or had been done one time, and one said that additional studies were planned. Officials from these systems also cite various other methods to consider when ensuring data accuracy and completeness, including reviewing established measures annually, identifying a point person at each facility to provide consistency, establishing channels for ongoing communication, and providing training on a continuous basis.

Other Reporting Systems Conduct Independent Audits

Most other reporting system officials we interviewed conduct independent audits that include a comparison of submitted data to medical records. Most other reporting systems that conduct independent audits incorporate three methods as part of their process that CMS does not use in its independent audit. Specifically, they (1) include an on-site visit as part of their independent audit, (2) focus their audits on a selected number of facilities or reporting entities, and (3) review a minimum of 50 patient medical records per reporting entity during the auditing process. During an on-site visit, auditors are able to review patient medical records for accuracy and interview staff when additional information is needed. Auditors are also able to check the data submitted to their system against other data sources at the facilities, including physician notes, patient or resident rosters, billing records, laboratory records, and pharmacy records. In addition, because auditors from other reporting systems may not visit every facility, the systems use various methods to focus the auditing process when selecting which facilities to visit. These include auditing a percentage of all eligible facilities, auditing facilities that did particularly well or poorly, and auditing a subset of facilities each year. Furthermore, most of the other reporting systems that conduct independent audits review a minimum of 50 patient medical records per audited entity as part of their independent auditing process. When selecting which patient medical records to review, some systems take a random sample of the patient population, one system reviews all deaths at the selected facility, and another reviews all instances where the patient died from shock as a result of percutaneous coronary intervention.

Officials at other reporting systems we interviewed and an expert in the field stressed the importance of the independent audit. For example, an official from one of the other reporting systems said that audits conducted by an independent third party are “the best way” to ensure data accuracy and completeness. An official from another reporting system said that having someone independently check the data is “one of the most important things” that an organization can do to check data accuracy and completeness. Additionally, an expert we interviewed said that independent, external audits are “essential.” Though most of the other reporting systems employ an independent auditing process, officials from one system that has yet to implement such a process said their organization recognizes the importance of independently checking the data and is currently designing and implementing an independent auditing process.

Conclusions

Data collected for the APU program affect the payment received by hospitals from Medicare and are used to inform the public about hospital quality. For both these purposes, it is important that CMS is able to ensure that the data are reliable in terms of both accuracy and completeness.

CMS has put in place an ongoing process for assessing the accuracy of quality data submitted by hospitals, but the process has limitations. Although CMS checks the accuracy of data electronically as they are submitted and through an independent audit conducted by CDAC, the latter process is limited by the selection of only five cases per quarter per hospital, regardless of the hospital’s size. Most hospitals had high baseline accuracy scores that were statistically certain. However, for about one- fourth to one-third of all the hospitals that CMS assessed for the first two calendar quarters of 2004, CMS’s determination as to whether the hospital met its accuracy standard was statistically uncertain. This was due primarily to the small number of cases selected for an audit. Although CMS has stated its intention to look at more cases by pooling reabstraction results from more than one calendar quarter, all of the hospital accuracy reports that it has generated to date have been based on a single quarter of data. Officials from other reporting systems that collect clinical performance data told us that they also use an independent audit to check data accuracy, but generally sample a larger number of patient medical records, either by sampling a percentage of total cases submitted or by identifying a minimum number of cases in the sample. In addition, most other reporting systems focused their audits on a selected number of facilities.

In contrast to CMS’s establishment of an ongoing process for assessing data accuracy, the agency has not put in place an ongoing process to check the completeness of the data that hospitals submit. Because of the purposes for which these data may be used, there could be an incentive for hospitals to selectively report data on cases that score well on the quality measures. With no ongoing way to check completeness, CMS does not know whether or how often hospitals submit incomplete data. We believe this is a significant gap in oversight. The process used for the fiscal year 2005 annual payment update compared hospital submissions to Medicare claims data, but as CMS has noted, this did not provide a comparable assessment of each hospital’s data, even for Medicare patients alone. Moreover, in its comparison of hospital quality data submissions with Medicare claims for the fiscal year 2006 update, CMS identified more than 100 hospitals that had treated eligible patients in a given quarter but had not submitted data on a single case for that quarter to the clinical warehouse. Yet CMS has not asked hospitals to certify that the data they have submitted constitute all, or a representative sample, of the eligible patient population. The various methods used by other reporting systems to check the completeness of data illustrate the variety of approaches that are available. These include conducting on-site visits as part of their independent audit, comparing data submissions to data from another source maintained by the facility or external to it, and performing such checks annually or planned at specified intervals.

Given CMS’s plans to continue public reporting efforts after the APU program ends, we believe that processes for checking the reliability of data should continue to be refined in order for the individuals and organizations that use the data to have confidence in the information.

Recommendations for Executive Action

In order for CMS to help ensure the reliability of the quality data it uses to produce information on hospital performance, we recommend that the CMS Administrator undertake the following three actions: focusing on the subset of hospitals for which it is statistically uncertain if they met CMS’s accuracy threshold in one or more previous quarters, increase the number of patient records reabstracted by CDAC in a subsequent quarter so that the proportion of hospitals with statistically uncertain results is reduced; require hospitals to certify that they took steps to ensure that they submitted data on all eligible patients, or a representative sample thereof; and assess the level of incomplete data submitted by hospitals for the APU program to determine the magnitude of underreporting, if any, in order to refine how completeness assessments may be done in future reporting efforts.

Agency Comments

In commenting on a draft of this report, CMS stated it appreciated our analysis and recommendations. (CMS’s comments appear in app. IV.) The agency noted that the APU program led to a dramatic increase in the number of hospitals that submitted data on the designated 10 quality measures, resulting in public reporting of quality data for about 3,600 hospitals on the agency’s Web site. In addition, CMS described the steps it had taken to ensure the accuracy and completeness of the quality data submitted by hospitals for the APU program. It said that the methods it had used were sound, but it agreed that the quality and completeness of the data must be improved.

With respect to reducing the statistical uncertainty of its assessments of the accuracy of hospital quality data submissions, CMS agreed that the quarterly accuracy assessments based on five patient charts can have considerable sampling error and stated that it would improve the stability of its accuracy assessments by using data from four calendar quarters when it assessed hospital eligibility for the fiscal year 2007 annual payment update. CMS stated a concern with having sufficient time within the current data submission schedule to increase the number of patient records reabstracted. However, we recommended in the draft report that hospitals with statistically uncertain results in one or more previous quarters have an increased number of records reabstracted. The assessment of statistical uncertainty for a hospital and the reabstraction of additional records do not need to occur within the same quarter. We have modified slightly the wording of the recommendation to clarify the intended timing of these additional reabstractions.

With respect to ensuring the completeness of quality data submitted by hospitals, CMS agreed that it needs to improve its methods. CMS noted that its comparison of hospital data quality submissions to the claims filed by those hospitals to be paid for treating Medicare beneficiaries uncovered numerous discrepancies. The agency agreed with our recommendation to require hospitals to formally attest to the completeness of the quality data that they submit quarterly. In addition, CMS stated that it would also require each hospital to report the total number of Medicare and non- Medicare patients who were eligible for quality assessment under the APU program.

In terms of assessing the level of incomplete data for the APU program, CMS said it had a process in place to accomplish this, but as we stated in the draft report, CMS’s process did not cover all patients and all hospitals because it lacked information on non-Medicare patients even though hospitals were required to submit data on both Medicare and non- Medicare patients. Additionally, the tests that CMS applied could detect incomplete data for only a limited subset of hospitals, in contrast to its assessment of data accuracy which covered all hospitals that submitted data on six or more cases in a quarter. CMS acknowledged it could assess completeness only for Medicare patients, but said that by requiring hospitals to report an aggregate count of all eligible patients, it would henceforth have the data needed to assess the completeness of both Medicare and non-Medicare quality data submissions. The agency stated it will use these data to provide quarterly feedback to hospitals about the accuracy and completeness of their data submissions, and require them to explain discrepancies between the data they have submitted for the APU program and the aggregate count of eligible patients they have reported. CMS has not said that it will determine the magnitude of underreporting for the program as a whole, as we recommended. Additionally, by relying on the hospitals themselves to supply data on the number of non-Medicare patients, CMS’s proposed approach lacks an independent verification of the completeness of submitted data. This contrasts with the practice of most of the other reporting systems we contacted, as well as experts in the field, who generally underscored the importance of independently checking both the accuracy and the completeness of the quality data.

As arranged with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies of this report to the Administrator of CMS and other interested parties. We will also make copies available to others on request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-7101 or BascettaC@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Scope and Methodology

To determine the processes used by the Centers for Medicare & Medicaid Services (CMS) to ensure the accuracy and completeness of data submitted by hospitals for the Annual Payment Update program (APU program), we interviewed both CMS officials and staff at DynKePRO— which operates the Clinical Data Abstraction Center (CDAC)—and the Iowa Foundation for Medical Care (IFMC), two contractors that perform data collection and data quality monitoring tasks for the APU program. In addition, we reviewed documentation on the program available publicly on the Quality Net Exchange Web site and the Web sites of several quality improvement organizations (QIO)—contractors to CMS that provide technical assistance to hospitals on the APU program—as well as documents on the APU program provided to us at our request by CMS. We also obtained access to CMS’s intranet system and searched for relevant memorandums and other documents regarding CMS’s policies and requirements for hospitals that participated in the APU program. To gain insights from other groups involved in the APU program, we interviewed officials from two or more QIOs, state hospital associations, and hospital data vendors that submitted data to the IFMC-operated database for their hospital clients.

Our assessment of the baseline accuracy of the initial APU program data depended on the availability of suitable information from CMS. We examined CMS’s reabstraction process to determine if the CDAC assessments of data accuracy would be appropriate for that purpose. Reabstraction is the re-collection of clinical data for the purpose of assessing the accuracy of data abstractions performed by hospitals. In the APU program, CDAC compares data reported by the hospitals to those it has independently obtained from the same medical records. CDAC has instituted a range of procedures, including training of its abstractors and continuous monitoring of interrater reliability, intended to ensure that its abstractors understand and follow its detailed guidance for arriving at abstraction determinations that are correct in terms of CMS’s data specifications. We interviewed CDAC staff and observed the implementation of these procedures during a site visit at the CDAC facility. On the basis of this information we concluded that it would be appropriate for us to use the results of the CDAC reabstractions to estimate baseline data accuracy for the APU program.

We obtained the results of the reabstractions that CDAC had conducted on samples of the patients for whom hospitals had submitted data from the first two quarters of 2004. These two quarters were the first two data submissions made by hospitals under the APU program and the most recent available when we conducted these analyses. They constituted 20,465 patient records for the first quarter and 20,259 for the second. These files showed, for each data element that CMS used in assessing abstraction accuracy, the correct entry as determined by the CDAC abstractors and whether this matched the value that the hospital had reported. We applied CMS’s algorithms for computing hospital scores on the expanded-measure set in order to determine the extent of missing or invalid data. We found that approximately 2 to 3 percent of patient records could not be scored on any given APU measure due to missing data. We excluded from the analysis records from critical access hospitals and acute care hospitals in Maryland and Puerto Rico (which are paid under different payment systems than other acute care hospitals and therefore are not subject to a reduced annual payment update under the APU program) and a small number of records not related to the three medical conditions covered by the APU program.

Next we applied the scoring rules developed by CMS to assess the accuracy of hospital abstractions. We calculated the accuracy score for each hospital in each quarter, using the data elements needed for the APU- measure set and, separately, for the expanded-measure set. Accuracy scores for the expanded-measure set are based on all the measures for which a hospital submitted data, which could range from the APU measures alone to a maximum of 17—the 10 measures in the APU- measure set plus the 7 additional measures adopted by the Hospital Quality Alliance for hospital discharges through the second calendar quarter of 2004. These scores represented the proportion of data elements where CDAC and the hospital agreed, summing across all the assessed data elements for the five sampled cases. We then calculated the distribution of those scores, and the proportion of hospitals that met or exceeded the 80 accuracy threshold that CMS had set. Next we calculated the confidence interval for each of those accuracy scores, using the formula that CMS had selected for that purpose. However, whereas CMS applied a one-tailed test—passing any hospital that had a confidence interval whose upper bound reached 80 or above—we applied a two-tailed test to assess the statistical uncertainty attached to both passing and failing the threshold. The one-tailed test that CMS applied prevented hospitals from losing their full annual payment update on the basis of their accuracy score if there was less than a 95 percent probability that a score below 80 would have remained below 80 in another sample. This meant that hospitals with large confidence intervals could have accuracy scores well below 80 and still pass the CMS accuracy requirement. Our analysis focused instead on assessing the level of statistical certainty for all the accuracy scores, both above and below the 80 percent threshold. We sought to identify passing as well as failing scores that could have changed with another sample. To do so, we applied a two-tailed test and observed whether a hospital’s confidence interval bracketed the 80 percent threshold.

To provide descriptive information about variation in the accuracy scores obtained by hospitals in different situations, we collected additional information about the hospitals from other sources. From the Medicare Provider of Services file we obtained the Social Security Administration metropolitan statistical area code (referred to as the SSA MSA code) and Social Security Administration metropolitan statistical area size code (referred to as the SSA MSA size code) to distinguish between urban and rural hospitals. We also obtained from that source the total number of Medicare-certified beds in order to categorize hospitals by size. To compare the accuracy scores of hospitals that employed different data vendors, we obtained from IFMC the identification codes (but not the names) of the various data vendors certified by the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) that had submitted to the clinical warehouse data for the APU program on behalf of hospitals they served. Those codes were also available in the case tracking information for the patient records in the CDAC database. We then identified for each CDAC reabstraction whether the case had originally been submitted by a JCAHO-certified data vendor, and if so, which one. These data were aggregated to generate accuracy scores for each hospital that consistently submitted its quality data through one data vendor in a given quarter. This allowed us to determine the proportion of hospitals served by each JCAHO data vendor that met CMS’s 80 percent accuracy threshold. We also calculated the proportion of hospitals that submitted their own quality data to CMS (identified in the CDAC case tracking information by the hospital’s Medicare provider ID number) that met the accuracy threshold. Although this analysis was limited to data vendors that were JCAHO-certified, those vendors collectively submitted data to the clinical warehouse for 78 to 79 percent of the hospitals we analyzed in the two baseline quarters. Another 13 to 14 percent of hospitals directly submitted their own data, and we do not have information on how the remaining hospitals submitted data to the clinical warehouse.

As was the case for our baseline accuracy assessment, our assessment of the baseline completeness of the data submitted for the APU program depended on the availability of suitable data from CMS. Specifically, we considered using CMS’s estimates of minimum expected cases derived from Medicare claims data to arrive at estimates of baseline completeness. The CMS officials we spoke with noted that there were numerous reasons why the two data sources—quality data submissions for the APU program and cases billed to Medicare—would be expected to diverge, apart from any underreporting of quality data by hospitals. The claims data were limited to Medicare fee-for-service patients, whereas the hospitals were obliged to submit quality data on all patients over 18 years of age (over 28 days old for most pneumonia measures), including patients belonging to Medicare health maintenance organizations. In addition, hospitals with large numbers of cases could draw samples for the quality data, but would bill for all patients. In making adjustments to its number of “expected cases” to take account of sampling, CMS found that it could not reliably identify the hospitals that should have followed the JCAHO sampling rules, which would result in larger-sized samples. Therefore, in calculating the number of cases it expected hospitals to have submitted to the clinical warehouse, CMS applied to all hospitals across the board the expectation of smaller samples based on rules that pertained to hospitals not accredited by JCAHO. Finally, the Medicare data used for the comparison was an average volume recorded over the previous 2 years, not claims filed for the quarter to which the quality data applied.

We found that these limitations precluded our using information from CMS’s Medicare claims analysis to assess the baseline completeness of the data submitted by hospitals for the APU program. CMS’s comparison of hospital quality data submissions to the clinical warehouse to its estimated number of “expected cases” might have served CMS’s purposes, by identifying at least some instances of significant discrepancy between the number of cases for which quality data were submitted and claims filed. However, we determined that it would not provide a reasonable estimate of the magnitude of data completeness for all hospitals. Because the limitations in the CMS analysis would affect some hospitals more than others, depending on how many non-Medicare patients a hospital treated and whether it applied the JCAHO sampling rules, we concluded that using information from this analysis to estimate baseline data completeness would lead to results that were distorted by the uneven impact of those factors on the information produced for different hospitals.

To obtain information on other processes that could be used to check data accuracy and completeness, we interviewed officials from organizations that administer reporting systems that collect clinical performance data. To select these organizations, we took several steps. We reviewed reports on reporting systems, including two issued by QIOs: IPRO’s 2003 Review of Hospital Quality Reports and Delmarva Foundation’s The State-of-the- Art of Online Hospital Public Reporting: A Review of Forty-Seven Websites. We solicited input from the authors of each report and interviewed academic researchers who have researched methods of assessing the reliability of performance data. We used on-line resources to obtain information on federal- and state-administered surveillance efforts. Our selection criteria focused on systems that collected clinical data, as opposed to administrative or claims data, and that were mentioned most often in the reports and interviews cited above. To ensure variation, we selected a mix of systems, including those run by public and private organizations, those receiving data from hospitals and those receiving data from other types of providers, and those collecting data across a range of medical conditions and those collecting data on specific medical conditions. Using a structured protocol, we interviewed officials from the following organizations: JCAHO, National Committee for Quality Assurance, Society of Thoracic Surgeons, California Office of Statewide Health Planning and Development, New York State Department of Health, CMS (the units responsible for monitoring nursing home care regarding the Data Assessment and Verification Project (DAVE) contract), and the American College of Cardiology. Each organization reviewed and confirmed the accuracy of the information presented in appendix II.

Our analysis is based on the quality measures established for the APU program and the information available as of September 2005 on the accuracy and completeness of data submitted by hospitals for that program. We did not evaluate the appropriateness of these quality measures relative to others that could have been selected. Nor did we examine the actual performance by hospitals on the measures (e.g., how often they provide a particular service or treatment). Our analysis of the baseline level of accuracy and completeness of data submitted for the APU program is based on the procedures developed by CMS to validate the data submitted. We have not independently compared the data submitted by hospitals to the original patient clinical records.

We conducted our work from November 2004 through January 2006 in accordance with generally accepted government auditing standards.

Appendix II: Other Reporting Systems

Data Assessment and Verification Project (DAVE)

Joint Commission on Accreditation of Healthcare Organizations (JCAHO)

National Committee for Quality Assurance (NCQA)

Society of Thoracic Surgeons (STS)

Facilities with at least one catheterization laboratory (includes in- hospital, freestanding, and/or mobile catheterization laboratories)

Hospitals that perform cardiac surgery and/or percutaneous coronary intervention (PCI)

Cardiac – coronary artery bypass grafting (CABG)

Cardiac - CABG, aortic and mitral valve 16 years Under Section 501(b) of the Medicare Prescription Drug, Improvement, and Modernization Act of 2003, hospitals shall submit data for a set of indicators established by the Department of Health and Human Services (HHS) as of November 1, 2003, related to the quality of inpatient care. Section 501 (b) also provides that any hospital that does not submit data on the 10 quality measures specified by the Secretary of Health and Human Services will have its annual payment update reduced by 0.4 percentage points for each fiscal year from 2005 through 2007.

Some states and insurance companies have started to require hospital participation.

The number of hospitals that submitted data to receive their annual payment update for fiscal year 2005.

The number of facilities enrolled in ACC’s National Cardiovascular Data Registry® as of July 13, 2005.

Data Assessment and Verification Project (DAVE)

Joint Commission on Accreditation of Healthcare Organizations (JCAHO)

National Committee for Quality Assurance (NCQA)

Society of Thoracic Surgeons (STS)

JCAHO performs independent audits of data vendors.

STS is planning to incorporate an independent audit into its system. STS officials plan on including an on-side audit and medical record review as part of their audit system.

STS plans to review a minimum of 30 records as a part of its independent auditing process.

STS plans on visiting 24 facilities per year as a part of its independent auditing process.

Data Assessment and Verification Project (DAVE)

Joint Commission on Accreditation of Healthcare Organizations (JCAHO)

National Committee for Quality Assurance (NCQA)

Society of Thoracic Surgeons (STS)

Under concurrent review, auditors assess data as they are being collected.

JCAHO performs independent audits of data vendors.

STS is planning to incorporate an independent audit into its system. STS officials plan on including an on-side audit as part of their audit system.

CMS conducted two separate one-time studies that compared Medicare claims data to submitted data.

Data completeness reviews are conducted annually for randomly selected sites as part of the on-site audit process and quarterly for data submissions.

Appendix III: Data Tables on Hospital Accuracy Scores

Rural hospitals and smaller hospitals generally received accuracy scores that differed minimally from those of urban hospitals and larger hospitals. (See tables 6 and 7.) To the extent there are small differences across categories, they do not show a consistent pattern based on geographic location or size.

Accuracy scores among hospitals whose data were submitted to CMS by different JCAHO-certified vendors varied more, especially in the percentage of the hospitals that failed to meet the 80 percent threshold. (See table 8.) Collectively, these data vendors submitted data to the clinical warehouse for approximately 78 to 79 percent of hospitals affected by the APU program in the two baseline quarters, while another 13 to 14 percent of hospitals directly submitted their own data. For large data vendors (serving more than 100 hospitals), medium vendors (serving between 20 and 100 hospitals), and small vendors (serving fewer than 20 hospitals), there was marked variation within each size grouping in the proportion of the vendors’ hospitals that did not meet the accuracy threshold. Such variation could reflect differences in the hospitals served by different vendors as well as differences in the services provided by those vendors.

Rank ordering hospitals by the breadth of the confidence intervals around their accuracy scores, from the narrowest to the widest intervals, shows the large variation that we found across both quarters and measure sets. Hospitals with the narrowest confidence intervals, shown in table 9 as the 10th percentile, had a range of no more than 6 percentage points between the lower and upper limits of their confidence interval. That meant that their accuracy scores from one sample to the next were likely to vary by no more than plus or minus 3 percentage points from the accuracy score obtained in the sample drawn by CMS. By contrast, hospitals with the widest confidence intervals, shown in table 9 as the 90th percentile, exceeded 36 percentage points from the lower limit to the upper limit of their confidence interval. The accuracy scores for these hospitals would likely vary from one sample to the next by 18 percentage points or more, up or down, relative to the accuracy score derived from the CMS sample. For hospitals whose confidence interval included the 80 percent threshold, it was statistically uncertain whether a different sample of cases would have altered their result from passing the 80 percent threshold to failing, or vice versa.

One-third to one-fourth of hospitals had statistically uncertain results because their confidence interval extended both above and below the 80 percent threshold. Some of these hospitals had accuracy scores of 80 or above and some had scores of less than 80. Table 10 separates these hospitals into (1) those that had accuracy scores equal to 80 or above and were statistically uncertain and (2) those that had accuracy scores below 80 and were statistically uncertain. The table shows that most of the statistical uncertainty involved hospitals that passed CMS’s accuracy threshold, but if a different sample of cases had been reabstracted by CDAC, there was a substantial possibility that they would not have passed.

Appendix IV: Comments from the Centers for Medicare & Medicaid Services

Appendix V: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the contact named above, Linda T. Kohn, Assistant Director; Ba Lin; Nkeruka Okonmah; Eric A. Peterson; Roseanne Price; and Jessica C. Smith made key contributions to this report.