Medicare Payment: CMS Methodology Adequate to Estimate National Error Rate
Released: 2006-03-24T00:00:00

Letter

Background

In fiscal year 2003, CMS assumed responsibility for estimating the national Medicare error rate, a responsibility that had previously been held by HHS OIG. OIG began estimating the national Medicare error rate in fiscal year 1996, and continued doing so for each subsequent fiscal year through 2002. The transfer of responsibilities for estimating the national Medicare error rate to CMS coincided with the implementation of the Improper Payments Information Act of 2002 (IPIA). The IPIA requires federal agencies to estimate and report annually on the extent of erroneous payments in their programs and activities. The IPIA defines an improper payment as any payment that should not have been made or that was made in an incorrect amount, including both under- and overpayments. All agencies that identify a program as susceptible to significant improper payments, defined by guidance from the Office of Management and Budget (OMB) in 2003 as exceeding both 2.5 percent of total program payments and $10 million, are required to annually report to Congress and the President an estimate of improper payments and report on corrective actions.

In addition to estimating the national Medicare error rate for purposes of compliance with the IPIA, CMS also began producing contractor-specific error rate estimates beginning in fiscal year 2003 to identify the underlying causes of errors and to adjust action plans for carriers, DMERCs, FIs, and QIOs. To produce these contractor-specific error rate estimates for fiscal year 2004, CMS sampled approximately 160,000 claims. The contractor- specific error rate information was then aggregated by the four contractor types (carrier, DMERC, FI, and QIO), which were ultimately combined to estimate the national Medicare error rate. Under the methodology previously used by OIG to estimate the national Medicare error rate, 6,000 claims were sampled. While the sample size used by OIG was sufficient to estimate the national Medicare error rate, it was not sufficient to reliably estimate the contractor-specific error rates. Additionally, the increased sample size improved precision of the national Medicare error rate estimate.

CMS Programs to Monitor the Payment Accuracy of Medicare FFS Claims

The objective of the CERT Program and the HPMP is to measure the degree to which CMS, through its contractors, is accurately paying claims. Through the CERT Program, CMS monitors the accuracy of Medicare FFS claims that are paid by carriers, DMERCs, and FIs. In fiscal year 2004, the Medicare error rates by contractor type as estimated through the CERT Program were 10.7 percent for the carrier contractor type, 11.1 percent for the DMERC contractor type, and 15.8 percent for the FI contractor type. (See table 1.)

Through the HPMP, CMS monitors the accuracy of paid Medicare FFS claims for acute care inpatient hospital stays—generally those that are covered under the prospective payment system (PPS). For fiscal year 2004, the Medicare error rate for the QIO contractor type, as estimated through the HPMP, was 3.6 percent. (See table 1.)

CERT Program

To estimate contractor-specific Medicare FFS error rates for the CERT Program, CMS reviews a sample of claims from each of the applicable contractors, which included 25 carriers, 4 DMERCs, and 31 FIs for the fiscal year 2004 error rates. These error rates are then aggregated by contractor type. (See fig. 1.) For fiscal year 2004, CMS contracted with AdvanceMed to administer the CERT Program. AdvanceMed sampled approximately 120,000 claims submitted from January 1, 2003, through December 31, 2003, to estimate the fiscal year 2004 contractor-specific and contractor-type error rates for the CERT Program.

For each of the approximately 120,000 sampled claims, AdvanceMed requested the medical records from the provider that rendered the service or from the contractor that processed the related claim, if the contractor previously performed a medical review on the claim. If a provider did not respond to the initial request for medical records after 19 days, AdvanceMed initiated a series of follow-up procedures in an attempt to obtain the information. The follow-up procedures with nonresponding providers for fiscal year 2004 included three written letters and three contacts by telephone. Additionally, in fiscal year 2004, OIG followed up directly with nonresponders on claims over a certain dollar amount. If medical records were not received within 55 days of the initial request, the entire amount of the claim was classified by AdvanceMed as an overpayment error.

When medical records were received from the provider or from the contractor, CERT medical review staff reviewed the claim (which billed for the services provided) and the supporting medical records (which detailed the diagnosis and services provided) to assess whether the claim followed Medicare’s payment rules and national and local coverage decisions. Claims that did not follow these rules were classified by AdvanceMed as being in error. Providers whose claims were reviewed were allowed to appeal these claims, and if the error determination for a claim was overturned through the appeals process, AdvanceMed adjusted the error rate accordingly. For the fiscal year 2004 error rate, AdvanceMed notified individual carriers, DMERCs, and FIs of their respective payment errors.

For the HPMP, CMS analyzes a sample of claims across QIOs to estimate Medicare error rates by state, because QIOs are organizations with state- based service areas. CMS estimated the QIO contractor-type error rate by aggregating the QIO error rate estimates for each of the 50 states, the District of Columbia, and Puerto Rico. (See fig. 2.) Through the HPMP, CMS sampled approximately 40,000 claims for acute care inpatient hospital discharges that occurred from July 1, 2002, through June 30, 2003, to estimate the fiscal year 2004 state-specific and contractor-type error rates for QIOs.

For fiscal year 2004, CMS contracted with two organizations known as Clinical Data Abstraction Centers (CDAC)—AdvanceMed and DynKePRO—that were responsible for requesting medical records from providers for each of the approximately 40,000 sampled claims. Each CDAC was responsible for reviewing the sampled claims, which were assigned on the basis of the geographic location where the discharge occurred. Upon receipt of the medical records, CDAC admission necessity reviewers screened the related claims for the appropriateness of the hospitalization and, with the exception of claims from Maryland, coding specialists independently recoded diagnosis-related groups (DRG) based on the records submitted. Because Maryland does not use DRG coding, nonphysician reviewers screened claims from Maryland to determine whether the length of the acute care inpatient hospital stay was appropriate.24, 25 Claims that failed the screening process, including those where the admission was determined to be unnecessary or where an inappropriate DRG code was used, were forwarded to the QIO responsible for the state where the discharge occurred for further review. Records not received by the CDACs within 30 days of the request for information were “canceled” and referred to the QIO to be processed as overpayment errors caused by nonresponse. The QIO referred these claims to the FI responsible for paying the claim for the necessary payment adjustments.

At the QIO, claims forwarded from the CDACs underwent further review, primarily medical necessity admission reviews and DRG validations. Determinations of error were made by QIO physician reviewers. Providers whose claims were reviewed were given the opportunity to provide comments or discuss the case and pursue additional review, which could result in an appeal to an administrative law judge. After the matter was resolved, resulting in a determination that a provider was either underpaid or overpaid, the QIO forwarded the claim to the FI for payment adjustment.

Maryland is the only state that does not use the PPS system for acute care inpatient hospitals. Maryland instead has an alternative payment system, known as an all-payer system, in which the state decides each hospital’s level of reimbursement and requires that all payers be charged the same rate for the same service. Medicare and Medicaid pay the state-approved rates.

Claims from Maryland with length of stay errors are considered medically unnecessary services. Length of stay reviews identified cases of potential delayed discharge. For example, the patient was medically stable, and continued hospitalization was unnecessary.

Estimation of the National Medicare FFS Error Rate

CMS estimated the national Medicare FFS error rate by combining the three contractor-type error rates (carrier, DMERC, and FI) from the CERT Program and the one contractor-type error rate (QIO) from the HPMP. (See fig. 3.)

Medicare FFS claims that were paid in error as identified by the CERT Program and the HPMP for the fiscal year 2004 error rates were sorted into one of five categories of error: Insufficient documentation: Provider did not submit sufficient documentation to support that the services billed were actually provided. Nonresponse: Provider did not submit any documentation to support that the services billed were actually provided.

Medically unnecessary services: Provider submitted sufficient documentation, but the services that were billed were deemed not medically necessary or the setting or level of care was deemed inappropriate. Incorrect coding: Provider submitted documentation that supported a different billing code that was associated with a lower or higher payment than that submitted for the services billed.

Other: Provider submitted documentation, but the services billed did not comply with Medicare’s benefit or other billing requirements.

See table 2 for the national Medicare FFS error rate by category of error for fiscal year 2004.

As reported in CMS’s fiscal year 2004 Medicare error rate report, the agency planned to use the error rates to help determine the underlying reasons for claim errors, such as incorrect coding or nonresponse, and implement corrective action plans for carriers, DMERCs, FIs, and QIOs. Draft statements of work, dated February and April 2005, for carriers, DMERCs, and FIs set goals for contractors to achieve a paid claims error rate of less than a certain percentage, to be determined by CMS. According to the standards for minimum performance on QIO statements of work that ended in 2005 for some QIOs and 2006 for other QIOs, QIOs are evaluated on 12 tasks, one of which is the HPMP. QIOs have to meet the performance criteria standards on 10 tasks set forth by CMS to be eligible for a noncompetitive contract renewal.

CMS’s use of the error rates is being done in the context of the agency’s current effort to significantly reform its contracting efforts for the payment of Medicare claims. By July 2009, CMS plans to reduce the total number of contractors responsible for paying Medicare claims to 23 total contractors, which the agency refers to as Medicare administrative contractors (MAC). CMS also plans to institute performance incentives in the new contracts, which will be based on a number of different factors, including the Medicare error rates. According to CMS’s report to Congress on Medicare contracting reform, CMS believes that the consolidation of Medicare contractors and the integration of processing for Medicare claims will lead to a reduced Medicare error rate.

CMS Methodology Adequate for Estimating the Error Rates in the CERT Program

The methodology used by CMS in the CERT Program to estimate error rates by contractor type (carrier, DMERC, and FI) in fiscal year 2004 was adequate. We found that the sample size and the use of systematic sampling with a random start were adequate to reliably estimate the Medicare error rates by contractor type. The CERT Program also had adequate processes in place to collect medical records and to accurately identify and categorize payment errors. The statistical methods that CMS used to estimate the contractor-type error rates were valid.

Sampling Methods

The sample size that CMS used in the CERT Program, approximately 120,000 claims, was sufficiently large to produce reliable estimates of the fiscal year 2004 Medicare error rates by contractor type (carrier, DMERC, and FI). CMS selected 167 claims each month on a daily basis from each of the 60 contractors, including 25 carriers, 4 DMERCs, and 31 FIs. This sample generated error rate estimates by contractor type within acceptable statistical standards, such as relative precision of no greater than 15 percent. Specifically, the error rate for the carrier contractor type was 10.7 percent with a relative precision of 3.7 percent, the error rate for the DMERC contractor type was 11.1 percent with a relative precision of 13.5 percent, and the error rate for the FI contractor type was 15.7 percent with a relative precision of 4.5 percent.

Further, we found that the sampling methods were adequate because CMS used a systematic sample with a random start. Sampling methods that employ a random start are designed to ensure that the sample selected is representative of the population from which it is drawn. We reviewed CERT Program documentation, which described the use of a systematic sample with a random start. The OIG contractor reviewed the computer program used for the CERT Program sample selection and verified that the claims were selected according to the documentation. CMS officials told us that the CERT Program conducts tests to compare the sampled claims to the population of claims. For example, CMS compared the percentage of claims sampled in each category of Medicare-covered service to the percentage of claims in the population by category of Medicare-covered service. CMS provided us with an example of this test for one contractor’s claims from January 2003 through June 2003.

While the relative precision of the fiscal year 2004 error rate estimates by contractor type for the CERT Program was within acceptable statistical standards of no greater than 15 percent, the relative precision of half of the contractor-specific error rate estimates was not. (See app. II for contractor-specific error rate information, including the estimates and corresponding relative precision, for carriers, DMERCs, and FIs.)

Thirty of the 60 contractor-specific error rates had relative precision that were not within acceptable statistical standards. Additionally, the relative precision of the contractor-specific error rates showed wide variation within each contractor type. Relative precision among carriers ranged from 8.9 percent to 17.0 percent; among DMERCs, relative precision ranged from 12.3 percent to 20.7 percent; and among FIs, relative precision ranged from 10.3 percent to 42.5 percent. As demonstrated by the range in relative precision among FIs, for example, the error rate estimate for one FI was nearly four times more reliable than the error rate estimate for another.

The variation in relative precision among the contractor-specific error rate estimates was due, in part, to the sampling method CMS used for the CERT Program. CMS took an equal sample size from each contractor despite the fact that individual contractors accounted for varied amounts of Medicare claim volumes and total payments. For example, the claim volume for carriers in 2003 ranged from a minimum of 5.3 million claims to a maximum of 206 million claims; total payments for carriers in 2003 ranged from a minimum of $168 million to about $6.7 billion.

CMS officials told us that they plan to reallocate the CERT Program sample at the contractor level by increasing the sample size for those contractors that are not reaching CMS’s targeted precision and by decreasing the sample size for those contractors that are reaching targeted precision and achieving low error rates. In September 2005, CMS officials reported that this change to the methodology is expected to be implemented for the fiscal year 2007 error rate estimation, which will be based on claims processed in parts of 2006 and 2007. We support CMS’s planned changes to its sampling methodology. We believe that reallocation of the sample as planned by CMS will improve the relative precision of these estimates. If future samples were based on the volume of claims or total payments of each contractor and the relative precision of the contractor-specific error rate rather than on the current basis of an equal allocation across contractors, relative precision would likely be improved for the contractor-specific error rates of those targeted contractors that were allocated a larger sample. This is because relative precision improves with increased sample size. There would also likely be decreased variation in relative precision across all contractor-specific error rates. These results could be achieved without increasing the overall sample size for the CERT Program.

Medical Record Collection Process

Based on our review of oversight work conducted by OIG, we found that the process CMS used to collect medical records from providers for the CERT Program was adequate. Staff of AdvanceMed, the CMS contractor responsible for administering the CERT Program, were responsible for requesting medical records for each of the approximately 120,000 sampled claims used to estimate the fiscal year 2004 error rates. According to an OIG review of CMS’s corrective actions to improve nonresponse in the CERT Program for fiscal year 2004, AdvanceMed conducted a timely and systematic follow-up with providers that did not respond to initial requests for medical records. For the medical records collection process for the fiscal year 2004 error rates, CMS implemented corrective actions in the CERT Program to address the factors associated with the high rate of nonresponse experienced during the medical records collection process for the prior fiscal year. According to the CMS fiscal year 2003 error rate report, for example, the agency found that some nonresponse in fiscal year 2003 was due to providers’ lack of familiarity with AdvanceMed. In previous years when OIG had responsibility for estimating the Medicare error rate, OIG requested medical records directly from providers; providers were familiar with OIG and understood the importance of complying with the requests. However, when the responsibility for estimating the Medicare error rate was transferred to CMS, many providers were unfamiliar with AdvanceMed and may have been reluctant to submit medical records to an unknown company. Another factor that caused provider nonresponse in fiscal year 2003, according to the CMS report, was providers’ confusion about the submission of medical records within the constraints of the privacy regulations issued by HHS under the Health Insurance Portability and Accountability Act of 1996, which limit the use and release of individually identifiable health information. According to the CMS report, CMS found that providers were sometimes unaware that sending medical records to the CERT Program contractor was permissible under the regulations. As reported in the OIG review cited previously, CMS implemented corrective actions that increased provider compliance with medical record requests in fiscal year 2004. According to the OIG report, CMS conducted educational efforts to clarify the role of AdvanceMed. Additionally, OIG further reported that CMS took action to address providers’ concerns about compliance with the privacy regulations by revising its request letters to providers to highlight AdvanceMed’s authorization, acting on CMS’s behalf, to obtain medical records as requested. OIG told us that CMS instructed carriers, DMERCs, and FIs to refer certain claims for nonresponding providers to OIG for follow-up.

These improvements in the process used to collect medical records in the CERT Program helped reduce nonresponse. According to information provided to us by CMS, the percentage of error caused by nonresponse in the CERT Program decreased from 61 percent for fiscal year 2003 to 34 percent in fiscal year 2004. According to CMS’s fiscal year 2005 error rate report, the agency continued several corrective actions to address nonresponse for sampled claims for the fiscal year 2005 error rates. Further, beginning with claims sampled to estimate the fiscal year 2006 Medicare error rates, CMS transferred the medical record collection duties to a second contractor, Lifecare Management Partners, which the agency refers to as the CERT Program documentation contractor. CMS officials told us that the CERT Program documentation contractor is automating the medical record collection process and eliminating paper copies of documentation.

Identification and Categorization of Payment Errors

Based on our review of OIG’s fiscal year 2004 CERT Program evaluation, we concluded that the processes used in the CERT Program to identify and categorize payment errors for fiscal year 2004 were adequate because the medical record reviews were performed appropriately and the CERT Program staff conducting the reviews were adequately trained and qualified. Staff of the CERT Program contractor, AdvanceMed, reviewed the medical records to verify that claims were processed according to Medicare payment rules; if not, a claim was found to be in error and assigned to one of five categories of error (insufficient documentation, nonresponse, medically unnecessary, incorrect coding, or other). We reviewed work conducted by OIG that found AdvanceMed, the CMS contractor responsible for administering the CERT Program, had appropriate controls in place to ensure that the medical record reviews were performed in accordance with established CERT Program procedures. We also reviewed work by OIG, which examined the educational and training requirements for medical record reviewers as established in the CERT Program and assessed selected training files for selected medical record reviewers. OIG officials told us that they found these selected CERT Program medical record reviewers to be adequately trained and qualified.

OIG found that AdvanceMed did not complete all required quality assurance reviews within the designated time frame. CMS told OIG that it planned to reduce AdvanceMed’s workload. AdvanceMed conducts quality assurance reviews on a sample of medically reviewed claims to validate the initial reviewer’s decision on whether a claim was paid in error. OIG found that for the fiscal year 2004 CERT Program, AdvanceMed completed 984 of the required 2,587 quality assurance reviews by the required date. To determine whether these quality assurance reviews ensured the reliability of the CERT Program claims review process, OIG randomly sampled 45 of the 2,587 claims selected for quality assurance reviews. Of these 45 claims, AdvanceMed had completed a quality assurance review on 5 claims. OIG reported that the results of the 5 quality assurance reviews confirmed the results of the initial medical record reviews. Further, OIG reported that AdvanceMed stated that a backlog of medical reviews prevented the completion of the required quality assurance reviews within the prescribed time frame. In response to the OIG report on the fiscal year 2004 CERT Program evaluation, CMS commented that with Lifecare Management Partners assuming responsibilities for medical record collection for the fiscal year 2006 Medicare error rate estimation, AdvanceMed’s workload would be reduced. As a result, CMS commented that this will free up the necessary resources for AdvanceMed to comply with the quality assurance requirements. Further, in its response to the OIG report, CMS commented that both AdvanceMed and Lifecare Management Partners are required to report to the agency on the results of the quality assurance activities conducted. According to OIG’s evaluation of the fiscal year 2005 CERT Program, OIG found that AdvanceMed completed all of the required quality assurance reviews.

Statistical Methods

We found that the statistical methods used to estimate the error rates and standard errors by contractor type (carrier, DMERC, and FI) for the CERT Program were adequate. Based on our review of the computer programming code that generated the error rate estimates and standard errors by the CERT Program subcontractor responsible for calculating the contractor-type error rates, The Lewin Group, we found that the statistical methods were based on standard statistical principles and were used appropriately. For each contractor type, the stratified combined ratio estimation method was used to calculate the error rate by taking the difference between the overpaid dollars and the underpaid dollars divided by the total dollars paid by Medicare for FFS claims of each contractor type. The payment errors from the sample were then extrapolated to the population for each contractor type to estimate total payment errors. Further, The Lewin Group used a standard statistical method to calculate the standard errors of each of the contractor-type error rates. This method is appropriate for obtaining the standard error of an estimate when the stratified combined ratio estimation method is used and is valid for large sample sizes, such as that used for the CERT Program.

CMS Methodology Adequate for Estimating the Error Rate in the HPMP

We found that the methodology used by CMS was adequate to produce a reliable estimate of the fiscal year 2004 Medicare error rate for the one contractor type (QIO) in the HPMP. We found the methodology adequate because the sample size was large enough to produce a reliable error rate estimate. Additionally, the sample was representative of the population. We found also that the methodology was adequate because the HPMP contractors responsible for collecting the medical records for the sampled claims, as well as for identifying and determining errors, had appropriate controls in place to ensure that established procedures were followed. Further, the statistical method that CMS used to calculate the contractor- type error rate was valid.

Sampling Methods

The sample size that CMS used for the HPMP, about 40,000 claims, was sufficiently large to produce a reliable estimate of the fiscal year 2004 error rate for the QIO contractor type. Using a systematic sample, CMS selected 62 discharge claims per month for the District of Columbia, Puerto Rico, and each state except Alaska. CMS selected 42 claims per month for Alaska. The QIO contractor-type error rate was 3.6 percent with a relative precision of 5.6 percent. The relative precision for the QIO contractor-type error rate estimate is within acceptable statistical standards (a relative precision of no greater than 15 percent).

For the QIO contractor-type error rate to be a reliable estimate, it was necessary that the sample of discharge claims from which the error rate was estimated be representative of the population from which it was drawn. CMS’s documentation stated that the HPMP used a systematic sample selection process with a random start, which is a generally accepted method of sampling that is designed to ensure that the sample drawn is representative of the population. Our review of the computer programming code that selected the sample, however, found that a random start was not used. To determine whether the HPMP sample was compromised by the lack of a random start and whether it represented the population from which it was drawn, we examined the OIG contractor’s comparison of the June 2003 sample to a re-created version of the June 2003 population file from which the sample was drawn. Based on our review, we found that the HPMP sample was representative of the population from which it was drawn in terms of average dollar amount per claim.

While relative precision of the fiscal year 2004 QIO contractor-type error rate estimate was within acceptable statistical standards, relative precision of most of the state-specific QIO error rate estimates was not. (See app. II for state-specific QIO error rate information, including the error rate estimates and corresponding relative precision.) Only three states’ error rate estimates—Kentucky, Massachusetts, and New Mexico— had relative precision of less than 15 percent. Additionally, there was wide variation in relative precision of the state-specific QIO error rate estimates. Relative precision of the state-specific QIO error rates ranged from 10.5 percent in Massachusetts to 83.3 percent in Mississippi. The differences in relative precision of these state-specific QIO error rate estimates indicate that the error rate estimate for the QIO that served Massachusetts was eight times more reliable than the error rate estimate for the QIO that served Mississippi. The variation in relative precision was due, in part, to the sampling methods used by CMS for the HPMP. CMS took an equal sample size for each state except Alaska, despite the fact that there was significant variation between states in the overall volume of discharge claims and total payments. The number of discharges per state varied from a low of 15,166 in Wyoming to a high of 825,845 in Florida. Similarly, total dollars paid for acute-care inpatient hospital stays varied from less than $100 million in Wyoming to a high of $7.5 billion in California.

Although in February 2006 a CMS official told us the agency has no plans to reallocate the HPMP sample, CMS could adopt a similar sampling strategy as it plans to do for the CERT Program. If future state samples were based on the volume of discharge claims or total payments per state and the relative precision of the state-specific QIO error rates, rather than on the current basis of an equal allocation per state, relative precision would likely be improved for the state-specific QIO error rates in those states that were allocated a larger sample since relative precision improves as sample size increases. There would also likely be decreased variation in relative precision across all state-specific QIO error rates. These results could be achieved without increasing the overall sample size for the HPMP.

In addition to issues with the wide variation of relative precision of the state-specific QIO error rate estimates, we also found large differences in the average dollar amount per claim between the state-specific samples for some states and the respective state populations. These differences suggest that the samples drawn for more than half of the states were not representative of each state’s population. Based on our examination of the OIG contractor’s comparison of the state samples and the state populations for June 2003, we found that the ratio of the average dollar amount per claim in a state’s sample to the average dollar amount per claim in a state’s population varied from 62 percent in Maryland to 143 percent in Kentucky. Twelve states had a ratio above 110 percent, and 16 states had a ratio below 90 percent. It is still possible for the national HPMP sample to be representative of the national HPMP population even if all of the state-specific samples are not representative of their state populations. The larger size of the HPMP sample overall mitigates the problems identified in the smaller state-specific samples.

Medical Record Collection Process

Based on our review of oversight work of the HPMP conducted by OIG, we found that the process CMS used for collecting medical records from providers was adequate. OIG selected 46 discharge claims that were sampled for the HPMP to determine if the CDACs, AdvanceMed and DynKePRO, followed established HPMP procedures for obtaining and reviewing medical records to identify payment errors. OIG found that the CDACs generally had appropriate controls in place to ensure that the medical records were obtained and reviewed according to established HPMP procedures. Of the 46 discharge claims reviewed, OIG found that in two instances a required follow-up letter to the provider was not sent due to an error by a substitute CDAC employee. However, the medical records for these two discharge claims were obtained within 30 days of the original request, which resulted in no adverse effect on the error rate estimates. Overall, nonresponse for fiscal year 2004 represented approximately 5.1 percent of the total QIO contractor-type error rate of 3.6 percent, or 0.2 percent of all discharge claims reviewed through the HPMP.

The issue with providers not responding to requests for medical records was not as significant an issue for the HPMP as it was for the CERT Program. According to the CMS report on the fiscal year 2005 error rate, nonresponse was less problematic in the HPMP because of several factors, including the following: (1) providers were more likely to respond to requests from the HPMP since the average claim value was higher than the average claim value in the CERT Program; (2) providers were more familiar with the HPMP than with the CERT Program; and (3) providers were paid the cost of providing medical records by the HPMP, but not by the CERT Program.

Identification and Categorization of Payment Errors

Based on our review of OIG’s fiscal year 2004 HPMP evaluation, we concluded that the CDACs (AdvanceMed and DynKePRO) generally had processes in place to adequately identify and categorize claims paid in error in the HPMP for fiscal year 2004. OIG officials told us that they found the medical record reviewers, both admission necessity reviewers and DRG coding specialists, at the two CDACs met CMS’s qualifications for these positions. As part of its review of the fiscal year 2004 HPMP, OIG reviewed 46 discharge claims that were part of the sample for estimating the QIO contractor-type error rate. Based on that review, OIG reported that the CDACs generally had appropriate controls in place to ensure that admission necessity and DRG validation reviews were performed in accordance with CMS established procedures and that the results of those reviews were adequately maintained, updated, and reported.

As part of the internal HPMP quality control process, two activities were conducted regularly to ensure the reliability and accuracy of CDAC reviews both within each CDAC and across the two CDACs. Each CDAC randomly chose 30 claims per month to be reviewed by two of its medical record reviewers for intra-CDAC tests. Each CDAC compared the results of the two medical record reviews to determine the reliability of reviews within the CDACs and reported the results of the comparisons to CMS. The CDACs performed inter-CDAC tests to assess the reliability of the reviews between the two CDACs. For these tests, an additional 30 claims were chosen at random per quarter by each of the CDACs for review by a medical records reviewer at the other CDAC. As part of its evaluation of the fiscal year 2004 HPMP, OIG selected 45 claims that went through the intra-CDAC process and 42 claims that went through the inter-CDAC process to determine if these quality control activities ensured the reliability of the CDAC review process. OIG reported that the quality control reviews were generally operating effectively to ensure the reliability of the review process and the consistency of the error rate determination decisions.

From the same evaluation of the fiscal year 2004 HPMP, OIG found that the CMS contractor tasked with calculating the dollar amounts paid in error, Texas Medical Foundation, used a method that produced an amount of dollars in error that in some cases differed from what OIG found to be the amount of dollars in error. For claims identified by a QIO as having errors caused by changes in DRG codes, Texas Medical Foundation used a method that produced different dollar amounts in error than would have been produced if it had used the software that FIs used to pay the original discharge claims. The Texas Medical Foundation calculated a different amount in error for about 76 percent of 200 incorrectly coded claims that OIG reviewed. However, OIG reported that the differences did not have a significant effect on the QIO contractor-type error rate estimate. A CMS official told us that the agency has not invested in modifying the software for use by the Texas Medical Foundation for technical and financial reasons. For example, the software requires modifications using a specific programming language for which CMS has limited personnel with the needed expertise.

Statistical Methods

We verified the statistical methods CMS used to estimate the QIO contractor-type error rate and standard error in the HPMP by reviewing the computer programming code that produced this information. We found that the methods CMS used were adequate because they were based on standard statistical methods and were applied appropriately. To estimate the QIO contractor-type error rate, CMS weighted each state- specific QIO error rate according to that state’s share of the total Medicare FFS payments for acute-care inpatient hospital claims nationwide. This method is referred to as a stratified mean per unit estimation. Like the CERT Program, CMS used a standard statistical method to calculate the standard error of the estimate. In our review of the computer programming code that generated the QIO contractor-type error rate estimate, we found that CMS used annual instead of monthly weights in its estimate of the annual total dollars paid in error. It would have been more appropriate for CMS to have used monthly weights because the HPMP sample was drawn on a monthly, not an annual, basis. However, when we reviewed the OIG contractor’s comparison of the estimate of annual dollars paid in error using annual weights to what the estimate would have been had CMS used monthly weights, we concluded that the use of annual weights did not significantly affect the QIO contractor-type error rate estimate. A CMS official told us and provided us with documentation that beginning with the HPMP’s fiscal year 2005 error rate estimation process, monthly weights are being used.

CMS Methodology Adequate for Estimating the National Error Rate

CMS appropriately combined the error rates under the CERT Program and the HPMP to estimate the fiscal year 2004 national Medicare error rate. CMS estimated the national Medicare error rate by averaging the error rates of the four contractor types (carrier, DMERC, FI, and QIO), weighted by each contractor type’s share of total Medicare FFS payments. Likewise, CMS calculated the standard error, or precision, of the national error rate based on the standard error of each of the four types of contractors’ error rate estimates, weighted by each contractor type’s proportion of total Medicare FFS payments. The methods CMS used to calculate the national error rate and the standard error were statistically valid, since the units of measurement, which in this case were Medicare claims, of the four error rates that were combined were mutually exclusive (independent) among contractor types. Each contractor type consisted of multiple individual contractors. These contractors were independent in that one contractor’s estimated error rate or standard error did not affect the estimates of other contractors, since the claims in the population and in the sample were not overlapping among contractors.

Concluding Observations

Since assuming responsibility for estimating the national Medicare error rate in fiscal year 2003, CMS has made changes to the methodology, which have provided CMS with more detailed information about the error, thereby allowing the agency to better identify the underlying causes of error and implement corrective action plans to address them. For example, CMS significantly increased the size of the sample used to estimate the Medicare FFS claims paid in error. The increased sample size allowed the agency to estimate not only the error rate at the national level, but also more detailed error rates at the contractor-type and contractor- specific levels. Further, CMS has made changes in the way it collects medical records from providers in an effort to reduce the rate of error caused by nonresponse and insufficient documentation. These changes may affect the error rate estimates and thus the comparability of the estimates over time. Consequently, users of the error rate information should exercise caution when making year-to-year comparisons.

Our work focused on the methodology CMS used to estimate the national Medicare error rate and contractor-type error rates for fiscal year 2004. For these error rates, we found the methodology adequate for that year. Under CMS’s contracting reform initiative, there will be fewer individual contractors (carriers, DMERCs, and FIs). If CMS maintains the same overall sample size, the sample sizes of the remaining individual contractors would be increased. Reliability of the contractor-specific error rate estimates is likely to improve with the larger sample sizes. Until then, the wide variation in reliability of the contractor-specific error rate estimates may preclude meaningful comparisons across individual contractors.

Agency Comments

We received written comments from HHS (see app. III.) In responding to our draft report, HHS noted that we found the CMS methodology adequate for estimating the fiscal year 2004 national Medicare FFS error rate. HHS also noted that CMS is continually committed to refining the processes to estimate, as well as lower, the level of improper payments in the Medicare FFS program.

In its comments, HHS noted improvement in the national Medicare error rate from fiscal years 2004 to 2005. The department attributed the decline in the error rate to marked improvement in the nonresponse (which CMS now calls “no documentation”) and the insufficient documentation error rates. Commenting on the adequacy of the fiscal year 2005 methodology was beyond the scope of our work; however, as we noted in the draft report, changes in the methodology may affect the estimation of the error rates and thus the comparability of these error rates over time. For example, we discussed in the draft report that CMS has made changes in the way it collects medical records from providers in an effort to reduce the rate of error caused by nonresponse and insufficient documentation. These changes primarily affected HHS’s processes for calculating an annual error rate estimate for the Medicare FFS program. This may represent a refinement in the program’s estimation methodology rather than improved accountability over program dollars.

The national Medicare error rates for fiscal years 2004 and 2005 provided by HHS in its comments are not comparable to the error rates cited in this report for fiscal years 2004 and 2005. HHS provided gross error rates, which were calculated using gross dollars paid in error. Gross dollars paid in error were calculated by adding dollars paid in error that were due to underpayments to those that were due to overpayments. As noted in the draft report, we reported net error rates. Net error rates were calculated using net dollars paid in error. Net dollars paid in error were calculated by subtracting dollars paid in error that were due to underpayments from those that were due to overpayments.

HHS also provided technical comments, which we have addressed as appropriate.

We are sending copies of this report to the Secretary of Health and Human Services, the HHS Inspector General, the Administrator of CMS, and appropriate congressional committees. We will also provide copies to others upon request. In addition, the report is available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7101 or steinwalda@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Scope and Methodology

We reviewed the following components of the Centers for Medicare & Medicaid Services’s (CMS) methodology for estimating the fiscal year 2004 error rate: Sampling methods, including sample size, sample selection, sample representation, and precision of the estimates.

The medical records collection process.

Identification and categorization of claims payment error, including the medical record review process and quality assurance reviews.

Statistical methods used to estimate the error rates and precision.

To conduct our analysis of CMS’s sampling methods, we reviewed work performed by the Department of Health and Human Services (HHS) Office of Inspector General (OIG) contractor that assessed these methods and CMS documentation for the fiscal year 2004 Medicare error rate. For the Comprehensive Error Rate Testing (CERT) Program, we reviewed the program manual, which described the CERT Program sampling methods as well as CMS’s Medicare error rate reports for fiscal years 2003 and 2004. For the Hospital Payment Monitoring Program (HPMP), we reviewed the program manual and the HPMP computer programming code that generated the sample to verify that the sample was taken in accordance with the procedures outlined in the manual. Additionally, we reviewed the OIG contractor’s comparison of the June 2003 sample and a re-created version of the June 2003 sampling frame, or population, for the HPMP. It was not possible for the OIG contractor to obtain the exact June 2003 population file because the file is continuously updated and previous versions are not retained. We did not believe it was necessary to compare every month’s sample to the population from which it was drawn because of the large size of the sample (approximately 40,000 discharge claims) and population (approximately 11.5 million discharge claims), and the fact that the sample was drawn in the same manner each month.

To conduct our analysis of CMS’s medical record collection and review processes and identification and categorization of payment errors, we relied primarily on reports published by OIG. Since 2003, OIG has conducted annual reviews of the CERT Program and the HPMP as part of its review of work performed for HHS by contractors. These annual reviews examine whether the CERT Program and HPMP contractors have appropriate controls in place to ensure that the medical record reviews and quality assurance reviews were performed in accordance with established procedures. We reviewed OIG’s annual reviews of the CERT Program and the HPMP for fiscal year 2004. Our analysis of provider nonresponse within the CERT Program relied on two OIG studies of CMS’s actions to reduce nonresponse implemented for the CERT Program for fiscal year 2004. For the HPMP, we also reviewed four intra-Clinical Data Abstraction Center (CDAC) reports and two inter-CDAC reports, which were quality assurance reviews intended to assess the consistency of review decisions both within and across CDACs.

To conduct our analysis of CMS’s statistical methods, we reviewed the OIG contractor’s computer programming code, which replicated CMS’s estimation of the error rates for carriers, durable medical equipment regional carriers (DMERC), and fiscal intermediaries (FI), as calculated by the CERT Program subcontractor responsible for statistical analysis of the error rates for fiscal year 2004. We reviewed CMS’s computer programming code, which calculated the HPMP error rate for quality improvement organizations (QIO). In conducting these reviews of the computer programming codes for both the CERT Program and the HPMP, we verified that each code appropriately implemented a methodology that employed standard statistical principles and was used appropriately.

To inform all aspects of our study, we interviewed OIG officials with oversight responsibility for the error rate estimation, OIG contractor staff who conducted the evaluation of the statistical methodology, CMS officials with programmatic responsibilities for the CERT Program and the HPMP, and staff of the CERT Program subcontractor for statistical analysis.

We performed our work from April 2005 through March 2006 in accordance with generally accepted government auditing standards.

Appendix II: Fiscal Year 2004 Error Rate Information by Contractor Type—Carriers, DMERCs, FIs, and QIOs

Appendix II: Fiscal Year 2004 Error Rate Information by Contractor Type—Carriers, DMERCs, FIs, and QIOs year 2004(in dollars)

CMS estimated paid claims error rate (percentage)

CMS estimated standard error(percentage) precision(percentage) 13.5 year 2004(in dollars)

CMS estimated paid claims error rate (percentage)

CMS estimated standard error(percentage) precision(percentage) 4.5 year 2004(in dollars)

CMS estimated paid claims error rate (percentage)

CMS estimated standard error(percentage) precision(percentage) 23.8 year 2004(in dollars)

CMS estimated paid claims error rate (percentage)

CMS estimated standard error(percentage) precision(percentage)

Carriers are health insurers and pay claims submitted by physicians, diagnostic laboratories and facilities, and ambulance service providers.

DMERCs are health insurers and pay claims submitted by durable medical equipment suppliers.

For the fiscal year 2004 error rate,TriCenturion, a program safeguard contractor, was responsible for medical review in one of the four DMERC regions. Program safeguard contractors are Medicare contractors that conduct activities to address or prevent improper payments. As such, it was TriCenturion, not the DMERC, which was responsible for lowering the error rates in its region.

FIs are almost exclusively health insurers and pay claims submitted by home health agencies, non- prospective payment system (PPS) hospitals, hospital outpatient departments, skilled nursing facilities, and hospices. PPS is a reimbursement method used by Medicare where the payment is made based on a predetermined rate and is unaffected by the provider’s actual costs.

QIOs (formally known as peer review organizations) are responsible for ascertaining the accuracy of coding and payment of paid Medicare FFS claims for acute care inpatient hospital stays—generally those that are covered by PPS—for Medicare beneficiaries in all 50 states, the District of Columbia, and Puerto Rico. Unlike carriers, DMERCs, and FIs, however, QIOs do not process and pay claims. These activities are conducted by FIs.

Appendix III: Comments from the Department of Health and Human Services

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Debra Draper, Assistant Director; Lori Achman; Jennie Apter; Dae Park; and Ann Tynan made key contributions to this report.