Medicare Payment: CMS Methodology Adequate to Estimate National Error Rate
Published: 2006-03-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Centers for Medicare & Medicaid Services (CMS) estimated that the Medicare program paid approximately $20 billion (net) in error for fee-for-service (FFS) claims in fiscal year 2004. CMS established two programs--the Comprehensive Error Rate Testing (CERT) Program and the Hospital Payment Monitoring Program (HPMP)--to measure the accuracy of claims paid. The Medicare Prescription Drug, Improvement, and Modernization Act of 2003 directed GAO to study the adequacy of the methodology that CMS used to estimate the Medicare FFS claims paid in error. GAO reviewed the extent to which CMS's methodology for estimating the fiscal year 2004 error rates was adequate by contractor type for (1) the CERT Program, (2) the HPMP, and (3) the combined national error rate (including both the CERT Program and the HPMP). GAO reviewed relevant CMS documents and reports related to the CERT Program and the HPMP. In addition, GAO reviewed work performed by the Department of Health and Human Services (HHS) Office of Inspector General (OIG) and its contractor that evaluated CMS's fiscal year 2004 statistical methods and other aspects of the error rate estimation process. GAO also conducted interviews with officials from CMS, HHS's OIG, and their contractors.
section_title: What GAO Found
paragraphs: The methodology used by CMS for the CERT Program was adequate to estimate the fiscal year 2004 error rates by contractor type--carrier, durable medical equipment regional carrier (DMERC), and fiscal intermediary (FI). Carriers pay claims submitted by physicians, diagnostic laboratories and facilities, and ambulance service providers. DMERCs pay claims submitted by durable medical equipment suppliers. FIs pay claims submitted by hospitals, home health agencies, hospital outpatient departments, skilled nursing facilities, and hospices. The methodology was adequate because CMS used a large sample--about 120,000 claims--and an appropriate sample selection strategy. For these fiscal year 2004 error rate estimates, CMS made improvements in the collection of medical records that supported the sampled claims. These medical records were appropriately reviewed to determine whether there were errors in payment. CMS used valid statistical methods to estimate the fiscal year 2004 error rates for the carrier, DMERC, and FI contractor types. The methodology used by CMS for the HPMP was adequate to estimate the fiscal year 2004 error rate by quality improvement organizations (QIO), which are responsible for ascertaining the accuracy of coding and payment of Medicare FFS paid claims for acute care inpatient hospital stays. CMS's sampling methods were adequate because the agency used a large sample, approximately 40,000 claims, that was representative of the population from which it was drawn in terms of average dollar amount per claim. Also, the HPMP had adequate processes in place to ensure appropriate determinations of error. CMS used valid statistical methods to estimate the fiscal year 2004 error rate for the QIO contractor type. The fiscal year 2004 contractor-type error rate estimates for the CERT Program and the HPMP were appropriately combined to determine the national Medicare error rate through the use of a valid statistical method. CMS estimated the national Medicare error rate by averaging the carrier, DMERC, and FI contractor-type error rates in the CERT Program and the QIO contractor-type error rate in the HPMP, weighted by each contractor type's share of total Medicare FFS payments. In written comments, HHS noted that GAO found CMS's methodology adequate for estimating the fiscal year 2004 national Medicare FFS error rate. HHS also noted that CMS is continually committed to refining the processes to estimate, as well as lower, the level of improper payments in the Medicare FFS program.

section_title: Letter
paragraphs: The Centers for Medicare & Medicaid Services (CMS), the agency that  administers the Medicare program, monitors the accuracy of claims paid  for services provided to Medicare beneficiaries. Each fiscal year, CMS  reports an estimate of the claims paid in error based on a sample of claims  from previous years. In fiscal year 2004, CMS reported an error rate of   9.3 percent, which represented approximately $20 billion in error out of  the approximately $214 billion in fee-for-service (FFS) payments. The  fiscal year 2004 error rate estimated the percentage of FFS payments that  did not comply with Medicare’s payment rules for a sample of claims that  included inpatient discharges that occurred from July 1, 2002, through  June 30, 2003, as well other services that were paid in 2003. The fiscal   year 2004 Medicare FFS error rate was significantly higher than the goal of  4.8 percent for that fiscal year, which CMS set under the Government  Performance and Results Act of 1993 (GPRA).
CMS uses several types of contractors to ensure the payment accuracy of  Medicare claims, including carriers, durable medical equipment regional  carriers (DMERC), fiscal intermediaries (FI), and quality improvement  organizations (QIO). Using contractor-specific error rate information,  CMS estimates an error rate for each type of contractor; the agency  produces a national Medicare error rate by aggregating the four  contractor-type error rates. In its fiscal year 2004 Medicare error rate  report, CMS stated that it planned to use error rate information to help  determine the underlying reasons for claim errors, such as incorrect  coding, and implement corrective actions. In a congressional testimony in  July 2005, the Director of CMS’s Office of Financial Management stated  that CMS plans to create performance incentives for contractors. CMS is  also implementing a multiyear contractor reform initiative, which will  reduce the number of contractors responsible for paying claims.
To monitor the accuracy of Medicare FFS claims paid by contractors, CMS  established two programs—the Comprehensive Error Rate Testing  (CERT) Program and the Hospital Payment Monitoring Program (HPMP).  Through the CERT Program, CMS monitors payment decisions made by  three types of contractors—carriers, DMERCs, and FIs. It does this  through a review of the claims and submitted medical record  documentation to ensure that there is support for the payment based on  the information reviewed for a sample of paid claims. CMS uses a similar  process for the HPMP for a sample of claims that are reviewed by QIOs for  accuracy of payment.
The Department of Health and Human Services (HHS) Office of Inspector  General (OIG) estimated the error rate for each fiscal year from 1996  through 2002. CMS made significant changes to the methodology,  including substantially increasing the size of the sample used to estimate  the error rate, when it assumed responsibility for estimating the Medicare  error rate in fiscal year 2003. CMS has continued to make changes to the  methodology in subsequent years.
The Medicare Prescription Drug, Improvement, and Modernization Act of  2003 requires that we study the adequacy of the methodology that CMS  used to estimate Medicare error rates and to make recommendations as  deemed appropriate. Specifically, we report on the extent to which the  methodology used by CMS to estimate the fiscal year 2004 Medicare error  rates was adequate (1) by contractor type (carrier, DMERC, and FI) for the  CERT Program, (2) by contractor type (QIO) for the HPMP, and (3) for the  combined national error rate (including the CERT Program and the  HPMP).
To conduct our analysis of the adequacy of the methodology that CMS  used to estimate fiscal year 2004 Medicare error rates, we reviewed  relevant documents, including CMS’s Medicare error rate reports for fiscal  years 2003 and 2004, CERT and HPMP program documentation, and HHS  OIG reports evaluating the fiscal year 2004 CERT Program and HPMP. In  addition, we reviewed work performed by an OIG contractor that  evaluated CMS’s statistical sampling and estimation methodology for the  fiscal year 2004 Medicare error rate, including the contractor’s report and  supporting workpapers. We interviewed OIG officials; OIG contractor  staff; CMS officials; and staff of the CERT subcontractor responsible for  calculating the error rates for carriers, DMERCs, and FIs and the national  error rate for fiscal year 2004. Commenting on the adequacy of the  methodology used in any other years was beyond the scope of our work.  However, it is important to note that changes in the methodology may  affect the estimation of the error rates and thus the comparability of these  rates over time.
As part of our assessment of the adequacy of the methodology that CMS  used to estimate the Medicare error rates for fiscal year 2004, we reviewed  the reliability of these estimates by examining the precision of the  contractor-specific error rates, the contractor-type error rates, and the  national error rate. Precision is the amount of variation between an  estimate (such as the error rate for a sample of Medicare FFS paid claims)  and the result that would be obtained from measuring the entire  population (such as the error rate for all Medicare FFS paid claims). We  examined precision of the error rate estimates by assessing relative  precision, which is the standard error of the error rate estimate divided  by the estimate itself. Estimates with lower relative precision are more  reliable. For the purposes of this report, we established that a relative  precision of no greater than 15 percent was within the acceptable  statistical standard for precision. We chose relative precision because it  allows for better comparison of the reliability of the range of error rates  across contractors.
During the course of our work, CMS published a report in November 2005  that included its fiscal year 2005 error rates. While an evaluation of the  methodology used to estimate the fiscal year 2005 error rates was outside  the scope of our work, we reviewed the report and included references in  this report where appropriate.
For more information on our scope and methodology, see appendix I. We  performed our work from April 2005 through March 2006 in accordance  with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: In fiscal year 2003, CMS assumed responsibility for estimating the national  Medicare error rate, a responsibility that had previously been held by HHS  OIG. OIG began estimating the national Medicare error rate in fiscal year  1996, and continued doing so for each subsequent fiscal year through  2002. The transfer of responsibilities for estimating the national Medicare  error rate to CMS coincided with the implementation of the Improper  Payments Information Act of 2002 (IPIA). The IPIA requires federal  agencies to estimate and report annually on the extent of erroneous  payments in their programs and activities. The IPIA defines an improper  payment as any payment that should not have been made or that was  made in an incorrect amount, including both under- and overpayments. All  agencies that identify a program as susceptible to significant improper  payments, defined by guidance from the Office of Management and Budget  (OMB) in 2003 as exceeding both 2.5 percent of total program payments  and $10 million, are required to annually report to Congress and the  President an estimate of improper payments and report on corrective  actions.
In addition to estimating the national Medicare error rate for purposes of  compliance with the IPIA, CMS also began producing contractor-specific  error rate estimates beginning in fiscal year 2003 to identify the underlying  causes of errors and to adjust action plans for carriers, DMERCs, FIs, and  QIOs. To produce these contractor-specific error rate estimates for fiscal  year 2004, CMS sampled approximately 160,000 claims. The contractor- specific error rate information was then aggregated by the four contractor  types (carrier, DMERC, FI, and QIO), which were ultimately combined to  estimate the national Medicare error rate. Under the methodology  previously used by OIG to estimate the national Medicare error rate, 6,000  claims were sampled. While the sample size used by OIG was sufficient to  estimate the national Medicare error rate, it was not sufficient to reliably  estimate the contractor-specific error rates. Additionally, the increased  sample size improved precision of the national Medicare error rate  estimate.
subsections: section_title: CMS Programs to Monitor the Payment Accuracy of Medicare FFS Claims
paragraphs: The objective of the CERT Program and the HPMP is to measure the  degree to which CMS, through its contractors, is accurately paying claims.  Through the CERT Program, CMS monitors the accuracy of Medicare FFS  claims that are paid by carriers, DMERCs, and FIs. In fiscal year 2004, the  Medicare error rates by contractor type as estimated through the CERT  Program were 10.7 percent for the carrier contractor type, 11.1 percent for  the DMERC contractor type, and 15.8 percent for the FI contractor type.  (See table 1.)
Through the HPMP, CMS monitors the accuracy of paid Medicare FFS  claims for acute care inpatient hospital stays—generally those that are  covered under the prospective payment system (PPS). For fiscal year 2004,  the Medicare error rate for the QIO contractor type, as estimated through  the HPMP, was 3.6 percent. (See table 1.)
section_title: CERT Program
paragraphs: To estimate contractor-specific Medicare FFS error rates for the CERT  Program, CMS reviews a sample of claims from each of the applicable  contractors, which included 25 carriers, 4 DMERCs, and 31 FIs for the  fiscal year 2004 error rates. These error rates are then aggregated by  contractor type. (See fig. 1.) For fiscal year 2004, CMS contracted with  AdvanceMed to administer the CERT Program. AdvanceMed sampled  approximately 120,000 claims submitted from January 1, 2003, through  December 31, 2003, to estimate the fiscal year 2004 contractor-specific and  contractor-type error rates for the CERT Program.
For each of the approximately 120,000 sampled claims, AdvanceMed  requested the medical records from the provider that rendered the service  or from the contractor that processed the related claim, if the contractor  previously performed a medical review on the claim. If a provider did not  respond to the initial request for medical records after 19 days,  AdvanceMed initiated a series of follow-up procedures in an attempt to  obtain the information. The follow-up procedures with nonresponding  providers for fiscal year 2004 included three written letters and three  contacts by telephone. Additionally, in fiscal year 2004, OIG followed up  directly with nonresponders on claims over a certain dollar amount. If  medical records were not received within 55 days of the initial request, the  entire amount of the claim was classified by AdvanceMed as an  overpayment error.
When medical records were received from the provider or from the  contractor, CERT medical review staff reviewed the claim (which billed  for the services provided) and the supporting medical records (which  detailed the diagnosis and services provided) to assess whether the claim  followed Medicare’s payment rules and national and local coverage  decisions. Claims that did not follow these rules were classified by  AdvanceMed as being in error. Providers whose claims were reviewed  were allowed to appeal these claims, and if the error determination for a  claim was overturned through the appeals process, AdvanceMed adjusted  the error rate accordingly. For the fiscal year 2004 error rate, AdvanceMed  notified individual carriers, DMERCs, and FIs of their respective payment  errors.
For the HPMP, CMS analyzes a sample of claims across QIOs to estimate  Medicare error rates by state, because QIOs are organizations with state- based service areas. CMS estimated the QIO contractor-type error rate by  aggregating the QIO error rate estimates for each of the 50 states, the  District of Columbia, and Puerto Rico. (See fig. 2.) Through the HPMP,  CMS sampled approximately 40,000 claims for acute care inpatient  hospital discharges that occurred from July 1, 2002, through June 30, 2003,  to estimate the fiscal year 2004 state-specific and contractor-type error  rates for QIOs.
For fiscal year 2004, CMS contracted with two organizations known as  Clinical Data Abstraction Centers (CDAC)—AdvanceMed and  DynKePRO—that were responsible for requesting medical records from  providers for each of the approximately 40,000 sampled claims. Each  CDAC was responsible for reviewing the sampled claims, which were  assigned on the basis of the geographic location where the discharge  occurred. Upon receipt of the medical records, CDAC admission necessity  reviewers screened the related claims for the appropriateness of the  hospitalization and, with the exception of claims from Maryland, coding  specialists independently recoded diagnosis-related groups (DRG) based  on the records submitted. Because Maryland does not use DRG coding,  nonphysician reviewers screened claims from Maryland to determine  whether the length of the acute care inpatient hospital stay was  appropriate.24, 25 Claims that failed the screening process, including those  where the admission was determined to be unnecessary or where an  inappropriate DRG code was used, were forwarded to the QIO responsible  for the state where the discharge occurred for further review. Records not  received by the CDACs within 30 days of the request for information were  “canceled” and referred to the QIO to be processed as overpayment errors  caused by nonresponse. The QIO referred these claims to the FI  responsible for paying the claim for the necessary payment adjustments.
At the QIO, claims forwarded from the CDACs underwent further review,  primarily medical necessity admission reviews and DRG validations.  Determinations of error were made by QIO physician reviewers. Providers  whose claims were reviewed were given the opportunity to provide  comments or discuss the case and pursue additional review, which could  result in an appeal to an administrative law judge. After the matter was  resolved, resulting in a determination that a provider was either underpaid  or overpaid, the QIO forwarded the claim to the FI for payment  adjustment.
Maryland is the only state that does not use the PPS system for acute care inpatient  hospitals. Maryland instead has an alternative payment system, known as an all-payer  system, in which the state decides each hospital’s level of reimbursement and requires that  all payers be charged the same rate for the same service. Medicare and Medicaid pay the  state-approved rates.
Claims from Maryland with length of stay errors are considered medically unnecessary  services. Length of stay reviews identified cases of potential delayed discharge. For  example, the patient was medically stable, and continued hospitalization was unnecessary.
section_title: Estimation of the National Medicare FFS Error Rate
paragraphs: CMS estimated the national Medicare FFS error rate by combining the  three contractor-type error rates (carrier, DMERC, and FI) from the CERT  Program and the one contractor-type error rate (QIO) from the HPMP.  (See fig. 3.)
Medicare FFS claims that were paid in error as identified by the CERT  Program and the HPMP for the fiscal year 2004 error rates were sorted  into one of five categories of error:  Insufficient documentation: Provider did not submit sufficient  documentation to support that the services billed were actually provided.    Nonresponse: Provider did not submit any documentation to support that  the services billed were actually provided.
Medically unnecessary services: Provider submitted sufficient  documentation, but the services that were billed were deemed not  medically necessary or the setting or level of care was deemed  inappropriate.  Incorrect coding: Provider submitted documentation that supported a  different billing code that was associated with a lower or higher payment  than that submitted for the services billed.
Other: Provider submitted documentation, but the services billed did not  comply with Medicare’s benefit or other billing requirements.
See table 2 for the national Medicare FFS error rate by category of error  for fiscal year 2004.
As reported in CMS’s fiscal year 2004 Medicare error rate report, the  agency planned to use the error rates to help determine the underlying  reasons for claim errors, such as incorrect coding or nonresponse, and  implement corrective action plans for carriers, DMERCs, FIs, and QIOs.  Draft statements of work, dated February and April 2005, for carriers,  DMERCs, and FIs set goals for contractors to achieve a paid claims error  rate of less than a certain percentage, to be determined by CMS. According  to the standards for minimum performance on QIO statements of work  that ended in 2005 for some QIOs and 2006 for other QIOs, QIOs are  evaluated on 12 tasks, one of which is the HPMP. QIOs have to meet the  performance criteria standards on 10 tasks set forth by CMS to be eligible  for a noncompetitive contract renewal.
CMS’s use of the error rates is being done in the context of the agency’s  current effort to significantly reform its contracting efforts for the  payment of Medicare claims. By July 2009, CMS plans to reduce the total  number of contractors responsible for paying Medicare claims to 23 total  contractors, which the agency refers to as Medicare administrative  contractors (MAC). CMS also plans to institute performance incentives in  the new contracts, which will be based on a number of different factors,  including the Medicare error rates. According to CMS’s report to Congress  on Medicare contracting reform, CMS believes that the consolidation of  Medicare contractors and the integration of processing for Medicare  claims will lead to a reduced Medicare error rate.
section_title: CMS Methodology Adequate for Estimating the Error Rates in the CERT Program
paragraphs: The methodology used by CMS in the CERT Program to estimate error  rates by contractor type (carrier, DMERC, and FI) in fiscal year 2004 was  adequate. We found that the sample size and the use of systematic  sampling with a random start were adequate to reliably estimate the  Medicare error rates by contractor type. The CERT Program also had  adequate processes in place to collect medical records and to accurately  identify and categorize payment errors. The statistical methods that CMS  used to estimate the contractor-type error rates were valid.
subsections: section_title: Sampling Methods
paragraphs: The sample size that CMS used in the CERT Program, approximately  120,000 claims, was sufficiently large to produce reliable estimates of the  fiscal year 2004 Medicare error rates by contractor type (carrier, DMERC,  and FI). CMS selected 167 claims each month on a daily basis from each of  the 60 contractors, including 25 carriers, 4 DMERCs, and 31 FIs. This  sample generated error rate estimates by contractor type within  acceptable statistical standards, such as relative precision of no greater  than 15 percent. Specifically, the error rate for the carrier contractor type  was 10.7 percent with a relative precision of 3.7 percent, the error rate for  the DMERC contractor type was 11.1 percent with a relative precision of  13.5 percent, and the error rate for the FI contractor type was 15.7 percent  with a relative precision of 4.5 percent.
Further, we found that the sampling methods were adequate because CMS  used a systematic sample with a random start. Sampling methods that  employ a random start are designed to ensure that the sample selected is  representative of the population from which it is drawn. We reviewed  CERT Program documentation, which described the use of a systematic  sample with a random start. The OIG contractor reviewed the computer  program used for the CERT Program sample selection and verified that the  claims were selected according to the documentation. CMS officials told  us that the CERT Program conducts tests to compare the sampled claims  to the population of claims. For example, CMS compared the percentage  of claims sampled in each category of Medicare-covered service to the  percentage of claims in the population by category of Medicare-covered  service. CMS provided us with an example of this test for one contractor’s  claims from January 2003 through June 2003.
While the relative precision of the fiscal year 2004 error rate estimates by  contractor type for the CERT Program was within acceptable statistical  standards of no greater than 15 percent, the relative precision of half of the  contractor-specific error rate estimates was not. (See app. II for  contractor-specific error rate information, including the estimates and  corresponding relative precision, for carriers, DMERCs, and FIs.)
Thirty of the 60 contractor-specific error rates had relative precision that  were not within acceptable statistical standards. Additionally, the relative  precision of the contractor-specific error rates showed wide variation  within each contractor type. Relative precision among carriers ranged  from 8.9 percent to 17.0 percent; among DMERCs, relative precision  ranged from 12.3 percent to 20.7 percent; and among FIs, relative precision  ranged from 10.3 percent to 42.5 percent. As demonstrated by the range in  relative precision among FIs, for example, the error rate estimate for one  FI was nearly four times more reliable than the error rate estimate for  another.
The variation in relative precision among the contractor-specific error rate  estimates was due, in part, to the sampling method CMS used for the  CERT Program. CMS took an equal sample size from each contractor  despite the fact that individual contractors accounted for varied amounts  of Medicare claim volumes and total payments. For example, the claim  volume for carriers in 2003 ranged from a minimum of 5.3 million claims to  a maximum of 206 million claims; total payments for carriers in 2003  ranged from a minimum of $168 million to about $6.7 billion.
CMS officials told us that they plan to reallocate the CERT Program  sample at the contractor level by increasing the sample size for those  contractors that are not reaching CMS’s targeted precision and by  decreasing the sample size for those contractors that are reaching targeted  precision and achieving low error rates. In September 2005, CMS officials  reported that this change to the methodology is expected to be  implemented for the fiscal year 2007 error rate estimation, which will be  based on claims processed in parts of 2006 and 2007. We support CMS’s  planned changes to its sampling methodology. We believe that reallocation  of the sample as planned by CMS will improve the relative precision of  these estimates. If future samples were based on the volume of claims or  total payments of each contractor and the relative precision of the  contractor-specific error rate rather than on the current basis of an equal  allocation across contractors, relative precision would likely be improved  for the contractor-specific error rates of those targeted contractors that  were allocated a larger sample. This is because relative precision improves  with increased sample size. There would also likely be decreased variation  in relative precision across all contractor-specific error rates. These  results could be achieved without increasing the overall sample size for  the CERT Program.
section_title: Medical Record Collection Process
paragraphs: Based on our review of oversight work conducted by OIG, we found that  the process CMS used to collect medical records from providers for the  CERT Program was adequate. Staff of AdvanceMed, the CMS contractor  responsible for administering the CERT Program, were responsible for  requesting medical records for each of the approximately 120,000 sampled  claims used to estimate the fiscal year 2004 error rates. According to an  OIG review of CMS’s corrective actions to improve nonresponse in the  CERT Program for fiscal year 2004, AdvanceMed conducted a timely and  systematic follow-up with providers that did not respond to initial requests  for medical records. For the medical records collection process for the  fiscal year 2004 error rates, CMS implemented corrective actions in the  CERT Program to address the factors associated with the high rate of  nonresponse experienced during the medical records collection process  for the prior fiscal year. According to the CMS fiscal year 2003 error rate  report, for example, the agency found that some nonresponse in fiscal year  2003 was due to providers’ lack of familiarity with AdvanceMed. In  previous years when OIG had responsibility for estimating the Medicare  error rate, OIG requested medical records directly from providers;  providers were familiar with OIG and understood the importance of  complying with the requests. However, when the responsibility for  estimating the Medicare error rate was transferred to CMS, many  providers were unfamiliar with AdvanceMed and may have been reluctant  to submit medical records to an unknown company. Another factor that  caused provider nonresponse in fiscal year 2003, according to the CMS  report, was providers’ confusion about the submission of medical records  within the constraints of the privacy regulations issued by HHS under the  Health Insurance Portability and Accountability Act of 1996, which limit  the use and release of individually identifiable health information.  According to the CMS report, CMS found that providers were sometimes  unaware that sending medical records to the CERT Program contractor  was permissible under the regulations. As reported in the OIG review cited  previously, CMS implemented corrective actions that increased provider  compliance with medical record requests in fiscal year 2004. According to  the OIG report, CMS conducted educational efforts to clarify the role of  AdvanceMed. Additionally, OIG further reported that CMS took action to  address providers’ concerns about compliance with the privacy  regulations by revising its request letters to providers to highlight  AdvanceMed’s authorization, acting on CMS’s behalf, to obtain medical  records as requested. OIG told us that CMS instructed carriers, DMERCs,  and FIs to refer certain claims for nonresponding providers to OIG for  follow-up.
These improvements in the process used to collect medical records in the  CERT Program helped reduce nonresponse. According to information  provided to us by CMS, the percentage of error caused by nonresponse in  the CERT Program decreased from 61 percent for fiscal year 2003 to   34 percent in fiscal year 2004. According to CMS’s fiscal year 2005 error  rate report, the agency continued several corrective actions to address  nonresponse for sampled claims for the fiscal year 2005 error rates.  Further, beginning with claims sampled to estimate the fiscal year 2006  Medicare error rates, CMS transferred the medical record collection duties  to a second contractor, Lifecare Management Partners, which the agency  refers to as the CERT Program documentation contractor. CMS officials  told us that the CERT Program documentation contractor is automating  the medical record collection process and eliminating paper copies of  documentation.
section_title: Identification and Categorization of Payment Errors
paragraphs: Based on our review of OIG’s fiscal year 2004 CERT Program evaluation,  we concluded that the processes used in the CERT Program to identify  and categorize payment errors for fiscal year 2004 were adequate because  the medical record reviews were performed appropriately and the CERT  Program staff conducting the reviews were adequately trained and  qualified. Staff of the CERT Program contractor, AdvanceMed, reviewed  the medical records to verify that claims were processed according to  Medicare payment rules; if not, a claim was found to be in error and  assigned to one of five categories of error (insufficient documentation,  nonresponse, medically unnecessary, incorrect coding, or other). We  reviewed work conducted by OIG that found AdvanceMed, the CMS  contractor responsible for administering the CERT Program, had  appropriate controls in place to ensure that the medical record reviews  were performed in accordance with established CERT Program  procedures. We also reviewed work by OIG, which examined the  educational and training requirements for medical record reviewers as  established in the CERT Program and assessed selected training files for  selected medical record reviewers. OIG officials told us that they found  these selected CERT Program medical record reviewers to be adequately  trained and qualified.
OIG found that AdvanceMed did not complete all required quality  assurance reviews within the designated time frame. CMS told OIG that it  planned to reduce AdvanceMed’s workload. AdvanceMed conducts quality  assurance reviews on a sample of medically reviewed claims to validate  the initial reviewer’s decision on whether a claim was paid in error. OIG  found that for the fiscal year 2004 CERT Program, AdvanceMed completed  984 of the required 2,587 quality assurance reviews by the required date.  To determine whether these quality assurance reviews ensured the  reliability of the CERT Program claims review process, OIG randomly  sampled 45 of the 2,587 claims selected for quality assurance reviews. Of  these 45 claims, AdvanceMed had completed a quality assurance review on  5 claims. OIG reported that the results of the 5 quality assurance reviews  confirmed the results of the initial medical record reviews. Further, OIG  reported that AdvanceMed stated that a backlog of medical reviews  prevented the completion of the required quality assurance reviews within  the prescribed time frame. In response to the OIG report on the fiscal year  2004 CERT Program evaluation, CMS commented that with Lifecare  Management Partners assuming responsibilities for medical record  collection for the fiscal year 2006 Medicare error rate estimation,  AdvanceMed’s workload would be reduced. As a result, CMS commented  that this will free up the necessary resources for AdvanceMed to comply  with the quality assurance requirements. Further, in its response to the  OIG report, CMS commented that both AdvanceMed and Lifecare  Management Partners are required to report to the agency on the results of  the quality assurance activities conducted. According to OIG’s evaluation  of the fiscal year 2005 CERT Program, OIG found that AdvanceMed  completed all of the required quality assurance reviews.
section_title: Statistical Methods
paragraphs: We found that the statistical methods used to estimate the error rates and  standard errors by contractor type (carrier, DMERC, and FI) for the CERT  Program were adequate. Based on our review of the computer  programming code that generated the error rate estimates and standard  errors by the CERT Program subcontractor responsible for calculating the  contractor-type error rates, The Lewin Group, we found that the statistical  methods were based on standard statistical principles and were used  appropriately. For each contractor type, the stratified combined ratio  estimation method was used to calculate the error rate by taking the  difference between the overpaid dollars and the underpaid dollars divided  by the total dollars paid by Medicare for FFS claims of each contractor  type. The payment errors from the sample were then extrapolated to the  population for each contractor type to estimate total payment errors.  Further, The Lewin Group used a standard statistical method to calculate  the standard errors of each of the contractor-type error rates. This  method is appropriate for obtaining the standard error of an estimate  when the stratified combined ratio estimation method is used and is valid  for large sample sizes, such as that used for the CERT Program.
section_title: CMS Methodology Adequate for Estimating the Error Rate in the HPMP
paragraphs: We found that the methodology used by CMS was adequate to produce a  reliable estimate of the fiscal year 2004 Medicare error rate for the one  contractor type (QIO) in the HPMP. We found the methodology adequate  because the sample size was large enough to produce a reliable error rate  estimate. Additionally, the sample was representative of the population.  We found also that the methodology was adequate because the HPMP  contractors responsible for collecting the medical records for the sampled  claims, as well as for identifying and determining errors, had appropriate  controls in place to ensure that established procedures were followed.  Further, the statistical method that CMS used to calculate the contractor- type error rate was valid.
subsections: section_title: Sampling Methods
paragraphs: The sample size that CMS used for the HPMP, about 40,000 claims, was  sufficiently large to produce a reliable estimate of the fiscal year 2004  error rate for the QIO contractor type. Using a systematic sample, CMS  selected 62 discharge claims per month for the District of Columbia,  Puerto Rico, and each state except Alaska. CMS selected 42 claims per  month for Alaska. The QIO contractor-type error rate was 3.6 percent with  a relative precision of 5.6 percent. The relative precision for the QIO  contractor-type error rate estimate is within acceptable statistical  standards (a relative precision of no greater than 15 percent).
For the QIO contractor-type error rate to be a reliable estimate, it was  necessary that the sample of discharge claims from which the error rate  was estimated be representative of the population from which it was  drawn. CMS’s documentation stated that the HPMP used a systematic  sample selection process with a random start, which is a generally  accepted method of sampling that is designed to ensure that the sample  drawn is representative of the population. Our review of the computer  programming code that selected the sample, however, found that a  random start was not used. To determine whether the HPMP sample was  compromised by the lack of a random start and whether it represented the  population from which it was drawn, we examined the OIG contractor’s  comparison of the June 2003 sample to a re-created version of the   June 2003 population file from which the sample was drawn. Based on  our review, we found that the HPMP sample was representative of the  population from which it was drawn in terms of average dollar amount per  claim.
While relative precision of the fiscal year 2004 QIO contractor-type error  rate estimate was within acceptable statistical standards, relative  precision of most of the state-specific QIO error rate estimates was not.  (See app. II for state-specific QIO error rate information, including the  error rate estimates and corresponding relative precision.) Only three  states’ error rate estimates—Kentucky, Massachusetts, and New Mexico— had relative precision of less than 15 percent. Additionally, there was wide  variation in relative precision of the state-specific QIO error rate  estimates. Relative precision of the state-specific QIO error rates ranged  from 10.5 percent in Massachusetts to 83.3 percent in Mississippi. The  differences in relative precision of these state-specific QIO error rate  estimates indicate that the error rate estimate for the QIO that served  Massachusetts was eight times more reliable than the error rate estimate  for the QIO that served Mississippi. The variation in relative precision was  due, in part, to the sampling methods used by CMS for the HPMP. CMS  took an equal sample size for each state except Alaska, despite the fact  that there was significant variation between states in the overall volume of  discharge claims and total payments. The number of discharges per state  varied from a low of 15,166 in Wyoming to a high of 825,845 in Florida.  Similarly, total dollars paid for acute-care inpatient hospital stays varied  from less than $100 million in Wyoming to a high of $7.5 billion in  California.
Although in February 2006 a CMS official told us the agency has no plans  to reallocate the HPMP sample, CMS could adopt a similar sampling  strategy as it plans to do for the CERT Program. If future state samples  were based on the volume of discharge claims or total payments per state  and the relative precision of the state-specific QIO error rates, rather than  on the current basis of an equal allocation per state, relative precision  would likely be improved for the state-specific QIO error rates in those  states that were allocated a larger sample since relative precision  improves as sample size increases. There would also likely be decreased  variation in relative precision across all state-specific QIO error rates.  These results could be achieved without increasing the overall sample size  for the HPMP.
In addition to issues with the wide variation of relative precision of the  state-specific QIO error rate estimates, we also found large differences in  the average dollar amount per claim between the state-specific samples for  some states and the respective state populations. These differences  suggest that the samples drawn for more than half of the states were not  representative of each state’s population. Based on our examination of the  OIG contractor’s comparison of the state samples and the state  populations for June 2003, we found that the ratio of the average dollar  amount per claim in a state’s sample to the average dollar amount per  claim in a state’s population varied from 62 percent in Maryland to   143 percent in Kentucky. Twelve states had a ratio above 110 percent, and  16 states had a ratio below 90 percent. It is still possible for the national  HPMP sample to be representative of the national HPMP population even  if all of the state-specific samples are not representative of their state  populations. The larger size of the HPMP sample overall mitigates the  problems identified in the smaller state-specific samples.
section_title: Medical Record Collection Process
paragraphs: Based on our review of oversight work of the HPMP conducted by OIG,  we found that the process CMS used for collecting medical records from  providers was adequate. OIG selected 46 discharge claims that were  sampled for the HPMP to determine if the CDACs, AdvanceMed and  DynKePRO, followed established HPMP procedures for obtaining and  reviewing medical records to identify payment errors. OIG found that the  CDACs generally had appropriate controls in place to ensure that the  medical records were obtained and reviewed according to established  HPMP procedures. Of the 46 discharge claims reviewed, OIG found that in  two instances a required follow-up letter to the provider was not sent due  to an error by a substitute CDAC employee. However, the medical records  for these two discharge claims were obtained within 30 days of the  original request, which resulted in no adverse effect on the error rate  estimates. Overall, nonresponse for fiscal year 2004 represented  approximately 5.1 percent of the total QIO contractor-type error rate of   3.6 percent, or 0.2 percent of all discharge claims reviewed through the  HPMP.
The issue with providers not responding to requests for medical records  was not as significant an issue for the HPMP as it was for the CERT  Program. According to the CMS report on the fiscal year 2005 error rate,  nonresponse was less problematic in the HPMP because of several factors,  including the following: (1) providers were more likely to respond to  requests from the HPMP since the average claim value was higher than the  average claim value in the CERT Program; (2) providers were more  familiar with the HPMP than with the CERT Program; and (3) providers  were paid the cost of providing medical records by the HPMP, but not by  the CERT Program.
section_title: Identification and Categorization of Payment Errors
paragraphs: Based on our review of OIG’s fiscal year 2004 HPMP evaluation, we  concluded that the CDACs (AdvanceMed and DynKePRO) generally had  processes in place to adequately identify and categorize claims paid in  error in the HPMP for fiscal year 2004. OIG officials told us that they found  the medical record reviewers, both admission necessity reviewers and  DRG coding specialists, at the two CDACs met CMS’s qualifications for  these positions. As part of its review of the fiscal year 2004 HPMP, OIG  reviewed 46 discharge claims that were part of the sample for estimating  the QIO contractor-type error rate. Based on that review, OIG reported  that the CDACs generally had appropriate controls in place to ensure that  admission necessity and DRG validation reviews were performed in  accordance with CMS established procedures and that the results of those  reviews were adequately maintained, updated, and reported.
As part of the internal HPMP quality control process, two activities were  conducted regularly to ensure the reliability and accuracy of CDAC  reviews both within each CDAC and across the two CDACs. Each CDAC  randomly chose 30 claims per month to be reviewed by two of its medical  record reviewers for intra-CDAC tests. Each CDAC compared the results  of the two medical record reviews to determine the reliability of reviews  within the CDACs and reported the results of the comparisons to CMS.  The CDACs performed inter-CDAC tests to assess the reliability of the  reviews between the two CDACs. For these tests, an additional 30 claims  were chosen at random per quarter by each of the CDACs for review by a  medical records reviewer at the other CDAC. As part of its evaluation of  the fiscal year 2004 HPMP, OIG selected 45 claims that went through the  intra-CDAC process and 42 claims that went through the inter-CDAC  process to determine if these quality control activities ensured the  reliability of the CDAC review process. OIG reported that the quality  control reviews were generally operating effectively to ensure the  reliability of the review process and the consistency of the error rate  determination decisions.
From the same evaluation of the fiscal year 2004 HPMP, OIG found that  the CMS contractor tasked with calculating the dollar amounts paid in  error, Texas Medical Foundation, used a method that produced an amount  of dollars in error that in some cases differed from what OIG found to be  the amount of dollars in error. For claims identified by a QIO as having  errors caused by changes in DRG codes, Texas Medical Foundation used a  method that produced different dollar amounts in error than would have  been produced if it had used the software that FIs used to pay the original  discharge claims. The Texas Medical Foundation calculated a different  amount in error for about 76 percent of 200 incorrectly coded claims that  OIG reviewed. However, OIG reported that the differences did not have a  significant effect on the QIO contractor-type error rate estimate. A CMS  official told us that the agency has not invested in modifying the software  for use by the Texas Medical Foundation for technical and financial  reasons. For example, the software requires modifications using a specific  programming language for which CMS has limited personnel with the  needed expertise.
section_title: Statistical Methods
paragraphs: We verified the statistical methods CMS used to estimate the QIO  contractor-type error rate and standard error in the HPMP by reviewing  the computer programming code that produced this information. We  found that the methods CMS used were adequate because they were based  on standard statistical methods and were applied appropriately. To  estimate the QIO contractor-type error rate, CMS weighted each state- specific QIO error rate according to that state’s share of the total Medicare  FFS payments for acute-care inpatient hospital claims nationwide. This  method is referred to as a stratified mean per unit estimation. Like the  CERT Program, CMS used a standard statistical method to calculate the  standard error of the estimate. In our review of the computer  programming code that generated the QIO contractor-type error rate  estimate, we found that CMS used annual instead of monthly weights in its  estimate of the annual total dollars paid in error. It would have been  more appropriate for CMS to have used monthly weights because the  HPMP sample was drawn on a monthly, not an annual, basis. However,  when we reviewed the OIG contractor’s comparison of the estimate of  annual dollars paid in error using annual weights to what the estimate  would have been had CMS used monthly weights, we concluded that the  use of annual weights did not significantly affect the QIO contractor-type  error rate estimate. A CMS official told us and provided us with  documentation that beginning with the HPMP’s fiscal year 2005 error rate  estimation process, monthly weights are being used.
section_title: CMS Methodology Adequate for Estimating the National Error Rate
paragraphs: CMS appropriately combined the error rates under the CERT Program and  the HPMP to estimate the fiscal year 2004 national Medicare error rate.  CMS estimated the national Medicare error rate by averaging the error  rates of the four contractor types (carrier, DMERC, FI, and QIO), weighted  by each contractor type’s share of total Medicare FFS payments. Likewise,  CMS calculated the standard error, or precision, of the national error rate  based on the standard error of each of the four types of contractors’ error  rate estimates, weighted by each contractor type’s proportion of total  Medicare FFS payments. The methods CMS used to calculate the national  error rate and the standard error were statistically valid, since the units of  measurement, which in this case were Medicare claims, of the four error  rates that were combined were mutually exclusive (independent) among  contractor types. Each contractor type consisted of multiple individual  contractors. These contractors were independent in that one contractor’s  estimated error rate or standard error did not affect the estimates of other  contractors, since the claims in the population and in the sample were not  overlapping among contractors.
section_title: Concluding Observations
paragraphs: Since assuming responsibility for estimating the national Medicare error  rate in fiscal year 2003, CMS has made changes to the methodology, which  have provided CMS with more detailed information about the error,  thereby allowing the agency to better identify the underlying causes of  error and implement corrective action plans to address them. For  example, CMS significantly increased the size of the sample used to  estimate the Medicare FFS claims paid in error. The increased sample size  allowed the agency to estimate not only the error rate at the national level,  but also more detailed error rates at the contractor-type and contractor- specific levels. Further, CMS has made changes in the way it collects  medical records from providers in an effort to reduce the rate of error  caused by nonresponse and insufficient documentation. These changes  may affect the error rate estimates and thus the comparability of the  estimates over time. Consequently, users of the error rate information  should exercise caution when making year-to-year comparisons.
Our work focused on the methodology CMS used to estimate the national  Medicare error rate and contractor-type error rates for fiscal year 2004.  For these error rates, we found the methodology adequate for that year.  Under CMS’s contracting reform initiative, there will be fewer individual  contractors (carriers, DMERCs, and FIs). If CMS maintains the same  overall sample size, the sample sizes of the remaining individual  contractors would be increased. Reliability of the contractor-specific error  rate estimates is likely to improve with the larger sample sizes. Until then,  the wide variation in reliability of the contractor-specific error rate  estimates may preclude meaningful comparisons across individual  contractors.
section_title: Agency Comments
paragraphs: We received written comments from HHS (see app. III.) In responding to  our draft report, HHS noted that we found the CMS methodology adequate  for estimating the fiscal year 2004 national Medicare FFS error rate. HHS  also noted that CMS is continually committed to refining the processes to  estimate, as well as lower, the level of improper payments in the Medicare  FFS program.
In its comments, HHS noted improvement in the national Medicare error  rate from fiscal years 2004 to 2005. The department attributed the decline  in the error rate to marked improvement in the nonresponse (which CMS  now calls “no documentation”) and the insufficient documentation error  rates. Commenting on the adequacy of the fiscal year 2005 methodology  was beyond the scope of our work; however, as we noted in the draft  report, changes in the methodology may affect the estimation of the error  rates and thus the comparability of these error rates over time. For  example, we discussed in the draft report that CMS has made changes in  the way it collects medical records from providers in an effort to reduce  the rate of error caused by nonresponse and insufficient documentation.  These changes primarily affected HHS’s processes for calculating an  annual error rate estimate for the Medicare FFS program. This may  represent a refinement in the program’s estimation methodology rather  than improved accountability over program dollars.
The national Medicare error rates for fiscal years 2004 and 2005 provided  by HHS in its comments are not comparable to the error rates cited in this  report for fiscal years 2004 and 2005. HHS provided gross error rates,  which were calculated using gross dollars paid in error. Gross dollars paid  in error were calculated by adding dollars paid in error that were due to  underpayments to those that were due to overpayments. As noted in the  draft report, we reported net error rates. Net error rates were calculated  using net dollars paid in error. Net dollars paid in error were calculated by  subtracting dollars paid in error that were due to underpayments from  those that were due to overpayments.
HHS also provided technical comments, which we have addressed as  appropriate.
We are sending copies of this report to the Secretary of Health and Human  Services, the HHS Inspector General, the Administrator of CMS, and  appropriate congressional committees. We will also provide copies to  others upon request. In addition, the report is available at no charge on the  GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-7101 or steinwalda@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: We reviewed the following components of the Centers for Medicare &  Medicaid Services’s (CMS) methodology for estimating the fiscal year 2004  error rate:    Sampling methods, including sample size, sample selection, sample  representation, and precision of the estimates.
The medical records collection process.
Identification and categorization of claims payment error, including the  medical record review process and quality assurance reviews.
Statistical methods used to estimate the error rates and precision.
To conduct our analysis of CMS’s sampling methods, we reviewed work  performed by the Department of Health and Human Services (HHS) Office  of Inspector General (OIG) contractor that assessed these methods and  CMS documentation for the fiscal year 2004 Medicare error rate. For the  Comprehensive Error Rate Testing (CERT) Program, we reviewed the  program manual, which described the CERT Program sampling methods  as well as CMS’s Medicare error rate reports for fiscal years 2003 and  2004. For the Hospital Payment Monitoring Program (HPMP), we  reviewed the program manual and the HPMP computer programming code  that generated the sample to verify that the sample was taken in  accordance with the procedures outlined in the manual. Additionally, we  reviewed the OIG contractor’s comparison of the June 2003 sample and a  re-created version of the June 2003 sampling frame, or population, for the  HPMP. It was not possible for the OIG contractor to obtain the exact   June 2003 population file because the file is continuously updated and  previous versions are not retained. We did not believe it was necessary to  compare every month’s sample to the population from which it was drawn  because of the large size of the sample (approximately 40,000 discharge  claims) and population (approximately 11.5 million discharge claims), and  the fact that the sample was drawn in the same manner each month.
To conduct our analysis of CMS’s medical record collection and review  processes and identification and categorization of payment errors, we  relied primarily on reports published by OIG. Since 2003, OIG has  conducted annual reviews of the CERT Program and the HPMP as part of  its review of work performed for HHS by contractors. These annual  reviews examine whether the CERT Program and HPMP contractors have  appropriate controls in place to ensure that the medical record reviews  and quality assurance reviews were performed in accordance with  established procedures. We reviewed OIG’s annual reviews of the CERT  Program and the HPMP for fiscal year 2004. Our analysis of provider  nonresponse within the CERT Program relied on two OIG studies of CMS’s  actions to reduce nonresponse implemented for the CERT Program for  fiscal year 2004. For the HPMP, we also reviewed four intra-Clinical Data  Abstraction Center (CDAC) reports and two inter-CDAC reports, which  were quality assurance reviews intended to assess the consistency of  review decisions both within and across CDACs.
To conduct our analysis of CMS’s statistical methods, we reviewed the  OIG contractor’s computer programming code, which replicated CMS’s  estimation of the error rates for carriers, durable medical equipment  regional carriers (DMERC), and fiscal intermediaries (FI), as calculated by  the CERT Program subcontractor responsible for statistical analysis of the  error rates for fiscal year 2004. We reviewed CMS’s computer  programming code, which calculated the HPMP error rate for quality  improvement organizations (QIO). In conducting these reviews of the  computer programming codes for both the CERT Program and the HPMP,  we verified that each code appropriately implemented a methodology that  employed standard statistical principles and was used appropriately.
To inform all aspects of our study, we interviewed OIG officials with  oversight responsibility for the error rate estimation, OIG contractor staff  who conducted the evaluation of the statistical methodology, CMS  officials with programmatic responsibilities for the CERT Program and the  HPMP, and staff of the CERT Program subcontractor for statistical  analysis.
We performed our work from April 2005 through March 2006 in  accordance with generally accepted government auditing standards.
section_title: Appendix II: Fiscal Year 2004 Error Rate Information by Contractor Type—Carriers, DMERCs, FIs, and QIOs
paragraphs: Appendix II: Fiscal Year 2004 Error Rate  Information by Contractor Type—Carriers,  DMERCs, FIs, and QIOs year 2004(in dollars)
CMS estimated paid  claims error rate  (percentage)
CMS estimated  standard error(percentage) precision(percentage) 13.5 year 2004(in dollars)
CMS estimated paid  claims error rate  (percentage)
CMS estimated  standard error(percentage) precision(percentage) 4.5 year 2004(in dollars)
CMS estimated paid  claims error rate  (percentage)
CMS estimated  standard error(percentage) precision(percentage) 23.8 year 2004(in dollars)
CMS estimated paid  claims error rate  (percentage)
CMS estimated  standard error(percentage) precision(percentage)
Carriers are health insurers and pay claims submitted by physicians, diagnostic laboratories and  facilities, and ambulance service providers.
DMERCs are health insurers and pay claims submitted by durable medical equipment suppliers.
For the fiscal year 2004 error rate,TriCenturion, a program safeguard contractor, was responsible for  medical review in one of the four DMERC regions. Program safeguard contractors are Medicare  contractors that conduct activities to address or prevent improper payments. As such, it was  TriCenturion, not the DMERC, which was responsible for lowering the error rates in its region.
FIs are almost exclusively health insurers and pay claims submitted by home health agencies, non- prospective payment system (PPS) hospitals, hospital outpatient departments, skilled nursing  facilities, and hospices. PPS is a reimbursement method used by Medicare where the payment is  made based on a predetermined rate and is unaffected by the provider’s actual costs.
QIOs (formally known as peer review organizations) are responsible for ascertaining the accuracy of  coding and payment of paid Medicare FFS claims for acute care inpatient hospital stays—generally  those that are covered by PPS—for Medicare beneficiaries in all 50 states, the District of Columbia,  and Puerto Rico. Unlike carriers, DMERCs, and FIs, however, QIOs do not process and pay claims.  These activities are conducted by FIs.
section_title: Appendix III: Comments from the Department of Health and Human Services
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Debra Draper, Assistant Director;  Lori Achman; Jennie Apter; Dae Park; and Ann Tynan made key  contributions to this report.