Medicare Advantage: CMS Should Fully Develop Plans for Encounter Data and Assess Data Quality before Use
Published: 2014-07-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Medicare Advantage—the private plan alternative to the traditional Medicare program—provides health care for nearly 15.5 million enrollees, about 30 percent of all Medicare beneficiaries. After a multiyear rollout, CMS began collecting encounter data in January 2012. GAO was asked to review CMS's plans for using MA encounter data and its efforts to validate the data's quality.
This report examines (1) how the scope of MA encounter data compare with CMS's current risk adjustment data, (2) the extent to which CMS has specified plans and time frames to use encounter data for risk adjustment and other purposes, and (3) the extent to which CMS has taken appropriate steps to ensure MA encounter data's completeness and accuracy. In addition to reviewing laws, regulations, and guidance on MA encounter data collection and reporting, GAO interviewed CMS officials and representatives of MAOs. GAO also compared CMS's activities to the protocol CMS developed to validate Medicaid encounter data—comparable data collected and submitted by entities similar to MAOs.
section_title: What GAO Found
paragraphs: The Centers for Medicare & Medicaid Services (CMS) is collecting Medicare Advantage (MA) encounter data—information on the services and items furnished to enrollees—that are more comprehensive than the beneficiary diagnosis data the agency currently uses to risk adjust capitated payments to MA organizations (MAO). CMS, an agency within the Department of Health and Human Services (HHS), makes these adjustments to reflect the expected health care costs of MA enrollees. Encounter data have many more elements—including procedure codes and provider payments—from a wider range of provider types—such as home health agencies and skilled nursing facilities—thus expanding the scope of sources for diagnosis and other information.
CMS has not fully developed plans for using MA encounter data. The agency announced that it will begin using diagnoses from both encounter data and the data it currently collects for risk adjustment to determine payments to MAOs in 2015. However, CMS has not established time frames or specific plans to use encounter data for other potential purposes.
CMS has taken some, but not yet all, appropriate actions to ensure that MA encounter data are complete and accurate. (See figure.) The agency has established timeliness and frequency requirements for data submission, but has not yet developed requirements for completeness and accuracy. Also, the agency has certified nearly all MAOs to transmit encounter data. Although CMS performs automated checks to determine whether key data elements are completed and values are reasonable, it has not yet performed statistical analyses that could detect more complex data validity issues. For example, CMS has not yet generated basic statistics from the data by demographic group or provider type to identify inconsistencies or gaps in the data. Also, it has not yet reviewed medical records to verify diagnoses and services listed in encounter data or reported what it has learned about data quality to MAOs. Agency officials told GAO they intend to perform these additional quality assurance activities but have not established time frames to do so.
section_title: What GAO Recommends
paragraphs: CMS should establish specific plans for using MA encounter data and thoroughly assess data completeness and accuracy before using the data to risk adjust payments or for other purposes. While in general agreement, HHS did not specify a date by which CMS will develop plans for all authorized uses of encounter data and did not commit to completing data validation before using the data for risk adjustment in 2015.

section_title: Letter
paragraphs: Since 1990, we have designated Medicare as a high-risk program, given  its size, complexity, and susceptibility to payment errors from various  causes. Medicare’s serious long-term financial challenges heighten the  need for the Centers for Medicare & Medicaid Services (CMS) to closely  examine expenditures in the Medicare Advantage (MA) program—the  private plan alternative to the Medicare fee-for-service (FFS) program.  For the MA program, CMS contracts with MA organizations (MAO) to  provide covered services to beneficiaries who enroll in one of their plans.  As of April 2014, CMS had 571 contracts with MAOs that served nearly  15.5 million enrollees, accounting for approximately 30 percent of all  Medicare beneficiaries. The Congressional Budget Office has projected  that enrollment in MA plans will increase to about 21 million enrollees by  2023 and that Medicare payments to MAOs will grow from about   $154 billion in 2014 to about $250 billion in 2023. As the MA program  expands, setting appropriate payments to MAOs and making Medicare a  more prudent purchaser of health care services will remain critical.
Whereas Medicare pays FFS providers who submit claims for  reimbursement after services have been provided, Medicare pays MAOs  a fixed monthly amount per enrollee to cover all the services enrollees  use. CMS adjusts payments to MAOs to reflect enrollees’ expected health  care costs, with higher payments for enrollees who are predicted to be  sicker and lower payments for those predicted to be healthier—a process  known as risk adjustment. MAOs can incur losses if aggregate costs  exceed payments but can retain savings if aggregate costs are less than  payments.
Since the 1980s, the methodology for setting MAO payments has  changed several times to better reflect the expected health care costs of  plan enrollees and to mitigate the effects of potential favorable selection.  The Tax Equity and Fiscal Responsibility Act of 1982 authorized payment  for full-risk Medicare plans on the basis of a fixed per beneficiary rate set  at 95 percent of the estimated average cost of beneficiaries in Medicare  FFS. We and other researchers criticized the accuracy of the  adjustment, which resulted in excess payments to plans that enrolled  healthier beneficiaries with below-average health care costs. In an effort  to refine the payment methodology, the Balanced Budget Act of 1997  required CMS to use health status measures to adjust plan payments and  required the agency to collect inpatient hospital data for this purpose. In  implementing these requirements in 2000, CMS selected a risk  adjustment method that relied on enrollee demographics augmented with  hospital inpatient data to measure health status. This risk adjustment  method more accurately matched MA payments to enrollees’ expected  total Medicare costs compared with the previous method.
In 2002, CMS transitioned to collecting a larger set of medical data—via  the Risk Adjustment Processing System (RAPS)—to risk adjust  payments. As required by law, the agency began using information on  enrollee demographics and diagnoses collected from MAOs for services  provided in physician office, hospital inpatient, and hospital outpatient  settings to risk adjust payments in 2004. Using these data in conjunction  with FFS beneficiaries’ cost and diagnosis information, CMS developed  the hierarchical condition categories risk adjustment model that uses one  year’s diagnoses to predict the following year’s health care costs for each  MA enrollee. Although this model is better than the previous model— which estimated an MA enrollee’s health status from certain inpatient  diagnosis and demographic factors—at predicting an enrollee’s costs, it  has led to overpayments to MAOs because of different diagnostic coding  patterns between the FFS and MA programs.
In a 2008 final rule, CMS reasserted its authority to collect more detailed  data, referred to as encounter data. Such data include diagnosis and  treatment information for all medical services and items provided to an  enrollee, with a level of detail similar to FFS claims. In January 2012, after  a multiyear rollout, the agency began collecting such data from MAOs via  the Encounter Data System. In the 2008 final rule, CMS stated it would  use these data for risk adjustment and may also use them for additional  payment and oversight purposes. The agency plans to discontinue RAPS  after it begins using MA encounter data for risk adjustment purposes.
Realizing the benefits of encounter data will depend on the quality of the  data and the data’s suitability for various uses. CMS has recognized the  importance of ensuring that the data collected are complete— representing all encounters for all enrollees—and accurate—representing  a correct record of all encounters that occurred—given the important  functions to which the data will be put. Furthermore, CMS has stated that  the encounter data submitted by MAOs would be used once the agency  has reviewed and analyzed the submissions to identify and address any  data issues.
You asked us about CMS’s plans for using MA encounter data and its  efforts to validate the completeness and accuracy of the data. In this  report, we examine  1.  how the scope and reporting frequency of MA encounter data  compare with CMS’s current MA risk adjustment data,  2.  the extent to which CMS has gathered MA encounter data and  specified plans and time frames to use encounter data for risk  adjusting MA payments and other purposes, and  3.  the extent to which CMS has taken appropriate steps to ensure the  completeness and accuracy of MA encounter data.
For all objectives, we reviewed relevant laws, regulations, and guidance  pertaining to MA encounter data collection and reporting and RAPS data  submissions. We also reviewed documents from meetings that CMS has  held with MAOs and from public presentations on MA encounter data  collection requirements, the implementation schedule, and specific MAO  concerns. In addition, we interviewed CMS officials in the Medicare Plan  Payment Group and other agency components, as well as CMS  contractors. We also interviewed representatives of America’s Health  Insurance Plans and its member MAOs.
To determine the extent to which CMS has taken appropriate steps to  ensure the completeness and accuracy of MA encounter data, we  compared the agency’s activities to the principal activities identified in its  2012 protocol for validating Medicaid encounter data that states receive  from managed care organizations (MCO)—entities that provide Medicaid  benefits in exchange for a fixed monthly payment. The protocol specifies  a procedure for assessing the completeness and accuracy of encounter    data that Medicaid MCOs are required to submit to state agencies. Because of the similarities in the type of data—encounter data—and  entities gathering the data—managed care entities—we refer to the  protocol in assessing CMS’s actions to ensure MA encounter data  quality.external quality review organizations with experience using the protocol  and reviewed their documents and other reports on Medicaid encounter  data validation.
To learn about the practical uses of the protocol, we interviewed  We conducted this performance audit from July 2013 to June 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: MA Encounter Data Collection Process
paragraphs: Although MAOs are responsible for collecting and transmitting encounter  data to CMS, they rely on the cooperation of providers to submit complete  and accurate data that conform to agency requirements. MAOs gather  encounter data—which originate from the information in an enrollee’s  medical record—for their own management and payment purposes. While  Medicare pays MAOs a fixed monthly amount per enrollee, MAOs may  compensate providers for services rendered to enrollees under different  contractual arrangements. Providers paid by their plans on a FFS basis  submit claims for payment that include the amounts to be reimbursed by  the MAO.basis—that is, that receive a designated amount to cover all services for  an assigned enrollee—record a zero if no payment amount information is  available. MAOs may include provisions in their contracts with providers  that require submission of complete and accurate encounter data.
In contrast, providers paid by their plans on a capitated  After collecting and reviewing these data, MAOs transmit the data to  CMS. CMS adopted the health insurance industry’s standard claims  format for reporting MA encounter data. MAOs must use the Health  Insurance Portability and Accountability Act of 1996-compliant Accredited  Standards Committee X12 Version 5010 format, which all providers and  private and public health plans are required to use for electronic claim   By using this standard format, CMS generally avoided  submissions.placing a new requirement on MAOs.
In January 2012, CMS began phasing-in MA encounter data collection by  type of provider. It began by receiving data on professional encounters  (such as services performed by physicians), then institutional encounters  (such as services performed during an inpatient hospital stay), and finally  durable medical equipment (such as hospital beds and wheelchairs). By  August 2012, MAOs were submitting data from all types of providers.  Although CMS requires that MAOs submit encounter data for all items  and services provided to enrollees, the file formats that CMS currently  uses preclude MAOs from transmitting data on the utilization of certain  supplemental MA benefits, such as dental and vision services. Agency  officials told us they are considering how to address this discrepancy.
section_title: Risk Adjustment
paragraphs: To risk adjust payments to MAOs, CMS calculates a risk score—the  expected health care expenditures for an enrollee compared with the  average health care expenditures of all beneficiaries—for each MA  enrollee and Medicare FFS beneficiary. CMS calculates risk scores for  MA enrollees using diagnosis information provided by MAOs along with  cost and utilization information from Medicare’s FFS claims systems. This  approach assumes that diagnostic coding practices are the same in MA  and FFS. Risk scores for beneficiaries with the same diagnoses and  characteristics should be identical, regardless of whether the beneficiaries  are in MA or Medicare FFS. However, MAOs have a greater incentive  than FFS providers to make sure that all diagnoses are coded, as this can  increase enrollees’ risk scores and ultimately the payments plans receive.  In part because of this incentive, risk scores for MA enrollees may tend to  be higher relative to the risk scores of FFS beneficiaries who are in  similar health and have identical characteristics.
To help ensure that MAOs are not overpaid as a result of these  differences in diagnostic coding, CMS makes a separate adjustment to  payments to MAOs. In 2012 and 2013, we reported inaccuracies in  CMS’s methodology for adjusting MA payments. Specifically, we  estimated in 2013 that MAOs received excess payments of between   $3.2 billion and $5.1 billion from 2010 through 2012 because CMS’s  adjustment to account for differences in risk scores was too low. We  recommended that the CMS Administrator take steps to improve the  accuracy of the adjustment made for differences in diagnostic coding  practices between MA and Medicare FFS. The agency did not formally  comment on this recommendation and, as of March 2014, it has not  improved the accuracy of this adjustment. Consistent with our finding that  this adjustment was too low, Congress has taken steps to increase the  statutory minimum annual adjustment.
section_title: State Medicaid Agency Uses of and Requirements for Encounter Data
paragraphs: State Medicaid agencies that contract with MCOs typically require them to  report encounter data to the state for various purposes.encounter data collection, the agencies may also establish requirements  regarding the timeliness, completeness, and accuracy of Medicaid  encounter data in contracts with MCOs. For example, to encourage timely  submission and promote data quality, they may require MCOs to submit  data within a certain time frame and attest that the data submitted are  complete and accurate. State Medicaid agencies may then use these  data for a variety of purposes, such as setting payment rates, evaluating  the performance of MCOs, and providing reports to their legislatures and  the public.
State Medicaid agencies may assess encounter data to determine  whether MCOs meet requirements for complete and accurate reporting.  For example, they may calculate the proportion of encounter data files  denied or examine the rates of services used per enrollee against specific  benchmarks—target rates. As another example, they may require MCOs  to list encounter data that were corrected or voided and provide reasons  for these actions. They may also have the medical records examined to  verify that the information submitted in encounter data is complete and  accurate. For example, this review may determine whether services  submitted in encounter data were performed or confirm that enrollees  have specific diagnoses.
section_title: MA Encounter Data Are More Comprehensive and Are Reported More Frequently than Current Risk Adjustment Data
paragraphs: Compared with information available in RAPS data, MA encounter data,  with more elements reported, provide CMS with more comprehensive  information on all enrollee diagnoses as well as the cost and types of  services and items provided to enrollees. Each RAPS data file—an  electronic record that contains an enrollee’s diagnosis information from  one or more providers—contains between 1 and 10 diagnosis groupings.  Although all diagnoses are included in these groupings, not all groupings  are used in CMS’s risk adjustment model. For each diagnosis grouping,  there are 9 RAPS data elements, including the “from” and “through” dates  of service, the diagnosis code, and the provider type. Each RAPS data  file also includes 31 other elements that are used for data processing and  other purposes. Thus, depending on the number of diagnosis groupings,  each RAPS data file contains between 40 and 121 data elements. In  contrast, each encounter data file—an electronic record that contains  detailed information for each medical service and item provided to an  enrollee—includes approximately 200 data elements. These elements  include information on the patients, providers, and payers of services and  items, as well as the dates of service and procedure codes. They also  include information on the primary and supplemental diagnoses  associated with the service or item. Whereas RAPS data are limited to 10  diagnosis groupings, encounter data can include up to 12 diagnoses for  professional services and 25 diagnoses for institutional services. Table 1  summarizes the types and number of encounter data elements.
In addition, MA encounter data are more comprehensive than RAPS data  because they include information that originates from a wider range of  provider types. Specifically, only physicians and hospital inpatient and  outpatient facilities report enrollee diagnoses for RAPS data collection. Of  these provider types, physicians provide approximately 80 percent of the  diagnostic information used for risk adjustment, according to one  estimate. As a result, CMS’s current risk adjustment model relies  primarily on diagnoses that physicians report. In contrast, more provider  types report encounter data, significantly expanding the scope of sources  for diagnosis and other information. In addition to physicians and hospital  inpatient and outpatient facilities, many other provider types—including  ambulance providers, clinical laboratories, durable medical equipment  suppliers, home health providers, mental health providers, rehabilitation  facilities, and skilled nursing facilities—report encounter data.
The fact that CMS has taken a comprehensive approach in collecting  encounter data has raised concerns for some MAOs. Because MAOs  generally gather such data from certain providers for payment and plan  management purposes, the CMS encounter data reporting requirements  may not have significantly added to their ongoing data gathering  activities. However, industry representatives noted that submitting  encounter data from a broad array of provider types may not add to  information on diagnoses CMS uses for risk adjustment. Agency officials  acknowledged that many encounter data elements MAOs report will not  be used for risk adjustment but will allow CMS to more completely identify  the services furnished to enrollees.
Finally, CMS requires MAOs to submit encounter data more frequently  than RAPS data. MAOs submit RAPS data at least quarterly to CMS, with  each submission representing approximately one-fourth of the RAPS data  an MAO submits during the year. In contrast, CMS requires MAOs to  submit encounter data weekly, biweekly, or monthly depending on their  number of enrollees. MAOs with more than 100,000 enrollees must  submit encounter data on a weekly basis, those with between 50,000 and  100,000 enrollees on a biweekly basis, and those with fewer than 50,000  enrollees on a monthly basis. CMS also requires MAOs to submit  encounter data within 13 months from the date of service.  Representatives of MAOs noted that they are able to meet CMS’s  encounter data frequency requirements, and one MAO told us that it  sometimes chooses to submit encounter data more frequently—three  times a week rather than once a week—to reduce the size of each  submission.
section_title: With MA Encounter Data at Expected Volume, CMS Will Begin Using Some Data for Risk Adjustment in 2015, but Its Plans for Other Uses Remain Undeveloped
paragraphs: CMS is receiving MA encounter data from nearly all MAOs on all types of  services at the monthly volume that CMS officials told us they expected.  After increasing significantly from August 2012—the first month that  MAOs submitted encounter data on services from all provider types— through May 2013, the volume of MA encounter data has had some  fluctuations from month to month. (See fig. 1.) Specifically, the volume of  encounter data files—electronic records containing detailed information  for each medical service and item provided to an enrollee—rose sharply  from 2.0 million data files in August 2012 to 49.6 million data files in May  2013. From June 2013 through April 2014, the volume has fluctuated  between 34.2 and 54.2 million data files each month. CMS officials told  us that they expect to continue receiving encounter data files at a rate of  approximately 40 to 50 million files per month throughout 2014. In total,  MAOs submitted about 497.9 million encounter data files in 2013  comprising approximately 416.5 million professional encounter data files,  64.9 million institutional files, and 16.5 million durable medical equipment  files.
In April 2014, CMS announced that it will use MA encounter data as part  of risk adjustment and no longer rely solely on RAPS data. The agency  will start using the diagnosis information from MA encounter data as well  as the diagnoses in RAPS data from 2014 dates of service when  calculating 2015 enrollee risk scores.both data sources to obtain diagnoses reported from hospital inpatient  and outpatient facilities and physicians. Further, the agency announced   Specifically, CMS intends to use  that it will use all diagnoses equally, whether they came from RAPS data  or MA encounter data, when calculating risk scores. CMS officials said  that RAPS data generally have all MA enrollees’ diagnoses, so they do  not anticipate receiving a significant amount of additional diagnoses from  MA encounter data. CMS officials described these efforts to begin using  diagnoses from MA encounter data as a key step in making full use of the  data in risk adjusting payments. Furthermore, officials stated that 2015— and perhaps subsequent years—will serve as a time to transition from  using RAPS data to using MA encounter data to calculate risk scores.  Once MA encounter data are deemed sufficiently complete and accurate  for use in risk adjusting payments, CMS plans to discontinue RAPS data  collection and transition entirely to using MA encounter data.
Although CMS identified a number of other potential uses for MA  encounter data in the 2008 final rule, how it would use the data for any of these additional purposes.  Adequately developing plans for encounter data, and communicating  them to MAOs, may improve CMS’s efforts to manage aspects of the  Medicare program. Agency officials told us that they have deferred  planning efforts to use MA encounter data for any purposes besides using  it as a different way to collect diagnosis information for the current risk  adjustment model. Accordingly, CMS’s plans remain undeveloped and it  has not established specific time frames for any of the following potential  uses outlined in 2008:   the agency has yet to determine
Revise CMS’s risk adjustment model for MA payments. CMS  intends to improve its model for risk adjustment using MA enrollee  cost, diagnoses, and utilization information. However, agency officials  noted that developing such a model is an involved process and would  take a number of years to complete.
In the 2008 final rule, CMS listed the following potential data uses without further  explanation: (i) updating risk adjustment models, (ii) calculating Medicare disproportionate  share percentages, (iii) conducting quality review and improvement activities, and (iv)  Medicare coverage purposes. 73 Fed. Reg. 48434 (Aug. 19, 2008).
Conduct quality review and improvement activities. Such activities  refer to assessing the performance of providers and MAOs in  delivering care. For example, CMS could use encounter data to  develop new MA quality measures.
Calculate Medicare disproportionate share hospital percentages.
CMS increases payments to hospitals serving a disproportionate  number of low-income patients through the disproportionate share  hospital adjustment. The agency currently incorporates the number of  hospital days for MA enrollees in its calculation of disproportionate  share hospital percentages. CMS officials have not specified how the  agency could use MA encounter data to modify how it calculates the  disproportionate share hospital percentages.
Monitor Medicare coverage. Such monitoring may refer to using  encounter data to determine whether an MA enrollee has reached the  maximum out-of-pocket limit for an enrollee’s cost sharing each year.
Without developing plans for these additional uses of encounter data,  CMS cannot determine whether it is gathering the proper amount and  types of information required for these purposes. Additionally, the agency  would not be able to establish management priorities, and therefore  cannot ensure that it is using the data to their full potential.
In a May 2014 proposed rule, CMS identified a number of additional uses  of encounter data designed to strengthen program management and  increase transparency in the MA program. These were the following:
Conduct program evaluations. CMS noted that it may use  encounter data to assess the MA program and demonstration  designs. The data could also be used for government-sponsored  public health initiatives and academic health care research.
Support program integrity efforts. CMS indicated that encounter  data could be used to conduct audits, investigations, and other efforts  to combat waste, fraud, and abuse undertaken by the Office of  Inspector General or CMS.
Aid program administration. CMS stated that it may use encounter  data in reviewing the validity of MAO bid submissions—projected  revenue for providing standard Medicare services to an average  enrollee in an MAOs’ service area. It may also use the data to verify  MAO information on medical loss-ratios—the percentage of revenue  used for patient care, quality improving activities, and reduced  premiums—to determine whether MAOs have met minimum  requirements.
section_title: CMS Has Performed Some Steps to Ensure That MA Encounter Data Are Complete and Accurate but Has Not Yet Fully Validated the Data
paragraphs: As of May 2014, CMS had taken some, but not yet all, appropriate  actions—as outlined in its Medicaid encounter data validation protocol— to ensure that MA encounter data are complete and accurate before they  are used. (See fig. 2.) CMS’s actions to date have focused on  communicating with MAOs about data submission requirements,  certifying MAOs’ capability to transmit the data, and conducting  automated checks for completeness and accuracy of the data. However,  the agency has not yet conducted statistical analyses, reviewed medical  records, and provided MAOs with summary reports on data quality  referenced in its Medicaid encounter data validation protocol. Although  CMS intends to perform these additional quality assurance activities,  CMS officials have not specified a time frame for doing so.
subsections: section_title: CMS Has Established and Publicized MA Encounter Data Submission Requirements but Has Not Developed Benchmarks for Completeness and Accuracy
paragraphs: CMS has established certain requirements for the MAOs’ collection and  submission of encounter data. The initial step in CMS’s Medicaid  encounter data validation protocol is to establish such features as the  data submission format, the data elements to be reported, the time  frames for data submission, and the benchmarks—target rates—for  completeness and accuracy of the data. CMS required that MAOs use the  reporting format that is standard in the health insurance industry, and  MAOs began submitting files in that format in January 2012. CMS also  required that MAOs submit encounter data—including any corrections to  the data—within specified time frames and at intervals determined by  their size.
To enhance compliance with its encounter data requirements and thus  promote complete and accurate data submissions, CMS has engaged in  regular and open communication with MAOs since 2010. Agency officials  have regularly held meetings with MAOs to discuss encounter data  system enhancements, changes in data submission requirements, and  concerns on specific encounter data topics. Although MAOs noted that  they would like to continue receiving detailed information frequently— particularly on the more complex aspects of the encounter data  submission process—CMS reduced the frequency of the encounter data  user group meetings during the summer of 2013. According to CMS  officials, MAOs’ increased familiarity with data submission requirements  reduced the need for ongoing guidance. Other outreach methods the  agency has used include newsletters and bulletins with information such  as the most common data errors, a technical assistance support desk to  address specific reporting circumstances, and a website for posting  summaries and questions from meetings with MAOs. To maintain detailed  guidance on data submission requirements, CMS produced and updates  an encounter data manual. (See app. I for more information about CMS’s  outreach efforts with MAOs.)
While it has focused its efforts on educating plans about reporting  requirements, CMS has not yet established benchmarks for encounter  data completeness and accuracy and does not have a timeline for  developing such benchmarks. As stated in the Medicaid encounter data  validation protocol, each data field submitted for each encounter type  should have an acceptable rate of completeness and accuracy. The  protocol states that each plan’s target error rate should be below   5 percent, composed of the percentage of missing, duplicate, or incorrect  records. In 2012, CMS stated it would develop error frequency  benchmarks but has yet to do so. In May 2014, agency officials told us  that as they gain more experience with the data submissions they will use  that information to develop such performance benchmarks. In contrast,  state Medicaid agencies have established various benchmarks for  Medicaid encounter data, which could help inform the development of  similar benchmarks for MA encounter data. For example, New Jersey  permits MCOs to have no more than 2 percent of encounter data  submissions denied each month. Alternatively, Arizona does not allow the  payment information in MCOs’ encounter data submissions to vary from the financial reports submitted by MCOs by more than 3 to 5 percent.Without benchmarks, as suggested by the protocol, the agency has no  objective standards against which it could hold MAOs accountable for  complete and accurate data reporting.
section_title: CMS Has Certified MAOs to Ensure Their Capability to Successfully Transmit Encounter Data
paragraphs: CMS has certified nearly all MAOs to verify their capability for collecting   Under CMS’s protocol for validating  and submitting encounter data.Medicaid encounter data, each MCO’s information system should be  assessed to see whether it is likely to successfully capture and transmit  the encounter data. Such a review would determine where the plan’s  information system may be vulnerable to incomplete or inaccurate data  capture, storage, or reporting. To become certified for submitting  encounter data, CMS requires an MAOs’ information system undergo  complete end-to-end testing. MAOs needed to separately demonstrate  that they can successfully transmit encounter data collected from  institutional, professional, and durable medical equipment providers.  During official testing, MAOs had to achieve at least a 95 percent  acceptance rate for each type of encounter. Also, MAOs were required to  sign an agreement with CMS attesting that they will ensure that data in  every submission can be supported by a medical record and that the data  are complete and accurate.
section_title: CMS Performs Automated Checks for Completeness and Accuracy of MA Encounter Data
paragraphs: CMS performs automated checks to determine encounter data quality  and identify submission issues, such as whether certain data elements  are missing, and sends automated notifications to MAOs. According to  CMS’s protocol for validating Medicaid encounter data, performing  electronic checks on encounter data is a key step in verifying data quality.  Such a data review should include basic checks of the data files by  provider type and for each of the data elements in the files. Under the  protocol, the standard data review process should be automated to verify  that, among other things,  critical data elements are not missing and are in the correct format, data values are consistent across data elements and are within  the volume of data aligns with enrollment figures.
CMS performs over 1,000 automated checks to verify that data elements  are present, the values of data elements are reasonable, and the data are  not duplicated. According to CMS, the number of errors returned to MAOs  and the number of duplicate data files submitted per MAO can illustrate,  at least in part, the completeness and accuracy of the encounter data  submitted. For example, CMS performs automated checks to flag missing  data elements—such as a contract ID number or a date of service—in  encounter data files and returns files with missing elements to MAOs.  CMS also determines whether encounter data elements have the correct  alphanumeric values (e.g., that zip codes contain only numeric  characters) and verifies whether the values are consistent across data  elements (e.g., the date of service is before the date of data submission).  In addition, CMS ensures that the same data file is not accepted into the  Encounter Data System more than once, which would result in duplicate  data files.
After performing these automated checks, CMS sends MAOs up to five  types of automated reports with details about the encounter data  submission, including information on the data received. These reports  provide information on whether errors occurred and whether encounter  data processing can continue so that MAOs can identify problems with  their data. CMS may provide additional reports if MAOs need to resubmit  the data to meet CMS’s requirements. (See app. II for a summary of  CMS’s automated reports for MAOs.) Although representatives from  MAOs noted that CMS has not always produced the automated reports  on a timely basis, CMS officials told us that they now produce the reports  within 1 week of encounter data submission. According to the MAO  representatives, the reports are generally useful, straightforward, and  easy to understand.
CMS plans to develop four other automated reports related to data  submission and processing but has not established a time frame for doing  so. These reports are expected to determine the number of encounter data files accepted and rejected  identify encounter data submission errors and summarize the type of  errors found, provide additional details on rejected encounter data submissions, ascertain diagnoses that CMS accepted and will use to calculate risk  scores.
Although CMS has had plans since at least October 2010 to develop a  report that identifies those diagnoses that it will use to calculate risk  scores, it still has not produced the report. MAOs we interviewed  expressed concern that CMS has not developed this report, which could  help them understand how encounter data will be incorporated into  CMS’s risk score calculation.
section_title: CMS Has Not Yet Conducted Statistical Analyses Using MA Encounter Data
paragraphs: Although CMS is aware of the need to perform statistical analyses of MA  encounter data, the agency has not yet conducted these analyses.  According to CMS’s protocol for validating Medicaid encounter data,  analyzing values in specific data elements, generating basic statistics on  the volume and consistency of data elements, and periodically compiling  and reviewing statistics on utilization rates can help detect data validity  issues. In March 2014, CMS officials told us that they were able to only  summarize the volume of encounter data files by MA plan. The officials  stated that they would like to be able to determine the frequency of  diagnoses per encounter data file, examine the dates of service on  encounter data files, and compare MA encounter data with FFS claims  data by the summer of 2014. However, the officials noted that they have  not developed a specific time frame for performing statistical analyses  and that determining which analyses to perform will be an iterative  process over the next year or two. It appears that CMS will use encounter  data to calculate 2015 risk scores and pay MAOs before it performs  statistical analyses of the data.
As CMS begins to analyze the quality of MA encounter data, several  types of statistical analyses outlined in its Medicaid validation protocol  may help detect potentially inaccurate or unreliable data. Such analyses  include the following:
Determining whether the frequency of values within an  encounter data element is reasonable. For example, a frequency  distribution for the place of service variable would be expected to  include a reasonable distribution between inpatient hospital,  outpatient hospital, and physician office visits. This type of analysis  can help detect whether a value is missing, underreported, or  overreported.
Generating basic statistics from the encounter data. For example,  analyzing the rates of outpatient services by provider zip code could  help discover that encounter data files are missing certain zip codes,  indicating a certain amount of underreporting. To examine the  completeness of Medicaid encounter data, a CMS contractor  assessed the average number of encounter data files per enrollee and  the percentage of enrollees with data files. As another example,  MAOs reporting an unexpectedly large number of a particular service  may suggest overreporting.
Analyzing encounter data elements by demographic group,  provider type, and service type. For example, analyzing encounter  data by enrollees’ gender would enable CMS to verify that data files  for gender-specific diagnoses and procedures are logical. Analyzing  encounter data by provider type helps identify missing or erroneous  data for specific provider types or discover fluctuations in enrollee  visits. For example, dramatic changes in utilization from one time  period to another may indicate erroneous data. Finally, analyzing  encounter data by service type also helps to examine the relationship  between (1) ancillary services (e.g., laboratory tests and x-rays)   and enrollee visits, (2) primary care and specialty care visits, or   (3) outpatient services and inpatient admissions.
Comparing encounter data with other Medicare data sources,  MAOs’ financial reports, or other benchmarks. Comparing MA  encounter data with Medicare FFS data may help determine whether  differences exist in hospital admission rates between the MA and FFS  programs. In addition, comparing the volume of encounter data to  financial reports helps reveal gaps in the data.monitoring MAOs’ encounter data volumes for various service types  against established benchmarks helps identify data submission  problems, according to an actuarial firm that assessed Medicaid   Furthermore,   Moreover, New Jersey examines the  encounter data for California.rate of services used per 1,000 MCO enrollees against benchmarks  for 28 service categories, such as laboratory services, to assess data  completeness.
section_title: CMS Has Not Yet Reviewed Medical Records to Verify MA Encounter Data
paragraphs: CMS has not yet performed MA enrollees’ medical record reviews, which  typically involve comparing a sample of encounter data with the clinical  information contained in enrollees’ medical records. According to CMS’s  protocol for validating Medicaid encounter data, medical record reviews  can help confirm the findings generated in the encounter data analyses.
Use of Medical Record Reviews in a Medicaid Encounter Data Validation Study  In a 2010 Medicaid encounter data validation study for Georgia, the Health Services  Advisory Group examined the extent to which services documented in enrollees’ medical  records were absent in their encounter data and the extent to which services  documented in enrollees’ encounter data were absent from their medical records. It also  evaluated the accuracy of the diagnosis and procedure codes in encounter data by  comparing them with documentation in enrollees’ medical records. The findings showed  that enrollees’ medical records generally supported encounter data files in Georgia.  Specifically, 93.9 percent of the dates of service, 86.7 percent of the diagnosis codes,  and 78.5 percent of the procedure codes from the encounter data had supporting  evidence in the medical records. However, not all services documented in the medical  records were found in encounter data files. Specifically, 20.5 percent of the dates of  service, 49.3 percent of the diagnosis codes, and 40.1 percent of the procedure codes  documented in enrollees’ medical records were omitted from encounter data files.
Acknowledging the necessity of medical record reviews, CMS officials  noted that the agency plans to review medical records as part of its MA  risk adjustment data validation auditing process when it begins using  encounter data for risk adjustment purposes. However, as of May 2014,  the agency has not developed specific plans or time frames to include  these data in this auditing process. CMS currently uses this auditing  process to validate the accuracy of RAPS data. Specifically, CMS  requires MAOs to provide medical records to substantiate the information  in a selected sample of MA enrollees’ RAPS data. On the basis of the  clinical information contained in enrollees’ medical records that  substantiates the information in their RAPS data, CMS calculates  corrected risk scores and calculates payment errors—which can  represent a net overpayment or underpayment. Some MAOs may have  been overpaid and may need to refund Medicare payments when  enrollees’ medical records do not provide evidence for the risk-adjusted  payment they had received from CMS.
section_title: CMS Has Not Yet Provided MAOs with Summary Reports on the Completeness and Accuracy of Encounter Data
paragraphs: Because CMS has not yet conducted statistical analyses and reviewed  medical records, it cannot report information from these activities to each  MA plan. CMS’s protocol for validating Medicaid encounter data highlights  the importance of summarizing information, reporting findings, and  providing recommendations to plans for improving the completeness and  accuracy of encounter data. Once the data are validated, MAOs could  use summary reports from CMS in several ways, such as monitoring  quality improvement and service utilization and managing their enrollees  and providers. For example, the Health Services Advisory Group’s 2010  encounter data validation study for Georgia evaluated the completeness  and accuracy of encounter data submitted by three MCOs. The study  included plan-specific tables on the number and percentage of encounter  files by place of service with statewide comparisons, the number and  percentage of valid encounter files for age- and gender-appropriate  diagnoses and services, and the number of encounter files by month of  service. Without this type of information, CMS and MAOs cannot ensure  the completeness and accuracy of MA encounter data.
section_title: Conclusions
paragraphs: The collection of MA encounter data provides CMS with the opportunity to  improve the accuracy of Medicare payments to MAOs and to monitor  specific health care services used by enrollees. For MA encounter data to  reach its full potential, it is critical that CMS develop a clearly defined  strategy—which includes specific actions, priorities, and milestones—for  using the information and validating the completeness and accuracy of  the data. Deciding how the data are to be used influences which data  elements need to be collected and, in turn, the extent of data reporting.  However, CMS has yet to develop specifics on how or when it will use  encounter data for a variety of program management purposes.
Although CMS has decided to use MA encounter data to supplement  RAPS data in calculating risk scores in 2015, this use may inappropriately  increase payments to MAOs. Using diagnoses from both sources can  only increase the number of diagnoses reported for MA enrollees. This  has the potential to widen the existing discrepancy between FFS and MA  in the coding of diagnoses. As we previously recommended, CMS should  take steps to improve the accuracy of the adjustment made for  differences in diagnostic coding practices between FFS and MA to help  ensure appropriate payments to MAOs. As of March 2014, CMS has not  acted on our recommendation. To the extent that CMS continues to pay  MAOs under its current risk adjustment system and not adequately  account for diagnostic coding differences, excess payments to MAOs  could grow.
Furthermore, CMS’s decision to use MA encounter data in 2015 may be  premature because the agency has not yet fully validated the data. CMS  has taken steps to validate the data, such as establishing certain  reporting requirements and performing automated data checks. However,  the agency has not yet completed other steps in assessing whether the  data are suitable for use, such as performing statistical analyses,  reviewing medical records, and providing MAOs with summary reports of  its findings. Without fully validating the completeness and accuracy of MA  encounter data, CMS and MAOs would be unable to confidently use  these data for risk adjustment or any other program management or  policy purposes. Until CMS determines that encounter data are  sufficiently complete and accurate to be used for risk adjustment and  other intended purposes, the potential benefits of using these data will be  delayed.
section_title: Recommendations for Executive Action
paragraphs: To ensure that MA encounter data are of sufficient quality for their  intended purposes, the Administrator of CMS should establish specific plans and time frames for using the data for all  intended purposes in addition to risk adjusting payments to MAOs and  complete all the steps necessary to validate the data, including  performing statistical analyses, reviewing medical records, and  providing MAOs with summary reports on CMS’s findings, before  using the data to risk adjust payments or for other intended purposes.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to HHS for comment. In its written  response, HHS agreed that establishing plans for using encounter data  and performing steps to validate the data are important activities.  However, it did not address the specific recommendations we made. (See  app. III.)
Regarding our first recommendation—to establish specific plans and time  frames for all uses of MA encounter data—HHS stated that it agreed that  the agency should establish plans for using the data. HHS noted that it  has developed lists of the purposes for which it intends to use the data.  Current regulations authorize four purposes—risk adjustment, quality  improvement, Medicare disproportionate share percentages, and  Medicare coverage monitoring. HHS recently published a proposed rule  that outlines several additional purposes—program evaluation, program  integrity, program administration, and others. HHS stated that once the  proposed rule is finalized, CMS will be in a position to establish specific  plans and time frames for the list of additional permissible uses of MA  encounter data. However, CMS has had the authority since 2008 to use  MA encounter data for the initial four purposes, thus it remains unclear if  and when CMS will develop detailed plans for these earlier authorized  uses.
Regarding our second recommendation—to fully validate MA encounter  data before use for any purpose—HHS stated that it will continue its data  validation activities while it uses the data for risk adjusting payments to  MAOs. HHS agreed that performing statistical analyses and providing  summary reports are important elements of data validation. HHS stated  that it will undertake medical record reviews—which are used to verify  encounter data—through its ongoing risk adjustment validation audits of  RAPS data. We remain concerned, however, that CMS intends to use  2014 MA encounter data to support its risk score calculations for 2015  before it completes all data validation activities suggested by the  Medicaid protocol. Furthermore, CMS’s reliance on risk adjustment  validation audits of RAPS data may not be adequate to confirm the  completeness and accuracy of encounter data given significant  differences—in the volume and breadth of data elements—between the  two data collection efforts.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Administrator of CMS,  interested congressional committees, and others. In addition, the report  will be available at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or cosgrovej@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.
section_title: Appendix I: Select Centers for Medicare & Medicaid Services (CMS) Outreach Efforts on Medicare Advantage (MA) Encounter Data
paragraphs: Description  CMS organized this meeting with Medicare Advantage  organizations (MAO) to disseminate information on the  requirements for encounter data submission, the transition  to submitting encounter data, and the schedule for  encounter data implementation.
CMS prepared this reference guide to provide MAOs with  information about collecting encounter data from health  care providers and submitting encounter data to CMS. The  reference guide also includes information about encounter  data processing that CMS performs.
CMS prepared these guides, which it regularly updates, to  provide technical assistance on conducting Health  Insurance Portability and Accountability Act of 1996- compliant electronic transactions.
December 2010 through March 2011  CMS organized these meetings between CMS and MAOs  to discuss specific MA encounter data topics and  concerns. CMS also published questions and answers  from these meetings.
CMS organized these meetings to provide MAOs with  continued guidance on submitting encounter data. CMS  also published questions and answers from these  meetings.
CMS organized these meetings to provide updates  regarding decisions the agency made regarding encounter  data implementation.
CMS prepares bulletins with updated information on  encounter data processing, such as the most common  errors in submitting encounter data. CMS also prepares  newsletters with information on policy updates.
The CSSC help line provides encounter data assistance to  MAOs by a toll-free telephone number.
MAOs receive updated information about encounter data  submission through the CSSC website.
Palmetto GBA is CMS’s contractor that manages the collection of MA encounter data and the CSSC  help line.
section_title: Appendix II: Summary of Automated Reports for Medicare Advantage (MA) Encounter Data Submissions
paragraphs: Appendix II: Summary of Automated Reports  for Medicare Advantage (MA) Encounter Data  Submissions Report developer  The Accredited Standards Committee X12  Notify MA plans about problems with the  submission of encounter data files.
Example  The Centers for Medicare & Medicaid  Services (CMS) rejects an encounter data  file that contains an error in a processing- related data element.
Determine the syntactical accuracy of  encounter data elements.
CMS alerts an MA plan that letters appear  in a numeric-only data element.
Acknowledge the acceptance or rejection of  encounter data files.
CMS notifies an MA plan about a problem  with an encounter data submission.
Provide information on whether encounter  data files and service-related data elements  have been accepted or rejected for further  processing.
CMS informs an MA plan that an encounter  data file was rejected for further processing  while another one was accepted.
Notify MA plans about duplicate encounter  data files and service-related data elements.
CMS tells an MA plan that an encounter  data file includes duplicate services or is  the duplicate of another data file.
section_title: Appendix III: Comments from the Department of Health and Human Services
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Rosamond Katz, Assistant  Director; Manuel Buentello; David Grossman; Elizabeth T. Morrison; and  Hemi Tewarson made key contributions to this report.
section_title: Related GAO Products
paragraphs: Medicare: Contractors and Private Plans Play a Major Role in  Administering Benefits. GAO-14-417T. Washington, D.C.: March 4, 2014.
Medicare Advantage: 2011 Profits Similar to Projections for Most Plans,  but Higher for Plans with Specific Eligibility Requirements. GAO-14-148.  Washington, D.C.: December 19, 2013.
High-Risk Series: An Update. GAO-13-283. Washington, D.C.: February  2013.
Medicare Advantage: Substantial Excess Payments Underscore Need for  CMS to Improve Accuracy of Risk Score Adjustments. GAO-13-206.  Washington, D.C.: January 31, 2013.
Medicare Advantage: CMS Should Improve the Accuracy of Risk Score  Adjustments for Diagnostic Coding Practices. GAO-12-51. Washington,  D.C.: January 12, 2012.
Medicare Advantage: Changes Improved Accuracy of Risk Adjustment for  Certain Beneficiaries. GAO-12-52. Washington, D.C.: December 9, 2011.