Fraud Detection Systems: Centers for Medicare and Medicaid Services Needs to Ensure More Widespread Use
Published: 2011-06-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO has designated Medicare and Medicaid as high-risk programs, in part due to their susceptibility to improper payments--estimated to be about $70 billion in fiscal year 2010. Improper payments have many causes, such as submissions of duplicate claims or fraud, waste, and abuse. As the administrator of these programs, the Centers for Medicare and Medicaid Services (CMS) is responsible for safeguarding them from loss. To integrate claims information and improve its ability to detect fraud, waste, and abuse in these programs, CMS initiated two information technology system programs: the Integrated Data Repository (IDR) and One Program Integrity (One PI). GAO was asked to (1) assess the extent to which IDR and One PI have been developed and implemented and (2) determine CMS's progress toward achieving its goals and objectives for using these systems to help detect fraud, waste, and abuse. To do so, GAO reviewed system and program management plans and other documents and compared them to key practices. GAO also interviewed program officials, analyzed system data, and reviewed reported costs and benefits.
section_title: What GAO Found
paragraphs: CMS has developed and begun using both IDR and One PI, but has not incorporated into IDR all data as planned and has not taken steps to ensure widespread use of One PI to enhance efforts to detect fraud, waste, and abuse. IDR is intended to be the central repository of Medicare and Medicaid data needed to help CMS program integrity staff and contractors prevent and detect improper payments of Medicare and Medicaid claims. Program integrity analysts use these data to identify patterns of unusual activities or transactions that may indicate fraudulent charges or other types of improper payments. IDR has been operational and in use since September 2006. However, it does not include all the data that were planned to be incorporated by fiscal year 2010. For example, IDR includes most types of Medicare claims data, but not the Medicaid data needed to help analysts detect improper payments of Medicaid claims. IDR also does not include data from other CMS systems that are needed to help analysts prevent improper payments, such as information about claims at the time they are filed and being processed. According to program officials, these data were not incorporated because of obstacles introduced by technical issues and delays in funding. Further, the agency has not finalized plans or developed reliable schedules for efforts to incorporate these data. Until it does so, CMS may face additional delays in making available all the data that are needed to support enhanced program integrity efforts. One PI is a Web-based portal that is to provide CMS staff and contractors with a single source of access to data contained in IDR, as well as tools for analyzing those data. While One PI has been developed and deployed to users, few program integrity analysts were trained and using the system. Specifically, One PI program officials planned for 639 program integrity analysts to be using the system by the end of fiscal year 2010; however, as of October 2010, only 41--less than 7 percent--were actively using the portal and tools. According to program officials, the agency's initial training plans were insufficient and, as a result, they were not able to train the intended community of users. Until program officials finalize plans and develop reliable schedules for training users and expanding the use of One PI, the agency may continue to experience delays in reaching widespread use and determining additional needs for full implementation of the system. While CMS has made progress toward its goals to provide a single repository of data and enhanced analytical capabilities for program integrity efforts, the agency is not yet positioned to identify, measure, and track benefits realized from its efforts. As a result, it is unknown whether IDR and One PI as currently implemented have provided financial benefits. According to IDR officials, they do not measure benefits realized from increases in the detection rate for improper payments because they rely on business owners to do so, and One PI officials stated that, because of the limited use of the system, there are not enough data to measure and gauge the program's success toward achieving the $21 billion in financial benefits that the agency projected.
section_title: What GAO Recommends
paragraphs: GAO is recommending that CMS take steps to finalize plans and reliable schedules for fully implementing and expanding the use of the systems and to define measurable benefits. In its comments, CMS concurred with GAO's recommendations.

section_title: Letter
paragraphs: For more than 20 years, GAO has designated Medicare as a high-risk  program due to its size and complexity, as well as its susceptibility to  mismanagement and improper payments. Improper payments are  overpayments or underpayments that should not have been made or were  made in an incorrect amount. Improper payments may be due to errors,  such as the inadvertent submission of duplicate claims for the same  service, or misconduct, such as fraud and abuse. Since 2003, we have also  designated Medicaid as a high-risk program because of concerns about the  adequacy of its fiscal oversight, which is necessary to prevent  inappropriate spending. The Department of Health and Human Services  (HHS) reported about $70 billion in improper payments for the Medicare  and Medicaid programs in fiscal year 2010.
The Centers for Medicare and Medicaid Services (CMS) within HHS is  responsible for administering the Medicare and Medicaid programs and  leading efforts to reduce improper payments. As part of these efforts, CMS  conducts reviews to prevent improper payments before claims are paid  and reviews of claims potentially paid in error. These activities are  predominantly carried out by contractors who, along with CMS personnel,  use various information technology (IT) solutions to consolidate and  analyze data in support of efforts to detect improper payments of claims.  For example, these analysts may use software tools to access data about  claims to identify patterns of unusual activities by attempting to match  services with patients’ diagnoses.
In 2006, CMS initiated efforts to centralize and make more accessible the  data needed to conduct these analyses, and to improve the analytical tools  available to its own and contractor analysts. Among these initiatives are  the Integrated Data Repository (IDR), which is intended to provide a  single source of data related to Medicare and Medicaid claims, and the  One Program Integrity (One PI) system, a Web-based portal and suite of  analytical software tools used to extract data from IDR and enable  complex analyses of these data.
You asked us to examine CMS’s efforts to develop and implement IDR and  One PI to improve the agency’s ability to detect fraud, waste, and abuse in  administering these programs. Specifically, our objectives were to  1.  assess the extent to which the IDR and One PI systems have been  developed and implemented, and 2.  determine the agency’s progress toward achieving defined goals and  objectives for using the systems to help detect fraud, waste, and abuse  in the Medicare and Medicaid programs.
To address these objectives, we reviewed IDR and One PI program  management and planning documentation and held discussions with  agency officials and system users. Specifically, to assess the extent to  which IDR and One PI have been developed and implemented, we  compared the functionality that has been implemented to date to  estimated schedule milestones and performance measures. We also  reviewed the programs’ requirements development and management plans  and other project management artifacts and assessed CMS’s documented  processes against criteria established by the Software Engineering  Institute. To supplement the information we collected from agency  documents, we met with agency officials to discuss plans for and  management of the IDR and One PI programs.
To determine the agency’s progress toward achieving goals and objectives  for improving outcomes of its program integrity initiatives, we reviewed  agencywide strategic plans and program planning documents to identify  the goals and objectives, and assessed the extent to which IDR and One PI  supported efforts to achieve them. We also interviewed agency officials  about steps the agency had taken to achieve the goals and objectives. To  determine the extent to which the use of IDR and One PI has enabled the  agency to meet goals for improving its ability to detect fraud, waste, and  abuse, we identified CMS program integrity personnel and contractors  who actively use the systems by analyzing training information and system  login data. We then held discussions with groups of these users to  determine the extent to which and for what purposes they used the  system. We also compared reported system costs and financial benefits to  those projected for both IDR and One PI.
We assessed the reliability of the agency’s data related to project  management practices; cost, schedule, and financial benefit estimates; and  system usage through interviews with agency officials knowledgeable of  the management of the programs, methods for tracking and reporting  costs of the IDR and One PI programs, the programs’ user community and  training plans, and mechanisms for accessing the systems. We determined  that the data we collected and assessed were sufficiently reliable for the  purposes of our study.
We conducted our work in support of this performance audit primarily at  CMS’s headquarters in Baltimore, Maryland, from June 2010 to June 2011,  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Detailed information about our  objectives, scope, and methodology is discussed in appendix I.
subsections: section_title: Background
paragraphs: Medicare is a federal program that provides health insurance coverage for  individuals aged 65 and older and for certain disabled persons. It is funded  by general revenues, payroll taxes paid by most employees, employers,  and individuals who are self-employed, and beneficiary premiums.  Medicare consists of four parts. Medicare Part A provides payment for  inpatient hospital, skilled nursing facility, some home health, and hospice  services, while Part B pays for hospital outpatient, physician, some home  health, durable medical equipment, and preventive services. In addition,  Medicare beneficiaries have an option to participate in Medicare  Advantage, also known as Part C, which pays private health plans to  provide the services covered by Medicare Parts A and B. Further, all  Medicare beneficiaries may purchase coverage for outpatient prescription  drugs under Medicare Part D, and some Medicare Advantage plans also  include Part D coverage. In 2010, Medicare covered 47 million elderly and  disabled beneficiaries and had estimated outlays of about $509 billion.
CMS uses contractors to help administer the claims processing and  payment systems for Medicare. These administrative contractors are  responsible for processing approximately 4.5 million claims per workday.  The contractors review the claims submitted by providers to ensure  payment is made only for medically necessary services covered by  Medicare for eligible individuals.
Medicaid is the federal-state program that provides health coverage for  acute and long-term care services for over 65 million low-income people. It  consists of more than 50 distinct state-based programs that each define  eligibility requirements and administer payment for health care services  for low-income individuals, including children, families, the aged, and the  disabled. Within broad federal requirements, each state operates its  Medicaid program according to a state plan. Low-income Americans who  meet their state’s Medicaid eligibility criteria are entitled to have payments  made on their behalf for covered services. States are entitled to federal  matching funds, which differ from state to state but can be up to three- fourths of their costs of this coverage. The amount paid with federal funds  is determined by a formula established in law.
CMS oversees the Medicaid program at the federal level, while the states  administer their respective programs’ day-to-day operations, such as  enrolling eligible individuals, establishing payment amounts for covered  benefits, establishing standards for providers and managed care plans,  processing and paying for claims and managed care, and ensuring that  state and federal health care funds are not spent improperly or diverted by  fraudulent providers. The estimated outlays for Medicaid for both the  federal and state governments were $408 billion in 2010. Of this cost,  approximately $275 billion was incurred by the federal government and  $133 billion by the states.
subsections: section_title: CMS Program Integrity Initiatives
paragraphs: The Health Insurance Portability and Accountability Act (HIPAA) of 1996  established the Medicare Integrity Program to increase and stabilize  federal funding for health care antifraud activities. The act appropriated  funds for the program as well as amounts for HHS and the Department of  Justice to carry out the health care fraud and abuse control program.  Subsequent legislation further outlined responsibilities under the Medicare  Integrity Program.
Under the Medicare Integrity Program, CMS staff and several types of  contractors perform functions to help detect cases of fraud, waste, and  abuse, and other payment errors, which include reviews of paid claims to  identify patterns of aberrant billing. Among these program integrity  contractors are program safeguard contractors, zone program integrity  contractors, and Medicare drug integrity contractors. The program  safeguard and zone program integrity contractors are responsible for  ensuring the integrity of benefit payments for Medicare Parts A and B  (including durable medical equipment), as well as the Medi-Medi data  match program. Medicare drug integrity contractors are responsible for  monitoring fraud, waste, or abuse in the Medicare prescription drug  program (i.e., Part D). These contractors work with the HHS Office of the  Inspector General (OIG) and law enforcement organizations, such as the  Department of Justice, to help law enforcement pursue criminal or civil  penalties when fraudulent claims are detected. Table 1 summarizes the  origin and responsibilities of the program integrity contractors who help  CMS to detect fraud, waste, and abuse.
In addition to provisions of HIPAA and other legislation intended to  strengthen Medicare program integrity functions, in 2006 Congress created  the Medicaid Integrity Program through the Deficit Reduction Act of 2005.  Its goals are to strengthen the national Medicaid audit program and to  enhance federal oversight of and support and assistance to state Medicaid  programs. The program provides states with technical assistance and  support to enhance the federal-state partnership as well as to expand  activities that involve data analysis, sharing algorithms of known improper  billings, and fraud awareness through education and outreach. Individual  states are responsible for ensuring the accuracy of Medicaid payments  within their state programs, which can involve using their own staff or  contractors to analyze claims to detect improper payments. In addition to  the states’ efforts, CMS employs Medicaid program integrity contractors to  perform specific activities as part of its efforts to detect fraud, waste, and  abuse in the Medicaid program, such as reviewing provider claims  payments that have been processed by the states.
Generally, each state Medicaid program integrity unit works  independently, using its own data models, data warehouses, and approach  to analysis. As a result, Medicaid data are stored in multiple disparate  systems and databases throughout the country. Because of the volumes of  work, states often augment their in-house capabilities by contracting with  companies that specialize in Medicaid claims and utilization reviews. State  Medicaid program integrity units target their activities to those providers  that pose the greatest financial risk to their Medicaid programs. However,  the states have limited methods of identifying Medicaid fraud in  neighboring jurisdictions or by providers who move from state to state.
As stated in a July 2007 report by the HHS OIG, the agency intends for  program integrity contractors to perform a significant amount of self- initiated, exploratory analysis to seek patterns or instances of fraud and  abuse. One of the specific activities undertaken by these contractors is the  analysis of claims data to identify improper billing that may indicate fraud  or abuse. If the billing appears to be potentially fraudulent or abusive, the  contractors take further actions, which can include requesting and  reviewing medical records associated with the claims and referring the  case to law enforcement.
In 2010, CMS created the Center for Program Integrity to serve as its focal  point for all national Medicare and Medicaid program integrity fraud and  abuse issues. The new center is responsible for, among other things,  collaborating with other CMS components to develop and implement a  comprehensive strategic plan, objectives, and measures to carry out the  agency’s program integrity mission and goals, and ensure program  vulnerabilities are identified and resolved. According to agency  documentation describing the program, the center was designed to    promote the integrity of the Medicare and Medicaid programs through  provider and contractor audits and policy reviews, identification and  monitoring of program vulnerabilities, and support and assistance to  states;   collaborate on the development and advancement of new legislative  initiatives and improvements to deter, reduce, and eliminate fraud, waste  and abuse;   oversee all CMS interactions and collaboration with key stakeholders  related to program integrity (e.g., the Department of Justice, HHS OIG, and  state law enforcement agencies) for the purposes of detecting, deterring,  monitoring, and combating fraud and abuse; and take action against those who commit or participate in fraudulent or other  unlawful activities.
section_title: CMS’s Use of IT to Help Detect Fraud, Waste, and Abuse
paragraphs: Like financial institutions, credit card companies, telecommunications  firms, and other private sector companies that take steps to protect  customers’ accounts, CMS uses automated software tools to help predict  or detect cases of improper claims and payments. For more than a decade,  CMS and its contractors have applied such tools to access data from  various sources to analyze patterns of unusual activities or financial  transactions that may indicate fraudulent charges or other types of  improper payments. For example, to identify unusual billing patterns and  to support referrals for prosecution or other action, CMS and program  integrity contractor analysts and investigators need, among other things,  access to information about key actions taken to process claims as they  are filed and specific details about claims already paid. This includes  information on claims as they are billed, adjusted, and paid or denied;  check numbers on payments of claims; and other specific information that  could help establish provider intent. These data, along with data on  regional or national trends on claims billing and payment, support the  investigation and potential prosecution of fraud cases. Upon completing  investigations, the contractors determine whether to refer the  investigations as cases to law enforcement officials.
CMS and its program integrity contractors currently use many different  means to store and manipulate data and, since the establishment of the  agency’s program integrity initiatives in the 1990s, have built multiple  databases and developed analytical software tools to meet their individual  and unique needs. However, according to CMS, these geographically  distributed, regional approaches to data analysis result in duplicate data  and limit the agency’s ability to conduct analyses of data on a nationwide  basis. Additionally, data on Medicaid claims are scattered among the states  in multiple disparate systems and data stores, and are not readily available  to CMS. Thus, CMS has been working for most of the past decade to  consolidate program integrity data and analytical tools for detecting fraud,  waste, and abuse. The agency’s efforts led to the initiation of the IDR  program and, subsequently, the One PI program, which are intended to  provide CMS and its program integrity contractors with a centralized  source that consolidates Medicare and Medicaid data from the many  disparate and dispersed legacy systems and databases and a Web-based  portal and set of analytical tools by which these data can be accessed and  analyzed to help detect cases of fraud, waste, and abuse.
subsections: section_title: CMS’s Initiative to Develop a Centralized Source of Medicare and Medicaid Data
paragraphs: The CMS Office of Information Services is responsible for agencywide IT  management. Its initiative to develop a centralized data warehouse began  in 2003 as an element of the agency’s Enterprise Data Modernization  strategy. According to agency documentation, the strategy was designed to  meet the increasing demand for higher quality and more timely data to  support decision making throughout the agency, including identifying  trends and discovering patterns of fraud, waste, and abuse. As part of the  strategy, the agency established the Data Warehouse Modernization  project to develop and implement the technology needed to store long- term data for analytical purposes, such as summary reports and statistical  analyses. CMS initially planned for the data warehouse project to be  complete by September 30, 2008.
However, in 2006 CMS expanded the scope of the project to not only  modernize data storage technology but also to integrate Medicare and  Medicaid data into a centralized repository. At that time, program officials   also changed the name to IDR, which reflected the expanded scope. The  Office of Information Services’ Enterprise Data Group manages the IDR  program and is responsible for the design and implementation of the  system. The program’s overall goal is to integrate Medicare and Medicaid  data so that CMS and its partners may access the data from a single  source. Specific goals for the program are to  transition from stove-piped, disparate sets of databases to a highly  integrated data environment for the enterprise; transition from a claim-centric orientation to a multi-view orientation that  includes beneficiaries, providers, health plans, claims, drug code data,   clinical data, and other data as needed;   provide uniform privacy and security controls;    provide database scalability to meet current and expanding volumes of    provide users the capability to analyze the data in place instead of relying  on data extracts.
According to IDR program officials, CMS envisioned that IDR would  become the single repository for the agency’s data and enable data  analysis within and across programs. Specifically, IDR was to establish the  infrastructure for storing data for Medicare Parts A, B, and D, as well as a  variety of other CMS functions, such as program management, research,  analytics, and business intelligence. CMS envisioned an incremental  approach to incorporating data into IDR. Specifically, program plans  provided to the Office of Management and Budget (OMB) by the Office of  Information Services in 2006 stated that all Medicare Part D data would be  incorporated into IDR by the end of that fiscal year. CMS’s 2007 plans  added the incorporation of Medicare Parts A and B data by the end of  fiscal year 2007, and Medicaid data for 5 states by the end of fiscal year  2009, 20 states by 2010, 35 by 2011, and all 50 states by the end of fiscal  year 2012.
Initial program plans and schedules also included the incorporation of  additional data from legacy CMS claims-processing systems that store and  process data related to the entry, correction, and adjustment of claims as  they are being processed, along with detailed financial data related to paid  claims. According to program officials, these data, called “shared systems”  data, are needed to support the agency’s plans to incorporate tools to  conduct predictive analysis of claims as they are being processed, helping  to prevent improper payments. Shared systems data, such as check  numbers and amounts related to claims that have been paid, are also  needed by law enforcement agencies to help with fraud investigations.  CMS initially planned to include all the shared systems data in IDR by July  2008. Figure 1 shows a timeline of initial plans for incorporating data into  IDR.
In 2006, CMS’s Office of Financial Management initiated the One PI  program with the intention of developing and implementing a portal and  software tools that would enable access to and analysis of claims,  provider, and beneficiary data from a centralized source. CMS’s goal for  One PI was to support the needs of a broad program integrity user  community, including agency program integrity personnel and contractors  who analyze Medicare claims data, along with state agencies that monitor  Medicaid claims. To achieve its goal, agency officials planned to  implement a tool set that would provide a single source of information to  enable consistent, reliable, and timely analyses and improve the agency’s  ability to detect fraud, waste, and abuse. These tools were to be used to  gather data about beneficiaries, providers, and procedures and, combined  with other data, find billing aberrancies or outliers. For example, as  envisioned, an analyst could use software tools to identify potentially  fraudulent trends in ambulance services. He or she could gather data  about claims for ambulance services and medical treatments, and then use  other software to determine associations between the two types of  services. If the analyst found claims for ambulance travel costs but no  corresponding claims for medical treatment, the analyst may conclude that  the billings for those services were possibly fraudulent.
According to agency program planning documentation, the One PI system  was to be developed incrementally to provide access to data, analytical  tools, and portal functionality in three phases after an initial proof of  concept phase. The proof of concept phase was reportedly begun in early  2007 and focused on integrating Medicare and Medicaid data into the  portal environment. After its completion, the first development phase  focused on establishing a development environment in CMS’s Baltimore,  Maryland, data center and, according to program officials, was completed  in April 2009. The second and third phases of development were planned  in January 2009 to run concurrently and to focus on the technical and  analytical aspects of the project, such as building the environment to  integrate the analytical tools using data retrieved from IDR, sourcing  claims data from the shared systems, conducting data analyses in  production, and training analysts who were intended users of the system.  CMS planned to complete these two phases and implement the One PI  portal and two analytical tools for use by program integrity analysts on a  widespread basis by the end of fiscal year 2009.
CMS’s Office of Financial Management engaged contractors to develop the  system. Responsibility for and management of the One PI program moved  from the Office of Financial Management to the Center for Program  Integrity in 2010. Figure 2 illustrates initial plans for One PI.
section_title: Prior GAO Reports on Fraud, Waste, and Abuse in the Medicare and Medicaid Programs
paragraphs: In our prior work, we have reported on CMS’s efforts to detect and  prevent fraudulent and improper payments in the Medicare and Medicaid  programs and on its management of IT to support its mission. For  example, as early as 1995, we reviewed IT systems used in the Medicare  program to detect and prevent fraud and discussed the availability of other  technologies to assist in combating fraudulent billing. We found it was  too early to fully document the cost-effectiveness of such systems,  although several potential fraud cases were detected by this technology,  indicating that these types of systems could provide net benefits in  combating fraud. We observed that such technology could ultimately be  utilized in the claims-processing environment to delay or even prevent the  payment of questionable claims submitted by suspect providers.
We have also reported on weaknesses in CMS’s processes for managing IT  investments based upon key practices established in our Information  Technology Investment Management framework. Specifically, in 2005, we  evaluated CMS’s capabilities for managing its internal investments,  described plans the agency had for improving these capabilities, and  examined the agency’s process for approving and monitoring state  Medicaid Management Information Systems. We found that CMS had not  established certain key practices for managing individual IT investments  and recommended that the CMS Administrator develop and implement a  plan to address the IT investment management weaknesses identified in  the report. We also recommended that at a minimum, the agency should  update its investment management guide to reflect current investment  management processes. CMS subsequently took actions to implement  each of our recommendations.
Additionally, our 2007 study of the Medicare durable medical equipment,  prosthetics, orthotics, and supplies benefit found that it was vulnerable to  fraud and improper payments. We recommended that CMS direct its  contractors to develop automated prepayment controls to identify  potentially improper claims and consider adopting the most cost-effective  controls of other contractors. CMS concurred with the recommendation,  but has not yet implemented the prepayment controls that we  recommended.
In 2009, we examined the administration of the Medicare home health  benefit, which we found to leave the benefit vulnerable to fraud and  improper payments. We made several recommendations to the  Administrator of CMS, including directing contractors to conduct post- payment medical reviews on claims submitted by home health agencies  with high rates of improper billing identified through prepayment review.  CMS stated it would consider two of our four recommendations—to  amend regulations to expand the types of improper billing practices that  are grounds for revocation of billing privileges, and to provide physicians  who certify or recertify plans of care with a statement of services received  by beneficiaries. CMS neither agreed nor disagreed with our other two  recommendations.
Finally, in testifying on Medicare and Medicaid fraud, waste, and abuse in  March 2011, we described steps that CMS could take to reduce improper  payments and the agency’s recent solicitation for proposals of contracts  for the development and implementation of automated tools that support  reviews of claims before they are paid. These predictive modeling tools  are intended to provide new capabilities to help prevent improper  payments of Medicare claims.
section_title: IDR and One PI Have Been Developed and Implemented but Without All Planned Data and Widespread Use
paragraphs: CMS has developed and implemented IDR and One PI for use by its  program integrity analysts, but IDR does not include all the data the  agency planned to have incorporated by the end of 2010, and One PI is  being used by a limited number of analysts. While CMS has developed and  begun using IDR, the repository does not include all the planned data,  such as Medicaid and shared systems data. Program officials attribute this  lack of data to insufficient planning, which did not consider unexpected  obstacles or allow time for contingencies. In addition, the agency has  developed and deployed One PI, but the system is being used by less than  7 percent of the intended user community and does not yet provide as  many tools as planned. According to agency officials, plans to train and  deploy the system to a broad community of users were disrupted when  resources dedicated to these activities were redirected to address a need  to improve the user training program. Further, plans and schedules for  completing the remaining work have not been finalized, and CMS has not  identified risks and obstacles to project schedules that may affect its  ability to ensure broad use and full implementation of the systems. Until  program officials finalize plans and develop reliable schedules for  providing all planned data and capabilities and ensuring that One PI gains  broader use throughout the program integrity community, CMS will  remain at risk of experiencing additional delays in reaching widespread  use and full implementation of the systems. Consequently, the agency may  miss an opportunity to effectively use these IT solutions to enhance its  ability to detect fraud, waste, and abuse in the Medicare and Medicaid  programs.
subsections: section_title: IDR Has Been Developed and Is in Use, but Does Not Yet Include All Data Needed to Enhance Program Integrity Efforts
paragraphs: IDR has been in use by CMS and contractor program integrity analysts  since September 2006 and currently incorporates data related to claims for  reimbursement of services under Medicare Parts A, B, and D. Specifically,  CMS incorporated Part D data into IDR in September 2006, as planned,  and incorporated Parts A and B data by the end of fiscal year 2008. The  primary source of these data is CMS’s National Claims History database,  from which data are extracted on a weekly basis. Other supplemental data  were incorporated into IDR that are used to conduct program integrity  analyses, including drug code data that are obtained from daily and weekly  updates of data from CMS’s Drug Data Processing System, and claims- related data about physicians that are retrieved from National Provider  Index databases on a daily basis. Additionally, IDR contains data about  beneficiaries that are extracted daily from the Medicare Beneficiary  Database and health plan contract and benefit data that are obtained on a  weekly basis from CMS’s Health Plan Management Systems. According to  IDR program officials with the Office of Information Services, the  integration of these data into IDR established a centralized source of data  previously accessed from multiple disparate system files.
CMS reported to OMB in 2010 that the agency had spent almost $48 million  to establish IDR and incorporate the existing data since the program was  initiated. Table 2 provides the actual costs of developing and implementing  IDR for each year since fiscal year 2006, as reported to us by CMS officials.
Although the agency has been incorporating data from various data  sources since 2006, IDR does not yet include all the data that were planned  to be incorporated by the end of 2010 and that are needed to support  enhanced program integrity initiatives. Specifically, the shared systems  data that are needed to allow predictive analyses of claims are not  incorporated. Without this capability, program integrity analysts are not  able to access data from IDR that would help them identify and prevent  payment of fraudulent claims. Additionally, IDR does not yet include the  Medicaid data that are critical to analysts’ ability to detect fraud, waste,  and abuse in the Medicaid program.
According to IDR program officials, the shared systems data were not  incorporated into IDR because, although initial program integrity  requirements included the incorporation of these data by July 2008,  funding for the development of the software and acquisition of the  hardware needed to meet this requirement was not approved until the  summer of 2010. Since then, IDR program officials have developed project  plans and identified users’ requirements, and plan to incorporate shared  systems data by November 2011.
With respect to Medicaid data, program officials stated that the agency has  not incorporated these data into IDR because the original plans and  schedules for obtaining Medicaid data did not account for the lack of a  mandate or funding for states to provide Medicaid data to CMS, or the  variations in the types and formats of data stored in disparate state  Medicaid systems. In this regard, program officials did not consider risks  to the program’s ability to collect the data and did not include additional  time to allow for contingencies. Consequently, the IDR program officials  were not able to collect the data from the states as easily as they expected  and, therefore, did not complete this activity as originally planned.
In addition to the IDR program, in December 2009, CMS initiated another  agencywide program intended to, among other things, identify ways to  collect Medicaid data from the many disparate state systems and  incorporate the data into a single data store. As envisioned by CMS, this  program, the Medicaid and Children’s Health Insurance Program Business  Information and Solutions program, or MACBIS, is to include activities in  addition to providing expedited access to current data from state Medicaid  programs. For example, the MACBIS initiative is also intended to result in  the development of a national system to address the needs of federal and  state Medicaid partners, along with technical assistance and training for  states on the use of the system. Once established, the MACBIS system data  would then be incorporated into IDR and made accessible to program  integrity analysts. According to program planning documentation, this  enterprisewide initiative is expected to cost about $400 million through  fiscal year 2016. However, plans for this program are not final, and funds  for integrating Medicaid data into IDR have not yet been requested.
According to agency planning documentation, as a result of efforts to be  initiated under the MACBIS program, CMS intends to incorporate  Medicaid data for all 50 states into IDR by the end of fiscal year 2014.  Program integrity officials stated that they plan to work with three states  during 2011 to test the transfer and use of Medicaid data to help CMS  determine the data that are available in those states’ systems. The Center  for Program Integrity is also working with Medicaid officials to establish a  test environment to begin integrating state Medicaid data into IDR. Despite  establishing these high-level milestones, the agency has not finalized  detailed plans for incorporating the Medicaid data that include reliable  schedules that identify all the necessary activities and resources for  completing these efforts or the risks associated with efforts to collect and  standardize data from 50 independent systems that differ in design,  technology, and other characteristics dictated by state policies.
Table 3 shows the original planned dates for incorporating the various  types of data and the data that were incorporated into IDR as of the end of  fiscal year 2010.
While CMS has identified target dates for incorporating the remaining  data, best practices, such as those described in our cost estimation guide,  emphasize the importance of establishing reliable program schedules that  include all activities to be performed, assign resources (labor, materials,  etc.) to those activities, and identify risks and their probability and build  appropriate reserve time into the schedule. However, the IDR schedule we  reviewed did not identify all activities and necessary resources or include  a schedule risk analysis. Such an analysis could have helped CMS identify  and prepare for obstacles, such as those previously encountered in trying  to incorporate Medicaid data into IDR and expected to be encountered as  CMS initiates efforts to collect and standardize data from 50 state systems.  Without establishing a reliable schedule for future efforts to incorporate  new data sources, the agency will be at greater risk of schedule slippages,  which could result in additional delays in CMS’s efforts to incorporate all  the data sources into IDR that are needed to support enhanced program  integrity efforts.
section_title: One PI Has Been Developed but Deployed to Few Users and With Less Functionality Than Planned
paragraphs: According to program officials, user acceptance testing of the One PI  system was completed in February 2009, and the system was deployed in  September 2009 as originally planned. This initial deployment of One PI  consisted of a portal that provided Web-based access to analytical tools  used by program integrity analysts to retrieve and analyze data stored in  IDR.
CMS reported to OMB that the agency had spent almost $114 million to  develop the existing features and functionality of the One PI system by the  end of fiscal year 2010. Table 4 provides information on the actual costs of  developing One PI since fiscal year 2006, as reported to us by CMS  officials.
As currently implemented, the system provides access to two analytical  tools—Advantage Suite and Business Objects. Documented specifications  of the One PI system described Advantage Suite as a commercial, off-the- shelf decision support tool that is used to perform data analysis to, for  example, detect patterns of activities that may identify or confirm  suspected cases of fraud, waste, or abuse. According to program officials  and the One PI users to whom we spoke, program integrity analysts use  Advantage Suite to analyze claims data retrieved from IDR and create  standard and custom reports that combine data about costs and quality of  services, providers, and beneficiaries. The results of this level of analysis  may be used to generate leads for further analysis with Business Objects,  which provides users extended capabilities to perform more complex  analyses of data by allowing customized queries of claims data across the  three Medicare plan types. It also allows the user to create ad hoc queries  and reports for nonroutine analysis.
For example, an analyst could use Advantage Suite to identify potentially  fraudulent trends in ambulance services. He or she could use the tool to  gather data about claims for ambulance services and medical treatments,  and then use Business Objects to conduct further analysis to determine  associations between the two types of services. If the analyst found claims  for ambulance travel costs but no corresponding claims for medical  treatment, the analyst may conclude that the billings for those services  were possibly fraudulent.
Figure 3 provides a simplified view of the IDR and One PI environment as  currently implemented.
While program officials deployed the One PI portal and two analytical  tools to CMS and contractor program integrity analysts, the system was  not being used as widely as planned. Program planning documentation  from August 2009 indicated that One PI program officials planned for 639  program integrity staff and analysts to be trained and using the system by  the end of fiscal year 2010; however, CMS confirmed that by the end of  October 2010 only 42 of those intended users were trained to use One PI,  and 41 were actively using the portal and tools. These users represent less  than 7 percent of the original intended users. Of these, 31 were contractors  and 10 were CMS staff who performed analyses of claims to detect  potential cases of fraud, waste, and abuse. Table 5 describes the analysts  planned to be and actually using One PI at the end of fiscal year 2010.
According to One PI program officials, the system was not being used by  the intended number of program integrity analysts because the office had  not trained a sufficient number of analysts to use the system. Similarly,  although CMS contractually requires Medicare program integrity  contractors to use the system, officials stated that they could not enforce  this requirement because they also had not trained enough of their  program integrity contractors.
Although One PI program plans emphasized the importance of effective  training and communications, program officials responsible for  implementing the system acknowledged that their initial training plans and  efforts were insufficient. According to the officials, they initially provided  training for the all the components of the system—the portal, tools, and  use of IDR data—in a 3-and-a-half-day course. However, they realized that  the trainees did not effectively use One PI after completing the training.  Consequently, program officials initiated activities and redirected  resources to redesign the One PI training plan in April 2010, and began to  implement the new training program in July of that year. The redesigned  program includes courses on each of the system components and allows  trainees to use the components to reinforce learning before taking  additional courses. For example, the redesigned plan includes a One PI  portal overview and data training webinars that users must complete  before attending instructor-led training on Advantage Suite and Business  Objects. The new plans also incorporate the use of “data coaches” who  provide hands-on help to analysts, such as assistance with designing  queries. Additionally, the plans require users to complete surveys to  evaluate the quality of the training and their ability to use the tools after  they complete each course.
As program officials took the initiative and time to redesign the training  program, this effort caused delays in CMS’s plans to train the intended  number of users. Since the new training program was implemented, the  number of users has not yet significantly increased, but the number of  contractor analysts requesting training has increased. Specifically, One PI  officials told us that 62 individuals had signed up to be trained in 2011, and  that the number of training classes for One PI was increased from two to  four per month. The officials also stated that they planned to reach out to  and train more contractors and staff from the HHS OIG and the  Department of Justice to promote One PI. They anticipated that 12  inspectors general and 12 law enforcement officials would be trained and  using One PI by the end of May 2011.
Nonetheless, while these activities indicate some progress toward  increasing the number of One PI users, the number of users expected to be  trained and to begin using the system represents a small fraction of the  population of 639 intended users. Additionally, One PI program officials  had not yet made detailed plans and developed schedules for completing  training of all the intended users. Further, although program officials had  scheduled more training classes, they have not established deadlines for  contractor analysts to attend training so that they are able to fulfill the  contractual requirement to use One PI. Unless the agency takes more  aggressive steps to ensure that its program integrity community is trained,  it will not be able to require the use of the system by its contractors, and  the use of One PI may remain limited to a much smaller group of users  than the agency intended. As a result, CMS will continue to face obstacles  in its efforts to deploy One PI to the intended number of program integrity  users as the agency continues to develop and implement additional  features and functionalities in the system.
Additionally, although efforts to develop and implement One PI were  initiated in 2006 and the Advantage Suite and Business Objects tools are  fully developed, implemented, and in use, the One PI system does not yet  include additional analytical functionality that CMS initially planned to  implement by the end of 2010. Program documentation for the system  includes plans for future phases of One PI development to incrementally  add new analytical tools, additional sources of data, and expanded portal  functionality, such as enhanced communications support, and specifically  included the integration of a third tool by the end of fiscal year 2010.  However, program officials have not yet identified users’ needs for  functionality that could be provided by another tool, such as the capability  to access and analyze more data from IDR than the current  implementation of the system provides. According to program officials,  they intend to determine users’ needs for additional functionality when the  system becomes more widely used by agency and contractor analysts who  are able to identify deficiencies and define additional features and  functionality needed to improve its effectiveness.
Additionally, as with IDR, in developing the One PI schedule estimate that  was provided to OMB in 2010, program officials did not complete a risk  assessment for the schedule that identified potential obstacles to the  program. As a result, they lacked information needed to plan for additional  time to address contingencies when obstacles arose. As the program office  makes plans for deploying the system to the wide community of program  integrity analysts and implementing additional tools, it is crucial that  officials identify potential obstacles to the schedules and the risks they  may introduce to the completion of related activities. For example, an  analysis that identified the risk that resources would need to be redirected  to other elevated priorities, such as user training, could have informed  managers of the need to include additional time and resources in the  schedule to help keep the development and deployment of One PI on  track. Unless program officials complete a risk assessment of schedules  for ongoing and future activities, CMS faces risks of perpetuating delays in  establishing widespread use of One PI and achieving full implementation  of the system for increased rates of fraud, waste, and abuse detection.
section_title: CMS Is Not Yet Positioned to Fully Meet Goals and Objectives for Detecting Fraud, Waste, and Abuse through the Use of IDR and One PI
paragraphs: Our prior work emphasized agencies’ need to ensure that IT investments  actually produce improvements in mission performance. As we have  reported, agencies should forecast expected benefits and then measure  actual financial benefits accrued through the implementation of IT  programs. Further, OMB requires agencies to report progress against  performance measures and targets for meeting them that reflect the goals  and objectives of the programs. To do this, performance measures should  be outcome-based, developed with stakeholder input, and monitored and  compared to planned results. Additionally, industry experts describe the  need for performance measures to be developed with stakeholders’ input  early in a project’s planning process to provide a central management and  planning tool and to monitor the performance of the project against plans  and stakeholders’ needs.
subsections: section_title: CMS Has Made Limited Progress toward Meeting Program Integrity Goals and Objectives through the Use of IDR
paragraphs: As stated in program planning documentation, IDR’s overall goal is to  integrate Medicare and Medicaid data so that CMS and its partners may  access the data from a single source. Specifically, the implementation of  IDR was expected to result in financial benefits associated with the  program’s goal to transition from a data environment of stove-piped,  disparate databases and systems to an integrated data environment.
Officials with the Office of Information Services stated that they developed  estimates of financial benefits expected to be realized through the use of  IDR. In 2006, program officials projected financial benefits from IDR of  $152 million at an estimated cost of $82 million, or a net benefit of about  $70 million. In 2007 these officials revised their projection of total financial  benefits to $187 million based on their estimates of the amount of  improper payments they expected to be recovered as a result of analyzing  data provided by IDR. The resulting net benefit expected from  implementing IDR was estimated to be $97 million in 2010 due to changes  in program cost estimates.
Table 6 includes CMS’s estimated financial benefits, costs, and net benefits  reported to OMB for the lifecycle of the program from fiscal year 2006 to  2010.
However, as of March 2011, program officials had not identified actual  financial benefits of implementing IDR based on the recovery of improper  payments. In our discussions with the Office of Information Services,  program officials stated they determined that deploying IDR led to the  avoidance of IT costs as a result of the retirement of several legacy  systems attributable to the implementation of IDR. However, they had not  quantified these or any other financial benefits. Until officials measure and  track financial benefits related to program goals, CMS officials cannot be  assured that the use of the system is helping the agency prevent or recover  funds lost as a result of improper payments of Medicare and Medicaid  claims.
Additionally, while program officials defined and reported to OMB  performance targets for IDR related to some of the program’s goals, they  do not reflect its goal to provide a single source of Medicare and Medicaid  data for program integrity efforts. Although progress made to date in  implementing IDR supports the program’s goals to transition CMS to an  integrated data environment, program officials have not defined and  reported to OMB performance measures to gauge the extent to which the  program is meeting this goal. Specifically, IDR officials defined  performance measures for technical indicators, such as incorporating  Medicare data into the repository, making the data available for analysis,  and reducing the number of databases CMS must support, but they have  not defined measures and targets that reflect the extent to which all the  data needed to support program integrity initiatives are incorporated into  a single source, including the Medicaid and shared systems data which  have not yet been incorporated into IDR. Further, the IDR performance  measures do not reflect indicators that may lead to the program’s ability to  achieve the financial benefits defined by the agency’s program integrity  initiatives.
In discussing this matter, IDR officials stated that the performance  measures for the program are only intended to track progress toward  implementing technical capabilities of the system, such as the amount of  data from specific sources incorporated into the repository and made  available through software tools to analysts. They do not define  performance indicators, measures, and targets for incorporating data from  future sources of data until plans are made and funds are provided by the  agency’s business offices to begin activities to implement new  functionalities into IDR. IDR program officials also stated that they do not  define or track business-related performance indicators for achieving  specific program integrity goals; rather, they depend upon business  owners to measure and track these indicators based upon the use of IDR  data to achieve business goals. However, without performance measures  that reflect business owners’ and other stakeholders’ needs for the  program to deliver a single source of all Medicare and Medicaid data  needed to conduct analyses, and lacking measures that reflect the success  of the program toward achieving financial benefits projected for program  integrity initiatives, program officials lack key management information  needed to ensure that the data and infrastructure components provided by  IDR enhance CMS’s ability to meet its program integrity goals and  objectives. Without this assurance, the effectiveness of the system’s  capability to increase rates of fraud, waste, and abuse detection and,  consequently, decrease the amount of money lost to improper payments of  claims will remain unknown.
section_title: CMS Is Not Yet Positioned to Demonstrate Improvements in Its Ability to Meet Goals and Objectives for Detecting Fraud, Waste, and Abuse through the Use of One PI
paragraphs: The Center for Program Integrity’s overall goal for One PI was to provide  robust tools for accessing a single source of information to enable  consistent, reliable, and timely analyses to improve the agency’s ability to  detect fraud, waste, and abuse. Achieving this goal was intended to result  in the recovery of significant funds lost each year from improper payments  of Medicare and Medicaid claims. In September 2007, program officials  projected financial benefits from implementing One PI—nearly $13 billion  over the 10-year lifecycle of the project. According to program officials,  these benefits were expected to accrue from the recovery of improper  payments of Medicare and Medicaid claims and reduced program integrity  contractor expenditures for supporting IT required to maintain separate  databases.
In September 2007, One PI officials projected and reported to OMB  benefits of nearly $13 billion. They subsequently revised this estimate to  approximately $21 billion. Program officials told us that increases in the  projected financial benefits were made based on assumptions that  accelerated plans to integrate Medicare and Medicaid data into a central  data repository would enable One PI users to identify increasing numbers  of improper payments sooner than previously estimated, thus allowing the  agency to recover more funds lost due to payment errors. Table 7 provides  data CMS reported to OMB on estimated benefits and costs, actual costs  as of the end of fiscal year 2010, and net benefits projected to be realized  as a result of implementing One PI from fiscal year 2007 through 2010.
However, the current implementation of One PI has not yet produced  outcomes that position the agency to identify or measure financial  benefits. Therefore, the net financial benefit of developing and  implementing One PI remains unknown. Center for Program Integrity  officials stated that at the end of fiscal year 2010—over a year after  deploying One PI—it was too early to determine whether the program has  provided any financial benefits because, since the program had not met its  goal for widespread use of One PI, there were not enough data available to  quantify financial benefits attributable to the use of the system. These  officials anticipated that as the user community is expanded, they will be  able to begin to identify and measure financial and other benefits of using  the system. However, the officials also indicated that they had not yet  defined mechanisms for determining the amount of money recovered as a  result of detecting improper payments through the use of One PI. As with  IDR, until the agency quantifies and tracks the progress it is making in  delivering benefits intended to be realized through widespread use of One  PI, CMS officials cannot be assured of the cost-effectiveness of  implementing One PI to help the agency meet its goal to enable consistent,  reliable, and timely analyses of data to improve the agency’s ability to  detect fraud, waste, and abuse.
Additionally, in discussion groups held with active One PI users, program  integrity analysts identified several issues that confirmed the agency’s  limited progress toward meeting the goals of the program. For example,  while several users told us that the One PI system can support their work,  they recognized limited progress toward the establishment of a single  source of information and analysis tools for all fraud, waste, and abuse  activities. Further, One PI users stated that the system enabled analysts to  access national data not otherwise accessible to them and supported  analysis across different Medicare programs. They also noted that the  tools offered by One PI provided more functionality than other tools they  use. However, of the analysts in the discussion groups, most did not use  One PI as their only source of information and analysis for detecting  improper payments. Rather, to help conduct their work, they relied on  other analysis tools provided by CMS or their companies, along with data  from CMS claims processing contractors or from private databases  created by other contractors.
One PI users in the discussion groups also told us that they use other tools  because they are more familiar with those tools. Additionally, they stated  that other databases sometimes provide data that are not currently  accessible through One PI and IDR, such as demographic data about  providers. Program integrity analysts further stated that they only use One  PI as a cross-check of data and analysis from their own systems because  they are not yet convinced that One PI can be used as a replacement for or  adjunct to those data sources and tools.
Further, CMS officials have not developed quantifiable measures for  meeting the program’s goals. CMS officials defined and reported to OMB  performance measures and targets toward meeting the program’s goals for  enabling timely analyses of data to detect cases of fraud, waste, and abuse,  but have not yet been able to quantify measures for these indicators. For  example, performance measures and targets for One PI include increases  in the detection of improper payments for Medicare Parts A and B claims.  However, according to program integrity officials, measures had not yet  been quantified because they had not yet identified ways to determine the  extent to which increases in the detection of errors could be attributed to  the use of One PI. Additionally, the limited use of the system has not  generated enough data to quantify the amount of funds recovered from  improper payments. Moreover, measures of One PI’s program  performance do not accurately reflect the current state of the program.  Specifically, indicators to be measured for the program include the  number of states using One PI (for Medicaid integrity purposes) and  decreases in the Medicaid payment error rate, but One PI does not have  access to those data because they are not yet incorporated into IDR.  Therefore, these performance indicators are not relevant to the current  implementation of the system.
Finally, CMS officials did not consult external system users (e.g., program  integrity contractors) in developing measures of One PI’s effectiveness.  According to industry experts,  developing performance measures with  stakeholder input early in the planning process can provide a mechanism  for gauging the effectiveness of outcomes toward meeting business needs  and achieving program goals as a program progresses. However, CMS  officials did not consult external users of the system about how they  would measure its effectiveness. According to program officials, program  integrity stakeholders within CMS were involved in the development of the  performance measures; however, external users of the system were not  asked to provide input when it may have been used to establish an  effective performance tracking tool, such as when defining ways to  determine whether One PI meets stakeholders’ needs. For example,  program officials told us that they intend to determine user satisfaction, a  performance measure reported to OMB, by conducting surveys at the end  of training sessions. However, these surveys were conducted before the  analysts actually used the system in their work and were focused on  satisfaction with the training itself. In this case, involvement of external  stakeholders when defining the measure could have led to more effective  ways to determine user satisfaction, such as surveying analysts based on  their experiences resulting from the use of One PI after a certain period of  time defined by stakeholders.
Until they define measurable performance indicators and targets that  reflect the goals and objectives of CMS’s program integrity initiatives,  agency officials will continue to lack the information needed to ensure  that the implementation of One PI helps improve the agency’s ability to  identify improper payments and to detect cases of fraud, waste, and abuse.  Additionally, when lacking stakeholders’ input into the process for  determining measures of successful performance, One PI program officials  may miss an opportunity to obtain information needed to define  meaningful measures that reflect the success of the program toward  meeting users’ and the agency’s needs. Because it lacks meaningful  outcome-based performance measures and effective methods for tracking  progress toward meeting performance targets, CMS does not have the  information needed to ensure that the system is useful to the extent that  benefits realized from the implementation of One PI help the agency meet  program integrity goals.
section_title: Conclusions
paragraphs: IDR and One PI program officials have made progress in developing and  implementing IDR and One PI to support CMS’s program integrity  initiatives, but the systems do not yet provide all the data and functionality  initially planned. Additionally, CMS program integrity officials have not yet  taken appropriate actions to ensure the use of IDR and One PI on a  widespread basis for program integrity purposes. Further, program  officials have not defined plans and reliable schedules for incorporating  the additional data into IDR that are needed to support its program  integrity goals. Until the agency takes these steps, it cannot ensure that  ongoing development, implementation, and deployment efforts will  provide the data and technical capabilities needed to improve program  integrity analysts’ capabilities for detecting potential cases of fraud, waste,  and abuse.
Furthermore, because the systems are not being used as planned, CMS  program integrity officials are not yet in a position to determine the extent  to which the systems are providing financial benefits or supporting the  agency’s initiatives to meet its program integrity goals and objectives.  Until it does so, CMS officials will lack the means to determine whether  the use of the systems contributes to the agency’s goal of reducing the  number and amounts of improper payments made as a result of fraudulent,  wasteful, or abusive claims for Medicare and Medicaid services.  Furthermore, the contribution of IDR and One PI to the agency’s efforts to  save billions of dollars each year attributable to improper payments made  due to fraud, waste, and abuse in the Medicare and Medicaid programs  will remain unknown.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that the development and implementation of IDR and One  PI are successful in helping the agency meet the goals and objectives of its  program integrity initiatives, we are recommending that the Administrator  of CMS take the following seven actions:  finalize plans and develop schedules for incorporating additional data into  IDR that identify all resources and activities needed to complete tasks and  that consider risks and obstacles to the IDR program; implement and manage plans for incorporating data in IDR to meet  schedule milestones;    establish plans and reliable schedules for training all program integrity  analysts intended to use One PI;   establish and communicate deadlines for program integrity contractors to  complete training and use One PI in their work;   conduct training in accordance with plans and established deadlines to  ensure schedules are met and program integrity contractors are trained  and able to meet requirements for using One PI;   define any measurable financial benefits expected from the  implementation of IDR and One PI; and    with stakeholder input, establish measurable, outcome-based performance  measures for IDR and One PI that gauge progress toward meeting program  goals.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by HHS’s Assistant  Secretary for Legislation and reprinted in appendix II, CMS stated that it  concurred with all of our recommendations and identified steps agency  officials were taking to implement them. Among these were actions to  further refine training plans to better ensure that program integrity  contractors are trained and able to meet requirements to use One PI, along  with efforts to define measurable financial benefits expected from  augmenting the data in IDR. If these and other identified actions are  implemented in accordance with our recommendations, CMS will be  better positioned to meet the goals and objectives of its program integrity  initiatives. The agency also provided technical comments, which were  incorporated as appropriate.
As we agreed with your offices, unless you publicly announce the contents  of this report earlier, we plan no further distribution of it until 30 days  from the date of this letter. At that time, we will send copies of this report  to appropriate congressional committees, the Administrator of CMS, and  other interested parties. The report will also be available at no charge on  the GAO Web site at http://www.gao.gov.
If you or your staffs have questions about this report, please contact me at  (202) 512-6304 or melvinv@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of our review were to (1) assess the extent to which the  Centers for Medicare and Medicaid Services (CMS) has developed and  implemented the Integrated Data Repository (IDR) and One Program  Integrity (One PI) systems and (2) determine the agency’s progress toward  achieving defined goals and objectives for using the systems to help detect  fraud, waste, and abuse in the Medicare and Medicaid programs.
To assess the extent to which IDR and One PI have been developed and  implemented, we collected and analyzed agency documentation that  described planning and management activities. Specifically, we assessed  project management plans and artifacts that described the status of the  systems, such as program management review briefings to technical  review boards, and memoranda approving continued development and  implementation of the systems at key decision points in the systems’  lifecycles. We observed the operation of CMS’s data center where IDR is  installed and viewed a demonstration of the One PI portal and analytical  tools. We also discussed with officials from CMS’s Office of Information  Services and Center for Program Integrity plans for and progress made  toward developing and implementing the systems. We focused our analysis  on the extent to which the development and implementation of IDR and  One PI met system and business requirements and plans for deploying the  systems to CMS’s program integrity analysts.
To assess the agency’s processes for defining system requirements, we  reviewed IDR and One PI requirements management plans, system  requirements, and documentation that traces requirements to functionality  provided by the systems at different stages of implementation. Program  documents we reviewed include the 2007 IDR Medicare Program Integrity  Requirements, the 2006 One PI Startup Findings Draft, the 2010 One PI  Requirements Management Plan, and detailed software requirements  specifications for One PI. In addition, we discussed with IDR and One PI  program officials their requirements development and management  processes and procedures. We then assessed the department’s current  approach to requirements development and management against best  practices identified in the Software Engineering Institute’s Capability  Maturity Model Integration.
To assess schedule estimates of the IDR and One PI programs, we used  criteria defined in GAO’s cost estimating and assessment guide to  determine the extent to which relevant schedules were prepared in  accordance with best practices that are fundamental to estimating reliable  schedules. We identified information reported to the Office of  Management and Budget (OMB) by CMS in fiscal year 2010 that defined  program schedule estimates for the remaining lifecycles of the programs  through 2016. We collected and analyzed program documentation that  supported these estimates, such as work breakdown structures and  staffing estimates.
To assess each program’s schedule estimates, we rated the IDR and One PI  program management offices’ implementation of nine scheduling best  practices defined in our guidance. Based on these criteria, we analyzed the  One PI integrated master schedule and the IDR validation, along with  supporting documentation, and used commercially available software  tools to assess the schedules. Specifically, we determined whether each  schedule was developed by identifying and including critical elements of  reliable scheduling best practices, such as identifying all resources needed  to conduct activities, and whether risk assessment and contingency plans  had been conducted for the schedules.
We shared our guidance, the criteria against which we evaluated the  program’s schedule estimates, as well as our preliminary findings with  program officials. We then discussed our preliminary assessment results  with the program management officials. When warranted, we updated our  analyses based on the agency response and additional documentation  provided to us. We also analyzed changes to the program schedules over  time.
To determine the reliability of the data used to assess schedule estimates,  we used a scheduling analysis software tool that identified missing logic  and constraints, and checked for specific problems that could hinder the  schedule’s ability to dynamically respond to changes. We examined the  schedule data to identify any open-ended activities (i.e., activities with no  predecessor or successors), and searched for activities with poor logic,  such as activities with constraints that keep the schedule rigid (e.g., start  no earlier than, finish no later than, etc.). We found the data sufficiently  reliable for the purposes of this review.
To determine the number of system end users for One PI, we identified the  universe of analysts trained to use One PI by examining documentation  provided by CMS. Specifically, we obtained a list of trained users from the  Center for Program Integrity. From that list, we selected program integrity  analysts whom CMS identified as using the system to conduct analyses of  IDR data to identify potential cases of fraud, waste, and abuse. We then  compared this selection of analysts to data generated by the One PI system  that recorded user login data from January 3, 2010, through October 16,  2010, to identify the current population of One PI users. Through this  analysis, we identified 41 trained program integrity analysts who had used  the system during the designated time period, including 8 Medicare drug  integrity contractors, 23 zone program integrity and program safeguard  contractors, and 10 CMS program integrity analysts.
To ensure that the data that we used to identify One PI users were reliable,  we held discussions with CMS officials who were knowledgeable of the  user community and mechanisms for accessing the system. We discussed  with them the list of trained end users and the computer-generated login  information provided by the system. We also discussed the reliability of  the computer-generated system login information. Specifically, agency  officials confirmed that the data reported by the system were complete  and accurate and that the method we used to identify active users—an  analysis of system login data—was valid.
To determine the extent to which the IDR and One PI programs have  achieved defined goals and objectives for using the systems to help detect  fraud, waste, and abuse, we collected CMS’s analyses of projected costs  and benefits for IDR and One PI. We also collected and assessed data  reported on the costs and benefits realized through the current  implementation of the systems. To do so, we compared (1) actual costs  and benefits attributed to each system through fiscal year 2010 and (2)  current estimated total lifecycle costs and benefits for each system. We  calculated the expected net benefit by subtracting estimated and actual  system costs from estimated and actual system benefits for each system.  To understand how costs and benefits for each system were derived, we  met with officials from the Office of Information Services and from the  Center for Program Integrity and discussed CMS’s processes for  estimating and tracking costs and benefits of both IDR and One PI. We  also obtained from agency officials documentation about and descriptions  of qualitative benefits provided by both systems. Additionally, we  reviewed planning documents that described the goals and objectives of  both programs, along with other documentation that described actions  taken to address program goals and objectives. We reviewed and assessed  supporting documentation for the measures, which the agency reported to  OMB as having been met.
To determine if CMS’s approach to developing performance measures for  IDR and One PI was consistent with federal guidance, we examined  documents describing CMS’s approach and held discussions with program  officials about practices they followed when defining performance  measures and targets. We compared program officials’ practices to  guidance defined by OMB. We also compared the performance measures  defined for the two programs to CMS’s goals and objectives for program  integrity initiatives to determine if the IDR and One PI measures supported  intended outcomes of agencywide efforts to better detect fraud, waste,  and abuse. We supplemented our documentation review with interviews of  officials from the Center for Program Integrity and the Office of  Information Services to obtain additional information about the  development of current and future performance measures for IDR and One  PI. During our interviews, we discussed performance measures and  strategic goals and initiatives for One PI and IDR, and the extent to which  the agency involved internal and external stakeholders in the development  of performance measures.
To obtain information about the extent to which One PI has been deployed  and is being used by a broad community of program integrity analysts to  meet CMS’s goals and objectives, we invited the 41 users we identified in  addressing the first objective of this engagement to participate in  facilitated discussions about the data and tools needed to support fraud,  waste, and abuse detection. Thirty-two of those 41 users attended the  discussion group meetings. During those meetings, we discussed the  following topics: usage of One PI tools and data from IDR, comparison and  contrasting of One PI and IDR with other tools and data sets, and benefits  and challenges of using One PI and IDR for detecting fraud, waste, and  abuse. We also discussed users’ needs for analytical tools and data and for  systems training. After those discussions, we sent written questions to all  32 discussion group participants to obtain more detailed information  about their use of analytical tools and data sources. Thirty-one  participants responded and provided additional supplementary  information about their use of One PI and IDR.
For each of the objectives, we assessed the reliability of the data we  analyzed through interviews with agency officials knowledgeable of the  user community and training program, mechanisms for accessing the  systems, and the methods for tracking and reporting costs and schedules  of the IDR and One PI programs. We found the data sufficiently reliable for  the purposes of this review.
We conducted this performance audit from June 2010 through June 2011 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix II: Comments from the Department of Health and Human Services
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Teresa F. Tucker (Assistant  Director), Sheila K. Avruch (Assistant Director), April W. Brantley, Clayton  Brisson, Neil J. Doherty, Amanda C. Gill, Kendrick M. Johnson, Lee A.  McCracken, Terry L. Richardson, Karen A. Richey, and Stacey L. Steele    made key contributions to this report.