Health Insurance Marketplaces: CMS Needs to Improve Its Oversight of State IT Systems' Sustainability and Performance
Published: 2017-08-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Patient Protection and Affordable Care Act required the establishment of health insurance exchanges—or marketplaces—to allow consumers to compare, select, and purchase health insurance plans. States can elect to establish a state-based marketplace, or cede this authority to CMS to establish a federally facilitated marketplace. Some states had difficulties with the rollout and operation of their marketplaces, and some states that struggled with IT implementation are now using the federal marketplace IT platform.
GAO was requested to review CMS's and states' actions to implement the marketplaces. This report (1) describes CMS's actions to assist states that have chosen to transition to a different marketplace IT platform and identify costs and challenges those states incurred in making this transition; (2) assesses CMS's actions taken to assist selected states to ensure that the development and operations of marketplace IT systems can be financially self-sustained; and (3) assesses CMS's steps to monitor the performance of the states' marketplace IT systems. GAO reviewed documentation from CMS and four states selected based on different types of marketplaces, federal grants provided, and enrollment numbers, and interviewed CMS and the states' officials.
section_title: What GAO Found
paragraphs: The Department of Health and Human Services' (HHS) Centers for Medicare & Medicaid Services (CMS) has offered assistance through providing periodic oversight and issuing regulation and guidance to states transitioning from state-based marketplaces to the federally based marketplace IT platform, including two states that GAO reviewed—Hawaii and Oregon—that had made that transition. While CMS provided these states with assistance, documented CMS transition guidance was not finalized until after the two states had completed their transition. The two states incurred costs of approximately $84.3 million, collectively, to transition to the federal platform. The two states' transition efforts included making changes to their Medicaid systems, with these states mainly relying on Medicaid matching funds from CMS to do this. While the selected states successfully transitioned, they encountered challenges during their transitions, due to accelerated transition time frames, difficulties reassigning marketplace responsibilities, delays in receiving approvals from CMS, and trouble accessing historical consumer data in previous vendor-developed marketplace IT systems.
CMS took steps to assist Hawaii and Oregon, as well as two states that GAO selected for review that operated state-based marketplaces, Minnesota and New York, in developing plans for marketplace IT system sustainability. CMS assisted these four states by consulting with the states' officials and providing oversight of their sustainability plans, financial audit reports, and risk assessments. However, CMS did not fully ensure the states provided complete sustainability plans and financial audit reports. Further, CMS did not base its risk assessments on fully defined processes. These weaknesses limit CMS's oversight and assurance that it can be informed of the state marketplaces' sustainability efforts.
Although CMS established a process to monitor the performance of state-based marketplaces, CMS did not consistently follow its processes. For example, CMS did not ensure that the two selected states, Minnesota and New York, had developed, updated, and followed their performance measurement plans. Also, CMS did not conduct reviews to analyze the operational performance of these states' marketplace IT systems against an established set of parameters. Further, while CMS collected IT performance metrics from the two states, such as the number of electronic enrollments and website traffic volume, it did not link state metrics to goals or establish targets for performance. These weaknesses limit CMS's ability to determine if states' marketplace systems are performing efficiently, effectively, and to provide early warnings of potential problems (see table).
section_title: What GAO Recommends
paragraphs: GAO recommends that CMS take six actions: ensure that states provide complete sustainability plans; complete financial audit reports; fully define its risk assessment process; complete updated performance measurement plans; align metrics with goals; and conduct operational analysis reviews. HHS concurred with two, partially concurred with two, and did not concur with two of GAO's recommendations, which GAO continues to believe are valid.

section_title: Letter
paragraphs: The Patient Protection and Affordable Care Act (PPACA), signed into law  on March 23, 2010, included provisions to reform aspects of the private  health insurance market and expand the availability and affordability of  health care coverage. The act required the establishment of health  insurance exchanges, now commonly referred to as “marketplaces,” in  each state by January 1, 2014. Marketplaces are required to allow  consumers and small employers to compare, select, and purchase health  insurance plans offered by participating private issuers of qualified health  plans. The Department of Health and Human Services’ (HHS) Centers  for Medicare & Medicaid Services (CMS) is responsible for overseeing  the establishment and operation of these marketplaces, including a  federally facilitated marketplace in states that do not choose to operate  their own. For their part, states are responsible for undertaking various  efforts, including information technology (IT) projects needed to support  the establishment and operation of their marketplaces or connections to  the federal marketplace.
PPACA places certain requirements on the design and function of the  marketplaces. For example, regardless of the type of marketplace  established, the marketplace must be able to determine individuals’  eligibility and enroll them in health insurance plans, ensure qualified  health plans are certified, conduct consumer assistance and outreach,  and have the necessary IT infrastructure in place. In addition, the  marketplaces must be able to determine eligibility for other health  coverage programs, such as Medicaid or the State Children’s Health  Insurance Program (CHIP).
These marketplaces began facilitating enrollment on October 1, 2013, as  required by a regulation implementing PPACA. In addition, the  marketplaces were to be self-sustaining beginning January 1, 2015.  However, we found that in the first few enrollment periods, all states faced  difficulties with the rollout and operation of their marketplace IT, and a  number of states that struggled with the implementation of their  marketplaces are now using the federal platform.
Recognizing states’ marketplace efforts, you requested that we review  CMS’s and states’ actions related to implementing the marketplaces. Our  objectives for this review were to: (1) describe what actions CMS has  taken, if any, to assist states that have chosen to transition to a  marketplace IT platform different from the one they originally used and  identify the costs and challenges for states in making this transition; (2)  assess what actions CMS has taken to assist selected states’ plans to  ensure that the development and operations of marketplace IT systems  can be financially self-sustained; and (3) assess the steps that CMS has  taken to monitor the performance of the states’ marketplace IT systems.
To address the objectives, we reviewed marketplace activities conducted  by a selection of the 17 states that operated their own marketplaces as of  March 2016. To make the state selections, we considered four factors for  the plan year 2016 enrollment period: total enrollment, total federal  marketplace grant dollars, a previous GAO review, and whether or not the  state transitioned its marketplace to the federal platform. State  enrollment numbers were sorted from highest to lowest for each of the 17  states and then used to divide states into four groups. Within each  group, we selected the states with the highest total federal marketplace  grant dollars awarded. We excluded states that had been included in a  recent GAO review of state health insurance marketplace IT. We also  ensured the selection included states that transitioned to the federal  platform. The selection resulted in two states using state-based  marketplaces (Minnesota and New York) and two states that had  transitioned from state-based marketplaces to using the federal platform  (Hawaii and Oregon). The four selected states were based on a  nongeneralizable sample and, thus, findings from our assessments of  these states cannot be used to make inferences about the full population  of all state marketplaces.
To address the first objective, we obtained and analyzed CMS’s transition  guidance that was distributed to all states. We also reviewed the actions  CMS and states performed, such as communications and transition planning for the two selected states—Hawaii and Oregon—that  transitioned from state-based to federal marketplace IT systems. Further,  to determine associated transition costs for the states, we reviewed  CMS’s and the selected states’ relevant budget and grant documentation.  We interviewed marketplace officials within the two selected states to  further identify reported transition costs and challenges faced during their  transitions. In addition, we interviewed CMS officials regarding identified  challenges and their actions to assist the two states in addressing them.
To address the second objective, we reviewed CMS’s guidance and  assistance to states regarding financial self-sustainability. Specifically, we  reviewed CMS’s procedures for financial audit and sustainability plan  collection, risk assessments, and sustainability consults. We also  reviewed the four selected states’ development plans, plans for  sustainability, financials audits, and grant documentation. We then  compared the states’ actions to applicable laws, regulation, leading  practices, and CMS guidance. We analyzed CMS’s sustainability  guidance provided to all states, as well as the four selected states’  sustainability plans. To understand how CMS monitors the four selected  states’ efforts in sustaining the marketplaces, we observed CMS’s and  these states’ web-based management tools for reporting and tracking  marketplace self-sustainability.
For the third objective, we analyzed CMS guidance provided to state- based marketplaces which called for the monitoring and tracking of the  performance of their marketplace IT systems. We also analyzed the  selected states’ system performance measurement plans and reports,  where available. In this regard, our evaluation included Minnesota and  New York, which operated state-based marketplace IT systems, but did  not include Hawaii and Oregon, which relied on the federal marketplace  IT platform operated by CMS and, thus, did not independently collect  system performance metrics. We reviewed CMS’s and the two selected  states’ (Minnesota and New York) use of tools, such as CMS’s  Collaborative Application Lifecycle Tool, to facilitate the monitoring of the  states’ marketplace operations and performance. In addition, we  determined if CMS followed leading practices for IT performance  measurement by comparing its marketplace performance guidance and  the selected states’ plans and reporting to leading practices.
For all three objectives, we supplemented the information and knowledge  obtained from our assessments of marketplace IT program, project, and  technical documentation by holding discussions with relevant CMS  officials and conducting interviews with officials at the four selected state  sites about their marketplaces. Additional details on our objectives,  scope, and methodology are provided in appendix I.
We conducted this performance audit from February 2016 to August 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: PPACA directed each state to establish and operate a health insurance  marketplace by January 1, 2014. In states electing not to establish a  marketplace, the law required HHS (which delegated this role to CMS) to  do so. These marketplaces were intended to provide a seamless, single  point of access for individuals to enroll in private health plans and apply  for income-based financial assistance established under the law. CMS  reported that around 12.7 million individuals applied to enroll in healthcare  coverage for plan year 2016.
PPACA and HHS regulations and guidance required each marketplace to  be able to carry out four key functions, among others:  1.  Eligibility and enrollment. Assess and determine an individual’s  enrollment eligibility, enroll eligible individuals in coverage, and certify  private health insurance plans for participation in the marketplace.  2.  Plan management. Provide services for activities such as submitting,  monitoring, and renewing qualified health plans.  3.  Financial management. Facilitate payments to health insurance  issuers as well as provide services such as payment calculation for  risk-adjustment analysis and cost-sharing reductions for individual  enrollments.  4.  Consumer assistance. Provide assistance to consumers in  completing an application, obtaining eligibility determinations,  comparing coverage options, and enrolling in coverage.
To provide these capabilities, PPACA further required the states  establishing marketplaces, as well as CMS, to design, develop,  implement, and operate health insurance marketplace IT systems.
States have some flexibility as to the marketplace types, such as:
State-based marketplaces. These states developed their own  marketplaces with final decision-making authority, and were provided  full autonomy in setting user fees and establishing sustainability  plans. The marketplaces varied by state, depending on each state’s  current and previous health care systems environment, governance  and business models, applicable laws and regulations, and other  factors.
State-based marketplaces using the federal platform. These  states initially elected to develop their own state marketplace systems,  but due to IT, financial, or other challenges, subsequently decided to  use the federal platform to perform certain eligibility and enrollment  functionalities. Issuers of health insurance in these states offer plans  through the federal platform and are charged a user fee by CMS for  the services and benefits they provide. These states also have the  authority to collect user fees beyond those collected by CMS. Federal  responsibilities, carried out by CMS, include managing the federal IT  platform, call center infrastructure, and eligibility determinations. The  states must maintain websites that provide information to consumers  and direct them to HealthCare.gov to apply for and enroll in  coverage. In addition, these states must coordinate with CMS on  outreach strategies to reach existing and new consumers; maintain  data and timely reporting for all coverage years prior to the transition;  and work with issuers to ensure they are prepared to transition to the  federal platform and exchange enrollment data with CMS.
Federally facilitated marketplaces. These states elected not to  develop their own platform and use the federally facilitated  marketplace—the federal IT platform, including the website,  HealthCare.gov.
Federally facilitated partnerships. These states are a variation of  the federally facilitated marketplaces, whereby CMS establishes and  generally operates the marketplaces and the states assist CMS with  operating various functions, such as plan management and consumer  assistance. In these cases, states rely to varying degrees on the  systems developed by CMS to support the federally facilitated  marketplaces and the federal government keeps the user fees paid by  the insurers.
For plan year 2016, utilizing state and CMS documentation, we  categorized 27 states as having federally facilitated marketplaces, 7 as  having federally facilitated partnerships, 13 as having state-based  marketplaces, and 4 as having state-based marketplaces using the  federal platform. Figure 1 shows the type of health insurance marketplace  used by each state for plan year 2016.
subsections: section_title: IT Funding for States to Establish, Operate, and Sustain Their Marketplaces
paragraphs: To establish, operate, and sustain health insurance marketplaces, states  have used several IT funding sources, which vary based on the  marketplaces’ operational model and the IT work being performed.
Marketplace grants: PPACA authorized CMS to award federal exchange  (marketplace) grants through December 2014 for planning and  implementation activities, as well as for the first year of a marketplace’s  operation. The federal marketplace grant stages include:
Pre-award. A funding announcement notified states about the grant  opportunity. States then submitted applications for CMS review.
Award. CMS identified successful applicants and awarded funding to  be used during specific time periods.
Implementation. Grantees draw down funds from preauthorized grant  accounts monitored by CMS, and report financial and performance  information to the agency. States are allowed to request a no-cost  grant extension in order to use remaining approved federal funds to  complete project goals and objectives. CMS reviews each no-cost  extension request to ensure that expenses are allowable, correctly  allocated, and reasonable based on PPACA and CMS grant rules and  policies. States also can resubmit budget requests to CMS to  authorize reallocation of grant funding among previously awarded  budget categories.
Closeout. This occurs after the period of performance ends and  includes preparation of final reports, financial reconciliation, and any  required accounting for property and funds. After a grant is closed,  CMS deobligates any remaining amounts. (In this report, we use  “deobligated funds” to describe amounts that are no longer available  to CMS for new obligations, although they may remain available for  certain limited other purposes.)
Medicaid funding: PPACA enactment required changes to Medicaid  eligibility and enrollment systems so that the program could operate  seamlessly with the marketplaces, as well as to implement new Medicaid  eligibility policies. Under federal law, states are eligible for an enhanced  federal matching rate of 90 percent (referred to as 90/10 funding—states  contribute 10 percent of the cost) for the design, development, or  installation of Medicaid claims processing and information retrieval  systems. Because states’ Medicaid systems had to be replaced or  modernized to meet PPACA requirements, CMS expanded the availability  of the 90/10 funding for states to make changes to improve Medicaid  eligibility IT systems, including the connection to the federal  marketplace. In addition, a state could receive funding in the form of a  75 percent federal matching rate for the maintenance and any ongoing  costs of operating its upgraded Medicaid eligibility and enrollment system.  The funding is generally available when the upgraded system becomes  operational. States were also allowed to use Medicaid funds when they  transitioned to a different marketplace type since the transition costs were  related to updating state Medicaid eligibility systems to be compatible with  the federal platform.
Revenue through user fees: PPACA authorized state-based  marketplaces to charge assessments, or user fees, such as a percentage  or a flat monthly rate to participating insurance issuers. For the selected  states to sustain their marketplaces, Hawaii charged a user fee of 3.5  percent per insurance plan, Minnesota charged a 3.5 percent user fee,  and Oregon charged a user fee of $9.66 per member per month. New  York does not impose a user fee.
Additional revenue sources: In addition to user fees, states have  leveraged their own funds to support and sustain marketplaces, including  general funds, fund transfers from other agencies, and broad-based  assessments. Examples of these additional revenue sources for the four  selected state marketplaces include:
Hawaii’s marketplace received funding from state general funds.
Minnesota’s marketplace incurred costs associated with enrollment of  individuals into public programs by the Minnesota Department of  Human Services. Payments from the Minnesota Department of  Human Services represent its share (in both federal and state funds)  of enrollment costs in public programs.
New York’s marketplace received revenue from a broad-based  assessment funded by the Health Care Reform Act instead of from  user fees. According to New York officials, that revenue included  assessments of certain medical services and is used to support  several health care programs in the state, including the New York  marketplace.
Oregon’s marketplace received reimbursements from other state  agencies for shared software license fees.
Operating reserves: Some states also implemented dedicated state  financial reserves for marketplace operations, including IT. Reserves can  cover increases in expenditures, such as unexpected development costs,  or decreases in revenue, such as decreased user fees due to lower than  expected enrollment. Minnesota and Oregon marketplaces were  legislatively allowed to use such reserves. Minnesota had less than 1  month of reserves and Oregon’s marketplace had 12 months of reserves  in 2016. Hawaii and New York did not allow their marketplaces to have  reserves.
Table 1 provides a summary of the four selected states’ marketplace  revenue sources and reserves.
section_title: CMS Oversight of State Marketplace IT Projects
paragraphs: To address the requirements of PPACA and its implementing policies,  HHS designated CMS to provide oversight of the IT supporting states’  marketplaces, and CMS assigned three key offices to do so—the Center  for Consumer Information and Insurance Oversight (CCIIO), the Center  for Medicaid and CHIP Services (CMCS), and the Office of Technology  Solutions (OTS). CMS relies on these offices to, among other things,  perform the following reviews:
Establishment reviews. CMS is to conduct establishment reviews of  states that receive federal marketplace grant and Medicaid funds,  following the Enterprise Life Cycle framework. These reviews are  intended to show the progress that the states made in using federal  funding to implement marketplace IT systems. The framework  requires states to provide specific artifacts for CMS review, such as  the concept of operations, system test documents, and project plans,  among others. Each review is incremental and states are expected to  show CMS an acceptable level of progress and maturity in their  projects’ development before proceeding to the next project phase.
Review of annual state financial/programmatic reports.
State-based marketplace annual reporting tool (SMART): CMS  collects and reviews state-based marketplace financial and  programmatic data regularly through SMART. The agency  collects and reviews annual financial/programmatic audit reports,  and attestations of compliance as part of this process. CMS staff  identify observations, action items, and ongoing monitoring  activities that could improve marketplace operations. For example,  they may identify areas where IT functionality needs improvement,  or more clarity on IT expenditures in marketplace financial audit  reports.
Oversight and program integrity standards for state exchanges:  Beyond SMART, CMS requires state-based marketplaces to also  annually provide financial statements, eligibility and enrollment  reports, and performance monitoring data.
Frequent, regular communication. CMS is to monitor and provide  assistance to states through frequent and regular communication,  including weekly telephone calls with state officials involved with  marketplace IT efforts. CMS includes subject matter experts in these  calls, as needed. State officials can report concerns or provide further  information during the weekly calls.
section_title: Establishment and Operations of Selected State Marketplaces
paragraphs: The selected state marketplaces—Hawaii, Minnesota, New York, and  Oregon—each had different experiences with regard to establishing and  operating their marketplaces. Each state’s establishment, operations, and  funding experiences are summarized in the discussions and figure 2  below, which are based, in part, on CMS budget summary documentation  for the states as of October 2016.
subsections: section_title: Hawaii
paragraphs: Since 2011, CMS had awarded Hawaii $205.3 million in federal  marketplace grants to establish a state-based marketplace. As of October  2016, the state had spent or planned to spend $140.4 million of those  grants, including $97.4 million for IT costs associated with building a  health insurance marketplace and developing infrastructure needed for  the marketplace’s ongoing operations.
The Hawaii Health Connector implemented the original state-based  marketplace, which began operation in 2013. In 2015, the marketplace  ceased operations and transitioned to a state-based marketplace using  the federal platform. According to CMS, the transition was due to a variety  of system issues and other factors that limited enrollment and had a  negative impact on the consumers’ use of the system. Hawaii decided to  undergo an additional transition in June 2016 in order to operate as a  federally facilitated marketplace for plan year 2017. As a result of the  state’s marketplace transition efforts, CMS deobligated $63 million of  Hawaii’s grant funds as of October 2016 for grants that had concluded  their period of performance and had been closed. As of March 2016,  approximately 15,000 individuals in Hawaii were enrolled in qualified  health plans.
section_title: Minnesota
paragraphs: Since 2011, CMS had awarded $189.4 million in federal marketplace  grants to Minnesota to build a health insurance marketplace and develop  infrastructure needed for the marketplace’s ongoing operations. MNsure  had implemented the state-based marketplace and spent or planned to  spend $159.6 million in grant funds as of October 2016, which included IT  expenditures of $48.2 million. Additionally, CMS had deobligated  $102,000 of Minnesota’s grant funds as of October 2016 for grants that  had concluded their period of performance and had been closed.
Minnesota is 1 of 12 states that have an integrated system for both  Medicaid and qualified health plans and is 1 of 2 states that have a Basic  Health Program. This allows Minnesota to combine eligibility  determinations for qualified health plans, Medicaid, and the state Basic  Health Program. As of March 2016, Minnesota had enrolled  approximately 84,000 individuals in qualified health plans.
section_title: New York
paragraphs: Since 2011, CMS had awarded $575.1 million in federal marketplace  grants to the state of New York to build a health insurance marketplace  and develop infrastructure needed for the marketplace’s ongoing  operations. The New York Health Benefit Exchange (later renamed the  New York State of Health) within the New York State Department of  Health implemented the state-based marketplace and had spent or  planned to spend $513.6 million in marketplace grants as of October 2016, to include $209.2 million for IT costs. Additionally, CMS deobligated  $4.5 million of New York’s grant funds as of October 2016 for grants that  had concluded their period of performance and had been closed.
New York is the other state, along with Minnesota, to have implemented a  Basic Health Program. As of March 2016, New York State of Health had  enrolled approximately 272,000 individuals in qualified health plans.
section_title: Oregon
paragraphs: Since 2011, CMS had awarded Oregon $305.2 million in federal  marketplace grants to build a health insurance marketplace and develop  infrastructure needed for the marketplace’s ongoing operations. Oregon  had spent $301.4 million in federal marketplace grants as of October  2016, which included IT spending of $79.7 million. Additionally, CMS had  deobligated $1.8 million of Oregon’s grant funds as of October 2016 for  grants that had concluded their period of performance and had been  closed.
Cover Oregon implemented the original marketplace, which operated  from 2013 to 2014 as a state-based marketplace. The state-based  marketplace then transitioned to a state-based marketplace using the  federal platform under the Oregon Department of Consumer and  Business Services, due to the technical challenges of IT development and  creating a functional website to enroll consumers. As of March 2016,  approximately 147,000 individuals had selected a qualified health plan.
The four selected states’ overall marketplace federal grant funding,  expenditures, including amounts planned to be spent, and deobligations,  as well as their enrollment as of plan year 2016 are reflected in figure 2.
section_title: Prior GAO Reports Highlighted Issues with and Areas to Improve Health Insurance Marketplaces
paragraphs: Over the past 4 years, we have issued various reports highlighting  challenges that CMS and the states have faced in implementing and  operating health insurance marketplaces. For example, in an April 2013  report, we described the actions of seven states that were in various  stages of developing an IT infrastructure to establish marketplaces,  including redesigning, upgrading, or replacing their outdated Medicaid  and CHIP eligibility and enrollment systems. Six of the seven states  were also building the IT infrastructure needed to integrate systems and  allow consumers to navigate among health programs. However, the  states had identified challenges with the complexity and magnitude of the  IT projects, time constraints, and guidance for developing their systems.
In December 2014, we reported that all states using the federal  marketplace IT solution had faced challenges transferring applications to  and from that system. We pointed out that none of the states using the  federal marketplace IT solution in the first enrollment period were able to  implement application transfers, which required the establishment of two  IT connections: one connection to transfer applications found ineligible for  Medicaid coverage from the state Medicaid agency to the federal  marketplace IT solution, and another connection to transfer applications  found ineligible for coverage from the federally facilitated marketplace to  the state Medicaid agency.
In March 2015, we reported that several problems with the initial  development and deployment of HealthCare.gov and its supporting  systems had led to consumers encountering widespread performance  issues when trying to create accounts and enroll in health plans. We  noted, for example, that CMS had not adequately conducted capacity  planning, adequately corrected software coding errors, or implemented all  planned functionality. In addition, the agency did not consistently apply  recognized best practices for system development, which contributed to  the problems with the initial launch of HealthCare.gov and its supporting  systems. In this regard, weaknesses existed in the application of  requirements, testing, and oversight practices. Further, we noted that  CMS had not provided adequate oversight of the HealthCare.gov initiative  through its Office of the Chief Information Officer.
We made seven recommendations aimed at improving requirements  management, system testing processes, and oversight of development  activities for systems supporting HealthCare.gov. CMS concurred with all  of our recommendations and subsequently took or planned steps to  address the weaknesses, including instituting a process to ensure that  functional and technical requirements are approved; developing and  implementing a unified standard set of approved system testing  documents and policies; and providing oversight for HealthCare.gov and  its supporting systems through the department-wide investment review  board.
Further, in September 2015, we reported that states had spent $3.22  billion in federal grant funding to establish their health insurance  marketplaces, of which approximately $1.45 billion was to establish IT  systems supporting their health insurance marketplaces. However, we  noted limitations in CMS’s oversight of the states’ IT system development,  such as a lack of clearly defined roles and responsibilities leading to  challenges with states receiving timely CMS guidance and relevant CMS  senior executives not always being involved in state IT funding decisions.  We also noted that CMS had allowed state systems to be launched  without being fully tested. This led to consumers in some states  experiencing long waits for eligibility determinations, websites freezing,  system failures, and manual enrollments. We made three  recommendations aimed at improving CMS oversight in those areas and  CMS concurred with all of the recommendations, noting actions they had  taken or planned to take in each of the areas. The agency’s plans to  implement the recommendations included steps to develop a  comprehensive communication plan, planned coordination between all  relevant business and IT units to review and approve state requests for  federal IT funds, and continued collaboration with states to test their  system functionality.
section_title: CMS Offered Assistance to Transitioning States, but Selected States Encountered Challenges in Their Efforts
paragraphs: CMS offered various types of assistance to states transitioning to a  different marketplace IT platform, including to the two selected states that  transitioned to the federal platform—Hawaii and Oregon. Among other  things, CMS’s assistance included periodic reviews of these two states’  transition plans and weekly calls to prioritize and address transition  issues. CMS also issued a regulation to all states that included  requirements regarding states’ transition plans and their coordination with  CMS and HHS; however, this regulation was not finalized until after  Hawaii and Oregon had initiated their transition efforts. States’ officials  reported costs of approximately $84.3 million, collectively, to transition to  the federal platform. The two states’ transition efforts included making  changes to their Medicaid systems and the states mainly relied on  Medicaid matching funds from CMS to do this. Further, the two states  encountered challenges during their transitions. These challenges were  related to accelerated transition time frames, difficulties reassigning  functional marketplace responsibilities, delays in receiving approvals and  decisions needed from CMS, and trouble accessing historical consumer  data in previous marketplace IT systems developed by vendors.
subsections: section_title: CMS Offered Assistance to Marketplaces, but Documented Guidance Was Issued after the Two Selected States Had Initiated Their Transitions
paragraphs: Among the assistance that CMS provided for states that sought to  transition to a different marketplace IT platform was conducting periodic  oversight activities. Specifically, for all state-based marketplaces, the  agency conducted oversight through the Open Enrollment Readiness  Reviews, as well as through sustainability consultations and weekly  telephone calls during which CMS reviewed states’ plans and milestones.  The Open Enrollment Readiness Review process involved CMS  reviewing each state-based marketplace’s readiness for the upcoming  open enrollment period by discussing topics such as IT system and  business function readiness, application processing and notices,  inconsistent data, enrollment transactions, and previous reviews’ findings.  After the discussions, follow-up written communication from CMS was  sent to the states highlighting outstanding action items that needed to be  completed.
In addition, CMS sustainability consultations evaluated state  marketplaces’ self-sustainability in the absence of federal funding from  marketplace establishment grants. These sustainability evaluations  included reviews of the state marketplaces’ revenue sources, cash  reserves, management team stability, and Medicaid and marketplace  systems’ integration. Lastly, weekly telephone calls with states included  discussions of the states’ plans and milestones associated with IT  releases and testing, identification of technical issues, and mitigation and  contingency plans.
Further, in March 2016, subsequent to Hawaii and Oregon’s transition to  the federal platform, CMS issued a regulation that set out more defined  transition guidance. The regulation, for the first time, provided generally  applicable guidance about the transition process and associated  requirements. It required state marketplaces that seek to utilize the  federal IT platform for selected functions to, among other things,  coordinate with CMS, including joint development of a transition plan,  which is to consist of a project plan with proposed milestones. CMS  officials are to review the plans and help the states prioritize their planned  transition milestone dates.
Also, in October 2016, CMS finalized an agreement for states seeking to  transition to the federal IT platform. That agreement became effective on  November 1, 2016, with 1-year renewal options. The agreement  established the mutual obligations and responsibilities of the transitioning  states and CMS in areas including eligibility and enrollment, maintenance  of related IT systems, call center operations, and casework support.
However, the issuance of the regulations and agreement discussed  above occurred after Hawaii and Oregon had already begun their April  2014 and June 2015 transitions to the federal platform. Thus, the  requirements and documented guidance included in the regulation and  the agreement were not available to these states when they initially  undertook their transition efforts.
Hawaii and Oregon officials described the assistance that CMS provided  to their states, as follows:
Hawaii began its year-long transition in June 2015 from a state-based  marketplace that was operated by the Hawaii Health Connector to a  state-run, state-based marketplace using the federal platform. The  state completed an additional transition to be a federally facilitated  marketplace in January 2017. At the time that Hawaii carried out its  transition in 2015, CMS had not yet provided the states with  documented guidance; however, CMS officials provided milestones to  Hawaii that needed to be completed to fully transition to the federal  platform. These milestones included dates when the marketplace plan  and issuer data should be submitted and transferred, Medicaid  account transfer functionality should be completed and tested, and  notices and other related enrollee communications should be  completed. In addition, Hawaii officials said that, throughout the  transition, they participated in weekly telephone calls with CMS  subject matter experts to discuss requirements such as consumer  outreach and the interface development needed for account transfers.
Oregon began its transition in April 2014 from a state-based  marketplace operated by the non-profit organization, Cover Oregon,  to a state-run marketplace using the federal platform, under the  state’s Department of Consumer and Business Services. According to  Oregon officials, CMS did not have guidance available at the time that  the state decided to transition. However, state officials said they  participated in many phone calls throughout the transition with CMS  officials within CCIIO and CMCS to identify whether the state  marketplace, state Medicaid, or CMS would be responsible for certain  marketplace functions after the transition from Cover Oregon to the  Department of Consumer and Business Services.
section_title: Transitioning to the Federal Platform Cost the Selected States Approximately $84.3 Million
paragraphs: The total reported IT costs associated with Hawaii’s and Oregon’s  transition to using the federal platform were approximately $84.3 million,  collectively. State officials said that the transition costs were incurred to  make changes to their Medicaid systems and connect their systems to  the federal IT platform. Hawaii and Oregon officials told us that the two  states primarily relied on federal Medicaid matching funds to cover the  cost of their transition efforts.
Hawaii officials reported to us that their total IT costs associated with the  initial transition to using the federal platform, begun in June 2015, were  approximately $27.0 million as of June 2016. According to the officials,  the transition costs primarily covered the development of functionality to  transfer account files between Hawaii’s existing state Medicaid system  and the federal platform. These included expenses for staff to carry out  project management, technical assistance, and independent verification  and validation activities. In addition, the costs covered funding for staff to  perform design, development, and implementation work in order to enable  the system to determine minimum essential coverage. The officials  stated that these transition costs were largely funded by the federal  government through Medicaid 90/10 and 75/25 matching funds.
With regard to the transition to become a federally facilitated marketplace  in June 2016, state officials said the primary reason Hawaii decided to do  so was because the state lacked additional federal grant funds and the  state legislature had denied subsequent requests for the additional  funding for marketplace development and operations. The officials said  that, in November 2016 they determined that, since the state was already  relying on the federal platform for the previous plan year, the only IT work  required to be completed would be a small update to their website. Thus,  there was no cost associated with the second transition in January 2017.
Oregon officials reported to us that their total IT transition costs to the  federal platform, begun in April 2014, were approximately $57.3 million,  as of November 2016. According to the officials, the transition costs  primarily covered the modernization of the state’s legacy Medicaid system  through the use of shared computer software code from the Kentucky  marketplace system and the development of functionality to transfer  account files between its Medicaid system and the federal platform.  Because Oregon’s previous state-based marketplace IT solution was  intended to integrate and modernize the state’s legacy healthcare  systems, the decision to switch to the federal IT solution left the state still  needing to modernize its legacy Medicaid system. In addition, the  transition effort included expenses for staffing, professional services,  computer hardware, and the software and service fees to host the new  Medicaid system.
The new Oregon Medicaid system went live in December 2015. Oregon  officials reported to us that of the $57.3 million in IT costs, approximately  $56.6 million were largely funded by the federal government through  Medicaid 90/10 and 75/25 matching funds and approximately $662,000  were funded by marketplace grants from April 2014 to November 2014.  The latter amount was needed to enable Oregon residents to apply for  coverage through the federal system, HealthCare.gov, for plan year 2015.
Hawaii’s and Oregon’s transition time frames and IT-related costs are  summarized in table 2.
section_title: Selected States Identified Numerous Challenges in Transitioning Their Marketplace IT Systems
paragraphs: While the two selected state-based marketplaces successfully  transitioned to the federal platform, officials from each state identified a  number of challenges they encountered in doing so.
Hawaii officials identified challenges that stemmed from difficulties  dealing with:
Accelerated transition time frames and reassigned marketplace  responsibilities. According to Hawaii officials, the state had originally  planned to transition to the federal platform in October 2016 but then  accelerated its efforts to address the unexpected shutdown of the  Hawaii Health Connector in December 2015, due to financial  problems. The accelerated and abrupt transition forced the state’s  officials into the difficult task of having to find a way to continue  operations and provide consumer support almost a year ahead of  their original plan, since that plan did not have the Hawaii Health  Connector transferring its marketplace responsibilities to state officials  until October 2016. The unexpected shutdown also meant that the  state had to quickly reassign functional marketplace responsibilities  among seven different state departments, which made it more  complicated to coordinate and find funding to continue marketplace- related operations to support the open enrollment for plan year 2016.  Additionally, the reassignment of responsibilities was complicated by  the fact that the Hawaii First Circuit Court appointed a receiver to  understand all of Hawaii Health Connector’s obligations, dissolve its  assets, and settle the organization’s financial dealings with creditors  and debtors.
Delays in communications with CMS. Hawaii officials stated that  the distance and time differences between CMS officials working in  the Eastern time zone and Hawaii officials working in the Hawaii- Aleutian time zone caused delays in communications between these  officials. State officials said the geographical distance and resulting  time difference of 6 hours in working with CMS officials primarily  based in the District of Columbia and Maryland caused delays in  receiving marketplace related information from CMS, such as  notifications about when service disruptions would occur. However,  Hawaii officials noted that, for the most part, this challenge  subsequently was overcome by improved communications with the  agency’s officials, thereby resolving former marketplace service  disruption issues, such as planned federal platform system outages  and automatic re-enrollment of individuals.
Oregon officials identified challenges that stemmed from difficulties  dealing with:
Transitioning without the benefit of a transition guide to follow.
According to Oregon officials, since their state was the first to  transition to the federal platform, there was no other state model or  guide to follow. In addition, CMS at the time, did not have documented  guidance and requirements for states that wanted to transition to a  different marketplace type. This required Oregon officials to spend  many hours in discussions with CMS officials to figure out what  functions CMS and the state each would be responsible for  performing.
Delays in receiving approvals from CMS. According to Oregon  officials, CMS did not always make decisions related to Oregon’s  transition in a timely manner. Oregon officials noted that there were  delays with CMS issuing the approvals needed to allow the state’s  residents to create accounts with HealthCare.gov before open  enrollment started in November 2014. The officials said that being  able to create those accounts in advance would have made  enrollment easier for Oregon residents.
In addition, the federal call center scripts initially were not customized with  the HealthCare.gov system, so the responses Oregon residents received  from HealthCare.gov were not always clear or accurate, according to the  officials. CMS officials within CCIIO and CMCS stated that, at the time  when Oregon began discussions with the agency about transitioning to  the federal platform, CMS was also addressing priorities from the 2014  open enrollment and needed to balance resources spent on Oregon’s  transition with other IT development and enhancement activities that were  needed.
Correcting errors with account transfers in the Medicaid system.
Oregon officials noted that, in December 2015, initial account  transfers between Oregon and CMS encountered errors due to a  configuration issue related to outdated CMS guidance. Oregon had to  make changes to its eligibility system in order to process account  transfers because the actual transfers varied from CMS’s previously  published specifications in certain areas.
Meeting CMS’s 60-day notice for technical changes. State officials  said CMS’s practice of providing approximately 60 days’ notice for  technical changes to account transfer specifications was challenging  to comply with in a timely manner due to the length of time that was  needed for Oregon’s system integrator to make the requisite system  changes.
As noted above, these state transition challenges stemmed from a variety  of issues including compressed time frames, split priorities, and unclear  marketplace responsibilities. Some of these challenges may have been  alleviated if the states had the benefit of documented transition-related  guidance that exists now, available to them when they first initiated their  transitions.
In addition to the aforementioned challenges, for both selected states, a  common challenge involved accessing the historical marketplace IT  systems containing consumer data that had been developed by the initial  contractors for Cover Oregon and the Hawaii Health Connector. In  Hawaii, the court-appointed receiver for the state said the Hawaii Health  Connector system integrator contractor had possession of the previous  Hawaii marketplace IT system that contained the consumer data. As of  November 2016, Hawaii and the contractor were still discussing the terms  of compensation under that contract, and Hawaii officials did not know  when those issues would be resolved.
Oregon officials informed us in July 2016 that their state and its contractor  were in litigation, and the contractor was storing Oregon’s previous  marketplace IT system and the data within it in an archive. Oregon  officials within the Department of Consumer and Business Services  stated that they had a local archive of the data from the previous  marketplace IT system, but did not have access to the actual contractor- based IT system after March 2016.
With regard to this matter, CMS officials within CCIIO and CMCS noted  that their continued work had included extensive discussions with both  states on options and mitigation strategies to retain the data and maintain  compliance with the marketplace 10-year archival requirement for  consumer and enrollment data. The officials said that the historical data  are needed so that states can maintain the ability to process actions that  include marketplace eligibility appeals (which could span multiple years)  and submission of enrollment data.
The CMS officials also stated that they had continued to refine their  operational processes and policy associated with transitioning to the  federal platform, as well as provide guidance and regulatory requirements  for states that transitioned their marketplaces to other models, with  particular focus on those that transitioned to the federal platform. Further,  the officials stated that they continued to work with Hawaii and Oregon to  explore opportunities to enhance the federal platform’s functionality to  better support the state and other state-based marketplaces on the  federal platform, where feasible. They added that their continued work  also included frequent communications with both states to ensure a  smooth coverage transition for residents of each state.
section_title: CMS Assisted Selected States with Their Sustainability Plans, but Did Not Always Ensure Reporting Was Complete and Risk Assessment Processes Were Fully Defined
paragraphs: CMS had processes in place to assist the selected states’ with their  efforts to financially sustain the development and operations of their  marketplaces, including supporting IT systems. These processes included  reviewing sustainability plans, reviewing annual independent financial  audit reports, and conducting and responding to sustainability risk  assessments.
However, in providing its assistance, CMS did not always ensure that the  four selected states’ sustainability plans and financial audit reports were  complete, or that the states had complied with PPACA and CMS  requirements regarding financial audit reporting. Additionally, CMS did  not clearly define its risk assessment processes, as suggested by  Standards for Internal Control.
subsections: section_title: CMS Did Not Always Ensure That Selected States’ Sustainability Plans Were Complete
paragraphs: As previously noted, PPACA required state marketplaces to be self- sustaining as of January 1, 2015, and, in turn, CMS developed  marketplace blueprint requirements to assist states with meeting the act’s  requirements. As part of the blueprint for approval of a state-based  marketplace, the agency required states to submit an operational budget  and management plan for its oversight, and to include proposed budget  information for the upcoming 5 years from the initial year of operations,  and long-term strategies for financial sustainability. According to CMS  officials within CCIIO, the states’ plans are intended to inform CMS  regarding the state-based marketplaces’ sustainability. According to  agency guidance, the plans are also used by CMS to assess and respond  to marketplace sustainability risks.
To aid in the reporting of the sustainability plans, CMS created a 5-year  budget forecast template for states to complete and submit. The template  called for high-level reporting of sustainability factors, to include  marketplace enrollment, revenue, expenditures, reserves, and  marketplace status over 5 years. In addition, contractual spending and  questions related to ongoing IT costs were to be included in the plans.  The 5-year span of budgets and enrollments that were to be reported in  the plans included current-year projections and actuals, as well as  forecast projections for the forthcoming 4 years. Sustainability guidance,  dated October 2016, further required states to update and submit their  sustainability plans to CMS twice a year.
While CMS received complete 2016 sustainability plans from New York  and Oregon, it did not ensure that it had complete sustainability plans with  the full 5-year budgets from Hawaii and Minnesota. Specifically,  Minnesota and Hawaii did not include the entire 4 years of budget  forecasts in their plans. Hawaii’s sustainability plan dated May 2016 only  contained the 2016 budget and enrollment amounts, but forecasts for the  forthcoming 4 years were missing. Instead of forecasted amounts,  Minnesota’s 2016 sustainability plan used duplicated budget and  enrollment amounts, in which certain budget numbers, such as equipment  and supply costs, were the same from 2016 to 2019.
Hawaii, Minnesota, and New York state officials said that CMS did not  provide them policies or procedures to help guide state marketplaces on  aspects of sustainability planning, such as budget and enrollment  forecasts. Minnesota and Oregon officials also reported that completing  5-year budgets was difficult due to problems with forecasting, and that  budgets beyond the regular 3- and 2-year state budgetary cycles,  respectively, were difficult to formulate. CMS officials further stated that  Hawaii’s transition to a federally facilitated marketplace for plan year 2017  had precluded any future budgets or enrollment forecasts.
CMS officials within CCIIO and CMCS stated that they communicate with  states to resolve issues in providing complete budget forecasts. The  officials acknowledged that states face uncertainties because the  marketplaces are new programs, and stated that they are considering  whether they should ask for 3-year budgets verses 5-year budgets.
While asking states for a 3-year budget instead of a 5-year budget may  be less of a burden on states to provide complete budgets, the smaller  timeframe may not fully inform CMS oversight of the long-term financial  sustainability for marketplaces, which are new systems that face multiple  uncertainties. Further, if CMS does not take steps to ensure that states  provide sustainability plans with complete 5-year budget forecasts, per its  guidance, then CMS may not be fully informed of the state-based  marketplaces’ sustainability factors. Incomplete sustainability plans may  also limit the agency’s ability to assess and respond to state marketplace  sustainability risks.
section_title: CMS Did Not Always Ensure Selected State Marketplace Financial Audit Reports Were Completed
paragraphs: PPACA and HHS regulations require state-based marketplaces to provide  an annual independent financial audit report, to include activities,  receipts, and expenditures, for CMS’s oversight. Also, submission of the  independent financial audit report is a requirement of CMS’s annual  reporting SMART process. HHS regulations stipulate that audit reports,  in addition to accounting for receipts and expenditures, should follow  generally accepted government auditing standards (GAGAS).
According to GAGAS, audit reports should include information such as a  review of compliance, internal controls, and related financial policies and  procedures. Further, according to CMS guidance, the independent  financial audit reports are to provide CMS with insight into marketplace IT  self-sustainability efforts and compliance, since IT costs are a large part  of the marketplace budgets. This guidance calls for the agency to use the  reports to inform sustainability risk assessments and responses provided  to the states.
While CMS took steps to collect and review marketplace financial audit  reports, it did not ensure that the four selected states always provided  audit reports or that the reports were complete. CMS reviewed New  York’s and Oregon’s relevant financial audit reports for 2015, which  included the information that GAGAS required, such as information on  compliance and internal controls.
However, CMS did not ensure that Hawaii provided a financial audit  report for 2015. In addition, although Minnesota submitted a financial  audit report in 2015, CMS did not ensure that the report included all  necessary information. For example, the Minnesota report was not  specific to the marketplace business operation or its IT platform and,  instead, included financial activities for all state programs and activities.  While marketplace receipts and expenditures were included, a review of  marketplace compliance, internal controls, and financial policies and  procedures were not. Since these reviews were not included in  Minnesota’s submission of its financial audit report to CMS that year,  CMS did not have visibility into these aspects of the state’s marketplace- specific activities or complete insight into its sustainability efforts.
In discussing their reporting, Hawaii officials in the Office of Community  Services and Department of Human Services said they are not required  to develop a financial audit report now that the state has switched over to  being a federally facilitated marketplace. Further, according to the state’s  officials, a court ruled that an independent financial audit of the defunct  Hawaii Health Connector would be unfeasible and impractical since the  Connector’s records are not amenable to audit, nor is there a Connector  official available to sign off on the audit.
A Minnesota marketplace official responsible for compliance and program  integrity stated that MNsure was in compliance with PPACA and HHS  regulations requiring the submission of financial audit reports, since its  report included marketplace financial statements. In addition, the official  stated that the cost to complete an independent financial audit specific to  the marketplace would be an undue burden. The official added that CMS  had accepted the overall state financial audit report as sufficient.
CMS officials stated that the agency accepted the overall Minnesota state  audit report given Minnesota’s challenges with reporting and said they  were working with the state to mitigate challenges in providing the  required audit report. Further, these officials stated that relevant audit  information may be gathered through alternate channels and that they do  not plan to enforce Minnesota’s compliance in providing an independent  financial audit report specific to the marketplace in the short term.
CCIIO officials further stated that, while they can communicate a lack of  compliance to the states, they are statutorily limited in regard to  enforcement mechanisms for state marketplaces. The officials added  that, in the long term, they hope to address reporting limitations through  providing additional guidance to the states.
Nevertheless, although CMS took steps to ensure that the states  submitted annual financial audit reports, the agency did not ensure the  states followed regulations and guidance, which decreased CMS’s  visibility into state marketplace sustainability and could increase  sustainability risks. While individual states may have unique situations  that preclude the submission of complete independent financial audit  reports, the law and guidance are clear that states must submit annual  financial audit reports that follow GAGAS requirements, including a  review of compliance, internal controls, and related financial policies and  procedures.
Further, although CMS officials stated that alternate channels may be  used to gather audit information, the agency’s guidance specifically refers  to the annual financial audit reports as one of the primary sources for  evaluating state marketplace sustainability. If CMS does not ensure that  states provide complete annual financial audit reports, it may not have  visibility into marketplace IT-related financial activities such as receipts,  expenditures, internal controls, and financial policies and procedures. A  lack of relevant financial audit reports can lead to uninformed CMS  sustainability risk assessments and responses, which could increase  state marketplace sustainability risks.
section_title: CMS’s Sustainability Risk Assessments of State Marketplaces Were Not Always Based on Defined Risk Management Processes
paragraphs: Standards for Internal Control calls for agencies to assess and respond to  risk using clearly defined measurable terms, objectives, and risk  tolerances, to include a clear categorization process, reliable information,  and a clear response to risks. Clearly defined objectives state what is to  be achieved in specific, measurable terms, as well as how the objectives  will be achieved, who will achieve them, and in what time frames. In  addition, clearly defined risk tolerances set the acceptable level of  variation in performance relative to the objective. These risks should be  assessed using relevant data from reliable sources in a timely manner  based on identified information requirements. According to CMS policy,  these required information sources include state sustainability plans and  independent financial audit reports. Additionally, agencies should respond  to risk based on the significance of the risk. According to CMS guidance,  these processes address issues that may impact state marketplace IT  required financial self-sustainability.
In order to assess and respond to risk, CMS had established processes  for the following activities for all state-based marketplaces, including  those that use the federal platform:  collects sustainability information. CMS uses financial audit reports  and sustainability plans to create sustainability risk assessments and  conduct consults for each state-based marketplace. assesses state marketplace risk. The agency conducts risk  assessments by reviewing factors such as marketplace IT costs,  functionality, and operations stability. It then scores states’  marketplace self-sustainability risk factors, and categorizes  marketplaces based on a risk rating of low, medium, medium-high, or  high. Low risks indicate a stable IT infrastructure and budget while  higher risks indicate challenges to marketplace sustainability, such as  insufficient projected revenues, insurance issuer volatility, or IT  system challenges.  responds to assessed risk. CMS responds to identified marketplace  sustainability risk, including IT system sustainability risks, through  sustainability consults. The agency discusses marketplace  sustainability with the state, addressing specific issues that may  impact sustainability, and identify areas for technical assistance.  Agency officials said they also provide assistance as needed through  phone calls with state officials.  follows up with the state. CMS provides the results of consults to the  states in a site visit report. This report conveys action items  concerning sustainability compliance and/or recommendations for  self-sustainability based on industry best practices. In addition,  according to CCIIO officials, marketplaces assessed at medium-high  or high risk receive more intensive assistance from the agency.
While CMS took steps to establish a sustainability risk assessment  process, there were numerous shortcomings with the agency’s  implementation of that process for the four selected states. Specifically,  the agency’s sustainability risk assessment and consult activities were not  based on a fully defined risk process, to include having fully defined  measurable terms, a clear categorization process, reliable information,  and a clear response to risk. CMS’s procedures outlined the steps in the  agency’s sustainability risk assessment process, but the procedures did  not always define risk factors in clear and measurable terms. For  example, the procedures did not define the risk factor for enrollment  target tolerances in quantifiable terms. Instead, the agency used the  terminology “close to expected” or “lower than expected.” Additionally,  while the agency assigned the four selected state marketplaces in  categories of related sustainability risk, there was no clear categorization  process defining how risk assessment scores of low, medium, medium- high, or high risk were obtained. Specifically, CMS did not provide  documentation of defined score thresholds, such as what score out of the  maximum determined the sustainability risk categorization. It also did not  define a consistent baseline for risk scores in its documentation or  assessments—we identified three different baselines in CMS’s risk  assessments of the four selected states. As an example, we found that  CMS assessed one state based on a scale of 59 points of weighted risk  factors, while other states were assessed on scales of 67 or 76 points.
In addition, as mentioned previously, the sustainability plans and financial  audits from the selected states that were used for the risk assessments  were not always complete and, thus, were not always reliable sources of  information. Further, CMS did not provide documentation defining a  process for responding to assessed risks and its response to risk as  documented in the site visit reports was not based on the significance of  the risk. The site visit reports addressed no more than one risk factor per  each of the four states we reviewed, despite the agency categorizing  some of the selected states at a medium-high or high risk level, with  multiple assessed risk factors. For example, in one state’s site visit report,  CMS provided a recommendation for a lower weighted risk factor— marketplace reserves. However, the agency did not address other factors  that it designated in the state’s associated risk assessment as being of  highest risk, such as a limited revenue source or IT functionality  shortcomings. Moreover, the agency’s policy and procedure for risk  response did not vary based on risk, so the internal guidance for  sustainability assistance is the same for a high-risk marketplace as a low- risk marketplace.
CCIIO officials said that risk assessments were never meant to be  quantitative and that they used their best judgment after looking at a  number of areas to rate marketplace risks. Further, the officials said they  did not tailor responses to different levels of assessed risk ratings  because the assessments were considered an internal guide to provide  CMS leadership with a general idea of a marketplace’s sustainability risk  and were used as a tool to determine what technical assistance states  needed.
However, if CMS does not take steps to define sustainability objectives in  measurable terms, to include a clear marketplace risk categorization  process, use of relevant data from reliable sources, and responses based  on risk significance, then it is possible that risks will not be correctly  assessed and responded to by the agency. Accordingly, if marketplace  sustainability is not correctly assessed and responded to, CMS may not  be able to assist states in achieving their required financial self- sustainability.
section_title: CMS Did Not Consistently Monitor the Performance of Selected States’ Marketplace IT Systems
paragraphs: Among other things, leading practices emphasize the importance of  having performance metrics and developing performance plans to identify  the most important metrics to guide decisions and measure IT  performance. In addition, CMS’s guidance calls for it to ensure that  states have documented performance measurement plans and to conduct  operational analysis reviews to examine the operating status of state  marketplace IT systems using key performance indicators. Also, during  the open enrollment period, CMS requires states to submit a report of  weekly performance indicators, which includes some metrics related to  the operational performance of marketplace systems, such as the number  of applications completed electronically, the total number of website visits,  and website offline time, among others.
However, CMS did not consistently monitor the performance of IT  systems for Minnesota and New York—the two selected states that  operated state-based marketplace systems. Specifically, CMS did not  ensure that the two states had developed, updated, and followed  performance measurement plans. In addition, it did not conduct reviews  to analyze the operational performance of the selected states’  marketplace IT systems against an established set of performance  parameters to evaluate whether the states were performing in an efficient  and effective manner. As for IT metrics that were collected from the  states, CMS did not link these metrics to performance measurement  goals or establish targets for performance.
subsections: section_title: CMS Did Not Always Ensure That Selected States Had Developed, Updated, and Followed Performance Measurement Plans
paragraphs: In applying its Enterprise Life Cycle framework to monitor states’  marketplace IT systems, CMS is to ensure, as part of its operational  readiness review, that states have documented performance  measurement plans. These plans are to be used to assess the business  value of states’ marketplace IT. Further, the Enterprise Life Cycle  framework requires that states evaluate performance metrics and share  their results with responsible parties, such as federal officials and state  project and business managers. CMS also provides states a template that  they can use, which includes a section on project measurement  objectives, performance metrics, and thresholds, which set parameters  for target performance.
In addition, our previous work has emphasized the importance of  performance metrics to assess the actual results, effects, or impact of a  program or activity compared to its intended purpose. We also  emphasized that leading practices include the development of plans to  identify the most important metrics to guide decisions, and document  goals and metrics to measure IT performance. Our previous work noted  that the performance measurement approach should be holistic, or seen  in terms of the operation as a whole, in order to identify a comprehensive  suite of metrics. Further, our work has stated that the performance  measurement approach should be continuously assessed and followed  by regularly reviewing metrics, goals, and targets; and adjusting these as  necessary.
CMS reviewed the two selected states’ progress on marketplace IT  projects, but had not ensured that these states documented, updated,  and followed their performance measurement plans to demonstrate that  they had identified and selected the most important metrics to guide  decisions. In addition, CMS had not ensured the selected states  continuously assessed and adjusted performance metrics and targets as  appropriate.
Minnesota: CMS did not ensure that Minnesota updated and followed  its performance measurement plan to show that the state had  continuously assessed its performance measurement approach and  adjusted metrics and targets as necessary. Specifically, in the  September 2013 operational readiness review, CMS noted that  Minnesota officials had partially identified performance metrics in their  project planning documentation, and had partially evaluated  performance metrics and shared results with responsible parties.  CMS also noted that the state had not developed or identified its  operational metrics.
Subsequently, to address CMS’s observations, in December 2013,  Minnesota drafted a performance measurement plan to identify  performance measurement goals and targets. The December 2013  plan included technology goals such as ensuring that at least 90  percent of users have real-time, online access to the marketplace  website and decreasing code defects, or software errors, per release  by at least 60 percent. Information in the plan stated that it was to be  reviewed quarterly and updated as needed.
However, in June 2016, Minnesota officials from the state’s IT  Services organization said that the 2013 performance measurement  plan was likely developed by a contractor and had not since been  updated or followed. According to the officials, while the state did not  follow its performance measurement plan, the state was monitoring  marketplace IT metrics related to its technology goals, such as system  availability and unexpected down time. In addition, Minnesota  developed a service-level agreement that included target metrics for  system availability, although it did not require the reporting of  defects. Neither CMS nor state officials provided evidence that the  state was actively monitoring these metrics related to their technology  goals.
New York: CMS had not ensured that New York developed a  performance measurement plan. In August 2013, CMS provided New  York a satisfactory rating for evaluating performance metrics and  sharing results with responsible parties; however, it noted that the  state had only partially documented performance metrics in a plan.  During this review, CMS also stated that New York had not developed  or identified operational metrics.
Although New York did not have a performance measurement plan,  according to state officials, performance metrics were documented in  its June 2016 oversight and monitoring plan. This plan included  metrics for the contractor responsible for New York’s marketplace IT  operations. Information in the oversight and monitoring plan referred  to IT metrics such as system downtime, timeliness of file processing,  real-time transaction processing, backup and recovery  implementation, failover and fallback capability, and disaster recovery  infrastructure. In addition, New York developed a service-level  agreement that included targets. For example, the service-level  agreement required that New York’s marketplace system be available  at least 98.5 percent of each month.
However, the oversight and monitoring plan and service-level  agreement did not include IT performance goals or related metrics.  For example, the oversight and monitoring plan did not include  metrics that New York IT officials said they use to monitor the  performance of marketplace IT systems, such as metrics related to  defect creation and remediation, the number of batch jobs, electronic  data interface files that were rejected, and notices and related  backlogs. These metrics were not documented or tied to performance  goals or targets in a plan.
In addition, according to New York marketplace officials, the goal of  the IT service-level agreement metrics for its contractor is to ensure  that consumers can readily and easily access the health insurance  application and systems needed to enroll in coverage. However, these  stated goals were not documented in the oversight and monitoring  plan or clearly linked to the IT performance metrics.
CMS officials from OTS said states were expected to address the issues  that it identified in its reviews, such as partially completed performance  measurement plans, and that the agency monitors marketplace system  performance through daily calls with states. The officials said that  Minnesota had not updated its performance measurement plan as  requested and that it falls upon the state to ensure that service level  agreement metrics are met, and if not met, that corrective actions will be  taken.
In addition, the officials stated that, instead of a performance  measurement plan, the agency reviewed New York’s IT contractor service  level agreements, which represented the state’s operational metrics.  However, CMS did not ensure that these metrics were tied to  performance goals in a plan in accordance with leading practices.  Further, according to CMS officials, the Enterprise Life Cycle required  project measurement plans in the planning and design phase, but not in  the operations and maintenance phase, and operational states were not  required to submit updated performance measurement plans. However,  while CMS’s Enterprise Life Cycle guidance did not explicitly require  states to update performance measurement plans during operations and  maintenance, leading practices stress that it is important to have updated  performance measurement plans to properly assess effectiveness in  meeting stated operational goals.
Because CMS had not ensured that states documented, updated, and  followed performance measurement plans for their state marketplace  systems, the agency did not have the assurance that states had taken a  holistic approach to developing performance metrics to assess actual  results as compared to intended goals. Further, without reviewing states’  plans, CMS could not ensure that states had carefully identified and  selected the most important metrics to guide decision making and  organizational operations, or that states continuously assessed and  adjusted performance metrics and targets as appropriate.
section_title: CMS Had Not Conducted Operational Analysis Reviews as Required by Its Guidance to Monitor Selected States’ System Performance
paragraphs: As part of the Enterprise Life Cycle framework, to monitor system  performance, CMS and states are required to conduct an operational  analysis review to examine the operating status of the marketplace  system through a variety of key performance indicators and determine  whether the system is performing in an efficient and effective manner. In  addition, according to leading practices, operational analysis is a key  management tool to examine the performance of an operational initiative  and measure that performance against an established set of performance  parameters. The operational analysis should consider how objectives  could be better met and how costs could be saved.
CMS had not conducted operational analysis reviews for states, including  Minnesota and New York. Instead of conducting operational analysis  reviews, the agency developed OTS reports in November 2015 regarding  selected states’ systems to prepare for the 2016 open enrollment period.  The reports included discussions of these states’ system performance.  For example, the OTS report for Minnesota included a summary of the  state’s system functionality and performance. Specifically, the report  noted that Minnesota discovered quality issues with the software used for  eligibility determinations and put a plan in place to remedy quality issues.  In addition, the report noted that Minnesota was using scenarios to test  site performance and assessing whether the system had sufficient  capacity to meet business needs.
For New York, the OTS report included a summary of the state’s system  functionality, system performance, and stability of application bandwidth  to handle peak volumes. It discussed business value, such as the number  of eligibility determinations (2.7 million eligibility determinations, and over  78.9 percent enrolled in a health care plan). The report also identified  risks for the open enrollment and estimated operating costs for the  system.
Nevertheless, while these reports discussed metrics on New York’s  performance, such as the number of eligibility determinations, and  Minnesota’s consideration of performance issues, they did not include key  performance indicators to show whether the states were performing in an  efficient and effective manner. Specifically, the reports did not include  metrics or targets that might define performance success. For example,  the reports did not discuss metrics related to application processing time  or application backlogs in the marketplace IT system.
CMS officials from OTS and CCIIO said that they had not conducted  operational analysis reviews because they conduct open enrollment  readiness reviews instead. The officials also said that states were still  developing their systems for the 2017 enrollment period. Additionally,  CMS officials said that they had not established key performance  indicators because states are responsible for developing such measures  and adjusting performance, and that the key performance indicators  would be in the states’ service-level agreements. The officials added that  throughout the year, OTS meets with states biweekly to track their  progress on their software releases, and discuss and assist with  resolution of any issues they may be encountering.
However, the open enrollment readiness review included operational  areas such as IT system and business functions, but did not note  discussion of performance in terms of key performance indicators or other  elements of operational analysis, such as how objectives could be better  met or costs could be saved. In addition, while officials from the states in  our review said they were still developing or updating certain aspects of  their marketplace systems, their marketplaces were operational. Further,  while service-level agreements and other metrics can be useful  indicators, conducting an operational analysis is an opportunity to perform  qualitative analysis of the utilization of technology in a holistic and  strategic way in order to see where the states are relative to their  performance indicators. Because CMS had not conducted operational  analysis reviews to monitor the performance of the selected states’  marketplace IT systems in a systematic way, it had limited assurance that  these states’ systems were performing in an effective and efficient  manner.
section_title: CMS Had Not Linked IT Metrics Collected from States to Its Performance Measurement Goals
paragraphs: During the open enrollment period, CMS requires states to submit a  report of weekly performance indicators. This report is to include metrics  related to the operational performance of marketplace systems. For  example, CMS requests that states include:  the number of applications completed electronically and on paper  In addition, according to our prior work, performance metrics should be  linked to the program’s IT performance measurement goals and define  what is important to the organization and what the baseline and target  performance should be in order to determine how efficiently and  effectively the systems are performing. Metrics can be used for an  organization to define success, structure improvement efforts and identify  early warning indicators of problems.
CMS collected IT metrics from the two selected states that operated  marketplace systems in the open enrollment weekly indicator template.  The template included metrics related to marketplace systems’  operational performance, such as the number of applications completed  electronically and the total number of website visits, among other metrics.
However, the metrics were not clearly linked to performance  measurement goals and did not include baselines and targets to indicate  how effectively they are performing. Specifically, CMS did not define  performance measurement goals, such as timely processing of  applications, and therefore was unable to link metrics to those goals. In  addition, CMS did not set, nor require states to set, performance  baselines and targets for metrics such as the number of applications  completed by electronic means or on paper, or the duration of times when  the website was offline.
According to CCIIO officials, the agency collects data to effectively and  consistently monitor state-based marketplace performance and to identify  any barriers to eligibility and enrollment in the marketplaces. The data are  used to inform programmatic understanding of operations, but are not  collected to diagnose or interpret IT system performance. According to  these officials, because states have different internal goals, CMS assists  states to oversee their own marketplace IT systems based on each  state’s need, goals, and resources.
Because CMS and states did not clearly link metrics to their own  performance measurement goals and did not include baselines or targets  for the marketplaces, the agency is limited in its ability to monitor whether  states’ systems are performing efficiently and effectively. In addition,  without baselines or targets, the agency may not have data to accurately  monitor progress for continual improvement.
section_title: Conclusions
paragraphs: Through periodic oversight and guidance, CMS offered assistance to  states we selected for review that sought to transition their health  insurance marketplaces to the federal IT platform. This oversight included  reviews of transition plans and milestones and weekly calls between the  agency and state officials to discuss transition progress. However, CMS  guidance for states transitioning to the federal platform was not  documented and finalized until after two states had already initiated their  transitions. Those two states primarily utilized federal Medicaid funding to  make the associated changes to their Medicaid systems to connect with  the federal IT platform. The states encountered challenges in their  transitions in part because CMS had not issued its transition-related  guidance until after these states had transitioned.
CMS assisted selected states with their effort to financially sustain the  development and operations of their marketplaces (including supporting  IT systems) by reviewing sustainability plans; reviewing annual  independent financial audit reports; and conducting sustainability risk  assessments. However, CMS did not provide consistent oversight of the  four selected states’ programs because the agency did not take steps to  collect complete sustainability plans or financial audit reports from all of  these selected states. In addition, CMS did not clearly define its  sustainability risk assessment process to assist states. Until CMS  addresses these issues, the agency’s assistance with, and assessments  of states’ marketplace sustainability may not fully account for risks that  could impact or interrupt state marketplace IT operations.
CMS’s guidance includes steps to monitor the performance of state- based marketplace IT system operations, including the collection of  related IT performance metrics such as electronic enrollments and  website traffic volume. However, its oversight did not ensure selected  states’ systems performance was monitored in a way that was consistent  with its guidance to states and leading practices. Specifically, CMS did  not ensure that two selected states (Minnesota and New York) had  developed, updated, and followed performance measurement plans. In  addition, the agency did not conduct operational reviews to determine if  marketplace IT systems for the two selected states were operating in an  efficient and effective manner; it also did not establish performance  measurement goals or targets for certain metrics it collected from states.  As a result, CMS has been limited in its ability to determine whether state  marketplace IT systems are performing efficiently and effectively and to  provide early warning of potential problems for the overall state  marketplace IT systems’ service delivery to consumers.
section_title: Recommendations for Executive Action
paragraphs: We are making the following six recommendations to the Secretary of  Health and Human Services to direct the Administrator of the Centers for  Medicare & Medicaid Services to take action.  1.  The Administrator of CMS should take steps to ensure that state- based marketplace annual sustainability plans, to the extent possible,  have complete 5-year budget forecasts. (Recommendation 1)  2.  The Administrator of CMS should take steps to ensure that all state- based marketplaces provide required annual financial audit reports  which are in accordance with generally accepted government auditing  standards. (Recommendation 2)  3.  The Administrator of CMS should take steps to ensure that  marketplace IT self-sustainability risk assessments are based on fully  defined measurable terms, a clear categorization process, and a  defined response to high risks. (Recommendation 3)  4.  The Administrator of CMS should take steps to ensure that states  develop, update, and follow performance measurement plans that  allow the states to continuously identify and assess the most  important IT metrics for their state marketplaces. (Recommendation 4)  5.  The Administrator of CMS should take steps to conduct operational  analysis reviews and systematically monitor the performance of  states’ marketplace IT systems using key performance indicators.  (Recommendation 5)  6.  The Administrator of CMS should take steps to ensure that metrics  collected from states to monitor marketplaces’ operational  performance link to performance goals and include baselines and  targets to monitor progress. (Recommendation 6)
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to HHS for comment. In its written  comments (reproduced in appendix II), the department concurred with  two of our recommendations, partially concurred with two  recommendations, and did not concur with two recommendations.
HHS concurred with our second and third recommendations which,  respectively, called for CMS to ensure that all state-based marketplaces  provide required annual financial audit reports that are in accordance with  generally accepted government auditing standards, and ensure that the  marketplace IT self-sustainability risk assessments are based on fully  defined measurable terms, a clear categorization process, and a defined  response to high risks. HHS stated that it will continue to provide  technical assistance to state marketplaces regarding independent  financial audits. The department also stated that it will refine its  marketplace self-sustainability risk assessment processes to provide  greater insight into the state marketplace sustainability efforts and to  identify areas where states may need assistance. Taking steps to provide  technical assistance to the states is important and CMS’s efforts to refine  the risk assessment processes can provide greater insight into the state  marketplace sustainability efforts and areas of needed assistance.
HHS partially concurred with our fourth recommendation that CMS ensure  that states develop, update, and follow performance measurement plans  that allow the states to continuously identify and assess the most  important IT metrics for their state marketplaces. While HHS did not  specifically identify which aspects of our recommendation it concurred  with and which it did not concur with, the department stated that, as part  of its Enterprise Life Cycle framework, state marketplaces were required  to submit performance measurement plans during the planning and  design phases. The department also stated that it will continue to monitor  the state marketplaces’ IT metrics in the implementation phase. For the  state marketplaces that are in the operations and maintenance phase, it  stated that each marketplace is accountable for managing and reporting  its own IT metrics in accordance with federal and state law.
The department also emphasized its consideration of states’ variations in  marketplace systems and reporting capabilities and the associated  burden of reporting IT metrics. However, as we noted in our report, CMS  and certain states were not always able to provide evidence of  performance measurement plans that were in accordance with the  agency’s policy, nor was evidence always provided that these states  updated and followed their performance measurement plans according to  best practices. While CMS required submission of performance  measurement plans in the planning and design phases, best practices  state that performance measurement plans should be continuously  updated and followed. Without ensuring that the states documented,  updated, and followed performance measurement plans, CMS may not  have reasonable assurance that the states established IT metrics to  assess their results compared to their intended goals for their  marketplace systems.
HHS also partially concurred with our sixth recommendation that CMS  ensure that metrics collected from the states to monitor marketplaces’  operational performance link to performance goals and include baselines  and targets to monitor progress. HHS did not specifically identify which  aspects of our recommendation it concurred with and did not concur with;  however, the department stated that, while it requests performance  measures from the state marketplaces, once the marketplaces are  operational, states are responsible for monitoring their own performance  measures. HHS also stated that it will continue to review IT metrics of  state marketplaces in the implementation phase of their systems, but  emphasized the burden on states and variations in state system reporting  capabilities. However, as we noted in our report, CMS did not ensure that  the metrics it is collecting from the states are linked to performance goals  as suggested by best practices. Without this linkage, the agency may  continue to be limited in its ability to monitor whether the state systems  are performing efficiently and effectively. Additionally, CMS may miss the  opportunity to refine its current IT metrics collection to better balance its  need for visibility into states’ performance without unnecessarily  burdening states.
HHS did not concur with our first recommendation that CMS ensure that  state-based marketplace annual sustainability plans, to the extent  possible, have complete 5-year budget forecasts. The department stated  that it has updated its requirements, and is now requesting 2-year budget  forecasts instead of 5-year budget forecasts. It also stated that this is part  of a new streamlined and simplified process to collect timely, accurate,  and relevant data while taking into consideration the burden on states and  the variations in state budget cycles. However, CMS did not provide  documented evidence of this process or justification for stating that a 5- year budget is not a reasonable time frame for sustainability planning.  This also contradicts previous CMS blueprint guidance for state  marketplace approval. While asking states for a 2-year budget instead of  a 5-year budget may streamline the process and be less of a burden on  states to provide complete budgets, the shorter time frame may not fully  inform CMS oversight of the long-term financial sustainability and  associated risks for marketplaces, which are new systems that face  multiple uncertainties.
Further, if CMS does not take steps to ensure that states provide  sustainability plans with complete 5-year budget forecasts, per its 2016  sustainability guidance, then it may not be fully informed of the state- based marketplaces’ sustainability factors. Incomplete sustainability plans  may also limit the agency’s ability to assess and respond to state  marketplace sustainability risks. Thus, we continue to believe that CMS  should ensure that state-based marketplace sustainability plans have, to  the extent possible, complete 5-year budget forecasts.
Lastly, HHS did not concur with our fifth recommendation that CMS  conduct operational analysis reviews and systematically monitor the  performance of states’ marketplace IT systems using key performance  indicators. The department stated that it conducts Open Enrollment  Readiness Reviews to assess marketplace key performance indicators,  which, according to CMS officials, are similar to operational analysis  reviews. However, as we noted in our report, Open Enrollment Readiness  Reviews did not systematically report the key performance indicators or  include discussions of other elements of operational analysis reviews,  such as how objectives could be better met or costs could be saved. In  not ensuring that these reviews include clearly identified key performance  indicators, CMS may miss an opportunity to perform strategic analysis of  the states’ utilization of their marketplace systems and it may continue to  have limited assurance that these states’ systems are performing in an  effective and efficient manner. Therefore, we continue to believe that  CMS should conduct operational analysis reviews, as required by its  guidance, to systematically monitor the performance of states’  marketplace IT systems using key performance indicators.
HHS also provided technical comments, which we incorporated in the  report as appropriate.
We also provided relevant excerpts of this product to each of the four  states included in our review—Hawaii, Minnesota, New York, and  Oregon—and received responses, via e-mail or in writing, from all four  states.
In written comments, the State of Hawaii’s Department of Labor and  Industrial Relations noted that the Hawaii Health Connector’s IT  provider would be turning over consumer data to the state by the end  of June 2017. The department also indicated that the sustainability  budget discussed in our findings was prepared by the Hawaii Health  Connector and not the state. The department further commented that  the state submitted a sustainability budget in April 2016 that included  complete forecasts through 2019, and that this budget was updated in  May 2016. Further, it said that the updated budget only reflected 2016  as Hawaii planned to operate as a federally facilitated marketplace  and no longer needed to plan for financial sustainability.
We noted in our report that the Hawaii Health Connector’s IT provider  plans to turn over consumer data to the state by the end of June  2017. We also recognize that the April 2016 sustainability budget had  the required budget forecast. However, our discussion of the Hawaii  sustainability budget referred to the more recent and updated  sustainability budget that the state prepared in May 2016. With regard  to Hawaii’s April 2016 budget, the budget was based on revenue  assumptions, such as state funding from the legislature that had not  been approved at the time of its submission and, thus, required  revision once those uncertainties were resolved in May 2016.
With regard to the applicability of our findings for the May 2016  budget, Hawaii did not officially begin the transition to a federally  facilitated marketplace until June 2016, and was still operating as a  state-based marketplace at the time the May 2016 budget was  revised. In addition, as indicated by CMS’s 5-year budget template  and sustainability guidance, the sustainability budget should include  forecasted years. According to CMS officials, they use this data to  inform its sustainability risk assessments. By not showing the total  effect of certain assumptions or outcomes across all forecasted years,  such as the decision of the state legislature to not fund marketplace  operations, the marketplace missed an opportunity to demonstrate the  total negative impact on the marketplace sustainably and associated  budget numbers. In that way, a fully forecasted budget could have  served as evidence of Hawaii’s justification for transitioning to a  federally facilitated marketplace. The State of Hawaii Department of  Labor and Industrial Relations’ comments are reprinted in appendix  III.
In written comments, the state of Minnesota’s MNsure marketplace  noted that the organization continues to focus on making  improvements related to accountability and transparency, including in  areas such as federal and state audit reporting for the marketplace.  Minnesota provided additional comments in which the marketplace  noted that its 3-year budget sustainability plans show that the  marketplace is and will be sustainable. While Minnesota’s  marketplace may have policies that do not align with CMS’s  requirements for projected budget time frames, it is nonetheless  important that the states submit information as required to the agency  so that CMS officials can perform oversight of states’ marketplace  sustainability in a consistent manner. Further, the shorter time frame  may not fully inform CMS oversight, including risk assessments and  responses, of the long-term financial sustainability for marketplaces.
MNsure also disagreed with our characterization of Minnesota’s  independent financial audit as not specific to the state’s marketplace.  The marketplace provided details, including statements of the  sufficiency of its state financial audit report because it is in adherence  with Minnesota’s state statutes and financial policies. Additionally,  MNsure officials stated that alternate audits may detail relevant audit  information. However, CMS’s guidance specifically requires states to  develop an annual financial audit report specific to the marketplace as  one of the primary sources for evaluating a state marketplace’s  financial sustainability. As we stated in our report, if MNsure does not  provide complete financial audits specific to the marketplace, CMS  may not have the necessary transparency into marketplace IT-related  financial activities such as receipts, expenditures, internal controls,  and financial policies and procedures. MNsure’s comments are  reprinted in appendix IV. In addition, technical comments provided by  marketplace officials were incorporated into our final report as  appropriate.
In e-mail comments, the Executive Director of the New York State of  Health marketplace disagreed with our conclusion that the New York  marketplace did not have a performance measurement plan that  directly tied to goals. The Executive Director stated that the state’s  overall goals were to enroll New Yorkers in coverage and reduce the  rate of uninsured persons in the state. However, these stated goals  were not documented nor were they tied to any specific metrics, as  noted in our report. Until New York’s marketplace has documented  these goals in a performance measurement plan with clear ties to its  metrics, CMS may not have visibility into New York’s marketplace  performance metrics which state officials said guide their decision  making and operations, or be able to ascertain that the state  continuously assessed and adjusted performance metrics and targets  as appropriate.
The Executive Director also noted that the New York marketplace  could not reconcile the amounts provided by CMS for their state.  According to their records, the New York marketplace had spent or  planned to spend $487.2 million in marketplace grants that included  $182.8 million for IT costs. Additionally, the Executive Director stated  that CMS had deobligated $64.8 million. However, the amounts  provided by New York were as of April 2016. Our report included  more recent data obtained from CMS in October 2016 that provided a  consistent view of spending for the four states in our analysis. Other  technical comments provided by marketplace officials were  incorporated into our final report as appropriate.
In e-mail comments, the Oregon Interim Administrator provided  technical comments, which we incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 28 days from the  report date. At that time, we will send copies to the Secretary of Health  and Human Services and other interested parties. In addition, the report  will be available at no charge on the GAO website at http://www.gao.gov.
Should you or your staffs have questions about this report, please contact  me at (202) 512-9286 or pownerd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to: (1) describe what actions the Centers for  Medicare & Medicaid Services (CMS) has taken, if any, to assist states  that have chosen to transition to a marketplace IT platform different from  the one they originally used and identify the costs and challenges for  states in making this transition; (2) assess what actions CMS has taken to  assist selected states’ plans to ensure that the development and  operations of marketplace IT systems can be financially self-sustained;  and (3) assess the steps that CMS has taken to monitor the performance  of the states’ marketplace IT systems.
To address the objectives, we reviewed marketplace activities conducted  by CMS and four selected states: Hawaii, Minnesota, New York, and  Oregon. We selected these 4 states from the 17 states that operated their  own marketplaces as of March 2016. To make the state selections, we  considered four selection factors for the plan year 2016 enrollment period:  total enrollment, total federal marketplace grant dollars, a previous GAO  review, and whether or not the state transitioned its marketplace to the  federal platform.
Specifically, we first sorted states operating marketplaces by enrollment  levels, from highest to lowest, based on plan year 2016 numbers reported  by the Department of Health and Human Services (HHS). We then  divided the states into four groups. Within each group, we sorted the  states from highest to lowest by the total amount of federal marketplace  grant funds awarded and selected the state with the highest amount of  awarded grant funding. In two cases, we selected the state with the  second highest federal marketplace grant award in the group because the  states with the highest award levels had been included in a recent GAO  review of state health insurance marketplace IT security and privacy. We  also reviewed selected states to verify that the states used different  systems integrator contractors. The selection resulted in two states that  transitioned to using the federal platform (Hawaii and Oregon) and two  that remained the same (Minnesota and New York). The four selected  states were based on a nongeneralized sample and, thus, findings from  our assessments of these states cannot be used to make inferences  about to the full population of all state marketplaces.
To assess the reliability of CMS’s data on state marketplace enrollment  figures, we reviewed the agency’s data and interviewed state marketplace  officials for the four selected states and asked how the state-reported  enrollment figures were utilized in their sustainability plans. We  determined that the data were sufficiently reliable for our purposes.
To assess the reliability of CMS’s data on grant funds awarded and state- reported IT spending to establish, support, and connect to marketplaces,  we assessed the reliability of the systems used to collect the information.  We asked officials responsible for entering and reviewing the grants  information in these systems a series of questions about the accuracy  and reliability of the data.
Among the sources of data used for our study, we reviewed a  spreadsheet compiled by CMS’s Center for Consumer Information and  Insurance Oversight (CCIIO) officials that contained state-reported grant  funding data and marketplace IT project status information drawn from  two separate information systems: CMS’s Grant Solutions and the  Payment Management System. The spreadsheet was a consistent  source of information that reflected the same cost factors for all states as  of October 2016. Specifically, the spreadsheet tracked, among other  things, the type and total amount of grant funding provided and available  to each state, deobligated grant funding, as well as the time period for  expending those funds. We also reviewed the data to determine if there  were any outliers and other obvious errors in the data. For any anomalies  in the data, we followed up with CMS officials to either understand or  correct those anomalies. We determined that the data were sufficiently  reliable for our purposes and noted any limitations in our report. While our  report discusses state-reported IT spending based on CMS data, we did  not verify the accuracy of the data states reported to CMS.
To address the first objective, we obtained and analyzed CMS’s transition  guidance that was distributed to assist all states. We also reviewed the  actions CMS and states performed, such as communications and  transition planning for the two selected states—Hawaii and Oregon—that  transitioned from state-based to federal marketplace IT systems. To  identify transition guidance and transition costs for the states, we also  observed CMS’s and the selected state’s management tools, such as the  Collaborative Application Lifecycle Tool, State-based Marketplace Annual  Reporting Tool (SMART), Payment Management System, and Grant  Solutions, for reporting and tracking of grant funding. In addition, we  reviewed relevant CMS and state budget and grant documentation to  determine associated transition costs. We also interviewed state  marketplace officials within the two selected states to further identify  transition costs and challenges faced during their transitions. We also  interviewed CMS officials regarding identified challenges and their actions  to assist the states in addressing them.
To address the second objective, we reviewed the four selected states’  sustainability plans and CMS sustainability guidance provided to the  states. To identify states’ plans for self-sustainability we reviewed  development plans, financials audits, and grant documentation. To  identify CMS sustainability guidance we reviewed the agency’s  procedures for financial audit and sustainability plan collection, risk  assessments, and sustainability consults. We compared CMS financial  audit collection against applicable laws, regulation, and agency  guidance. We also compared sustainability plan collection and risk  assessments against leading practices. To understand how CMS  monitors state sustainability, we observed its and the four selected states’  web-based management tools, such as SMART, Payment Management System, and Grant Solutions, for reporting and tracking of state  marketplace self-sustainability. To determine the reliability of state  sustainability plans, we reviewed the plans and relevant source data for  anomalies and outliers, as well as related documentation including state  audits and budgets. We also interviewed CMS and selected state officials  regarding collection and processing of the data. We determined that the  data in the sustainability plans were sufficiently reliable except where  noted in our report.
For the third objective, we reviewed CMS guidance provided to the state- based marketplaces which called for the monitoring and tracking of the  performance of states marketplace IT systems. We identified steps CMS  established for monitoring the performance of states’ IT systems. We  compared the steps established by CMS to leading practices identified in  our prior work and by the Office of Management and Budget. In  addition, where available, we reviewed the two selected states’ system  performance measurement plans and reports. This included Minnesota  and New York—which operated state-based marketplace IT systems— and did not include Hawaii and Oregon—which relied on the federal  marketplace IT platform operated by CMS and did not collect system  performance metrics. We reviewed the use of tools such as CMS’s  Collaborative Application Lifecycle Tool and Open Enrollment Weekly  Indicators reports to facilitate the monitoring of state marketplace  operations and performance. We analyzed whether CMS ensured that  states followed leading practices for IT performance measurement  guidance by assessing evidence of CMS and the two selected states’  marketplace performance measurement plans and reporting.
To determine the reliability of state performance metrics reports, we  reviewed the selected states’ reports for anomalies or missing data and  conducted interviews with CMS and selected state officials regarding the  collection and processing of the data. We determined that the data in the  performance metrics reports were sufficiently reliable. We also assessed  whether CMS had conducted operational analysis reviews of the two  selected states’ marketplace IT systems using key performance indicators  to determine whether states’ systems were performing in an efficient and  effective manner.
For all three objectives, we supplemented the information and knowledge  obtained from our assessments of the program, project, and technical  documentation by holding discussions with relevant CMS officials and  interviews with state officials at selected state sites regarding their  marketplaces.
We conducted this performance audit from February 2016 to August 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Health and Human Services
section_title: Appendix III: Comments from the State of Hawaii’s Department of Labor and Industrial Relations
section_title: Appendix IV: Comments from State of Minnesota’s MNSure
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Tammi Kalugdan (Assistant  Director), David Hong (Analyst in Charge), Alexander Anderegg,  Christopher Businsky, Debra Conner, Sandra George, Conor McPolin,  Brian Palmer, Monica Perez-Nelson, Priscilla Smith, Merry Woo, and  Elizabeth Wood made key contributions to this report.