Healthcare.gov: CMS Has Taken Steps to Address Problems, but Needs to Further Implement Systems Development Best Practices
Released: 2015-03-04T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: The Patient Protection and Affordable Care Act required the establishment of health insurance marketplaces to assist individuals in obtaining health insurance coverage. CMS, a component of HHS, was responsible for establishing a federally facilitated marketplace for states that elected not to establish their own. This marketplace is supported by an array of IT systems, which are to facilitate enrollment in qualifying health plans. These include Healthcare.gov, the website that serves as the consumer portal to the marketplace, as well as systems for establishing user accounts, verifying eligibility, and facilitating enrollment.

GAO was asked to review CMS's management of the development of IT systems supporting the federal marketplace. Its objectives were to (1) describe problems encountered in developing and deploying systems supporting Healthcare.gov and determine the status of efforts to address deficiencies and (2) determine the extent to which CMS applied disciplined practices for managing and overseeing the development effort, and the extent to which HHS and OMB provided oversight. To do this, GAO reviewed program documentation and interviewed relevant CMS and other officials.

section_title: What GAO Found
paragraphs: Several problems with the initial development and deployment of Healthcare.gov and its supporting systems led to consumers encountering widespread performance issues when trying to create accounts and enroll in health plans:

Inadequate capacity planning: The Centers for Medicare & Medicaid Services (CMS) did not plan for adequate capacity to support Healthcare.gov and its supporting systems.

Software coding errors: CMS and its contractors identified errors in the software code for Healthcare.gov and its supporting systems, but did not adequately correct them prior to launch.

Lack of functionality: CMS had not implemented all planned functionality prior to the initial launch of Healthcare.gov and its supporting systems.

Since the initial launch, CMS has taken steps to address these problems, including increasing capacity, requiring additional software quality reviews, and awarding a new contract to complete development and improve the functionality of key systems. After it took these actions, performance issues affecting Healthcare.gov and its supporting systems were significantly reduced.

In addition, CMS did not consistently apply recognized best practices for system development, which contributed to the problems with the initial launch of Healthcare.gov and its supporting systems.

Requirements were not effectively managed: Requirements management helps ensure that a project's plans and work products are aligned with the needs of users. However, CMS did not always ensure that requirements were approved and were linked to source and lower-level requirements. As a result, CMS was hindered in ensuring that expected functionality for the system was delivered.

System testing was inconsistent. Testing is essential for ensuring that a system operates as intended. However, Healthcare.gov and its supporting systems were not fully tested prior to launch, and test documentation was missing key elements such as criteria for determining whether a system passed a test. Thus, CMS's assurance that these systems would perform as intended was limited.

Project oversight was not effective. Oversight includes monitoring a project's progress and taking corrective actions when its performance deviates from what is planned. However, CMS's oversight was limited by an unreliable schedule, lack of estimates of work needed to complete the project, unorganized and outdated project documentation, and inconsistent reviews of project progress.

As it has undertaken further development, CMS has made improvements in some of these areas, by, for example, establishing new requirements management processes and improving test documentation. However, weaknesses remain in its application of requirements, testing, and oversight practices. In addition, the Department of Health and Human Services (HHS) has not provided adequate oversight of the Healthcare.gov initiative through its Office of the Chief Information Officer. The Office of Management and Budget's (OMB) oversight role was limited, and GAO has previously recommended that it improve oversight of IT projects' performance.

section_title: What GAO Recommends
paragraphs: GAO is recommending that CMS take seven actions to implement improvements in its requirements management, system testing, and project oversight, and that HHS improve its oversight of the Healthcare.gov effort. HHS concurred with all of the recommendations.

section_title: Letter
paragraphs: The Patient Protection and Affordable Care Act (PPACA), signed into  law on March 23, 2010, is intended to reform aspects of the private health  insurance market and expand the availability and affordability of health  care coverage. It requires the establishment of a health insurance  marketplace in each state and the District of Columbia to assist  individuals and small businesses in comparing, selecting, and enrolling in  health plans offered by participating private issuers of qualified health  plans.

The Department of Health and Human Services’ (HHS) Centers for  Medicare & Medicaid Services (CMS) is responsible for overseeing the  establishment of these marketplaces, including creating a federally  facilitated marketplace for states not establishing their own. CMS was  responsible for designing, developing, and implementing the information  technology (IT) systems needed to support the federally facilitated  marketplace, to include Healthcare.gov—the website that provides a  consumer portal to this marketplace—and the related data systems  supporting eligibility and enrollment.

The federally facilitated marketplace began accepting applications for  enrollment on October 1, 2013. However, individuals attempting to access  the systems supporting the marketplace, including Healthcare.gov,  encountered numerous problems. In light of these problems, you asked  us to examine the IT management of the systems supporting the federally  facilitated marketplace operated by CMS.

Our objectives for this study were to (1) describe the problems  encountered in developing and deploying Healthcare.gov and its  supporting systems and determine the status in addressing these  deficiencies and (2) determine the extent to which CMS oversaw the  development effort and applied disciplined systems development  practices to manage requirements and conduct systems testing, as well  as the extent to which HHS and the Office of Management and Budget  (OMB) provided oversight of the effort.

To address the first objective, we reviewed independent verification and  validation (IV&V) reports on the development effort, testimony from CMS  officials, and contracting documentation describing problems encountered  by users after the launch of Healthcare.gov and when these problems  were first identified by CMS and its stakeholders. To determine the status  of efforts to address deficiencies, we reviewed data from relevant  program documentation, such as system monitoring metrics,  supplementary guidance to contractors, and independent, third-party  reviews. In addition, we interviewed CMS program officials responsible for  the development and oversight of Healthcare.gov and its supporting  systems.

To address the second objective, we reviewed documents describing  CMS’s oversight and application of system development practices. We  assessed the agency’s actions against best practices identified by us and  the Software Engineering Institute, the Institute of Electrical and  Electronics Engineers (IEEE), federal statutes on OMB and agency IT  investment management and oversight responsibilities, and CMS and  HHS guidance pertaining to the oversight of major information technology  programs. These included recognized practices for managing  requirements, systems testing documentation, and conducting program  oversight. These practices are identified in the Software Engineering  Institute’s Capability Maturity Model Integration for Development, Version  1.3; the IEEE Standard for Software and System Test Documentation; our  Schedule Assessment Guide Exposure Draft; and CMS and HHS  systems development life-cycle frameworks. We reviewed data from  relevant program documentation, such as requirements documentation,  independent verification and validation reports, test plans and test cases,  project schedules, project management and requirements management  plans, and project milestone review documentation. In addition, we  reviewed four non-generalizable, random samples of test cases and  functional requirements. We also interviewed relevant officials from CMS  responsible for the development and oversight of Healthcare.gov and its  supporting systems. Further, we interviewed HHS and OMB officials to  determine the extent to which HHS and OMB provided oversight of the  effort.

To determine the reliability of the data obtained from CMS information  systems used for managing requirements, conducting system testing, and  tracking system defects, we interviewed knowledgeable agency officials  within the Center for Consumer Information and Insurance Oversight and  Office of Information Services about these systems and asked specific  questions to understand the controls in place for ensuring the integrity  and reliability of the data they contain. Based on these efforts, we  determined that the data we used from these sources were sufficiently  reliable for the purposes of our audit.

We conducted this performance audit from December 2013 to March  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. A full description of our  objectives, scope, and methodology can be found in appendix I.
subsections: section_title: Background
paragraphs: PPACA directed the federal government to establish and operate a health  insurance marketplace, referred to as the federally facilitated  marketplace, on behalf of states electing not to establish and operate a  marketplace by January 1, 2014. CMS operated a federally facilitated  marketplace or partnership marketplace for 34 states for plan years2014 and 2015.

Marketplaces, both federal and state, were intended to provide a  seamless, single point of access for individuals to enroll in qualified health  plans, apply for income-based financial assistance established under the  law, and, as applicable, obtain an eligibility determination for other health  coverage programs, such as Medicaid or the State Children’s Health  Insurance Program (CHIP).

PPACA required federal and state marketplaces to be operational on or  before January 1, 2014. Healthcare.gov, the public interface for the  federally facilitated marketplace, began facilitating enrollments on  October 1, 2013, at the beginning of the first annual open enrollment  period established by CMS.

Since that time, CMS has reported that over 8 million individuals selected  a qualified health plan through the federally facilitated marketplace or a  state-based marketplace from October 1, 2013, through March 31, 2014.  As of October 15, 2014, 6.7 million individuals were enrolled and paying  for 2014 health coverage through the marketplaces. HHS estimated up to  9.9 million enrollees for the 2015 enrollment period, which began on   According to  November 15, 2014, and ended on February 22, 2015.HHS, over 8.4 million people had submitted applications for coverage  through the federally facilitated marketplace for the 2015 enrollment  period as of January 2, 2015.

HHS established the Office of Consumer Information and Insurance  Oversight in April 2010 as part of the HHS Office of the Secretary. In  January 2011, the office moved to CMS and was renamed the Center for  Consumer Information and Insurance Oversight. This office has overall  responsibility for providing guidance and oversight for the federal and  state systems supporting the establishment and operation of health  insurance marketplaces. The Office of Information Services, headed by  the CMS Chief Information Officer (CIO), is responsible for oversight of  the development and implementation of federal systems supporting the  establishment and operation of the federally facilitated marketplace,  including review, selection, implementation, and continual evaluation of  these systems.
subsections: section_title: Several Major CMS Systems Support Enrollment-Related Activities
paragraphs: The federally facilitated marketplace relies on the Healthcare.gov website  and several supporting systems to accomplish enrollment-related  activities. To do so, these systems interconnect multiple other systems  from a broad range of federal agencies, states, and other entities, such as  contractors and issuers of qualified health plans, creating a complex  system of systems. The CMS Consumer Information and Insurance  Systems Group within the Office of Information Services is tasked with  technical oversight of the development and implementation of these  systems. A description of each of the major systems for which CMS is  responsible for implementing follows.
subsections: section_title: Healthcare.gov Website
paragraphs: Healthcare.gov is the federal website that serves as the user interface for  individuals who wish to obtain coverage through the federal marketplace.  Individuals can use the website to obtain information about health  coverage, set up a user account, select a health plan, and apply for  coverage by the selected health plan. The site supports two major  functions: (1) providing information about PPACA health insurance  reforms and health insurance options (the “Learn” web page), and  (2) facilitating enrollment in coverage (the “Get Insurance” web page).  The “Learn” page provides basic information on how the marketplace  works, available health plans, and how to apply for coverage. It also  contains information on plan costs, ways to reduce out-of-pocket costs,  and how individuals can protect themselves from fraud. Individuals do not  have to provide personal information to access this section of the website.  In contrast to the information-oriented “Learn” page, the “Get Insurance”  page allows an individual to take steps to apply for health insurance and  other associated benefits.

section_title: Enterprise Identity Management System
paragraphs: Before an individual can apply for health care coverage or other benefits,  CMS must verify his or her identity to help prevent unauthorized  disclosure of personal information. The process of verifying an applicant’s  identity and establishing a login account is facilitated by CMS’s Enterprise  Identity Management system. This system is intended to provide identity  and access management services to protect CMS data while ensuring  that users’ identities are confirmed, as only authorized users are allowed  and capable of accessing CMS resources.

section_title: Federally Facilitated Marketplace System
paragraphs: The main system, the Federally Facilitated Marketplace (FFM) system,  contains several modules that perform key functions related to obtaining  health care coverage. The core of the FFM system is a transactional  database that was developed to facilitate the eligibility verification  process, enrollment process, plan management, financial management  services, and other functions, such as quality control and oversight. From  a technical perspective, the FFM leverages data processing and storage  resources that are available from private sector vendors over the Internet,  a type of capability known as cloud-based services. It consists of three  major modules: eligibility and enrollment, plan management, and financial  management.

Eligibility and enrollment module. Individuals seeking to apply for  health care coverage through the federally facilitated marketplace use  the eligibility and enrollment module to guide them through a step-by- step process to determine their eligibility for coverage and financial  assistance. Once eligibility is determined, the applicant is then shown  applicable coverage options and has the opportunity to enroll.

Throughout the eligibility and enrollment process, the applicant’s  information, such as name, address, Social Security number,  citizenship status, and employer name, is collected and stored in the  FFM system’s database. This information is compared with records  maintained by other federal agencies and a private entity to determine  whether the applicant is eligible to enroll in a qualified health plan and,  if so, to receive the advance payment of the premium tax credit and  cost-sharing reductionscost of this coverage.  established through PPACA to defray the  The module further allows an applicant to view, compare, select, and  enroll in a qualified health plan. Options are displayed to the applicant  on the Healthcare.gov webpage, and applicants can use the “Plan  Compare” function to view and compare plan details. The applicant  can customize and filter the plans according to various factors such as  plan type, maximum out-of-pocket expenses, deductible, availability of  cost-sharing reductions, or insurance company, among others. Once  an applicant has signed up for a qualified health plan on  Healthcare.gov, information about the enrollment is sent to the chosen  health plan issuer.

Plan management module. The plan management module is  intended to interact with and is primarily used by state agencies and  issuers of qualified health plans. The module is intended to provide a  suite of services used for submitting, certifying, monitoring, and  renewing qualified health plans, as well as managing the withdrawal  of these health plans. Specifically, using this module, states and  issuers submit “bids” detailing proposed health plans to be offered on  Healthcare.gov, including rate and benefits information. CMS then  uses the module to review, monitor, and certify or decertify the bids  submitted by issuers. Once a bid has been certified and approved for  inclusion in the marketplace, it is made available for applicants to  enroll through Healthcare.gov.

Financial management module. This module is intended to facilitate  payments to issuers through electronic transactions. Like plan  management, the financial management module is used primarily by  issuers of qualified health plans. This module also provides issuers  additional services, including payment calculation for reinsurance, risk  adjustment analysis, and the data collection required to support these  services. Transactions to be supported by the module include  payments of premiums and cost-sharing reductions subsidies for  individual enrollments, reinsurance, and risk adjustments.

section_title: Federal Data Services Hub
paragraphs: The federal Data Services Hub (DSH) acts as a single portal for  exchanging information between the FFM and CMS’s external partners,  including other federal agencies and state-based marketplaces, for  purposes such as facilitating eligibility determinations and transferring  plan enrollment information. The DSH was designed as a “private cloud”  service supporting various functions such as real-time eligibility queries,  transfer of application information, and exchange of enrollment  information with issuers of qualified health plans.

section_title: Many External Partners Connect with the FFM and DSH
paragraphs: In conducting Healthcare.gov-related activities, various entities, including  federal agencies, a private-sector credit agency, states, issuers of  qualified health plans, and agents and brokers connect to and exchange  information with the systems supporting the federally facilitated  marketplace.
subsections: section_title: Federal Agencies and a Private Entity
paragraphs: Federal agencies such as the Social Security Administration (SSA),  Department of Homeland Security (DHS), and Internal Revenue Service  (IRS), along with Equifax, Inc. (a private-sector credit agency that CMS  contracts with) provide or verify information used in making  determinations of a person’s eligibility for coverage and financial  assistance.

Social Security Administration. This agency’s primary role is to  assist CMS in confirming applicant-supplied information by comparing  it with information in SSA’s records related to individuals’ citizenship,  Social Security number, incarceration status, and death. SSA also  provides CMS information on monthly and annual Social Security  benefits paid to individuals under the Old Age, Survivors, and  Disability Insurance program, if necessary to determine eligibility.

Department of Homeland Security. The department assists CMS by  verifying the naturalized, acquired, or derivedimmigration status of applicants seeking eligibility to enroll in a  qualified health plan or participate in Medicaid, CHIP, or a state-based  health plan using information supplied by each applicant through the  website. DHS generally undertakes this role only if CMS is unable to  verify an applicant’s status with SSA using a Social Security number  or if the applicant indicates on the application that he or she is not a  U.S. citizen. DHS also assists CMS by verifying the status of  noncitizens who are lawfully present in the United States and seeking  eligibility to enroll in a qualified health plan or participate in Medicaid,  CHIP, or a state-based health plan, as well as current beneficiaries  who have had a change in immigration status or whose status may  have expired.

Internal Revenue Service. IRS provides federal tax information to be  used by CMS in determining or assessing income and family size and  determining an applicant’s eligibility for insurance affordability  programs, including the advance payment of the premium tax credit,  cost-sharing reductions, Medicaid, and CHIP.

Equifax, Inc. This entity verifies information about an applicant’s  current income and employment to assist CMS in making a  determination about an applicant’s qualification for insurance  affordability programs, such as the advance payment of the premium  tax credit and cost-sharing reductions.

In addition, several other federal agencies—the Departments of Defense  and Veterans Affairs, the Office of Personnel Management, and the  Peace Corps—support CMS in determining whether a potential applicant  is eligible for or enrolled in minimum essential coverage and therefore  may not be eligible to receive the advance payment of the premium tax  credit and cost-sharing reductions. For example, applicants that are  enrolled in or eligible for coverage under certain government programs  such as Medicare or Medicaid, or certain employer-sponsored programs,  such as the Federal Employees Health Benefits program, are ineligible for  these subsidies.

section_title: States
paragraphs: In most states, multiple government systems may need to connect to the  FFM system and DSH to carry out a variety of functions related to health  care enrollment. For example, most states need to connect their state  Medicaid and CHIP agencies to either the FFM system (through the DSH)  or their state-based marketplace to exchange data with CMS about  enrollment in these programs. In addition, states may need to connect  with the IRS (also through the DSH) in order to calculate the maximum  amount of advance payments of the premium tax credit. Finally, state- based marketplaces are to send enrollment confirmations to the FFM  system so that CMS can administer advance payments of the premium  tax credit and cost-sharing reductions and track overall marketplace  enrollment.

Further, in certain cases, known as partnership marketplaces, states may  elect to perform one or both of the plan management and consumer  assistance functions while the FFM system performs the rest. The  specific functions performed by each partner vary from state to state.

section_title: Issuers of Qualified Health Plans
paragraphs: Issuers of qualified health plans receive enrollment information from the  FFM system using CMS’s Health Insurance Oversight System when an  individual completes the application process. In this case, the FFM  system transmits the enrollment information to the DSH, which forwards it  to the issuer of qualified health plans. The issuer then replies with a  confirmation message. Plan issuers also interact with the FFM through  the plan management and financial management modules, as previously  described.

section_title: Agents and Brokers
paragraphs: In addition to applicants themselves, agents and brokers may access the  Healthcare.gov website to perform enrollment-related activities on behalf  of applicants. It is up to individual states to determine whether to allow  agents and brokers to carry out these activities, which can include  enrolling in health care plans and applying for the advance payment of  the premium tax credit and cost-sharing reductions.

Figure 1 illustrates the systems that make up the federally facilitated  marketplace and their connections with each other, as well as with  external partners.

section_title: GAO Has Previously Highlighted Improvements Needed in the IT Management of Healthcare.gov and Related Systems
paragraphs: In 2014, we reported on challenges CMS and its contractor faced in  developing, implementing, and overseeing the Healthcare.gov initiative.

We reported on CMS’s efforts to plan and oversee Healthcare.gov- related development contracts, as well as the agency’s efforts in  addressing contractor performance, in July 2014. We determined  that the agency undertook the development of Healthcare.gov and its  related systems without effective planning or oversight practices,  despite facing a number of challenges that increased both the level of  risk and the need for effective oversight. In addition, CMS incurred  significant cost increases, schedule slips, and delayed system  functionality for the FFM and DSH systems due primarily to changing  requirements that were exacerbated by oversight gaps. Lastly, CMS  identified major performance issues with the FFM contractor but took  only limited steps to hold the contractor accountable. Specifically,  CMS declined to pay about $267,000 in requested fees to the FFM  contractor, which was about 2 percent of the $12.5 million in fees  paid. We recommended that CMS take actions to assess increasing  contract costs and ensure that acquisition strategies are completed  and oversight tools are used as required, among other actions. CMS  concurred with most of the recommendations.

In September 2014 we reported on the planned exchanges of  information between the Healthcare.gov website and other  organizations, as well as the effectiveness of the programs and  controls implemented by CMS to protect the security and privacy of  the information and IT systems used to support Healthcare.gov. We  described how many systems and entities exchange information to  carry out functions that support individuals’ ability to use  Healthcare.gov to compare, select, and enroll in private health  insurance plans participating in the federal marketplace, as required  by the Patient Protection and Affordable Care Act. In addition, we  determined that CMS took many steps to protect security and privacy,  including developing required security program policies and  procedures, establishing interconnection security agreements with its  federal and commercial partners, and instituting required privacy  protections.

However, Healthcare.gov had weaknesses when it was first deployed,  including incomplete security plans, lack of a privacy risk analysis,  incomplete security tests, and the lack of an alternate processing site  to avoid major service disruptions. Further, we identified weaknesses  in the technical controls protecting the confidentiality, integrity, and  availability of the FFM. Specifically, CMS had not always required or  enforced strong password controls, adequately restricted access to  the Internet, consistently implemented software patches, and properly  configured an administrative network. We made 28 recommendations  to HHS to enhance the protection of systems and information related  to Healthcare.gov as well as to resolve technical weaknesses in  security controls. HHS partially agreed with 3 of the 28  recommendations, agreed with 25, and described plans to implement  our technical recommendations.

section_title: Initial Development and Deployment of Healthcare.gov and Its Supporting Systems Faced Problems with System Capacity, Software Code Issues, and Limited Functionality, but CMS Has Taken Steps to Address Them
paragraphs: Several problems occurred in the development and deployment of  Healthcare.gov and its supporting systems, which affected their  performance. These problems included inadequate system capacity,  numerous errors in software code, and limited system functionality.  Although CMS was aware of these problems prior to initial launch in  October 2013, it proceeded with deployment in order to meet this  deadline. Consequently, consumers attempting to enroll in health plans  were met with confusing error messages, slow load times for forms and  pages, and, in some cases, website outages. Since the initial launch of  Healthcare.gov and its supporting systems, CMS has taken a number of  steps to address these problems, to include increasing system capacity,  outlining a new approach for ensuring the quality of software code, and  further developing required system functionality. As a result of these  efforts, the performance of Heathcare.gov and its supporting systems has  improved significantly.
subsections: section_title: Healthcare.gov and Its Supporting Systems Were Hindered by Inadequate Capacity, Software Code Errors, and Limited Functionality
paragraphs: Systems supporting Healthcare.gov were initially launched without  adequate capacity to accommodate the number of visitors to the website.  In particular, when the system was launched on October 1, 2013, the  Enterprise Identity Management system was overwhelmed by the number  of users attempting to create accounts—nearly half a million in the first 2- and-a-half weeks of open enrollment—preventing the system from  functioning as intended.

CMS officials within the Office of Information Services stated that they  had incorrectly estimated the number of users that would visit the site  during the initial launch of the 2014 enrollment period. As a result, CMS  had not planned to provide a level of capacity that would ensure  uninterrupted service to users in a cost-effective manner.

Independent assessments conducted in December 2012 and June 2013  also identified weaknesses in CMS’s capacity planning in the months  prior to launch. Examples of these weaknesses included the following:

Capacity requirements for hardware for the FFM system were not  developed.

A plan for capacity for the cloud computing environment had not been  developed, and thus there were uncertainties as to whether new and  existing system hardware configurations and their performance were  adequate to meet existing and proposed system requirements.

Existing capacity in the cloud environment was not adequate, and did  not include an adequate number of virtual machinesprocessors.

Further, in a November 2013 testimony, the CMS Administrator  acknowledged that although CMS tried to project demand for the website,  the agency underestimated that demand. As a result, consumers  attempting to enroll in health plans were met with confusing error  messages, slow load times for forms and pages, and in some cases  website outages. In particular, due to inadequate system capacity, many  consumers experienced difficulty creating accounts, and those that were  able to create accounts had difficulty logging into them.
subsections: section_title: Software Coding Errors
paragraphs: Software code for systems supporting Healthcare.gov contained  numerous errors, resulting in difficulties in accessing and using the site.  For example, in September 2013 (less than 1 month before launch), an  IV&V assessment ordered by CMS identified 45 critical and 324 serious  code errors across the plan management, financial management, and  eligibility and enrollment FFM system modules, with services relating to  the eligibility and enrollment module having the highest numbers of  errors. Further, the IV&V assessment team reported that there was no  evidence that software coding errors were being addressed.

Other IV&V assessments of the FFM and DSH systems also noted  problems in coding practices used by systems development contractors  that indicated concerns about system code. For example, in March 2013,  the IV&V assessment team reviewing the FFM and DSH systems noted  multiple issues with application coding, including undesirable coding  practices that were known to potentially cause errors and the inability of  the assessment team to locate CMS or contractor coding standards.

CMS also identified concerns with system coding prior to launch. In  March 2013, a Director within the Consumer Information and Insurance  Systems Group, charged with overseeing the development effort,  expressed concerns about the quality of FFM system code during a  monthly status meeting. In addition, CMS conducted an assessment of  FFM system documentation and development processes in August 2013  and noted that late-stage coding conducted by the FFM system  development contractor did not follow expected standards and best  practices, resulting in code conflicts between FFM system modules. The  assessment further stated that system technical changes and  development were being conducted on an ad-hoc basis to resolve  production issues rather than being coordinated across development  teams.

In September 2013, the FFM system development contractor attributed  certain coding errors to the urgency of implementing system fixes as  quickly as possible. To mitigate these issues, the contractor stated that it  was revisiting its code review process to help identify coding errors.

However, this action was not timely, as open enrollment began shortly  thereafter. Further, in November 2013, the FFM system development  contractor, in response to a CMS contracting officer’s concerns about  defects and errors in the FFM system code, stated that it was not possible  to ensure that each code release addressed all defects because there  was not sufficient time to fix the code and retest it to confirm that issues  were resolved. CMS officials agreed that some defects were not  addressed prior to system launch due to the urgency in meeting the  October 1, 2013, deadline.

As with the capacity problems, these software code errors also  contributed to the problems applicants faced in attempting to enroll in  health care plans. For example, according to an HHS report summarizing  findings from an Obama administration assessment, for some weeks in  the month of October 2013, the Healthcare.gov website was down an  estimated 60 percent of the time. In the report, HHS noted that the  assessment team determined that hundreds of errors in software code  contributed to that downtime.

section_title: Limited System Functionality
paragraphs: As of initial launch, the functionality provided by the FFM system was  limited compared to what was planned, thus hindering users from  performing actions needed to compare health plans and small businesses  from purchasing plans, as well as requiring the use of a manual process  for paying issuers.

In September 2011, CMS issued the first FFM system statement of work,  which stated that the federal marketplace would provide all exchange  capability in states electing not to establish a state-based marketplace.  The statement of work identified system modules that were to encompass  all federal exchange requirements, including the eligibility and enrollment,  plan management, and financial management modules.

However, at the time of initial open enrollment in 2013, while parts of the  eligibility and enrollment module were completed, others were not.  Specifically, after creating an account through the website, consumers  could apply for health coverage, compare and select a plan for  enrollment, and receive an advance payment of the premium tax credit  and Medicaid/CHIP eligibility determination through the eligibility and  enrollment module. Nonetheless, consumers were not able to perform  other intended eligibility and enrollment functions such as (1) “window  shopping” (i.e., comparing different plans) for health plans prior to  providing personal information to CMS and signing up for coverage, or  (2) designating authorized representatives to apply for coverage on their  behalf or change their advance payment of the premium tax credit  election. Further, small businesses were unable to purchase health  coverage for their employees through the FFM eligibility and enrollment  module.

Other planned modules, including the plan management and financial  management modules, were also not complete and thus did not provide  intended functionality. For example, CMS could not use the system to  acquire, certify, and manage issuers offering qualified health plans  through the exchange’s plan management module. Additionally, the  system did not allow payments to be made to health issuers and did not  calculate payments for reinsurance through the financial management  module.

section_title: CMS Has Taken Steps to Address Identified System Problems
paragraphs: Since the troublesome launch of Healthcare.gov, CMS has taken various  actions to address the problems that impeded the initial use of the  website and its supporting systems. For example, beginning in October  2013, the agency initiated steps to mitigate the lack of adequate system  capacity. Specifically, among other things, it doubled the number of  servers for systems supporting Healthcare.gov, added virtual machines  for the Enterprise Identity Management and FFM systems, and replaced a  virtual database with a high-capacity physical database for the Enterprise  Identity Management system, allowing more efficient system processing  for both the identity management and FFM systems.

By taking these actions, CMS increased overall system capacity to  support Internet users—going from 25 to 400 Terabytes of monthly  capacity. According to an HHS website, by December 2013, the  increased system capacity allowed the system to accommodate more  than 1.8 million visits a day from consumers to the website and its  supporting systems. According to an HHS progress report issued in  December 2013 and other data provided by CMS, Healthcare.gov system  availability went from 42.9 percent to just over 93 percent during  November 2013, and the FFM system response time went from 8  seconds in late October 2013 to less than 1 second by December 2013.

In addition, in October 2013 CMS took steps to mitigate system coding  issues. For example, the agency directed its development contractors to,  among other things, modify system software to increase the efficiency in  system interactions and implement software fixes to address issues with  users logging into their accounts. In December 2013, HHS reported that  the number of errors encountered by individuals using the system  decreased by over 5 percent by the end of November 2013, going from a  6 percent error rate to under 1 percent.

Also, CMS documented data quality plans for the Enterprise Identity  Management system in March 2014 and the FFM system in June 2014  that outline an approach for improving the quality of the systems’ code.  The Enterprise Identity Management system plan calls for peer reviews to  ensure that contract requirements are met and product reviews are  performed on all deliverables. The FFM plan identifies three types of  quality reviews—Peer Reviews, Process and Product Quality Assurance  Reviews, and Quality Assessment Reviews—that are to be used to  ensure work products conform to documented processes and standards.

Peer Reviews. As the primary verification activity, peer reviews are to  be conducted to help facilitate early detection of problems, and thus  reduce the number of problems discovered in later stages of  development, which helps to minimize the cost associated with  rework. Peer reviews are to include a review of requirements, design,  code, and test planning work products. Peer reviews can be  conducted by peer members of the project team or team leads,  managers, and design review boards.

Process and Product Quality Assurance Reviews. These reviews  are intended to ensure that work products, project management  processes, high-level development processes, and day-to-day  practices adhere to documented CMS processes and standards.  These reviews are to be conducted by contractors not directly  responsible for the work product or process being reviewed.

Quality Assurance Review. The primary purpose of the quality  assurance review is to verify that the FFM IT program is progressing  based on expectations and is providing business value, and that  appropriate risks are identified and managed so that solutions can be  delivered on time and within budget. This review is conducted by a  contractor Managing Director who is also referred to as a quality  assurance Director. These directors are external to the FFM system  project, with technical and functional expertise in line with the  program.

Nonetheless, even with these efforts, IV&V assessments continued to  identify issues with software coding practices. For example, in July 2014  the assessment team identified over 11,000 critical code violations in the  eligibility and enrollment module of the FFM system which could cause  major issues in production or difficulties in maintaining the code. The  assessment team highlighted the need for CMS to ensure the FFM  system code is reviewed and that critical and major violations are  remediated.

CMS has also taken steps to develop additional system functionality for  the FFM system. In order to complete FFM system development and to  improve system functionality already provided by the original contractor  tasked with developing this system, the agency awarded a new contract  in January 2014. According to the statement of work, this new FFM  system development contract represents almost exclusively new  development and major fixes to software already developed. The contract  called for the new contractor to design, develop, test, and implement  services supporting the FFM system. This includes the financial  management module, the plan management module, and certain  eligibility and enrollment module functions that include eligibility  verification and determination.

Some FFM system development activities are still in progress, such as  the payment service to issuers for subsidy payments to issuers through  the financial management module, among others. However, CMS made  progress in developing and implementing services related to the FFM  eligibility and enrollment and plan management modules. For example,  consumers can now “window shop” using the eligibility and enrollment  module, and CMS can now use the plan management module to validate  plan application information and route the validated information to the  appropriate system supporting Healthcare.gov.

section_title: CMS Inadequately Applied Best Practices in Developing Systems Supporting Healthcare.gov, and Needs to Build on Recent Progress
paragraphs: In developing Healthcare.gov and its supporting systems, CMS did not  adhere to best practices for managing IT development projects, which  contributed to problems with the launch of Healthcare.gov and its  supporting systems. Such best practices include managing requirements  to ensure that delivered functionality meets the needs of users,  conducting adequate system testing to validate that systems function as  intended, and providing oversight to ensure that a project is progressing  as planned and that corrective actions are taken as needed. Specifically,  CMS did not effectively manage requirements of key systems supporting  Healthcare.gov, nor did it adequately test the system, or include key  information in system test plans and test cases. In addition, CMS’s  oversight of the initiative was limited by an unreliable schedule, lack of  estimates of work needed to complete the project, unorganized and  outdated project documentation, and inconsistent reviews of project  progress.

CMS program and contracting officials attributed weaknesses in these IT  management areas to the complexity of developing a first-of-its-kind  federal marketplace, which was exacerbated by changing requirements  and compressed time frames for completing and deploying the systems.  CMS has taken action to address deficiencies in applying systems  development best practices for the FFM system. However, deficiencies in  requirements management, systems testing, and oversight remain. By not  engaging in effective systems development practices, CMS lacks  essential mechanisms to ensure the successful delivery of IT systems  such as Healthcare.gov and its supporting systems. In addition, HHS has  not provided adequate oversight of the Healthcare.gov initiative through  its office of the CIO, while OMB’s oversight role was limited to facilitating  discussions with federal partners, providing federal policy guidance, and  overseeing the project’s budget.
subsections: section_title: Weaknesses in Requirements Management Limited CMS’s Ability to Ensure That System Functionality Was Implemented as Intended
paragraphs: Best practices developed by the Software Engineering Institute call for,  among other things, ensuring that requirements are understood and  approved by system stakeholders, including system owners and system  developers. Thus, as a project matures and requirements are derived,  the requirements should be clearly defined, agreed upon, and approved  by the system stakeholders, including system owners and system  developers. Consistent with best practices, CMS guidance also requires  this approval. Specifically, the CMS Requirements Management Plan  documented specifically for the FFM and DSH systems called for  functional requirements to be approved by a CMS official—the Center  for Consumer Information and Insurance Oversight business owner— before being sent to the development team. The plan further stated that  an agency official within the Office of Information Services was to  document this approval in the Collaborative Application Lifecycle Tool  (CALT), the agency’s project management system and requirements  repository. The system records the name of the approver and the date  and time at which the requirement was approved.

However, in many instances, functional requirements that had been  identified for the FFM and DSH systems were included in the  development effort prior to or without clear evidence of required CMS  approval. Specifically,

Of the 37 FFM eligibility and enrollment functional requirements that

9 were designated as having been approved prior to development,

8 were approved after the requirements were sent to

20 were never approved by CMS.

Of the 67 DSH functional requirements we selected,approved by a CMS official.

CMS officials within the Office of Information Services acknowledged that  approvals were not always obtained for functional requirements prior to  the development of the FFM and DSH systems. The officials stated that  they were unable to enforce consistent application of life-cycle processes  because they were trying to develop the system in an expedited fashion  to meet the October 2013 deadline.

By allowing functional requirements to move to development without  approval, CMS did not position itself to ensure that there was a common  understanding of requirements between CMS Center for Consumer  Information and Insurance Oversight business owners and the  contractors tasked with developing these systems, or that expected  functionality would be provided.

Since Systems Launch, CMS Has Developed a New Requirements  Approval Process, but It Is Not Fully Implemented  After the initial system launch, CMS documented and began  implementing a new IT governance process in June 2014 that calls for  business requirementsCMS business owner, the CMS approving authority, and the contract  organization’s approving authority—instead of one CMS official (the  business owner). In addition, CMS officials within the Office of Information  Services stated that functional and technical requirements also require  the same three stakeholders’ approval and that these stakeholders’  signatures be included on all requirements documentation, indicating their  approval.   to be approved by three key stakeholders—the  Even with its new requirements approval process, however, CMS has not  consistently and appropriately approved requirements. In particular, 1 of  18 FFM system requirements documents that we examined under the  new process contained all the necessary approvals for business,  functional, and technical requirements that had been documented as part  of the effort to improve and expand system functionality. Specifically:

Of the 13 business requirements documents, 1 had been fully  approved by all three stakeholders. On the other hand, 4 business  requirements documents included the signature of the FFM  contractor, but did not include the CMS approving authority and  business owner signatures; 2 documents were approved by the CMS  business owner, but were not approved by the CMS approving  authority and the FFM contractor; and the remaining 6 were approved  by the CMS approving authority and business owner, but were not  approved by the FFM contractor.

Of the four functional design documents, none were fully approved by  the required stakeholders. Two of the four were not approved by the  CMS approving authority and the FFM contractor. One was approved  by the CMS business owner and the CMS approving authority, but  was missing the approval of the FFM contractor. The remaining  functional design document was approved by CMS’s approving  authority, but was missing the approval of the FFM contractor and  CMS business owner.

The one technical design document included the signature of the  CMS approving authority, but was missing the signatures of the CMS  business owner and FFM contractor.

In addition, it was not always clear what requirements were being  approved. Specifically, while pages with approval signatures were  scanned and uploaded to CALT, 10 of the 18 signature pages were not  electronically attached or linked to documents specifying the  requirements being approved, making it difficult to determine what  requirements were actually approved. These conditions present  uncertainty as to whether CMS and its contractors can readily and always  determine if the requirements being developed had received the  appropriate approval.

CMS officials in the Office of Information Services acknowledged the lack  of approvals and stated that as of mid-October 2014 they had not yet fully  implemented the new IT governance process, which is to include the  complete documentation of requirements approvals. Specifically, while  CMS has documented the approval procedures for business  requirements, it has not yet documented procedures for approving  functional and technical requirements.

While acknowledging these weaknesses, officials within the Office of  Information Services added that CMS is currently tracking approvals  through a weekly management report. However, this is inconsistent with  the agency’s newly developed procedures, which require stakeholders’  signatures on requirements documentation to indicate approval. The  officials further noted that they intend to review all required  documentation to identify any signatures that may be missing after 2015  open enrollment is complete. However, this review would take place after  the requirements were developed and would not ensure that they were  clearly defined, agreed upon, and approved before development began.  Until it fully documents and implements its new requirements approval  process, CMS may not establish a shared understanding of requirements  with its contractors, potentially resulting in critical system functionally not  providing needed capabilities.
subsections: section_title: Requirements Lacked Traceability Prior to Initial Launch
paragraphs: Best practices developed by the Software Engineering Institute call for,  among other things, effectively managing requirements by maintaining  bidirectional traceability from the high-level original source, such as the  business and program requirements, to the lower-level more detailed  system and technical requirements, and from those lower-level  requirements back to their original source. Such bidirectional traceability  allows stakeholders to (1) understand any system-wide effects as a result  of changes to requirements, (2) determine whether all high-level  requirements have been completely addressed and whether all lower- level more detailed requirements can be traced to a valid source (i.e.,  maintain requirement dependencies to ensure that higher-level  requirements are being addressed by lower-level more detailed  requirements), and (3) update requirements documentation as necessary  for approved changes.

Business processes illustrate the interactions and information exchanges among  functional activities and stakeholders (e.g., states, federal agencies, insurers, and  employers) performing those activities. These associations provide information for  stakeholder relationships and information exchanges to facilitate coordination and  agreement among stakeholders concerning their respective roles, responsibilities, and  information exchange needs.

FFM eligibility and enrollment business process associations were not documented in  CALT as required by the Requirements Management Plan. According to CMS officials  within the Office of Information Services, these associations were documented in a  separate spreadsheet. However, the spreadsheet only included 1,137 of the 3,779  eligibility and enrollment functional requirements. We reviewed all of the 1,137 functional  requirements.

However, by not maintaining bidirectional traceability among  requirements, CMS could not ensure that key stakeholders had a clear  understanding of system-wide effects as a result of changes to  requirements, determine whether all source requirements had been  completely addressed and whether all lower-level requirements could be  traced to a valid source, and appropriately update requirements  documentation for approved changes.

CMS Has Taken Steps to Establish Bidirectional Traceability for  Requirements Developed After Initial System Launch  To help improve the bidirectional traceability of requirements, CMS  documented and began implementing a new FFM requirements  management process in June 2014. This process includes guidance on  documenting traceability in a new requirements management system— the Quality Center Application Lifecycle Management tool.

Since the fall of 2014, CMS and its FFM contractors have made a  concerted effort to provide bidirectional traceability within the life-cycle  management tool for approved business, functional, and technical  requirements for development efforts. In November 2014, FFM  contractors, along with CMS officials within the Office of Information  Services and Office of Legislation, demonstrated to us how the current  process is providing bidirectional traceability. Specifically, contractors  provided examples of business requirements and their associated  functional requirements using the tool. The contractors also provided  examples of how functional requirements and their associated business  requirements were linked. According to the FFM contractor, as of  November 2014, requirements for three increments within the financial  management module and nine increments within the eligibility and  enrollment module were fully traceable within the life-cycle management  tool.

Going forward, effective use of this life-cycle management tool should  assist CMS in maintaining bidirectional traceability and, thus, (1) facilitate  the understanding of system-wide effects as a result of changes to  requirements, (2) help determine whether all source requirements have  been completely addressed, and (3) help determine whether all lower- level requirements can be traced to a valid source.

section_title: Systems Supporting Healthcare.gov Were Not Fully Tested, and Test Documentation Was Missing Key Elements
paragraphs: Testing an IT system is essential to validate that the system will satisfy  the requirements for its intended use and user needs. Effective testing  facilitates early detection and correction of software and system  anomalies; provides an early assessment of software and system  performance; and provides factual information to key stakeholders for  determining the business risk of releasing the product in its current state.  Best practices developed by the Institute of Electrical and Electronics  Engineers (IEEE) suggest that systems testing should be conducted  early and often in the life cycle of a systems development project to allow  for the modification of products in a timely manner, thereby reducing the  overall project and schedule impacts.

In May 2011, CMS documented a testing framework that was to establish  a consistent, repeatable CMS testing life-cycle process for business  application and infrastructure testing. In statements of work, CMS  required its FFM and DSH system development contractors to use this  framework and perform testing and validation of all software releases  prior to implementation. This was to include integration and end-to-end   of both the FFM and DSH systems, which would test how, for  testing example, various modules of the FFM system work together. This testing  would also assess whether the individual systems that support the  federally facilitated marketplace work together as intended. Further, CMS  testing documentation stated that any critical defects discovered through  the testing process were to be corrected or mitigated before the system  was put into production.

Integration testing is preliminary testing performed by the system developer to assess  the interfaces, data, and interoperability of modules and systems within a single business  application. End-to-end testing is a type of integration testing that tests all of the business  application’s access or touch points, and data, across multiple business applications and  systems, front to back (horizontal) and top to bottom (vertical), to ensure business  processes are successfully completed. Testing is conducted on a complete, integrated set  of business applications and systems to evaluate their compliance with specified  requirements, and to evaluate whether the business applications and systems  interoperate correctly, pass data and control correctly to one another, and store data  correctly. prior to system launch—integration testing with plan issuers that were  expected to connect to the DSH to send health plan information to the  FFM plan management module had not been completed, with outstanding  defects remaining unaddressed for the FFM system eligibility and  enrollment module. In addition, end-to-end testing of Healthcare.gov and  its supporting systems did not occur prior to system launch as required.  Further, CMS did not always ensure that system defects found during the  testing were corrected prior to system launch; thus, many defective  system components were placed into production.

CMS staff within the Office of Information Services, including a Deputy  Director, as well as representatives of development contractors for the  DSH and FFM systems, stated that there was insufficient time to conduct  all the needed testing prior to system launch. This was, in part, because  requirements were still being defined in mid-2013 and there were delays  in developing software that was ready for testing.

Without complete integration and end-to-end testing of the system, CMS  lacked a basis for knowing if all Healthcare.gov interconnected systems  could operate correctly, pass data correctly to one another, and store  data correctly prior to system launch. In addition, without ensuring that  defects were corrected prior to placing the system into production, CMS  jeopardized its assurance that the system would function as intended.

CMS Has Begun Taking Steps to Improve Systems Testing, but Has  Not Documented Its New Processes  According to officials in the Office of Information Services, CMS has taken  steps aimed at improving its testing processes since the highly  problematic launch of Healthcare.gov. For example, it has implemented a  new tool that integrates systems development and systems testing, which  is intended to provide the agency and its contractors greater visibility into  the development and testing process. In addition, according to CMS  officials in the Office of Information Services, business owners and other  stakeholders are now to review key testing documentation to ensure  proper test coverage and to validate the results.

At the time of our review the agency had not documented this new testing  process. Going forward, without a clearly defined and documented  process for how CMS will implement the testing tool as well as  requirements for stakeholder reviews, CMS may not be able to ensure  testing processes are carried out as intended.
subsections: section_title: System Test Plans Lacked Recommended Elements
paragraphs: A key document needed to ensure that testing is carried out effectively is  a test plan. Test plans describe the technical and management approach  to be followed for testing a system or a component of a system. Best  practices, such as those identified by IEEE, call for test plans to  identify the test items (software or system) that are the object of  testing; provide a description of the overall approach for testing; identify the set of tasks necessary to prepare for and perform testing; identify how testing anomalies will be tracked and resolved;  identify roles and responsibilities for individuals or groups responsible  for testing;  identify the risk issues that may adversely impact successful  completion of the planned testing activities;  identify the means by which the quality of testing processes will be  assured;  specify the necessary test environment and test data, such as  hardware, software, and test support tools; and  specify the criteria to be used to determine whether each test item has  passed or failed testing.

Test plans we examined for the DSH and FFM systems included most,  but not all of the recommended key elements. For example, all 19 DSH  and 14 FFM system test plans documented prior to the systems launch in  October 2013 identified the test items that were the object of testing; the  overall approach for testing; the set of tasks necessary to prepare for and  perform the testing; how testing anomalies were to be tracked and  resolved; and the roles for individuals or groups responsible for testing.

However, a number of these test plans did not address key elements  called for by best practices, relating to the quality of testing and the  pass/fail testing criteria. Specifically:

None of the 19 DSH and 14 FFM system test plans included the  means by which quality of testing processes would be assured.

Eleven of the 19 DSH and all 14 FFM system test plans were missing  detailed criteria to be used to determine whether each test item has  passed or failed testing.

In addition, these plans varied in the extent to which they addressed risk  issues and the test environment information. Specifically:

While all 14 FFM system test plans identified risk issues that may  adversely impact successful completion of the planned testing  activities, 8 of 19 DSH test plans included this information.

While all 14 FFM test plans specified the necessary test environment  and test data, such as hardware, software, and test support tools, 8 of  the 19 DSH test plans included all of the information recommended by  best practices.

These weaknesses existed, in part, because CMS lacked key elements in  its framework. For example, the framework did not require test plans to  include  the risk issues that may adversely impact successful completion of the  planned testing activities;  the means by which the quality of testing processes will be assured;  or  the necessary test environment and test data, such as hardware,  software, and test support tools.

Further, CMS officials in the Office of Information Services acknowledged  the lack of certain key elements in the test plans that existed for systems  supporting Healthcare.gov, and attributed this, in part, to an incomplete  test plan template. Without including key information in the test plans,  CMS had less assurance that testing carried out prior to initial launch was  consistently executed and of sufficient quality to validate that systems  supporting Healthcare.gov satisfied requirements.

Since the initial system launch, CMS has continued to develop test plans  for additional FFM system functionality, and these included most, but not  all, key elements. Specifically, all 11 post-October 2013 FFM system test  plans included test items that are the object of testing; the overall  approach for testing; the set of tasks necessary to prepare for and  perform testing; how testing anomalies will be tracked and resolved; risk  issues that may adversely impact successful completion of the planned  testing activities; and, for the most part, specified the necessary test  environment and test data, such as hardware, software, and test support  tools.

Nonetheless, similar to the pre-October 2013 test plans, FFM test plans  had not identified all key elements called for by best practices.  Specifically, none of the 11 FFM post-October 2013 test plans specified  the means by which the quality of testing processes would be assured,  and 9 of the 11 test plans lacked criteria to be used to determine whether  each test item has passed or failed testing.

In addition, these plans varied in the extent to which they discussed roles  and responsibilities of individuals or groups responsible for testing.  Specifically, while all 11 FFM test plans included the identification of roles  for individuals or groups responsible for testing, 5 of these plans did not  include the details regarding what tasks these individuals or groups would  perform.

According to an Information Technology Specialist within the Office of  Information Services, the test plan template that was used for test plan  development was updated in November 2014 to include the missing key  elements we identified. While updating the test plan template with missing  elements is a positive step, this will not necessarily ensure key  information is included in the test plan. Specifically, although the test  plans we reviewed for FFM and DSH included a section for roles and  responsibilities, for example, the information included was not always  comprehensive and did not provide needed information. As a result, CMS  may continue to lack assurance that testing is consistently executed and  of sufficient quality to ensure that Healtcare.gov-related systems function  as intended.

section_title: System Test Cases Included Most, but Not All, Key Information
paragraphs: As another key type of testing documentation, test cases describe  scenarios that the system must perform to meet intended requirements.  Testing teams use these test cases to determine whether an application,  system, or a particular system feature is working as intended. Best  practices identified by IEEE call for each test case to  include a unique identifier so that each test case can be distinguished  from all other test cases;  specify all outputs and the expected behavior required of the test  items;  identify dependencies (i.e., other test cases that must be executed  before the current test case);  identify and describe the objective for the test case (e.g., what feature  is being tested);  specify the ordered description of the steps to be taken by each  participant for the execution of the test procedure; and  specify the inputs required to execute each test case (i.e., values,  files, databases, etc.).

Best practices also state that test cases should be linked to requirements  in order to help stakeholders ensure that there is a valid relationship  between a system’s requirements and the plans and procedures for  testing to ensure they are met.

Test cases for components of systems supporting Healthcare.gov  included some, but not all key elements. Specifically, all of the selected  test cases (42 DSH and 83 FFM) that were documented prior to system  launch in October 2013 included a unique identifier. However, these test  cases did not always identify two other key elements called for by best  practices—outputs and the expected behavior and test case  dependencies. Specifically:

One of 42 DSH test cases specified outputs and the expected  behavior required of the test items.

While all 83 of the FFM test cases included expected behavior  required of the test items, 12 of the 83 included outputs.

One of 42 DSH test cases and 4 of 83 FFM test cases included  dependencies.

In addition, among the test cases, results were mixed regarding the  extent to which they included the objective, the description of steps, and  the inputs required. Specifically:

While all FFM test cases included the identification and description of  the testing objective, 29 of 42 DSH test cases included that  information.

All the FFM test cases specified the ordered description of the steps  to be taken by each participant for the execution of the procedure, but  one of the DSH test cases included this information.

Among the FFM test cases, 58 of 83 specified all of the inputs  required to execute each test case, while none of the DSH test cases  did so.

In addition, many of the test cases did not include enough information to  allow the project team to determine whether the testing contractor had  performed the test and whether or not the system passed testing.  Specifically, while all 42 DSH test cases included information about  whether or not the test passed or failed, 58 of the 83 FFM system test  cases were missing pass/fail information.

Further, although CMS provided documents that were intended to link  requirements to their corresponding test cases, in many instances these  documents did not correspond to the test cases we reviewed. Specifically,  for 24 of 42 DSH system test cases and 50 of 83 FFM system test cases,  the documents did not include enough information to link the  requirements being tested and the corresponding test cases. For  example, certain documents included a list of test case unique identifiers,  but did not include any information about the requirements related to  those test cases. In other instances, the documents included test case  identifiers that did not use the same naming convention as the test cases  we received, so it was unclear as to what test cases those documents  were related to.

CMS officials in the Office of Information Services acknowledged that test  case documentation for systems supporting the initial rollout of  Healthcare.gov had been lacking and that there were gaps in the  documentation linking the requirements being tested to the corresponding  test cases. They attributed these weaknesses to not having always  followed required procedures for appropriately documenting test cases.  These officials added that the procedures were being followed for the  contract awarded in January 2014 for the implementation of additional  and enhanced functionality for the FFM system. However, we determined  that test cases documented under the new development contract also  lacked key elements (as described below). Without key information  included in test cases, CMS was limited in its ability to ensure that  documented scenarios were performed and thus that applications,  systems, or features supporting Healthcare.gov activities were working as  intended.

Improvements Were Made to Test Cases Developed After Initial  System Launch, but Many Still Lacked Key Elements  CMS took steps to improve the quality and content of its test cases  subsequent to the launch of Healthcare.gov. In particular, all 83 post- October 2013 test cases included a unique identifier, the objective for the  test case, the ordered description of steps to be taken by each participant  for the execution of the procedure, and expected behavior required of the  test items.

However, similar to the pre-October 2013 documentation, these test  cases did not always include outputs and exact values; test case  dependencies; and required inputs. Specifically:

61 of 83 FFM test cases lacked information on outputs and exact

77 of 83 FFM test cases did not include dependencies; and

37 of 83 FFM test cases did not specify all the inputs required to  execute each test case.

Further, although the newly developed test case documentation did not  contain all recommended information, the majority of the documentation  did include information to allow the project team to determine whether the  testing contractor had executed the test and whether or not the system  passed testing, which is a considerable improvement over the previous  process. Specifically, the test procedures for 56 of the 70 newly  developed test cases that we review were executed and included  information about whether the test case passed or failed, compared with  25 of 83 of the pre-launch test cases.

In addition, in November 2014 CMS officials in the Office of Information  Services and the Office of Legislation, along with representatives from the  FFM system development contractor, demonstrated that they were  documenting the linkage of requirements to their corresponding test  cases within the Quality Center Application Lifecycle Management tool.  Going forward, use of this tool should assist CMS in ensuring that there is  a valid relationship between test plans, test design, test cases, and test  procedures. Nonetheless, until CMS begins to standardize and require all  key elements in test case documentation, as recommended by best  practices, it may continue lack information needed to determine whether  an application, system, or one of its features is working as intended.

section_title: CMS, HHS, and OMB Did Not Adequately Oversee Healthcare.gov Initiative System Development
paragraphs: Best practices that we and the Software Engineering Instituteidentified emphasize the importance of project oversight as a means of  ensuring project progress and that appropriate corrective actions can be  taken when project performance deviates significantly from the plan. A  deviation is significant if, when left unresolved, it precludes the project  from meeting its objectives. Best practices call for, among other things,  (1) establishing well-constructed schedules that include the entire scope  of work activities; (2) estimating the level of effort to be expended by the  project team on each task to assist in monitoring the progress of the  project; (3) documenting and monitoring activities for managing project  documentation; and (4) conducting project progress and milestone   have  reviews to address performance shortfalls and understand how well  requirements are being met.

However, CMS did not always (1) ensure project schedules for  Healthcare.gov and its supporting systems were well-constructed;  (2) estimate level of effort for DSH and FFM functional requirements;  (3) implement data management and monitoring processes; and  (4) conduct all recommended and required project progress and  milestone reviews. CMS officials within the Center for Consumer  Information and Insurance Oversight and the Office of Information  Services attributed these weaknesses, in part, to challenges with  enforcing consistent application of life-cycle processes while trying to  develop the system in an expedited fashion to meet the October 2013  deadline. As a result, without adequate and comprehensive information  that would be key for understanding the project’s progress, CMS and  other oversight agencies may not have the data necessary to  appropriately evaluate the project and take corrective actions.
subsections: section_title: Healthcare.gov Schedules Were Not Well-Constructed
paragraphs: A project schedule is a fundamental management tool that specifies when  work will be performed in the future and allows for measuring project  performance against an approved plan. To this end, our Schedule  Assessment Guide states that a project should be guided by an  integrated master schedule that reflects the entire scope of work  activities. An integrated master schedule may be made up of several or  several hundred individual schedules that represent portions of work  within a program. These individual schedules are “subprojects” within the  larger program.

CMS did not always have a comprehensive integrated master schedule  prior to system launch in October 2013. For example, IV&V assessment  reports issued in December 2012, February 2013, and May 2013  identified weaknesses in project scheduling throughout the  Healthcare.gov development process. For example:

Activities related to FFM and DSH system implementation and the  timeline for the design of the DSH database were not included in the  integrated master schedule.

Certain key development activities were not included in the FFM  integrated project schedule.

The FFM testing schedule and the DSH planning schedule did not  contain resource assignments needed to complete the work as  planned.

Therefore, management’s ability to monitor productivity or make effective  decisions on the allocation of resources was severely limited.

CMS Took Steps to Improve Project Schedules after Initial Launch,  but Schedules Were Not Always Well-Constructed  After awarding the new FFM development contract in January 2014, CMS  re-evaluated project schedules for systems supporting Healthcare.gov.  However, project schedules developed since then were not always well- constructed.

Best practices identified by usschedules include the following:

Logically sequencing all work activities. The schedule should be  planned so that critical project dates can be met. To do this, activities  need to be logically sequenced—that is, listed in the order in which  they are to be carried out. In particular, activities that must be  completed before other activities can begin (predecessor work  activities), as well as activities that cannot begin until other activities  are completed (successor work activities), should be identified. Date  constraints and lags should be minimized and justified to help ensure that the interdependence of activities that collectively lead to  the completion of events or milestones can be established and used  to guide work and measure progress.

Confirming that the critical path is valid. The schedule should  identify the program critical paththrough the sequence of work activities. Establishing a valid critical  path is necessary for examining the effects of any activity’s slipping  along this path. The program critical path determines its earliest  completion date and focuses the project team’s energy and  management’s attention on the activities that will lead to the project’s  success. Because a critical path defines a project’s earliest  completion date, it must be a continuous sequence of activities from  the schedule’s status date to the finish milestone.  —the path of longest duration

Ensuring reasonable total float. The schedule should identify  reasonable total floatdetermined. Large total float on a work activity indicates that the work  activity can be delayed without jeopardizing the finish date. The length  of delay that can be accommodated without the finish date’s slipping  depends on a variety of factors, including the number of date  constraints within the schedule and the amount of uncertainty in the  duration estimates, but the work activity’s total float provides a  reasonable estimate of this value. As a general rule, activities along  the critical path have the least total float.   so that the schedule’s flexibility can be  CMS has made an effort to tie all subprojects into an integrated master  schedule and to capture all of the required effort for the Healthcare.gov  initiative. Specifically, the agency had documented at least 26 subproject  schedules within the integrated master schedule. However, our review of  schedules for 4 of 17 FFM subprojects  determined that these schedules  did not always include key characteristics of a well-constructed schedule.

The FFM integrated master schedule contained 17 subproject schedules. We selected 4  schedules that relate to the Plan Management, Small Business Health Options Program,  Financial Management, and Eligibility and Enrollment modules of the Federally Facilitated  Marketplace System. predecessor activities for 9 percent of its remaining activities. In addition,  a significant number of date constraints were reflected in the project  schedules, and for the majority of them the agency did not provide a  justification. For example, we identified date constraints on 26 percent of  the remaining work activities in both the Financial Management and Small  Business Health Options Program schedules.

CMS did not always ensure that project schedules had a valid critical  path. For example, two of the four selected schedules—for the Eligibility  Business Operations and the Financial Management projects—did not  have valid critical paths because there were several gaps of time where  no critical activities were scheduled. Specifically, the critical path for the  Eligibility Business Operations schedule had four gaps, ranging from 8 to  15 days, where no critical activities were scheduled. The Financial  Management schedule had a gap of nearly 6 months with no critical  activities scheduled.

In addition, the other two schedules—for the Small Business Health  Options Program and Plan Management projects—did not have valid  critical paths because the paths were determined by long-duration  support and management activities rather than discrete, well-defined  work. For example, the Small Business Health Options Program schedule  includes management activities such as “Operations Management” and  “Deployments Management” that appear in the schedule as critical  activities. However, a critical path cannot include these types of activities  because, by their very nature, they do not represent discrete effort.

CMS did not always ensure reasonable total float. Each of the four project  schedules we reviewed appeared to be overly flexible, allowing for many  activities to slip a significant number of days before impacting the dates of  key events. For example, the Plan Management schedule allowed 50  percent of its remaining activities to slip more than 98 working days  before impacting the key finish milestone. Additionally, according to the  schedules, remaining activities in the Small Business Health Options  Program, Financial Management, and Eligibility Business Operations  schedules could be delayed an average of 49 to 50 days before causing  the project finish dates to be delayed. Inaccurate values of total float  falsely depict true project status, which could lead to decisions that may  jeopardize the project.

Table 1 below summarizes how well the current subprojects’ schedules  met best practices.

Because these project schedules did not fully meet key practices for  ensuring that they are well-constructed, they are limited as tools for  gauging progress and providing reliable estimates of project timelines. In  addition, because the reliability of an integrated master schedule depends  in part on the reliability of its subordinate schedules, the weaknesses in  these schedules will be reflected in the overall schedule for the  Healthcare.gov effort.

section_title: Level of Effort Was Not Consistently Estimated
paragraphs: Level-of-effort estimates are used to estimate the amount of time a  project will take to develop. According to the Software Engineering  Institute, this involves estimating the amount of time and resources to  be spent on each work item, such as developing functional requirements  for a system. These estimates can then be compared to the actual time  and resources expended on each work item. This allows the project’s  stakeholders to determine how well the project is progressing and  whether schedules should be adjusted or additional resources need to be  applied.

Consistent with best practices, the CMS Requirements Management Plan  documented specifically for the FFM and DSH systems required system  development teams to estimate the level of effort for each functional  requirement and those estimates to be recorded in CALT. The level-of- effort estimates, according to the plan, were to be used to inform  velocity—that is, how quickly the project was being developed.

However, CMS and its contractors rarely documented levels of effort for  the FFM and DSH functional requirements prior to initial system launch in  October 2013. Specifically, nearly 100 percent of the FFM eligibility and  enrollment functional requirements and nearly 84 percent of the DSH  functional requirements documented prior to initial launch were missing  the estimated levels of effort.

According to agency officials in the Office of Information Services,  contractor earned value management and other financial reports were  used in the place of level of effort estimates to track contractor progress.  However, the officials agreed that, while these reports would allow them  to track the progress made on total project cost estimates, these reports  would likely not provide the full insight necessary on how project  development was progressing as could be provided with level-of-effort  estimates.

Due to the lack of level-of-effort estimation, all subsequent monitoring  mechanisms that depended on these estimates, including velocity  reports, would have provided minimal guidance to CMS and its  contractors in monitoring work status and the remaining time needed to  complete projects.

CMS Has Taken Steps to Estimate Level of Effort for Major System  Modules and Supporting Projects, but Has Not Developed or  Documented This Policy or Procedures  As part of CMS’s efforts to improve project management processes after  initial launch of Healthcare.gov and its supporting systems, agency  officials stated in August 2014 that they had begun the process of  estimating levels of effort and including that information in a system that  they historically used to track software defects. They stated that CMS  planned to use this system to track further FFM software development  efforts, in order to provide more visibility into progress being made by the  systems’ development contractors. In addition, the agency provided  documentation to demonstrate its progress in estimating level of effort for  the FFM system. Specifically, the documentation showed that FFM  contractors had begun estimating levels of effort for major system  modules and supporting projects.

However, current CMS policy does not address estimating level of effort,  including how it should be calculated and applied. Specifically, neither  CMS’s eXpedited Life Cycle (XLC) process nor its newly developed  Requirements Management Guide addresses estimating level of effort at  any level. As a result, it will be difficult for agency officials to have  reasonable assurance that level-of-effort estimates are developed and  calculated and applied in a consistent manner and, therefore, it may be  limited as a tool for accurately monitoring progress.

section_title: CMS Lacked Effective Data Management Monitoring Practices
paragraphs: state that  Best practices identified by the Software Engineering Instituteexplicit specifications should be made concerning what, how, where, and  when data should be collected and stored to ensure their validity and to  support later use for analysis and documentation purposes. In this case,  data are forms of documentation required to support a project in various  areas (e.g., administration, configuration management, and quality).  These documents, among other things, are then used by project  stakeholders to conduct project oversight. Best practices further call for  activities for managing these data to be documented and monitored to  ensure that data management requirements are being satisfied.  Depending on the results of monitoring and changes in project  requirements, situation, or status, it may be necessary to re-plan the  project’s data management activities.

To facilitate a consistent process for managing documents, including  those that define requirements, CMS developed a guide in April 2012 for  internal and external stakeholders (e.g., other federal agencies providing  eligibility determination information). This guide requires the use of CALTSpecifically, the guide calls for updates to the status of each requirement  as development progresses to help facilitate project oversight. In addition,  the guide provides and defines specific status designations, such as  “system requirement approved” and “ready for development.” Further, the  agency’s Requirements Management Plan documented specifically for  the FFM and DSH systems required that CALT be used for storing  various project management documentation, including requirements;  source code; network, hardware, and infrastructure descriptions; test  cases; test results; and system defects.   for managing project data and functional requirements.

However, CMS and its contractors did not effectively implement data  management processes. For example, they used status designations that  were not standardized or defined, which would have hindered CMS’s  ability to analyze project progress and effectively oversee the  development for the FFM and DSH systems. Specifically:  seven undefined status designations, such as “grooming in progress,”  were used for the DSH functional requirements; and  two undefined status designations, “artifact confirmed” and “planned  development completed,” were used for the FFM eligibility and  enrollment module functional requirements.

Further, key project management documentation was not always stored  in CALT as required, which impeded reviews of the development effort.  For example, documents needed for reviews by the IV&V assessment  team in September 2012 and December 2012, such as quality assurance  testing results and hardware and software requirements documents, were  located on a contractor’s SharePoint site and were not uploaded to CALT.  This would have made it difficult for the assessment team to conduct their  review.

CMS officials in the Office of Information Services stated that project  owners of each individual effort, to include the DSH and the FFM  systems, were given autonomy in managing the status of functional  requirements within CALT. Consequently, it was difficult for CMS officials  responsible for overseeing the entire project to ensure consistency in  managing project documentation across each individual project team, of  which there were over 200 during the initial development of  Healthcare.gov and its supporting systems. The CMS Deputy Chief  Information Officer added that because project teams were receiving new  requirements well into the development process, required documentation  was not always a high priority.

This lack of a consistent process for managing project data prior to initial  system launch increased the risk that CMS would not have been able to  appropriately and effectively (1) monitor the progress of functional  requirements as they were being developed, (2) ensure all key  documentation needed for overseeing project development activities was  documented and updated, and (3) monitor data management.

Weaknesses in Data Management Practices Continued after Initial  Launch, but CMS Has Plans to Address Them  Subsequent to initial system launch, problems in CMS’s data  management practices persisted. For example, contractor staff stated that  several documents we requested for our review had not yet been  uploaded to CALT. Instead, these documents were stored on contractor  systems, and thus were not readily available for project oversight. In  addition, folders within CALT were not always well-organized, making  locating relevant documentation difficult and time consuming. For  example, many of the folders were similarly named, or the names of the  folders were too vague to determine what documents were included  within them. To illustrate, three sub-folders within the same folder were  named “UAT.” In addition, while certain software release folders were  named by software release number, others were named using a calendar  date, making it difficult to know what documentation was relevant to each  release.

To help mitigate weaknesses in data management monitoring, CMS  developed a document management reference guide for the FFM system  in July 2014 to establish a process for managing documents created by  the FFM development contractor. The guide specified necessary steps for  uploading and tracking documents in CALT. In addition, CMS has revised  its procedures for tracking the status of requirements through design and  testing, and no longer uses undefined status designations.

CMS officials in the Office of Information Services stated that, once open  enrollment for 2015 has ended, they intend to perform a review of all  required CALT documentation, identify missing documents, and locate  and upload those documents into CALT. The officials said they expect  this effort to be completed by April 2015.

section_title: CMS Does Not Always Conduct Progress and Milestone Reviews
paragraphs: According to best practices outlined by the Software Engineering Institute,  the purpose of a progress review is to provide relevant stakeholders the  results and impacts of a project’s activities and to determine whether  there are significant issues or performance shortfalls to be addressed.  Milestone reviews are pre-planned events or points in time at which a  thorough review of status is conducted to understand how well  stakeholder requirements are being met. These reviews are important to  ensure that a project is progressing as planned and to identify corrective  actions needed.

Consistent with best practices, CMS requires progress and milestone  reviews for each newly developed system. According to the CMS XLC— its system development life-cycle process—the purpose of these reviews  is to provide management and stakeholders with the opportunity to  assess project work to date and identify any potential issues. The CMS  XLC calls for a project process agreement, which is to serve as an  agreement between CMS and its development contractors on the  progress and milestone reviews and artifacts (i.e., documentation)  required for a project. The agency has identified 11 different progress and  milestone reviews which vary depending on the complexity of the project.  These reviews are to be conducted by CMS governance boards, which  are to approve the project to continue with the next phase of the systems  development life cycle. Table 2 describes the progress and milestone  reviews documented in the CMS XLC.

The FFM, DSH, and Enterprise Identity Management systems were all  deemed highly complex by CMS; as such, CMS guidance recommends,  but does not require, that they undergo all of the reviews discussed  above. However, the three systems did not undergo all the recommended  reviews. CMS documented a project process agreement for the  Enterprise Identity Management system in January 2012 which stated  that it should undergo 10 of the 11 progress and milestone reviews (all  but the Investment Selection Review) and specified the required artifacts  for each review. However, the agency could not demonstrate that 5 of  these reviews were held. CMS officials stated that 4 of these 5 reviews  had been performed, but they could not provide any evidence to show  this performance. For the DSH and FFM systems, the agency did not  document project process agreements, and it provided evidence that  some, but not all, of the recommended reviews were held for each. Table  3 shows the recommended reviews for a highly complex system and  whether or not those reviews were held for each system.  reviews was compromised due to slippages in scheduled deliverables.  However, it is unclear whether or not the contractors were aware of the  required reviews since the FFM and DSH systems both lacked project  process agreements.

Regarding the missing review artifacts, the officials further stated that all  critical artifacts for each gate review were developed and that the missing  artifacts were non-critical. However, the CMS life-cycle framework does  not designate artifacts as critical or non-critical, nor does it define these  terms. By not ensuring that required progress and milestone reviews took  place and that all required artifacts were developed, CMS stakeholders  lacked full awareness of the results and impacts of the project’s activities  and significant issues or performance shortfalls to be addressed.

CMS Has Taken Steps to Improve Processes for Project and  Milestone Reviews, but All Required Reviews Have Not Been Held  In January 2014, CMS began taking steps to improve its oversight  processes for conducting progress and milestone reviews. These  improvements, according to officials in the Office of Information Services,  included requiring greater collaboration between CMS and its contractors;  increasing the number and frequency of contract deliverables, which  would include key artifacts provided during the reviews; and placing  greater emphasis on progress and milestone reviews as well as formal  signoffs prior to the next life-cycle phase. Additionally, in May 2014 and  June 2014, CMS documented project process agreements for the  portions of the FFM system that were to be developed under the new  contract.

Despite these efforts, CMS had not documented a project process  agreement for DSH as of December 2014. In addition, although Office of  Information Services officials stated that they had held all the required  reviews for the portions of the FFM system that had been placed into  production at the time of our review, they were unable to provide  evidence for 5 of 20 required reviews. Table 4 below shows the required  reviews for the FFM system and whether or not those reviews were held  for newly developed portions of the FFM system that were in production  as of July 2014.

In addition, CMS was not always following the FFM project process  agreement. For example, Office of Information Services officials stated  that production readiness reviews and operational readiness reviews  were combined for certain increments. However, these reviews have  different purposes, and the project process agreements stated that they  should occur separately.

This approach to conducting reviews puts CMS at continued risk that  stakeholders may not be provided sufficient information on the results and  impacts of Healthcare.gov-related activities, identify significant issues or  performance shortfalls that need to be addressed, and understand how  well requirements are being met. In addition, inconsistent application of  the project process agreements may lead to key reviews continuing to be  missed and approvals not being obtained.

We previously reported the lack of certain progress and milestone   We  reviews in a report on Healthcare.gov contract management.recommended that HHS direct CMS to ensure that information technology  projects adhere to requirements for governance board approvals before  proceeding with the next phase of the systems development life cycle.  HHS agreed with and had begun to take actions to address our  recommendation.

section_title: HHS Had a Limited Role in Overseeing the Development and Implementation of Healthcare.gov and Its Supporting Systems
paragraphs: The Secretary of HHS is required by law and OMB guidance to designate  a CIO to be responsible for the management of agency information and  information technology. and other assistance to agency heads and other senior management  personnel on IT acquisition and management, monitoring the  performance of IT programs (including whether to continue, modify, or  terminate a program or project), and ensuring compliance with  information security requirements. More recently, Congress has  reaffirmed the importance of CIOs having a strong role in overseeing IT at  executive branch agencies. Specifically, in December 2014, new federal  information technology acquisition reform requirements were included in  the National Defense Authorization Act, to ensure that the CIO has a  significant role in the management, governance, and oversight processes  related to their agency’s IT investments.

44 U.S.C. § 3506(a), as amended by Pub. L. No. 104-106, § 5125 (Feb. 10, 1996), and  40 U.S.C. § 11315 (Paperwork Reduction Act of 1995 and the Clinger-Cohen Act of  1996); 44 U.S.C. 3501 note (E-Government Act of 2002, Pub. L. No. 107-347, § 202), and  44 U.S.C. § 3544(a)(3) (Federal Information Security Management Act of 2002), which as  of Dec. 18, 2014, was superseded by 44 U.S.C. § 3554(a)(3) (Federal Information  Security Modernization Act of 2014, Pub. L. No. 113-283); and OMB, Memorandum for  Heads of Executive Departments and Agencies, M-11-29 (Washington, D.C.: Aug. 8,  2011).

In March 1996, the Secretary of HHS delegated the Secretary’s IT-related  authorities under the Clinger-Cohen Act to the HHS CIO. The CIO in turn  requested that operating division heads designate a CIO for their  respective divisions, and that the operating division CIOs serve as  members of the department’s IT Investment Review Board. This board,  which is chaired by the HHS CIO, is to review, validate, and approve  selected IT investments in the department’s portfolio. An IT investment  may be selected for review at any time during its life cycle if it is high risk  and high value, is a high-visibility initiative, or is performing poorly, among  other criteria. This is consistent with key practices outlined in our IT  investment management guide, which call for the establishment of an  enterprise-wide investment review board to be composed of senior  executives from IT and business units, who are to be given the  responsibility for defining and implementing the organization’s IT  investment governance process.

Beyond the actions taken by CMS, in August 2011, OMB issued a  memorandum to all agency heads, stating that the role of the CIO  should be moved away from just policymaking and infrastructure  maintenance to true portfolio management for all IT. The memo was  intended to clarify the primary responsibility for agency CIOs, to include  responsibility over the entire IT portfolio for the agency and for terminating  or turning around underperforming investments.

Although the Secretary of HHS appointed a CIO, this official had a limited  role in overseeing the development and implementation of Healthcare.gov  and its supporting systems. The HHS CIO stated that his office did not  conduct oversight of the initial design and development for  Healthcare.gov and its supporting systems. The CIO further stated that  the status of the Healthcare.gov development project was occasionally  discussed at regular monthly meetings with senior leadership from each  operating division. However, the CIO stated that no issues with  Healthcare.gov and its supporting systems were raised in these meetings  prior to initial system launch.

In addition, although HHS established a process through its IT Investment  Review Board that may have revealed technical issues with  Healthcare.gov and its supporting systems, the CIO stated that the board  has not been active for 2 to 3 years. The CIO also stated that the  department is large and federated and his office’s ability to oversee its  operating divisions, such as CMS, is limited. He added that oversight  reviews are conducted within the operating divisions by their own  investment review boards.

By not effectively monitoring the performance of the Healthcare.gov  initiative prior to the initial launch in October 2013, the HHS CIO was not  appropriately positioned to advise the Secretary on actions that should be  taken to improve the program.

The Office of the CIO Expanded Its Oversight Role after Initial  Launch, but a Key Review Board Is Still Not Active  The HHS Office of the CIO (OCIO) has expanded its oversight role for the  Healthcare.gov initiative since initial launch by convening regular  meetings and briefings discussing the Healthcare.gov initiative with  officials at various levels. The CIO stated that CMS now regularly shares  project documentation with OCIO, which allows them to have better  insight as to the status of the project and its development activities.

The HHS CIO also stated that although he now has greater insight into  the project’s development progress, he does not believe he has the  authority to manage IT investments at the operating division level, which  includes the Healthcare.gov initiative. However, as previously noted,  federal law and OMB guidance place responsibility for overseeing and  managing the department’s IT investments with the CIO. Thus, the CIO  should be positioned within the department to successfully exercise his  authority.

Further, the department-wide investment review board called for by HHS  policy would provide a mechanism for carrying out these responsibilities,  although it has not met for the past 2 to 3 years, according to the CIO.  Until the department-wide investment review board carries out its  assigned duties, the oversight that HHS provides for Heathcare.gov- related projects may continue to be limited, potentially resulting in missed  opportunities to take timely corrective actions on poorly performing  projects.

section_title: OMB Had a Limited Role in Overseeing the Development and Implementation of Healthcare.gov and Its Supporting Systems
paragraphs: By law, OMB oversees the management by federal agencies of  information and information technology. OMB’s responsibilities include  establishing processes to analyze, track, and evaluate the risks and  results of major capital investments in information systems made by  executive agencies, as well as issuing guidance on processes for  selecting and overseeing agency privacy and security protections for  information and information systems. OMB’s guidance under these  authorities has included directions to agencies on the roles and  responsibilities of CIOs and the establishment of IT investment  management processes.

In June 2009, OMB launched the Federal IT Dashboard as a public  website that reports performance and supporting data for major IT  investments. The dashboard is to provide transparency for these  investments in order to facilitate public monitoring of government  operations and accountability for investment performance by the federal  CIOs who oversee them. According to OMB, it began using the  dashboard to identify at-risk investments with its launch in June 2009.  These investments became the focus of joint OMB-agency TechStat  Accountability Sessions (TechStats)—evidence-based reviews intended  to increase accountability and transparency and to improve investment  performance through concrete actions.

In January 2010, OMB began conducting TechStat sessions to enable the  federal government to intervene by turning around, halting, or terminating  IT projects that are failing or are not producing results. OMB has identified  factors that may result in an investment being selected for a TechStat  session, such as—but not limited to— evidence of (1) poor performance,  (2) unmitigated risks, and (3) misalignment with policies and best  practices. Although OMB called for agencies to work with their CIOs to  conduct TechStat sessions at the agency level beginning in December  2010, OMB may still select investments for review. Agency CIOs or OMB  select these high-risk projects for evaluation, and conduct a review of the  proposed improvement plans, revised schedules, and potential changes  to budget requests.

Although OMB plays a key role in overseeing the implementation and  management of federal IT investments, its involvement in overseeing the  development efforts of Healthcare.gov and its supporting systems was  limited prior to the initial launch in October 2013. According to officials  within OMB’s Office of E-Government and Information Technology,  headed by the Federal CIO, OMB’s role in overseeing the development of  Healthcare.gov and its supporting systems was limited to bringing CMS  and its federal partners together to work across technical teams, clarifying  federal policy guidance, and overseeing the project’s budget.

In particular, OMB facilitated monthly meetings of an IT steering  committee consisting of CMS and other key stakeholders (e.g., other  federal agencies providing eligibility determination information) that were  held to coordinate inter-agency efforts on broader federal marketplace IT  work. The meetings, which began in March 2012 and ended in  September 2013, primarily focused on addressing key federal  marketplace information-sharing policies and identifying barriers to  implementation as well as working with federal departments and agencies  as necessary on the implementation and execution of the Patient  Protection and Affordable Care Act.

However, although the Healthcare.gov initiative was considered a high- risk project and independent evaluations and the IT Dashboard identified  problems well before its deployment, OMB officials did not select this  investment for a TechStat review. Specifically, the dashboard indicated a  high-risk evaluation status of Healthcare.gov in March 2013. Officials in  the Office of E-Government and Information Technology stated that it was  HHS’s responsibility to select the investment for TechStat, but agreed that  they retained the right to select investments themselves for review.However, in the case of the Healthcare.gov initiative, OMB did not do so  although the IT Dashboard indicated problems 7 months prior to the initial  launch of Healthcare.gov and its supporting systems.

We reported in 2011 that the Federal IT Dashboard has enhanced OMB’s  oversight of federal IT investments. Among other things, we noted that  performance data from the dashboard were being used to identify poorly  performing investments for executive leadership review sessions.  However, in taking steps to oversee the management of the  Healthcare.gov IT investment, OMB did not effectively use information  provided by this mechanism to analyze, track, and evaluate the risks of  this major investment.

OMB Took Additional Steps to Provide Oversight by Establishing the  U.S. Digital Service  Shortly after initial system launch on October 1, 2013, OMB, along with  the Federal CIO, assisted HHS and CMS with addressing the technical  issues that existed with Healthcare.gov and its supporting systems.  Officials in the Office of E-Government and Information Technology  stated that after technical issues were reported during initial launch of the  system, the role of the Federal CIO was primarily to explore ways to  improve the customer experience with the website.

In addition, in August 2014, the administration established the U.S. Digital  Service, in part to respond to issues with Healthcare.gov and its  supporting systems. This service is to collaborate with federal agencies to  identify and correct problems with government websites, among other  things. OMB’s Deputy Federal CIO serves as the Administrator of the  U.S. Digital Service. The mission of this service is to improve and simplify  the online experience that people and businesses have with the federal  government by establishing standards to bring the government’s digital services in  line with the best private sector services;  identifying common technology patterns that will help effectively scale  services;  collaborating with federal agencies to identify and address gaps in  their capacity to design, develop, deploy and operate public-facing  services; and providing accountability to ensure agencies see results.

According to OMB officials in the Office of E-Government and Information  Technology, the service is working closely with the CMS systems team  charged with developing systems supporting Healthcare.gov. For  example, in August 2014, the administration, in conjunction with the U.S.  Digital Service, released a set of best practices for effective digital service  delivery that are intended to serve as a guide for CMS in further  improving systems supporting Healthcare.gov. CMS is working with the  service to implement these practices.

In addition to its role in assisting CMS with improving the Healthcare.gov  initiative through the U.S. Digital Service, OMB’s Office of E-Government  and Information Technology continues its role in working with HHS and  CMS to oversee the project’s budget. Additionally, the Consolidated and  Further Continuing Appropriations Act, 2015 provides for funding to support the Digital Service’s enhanced oversight and guidance for major  IT investments.

section_title: Conclusions
paragraphs: Problems related to insufficient capacity planning, coding errors, and  incomplete implementation of planned functionality resulted in numerous  performance issues with Healthcare.gov and its supporting systems upon  initial launch in October 2013. Consequently, individuals faced significant  challenges when attempting to enroll for health insurance coverage. CMS  has addressed many of the initial problems by increasing capacity and  taking steps to reduce software code errors. Moreover, the agency has  been developing additional functionality for the FFM system.

Nevertheless, many of the issues arose from the inadequate  implementation of key practices for managing IT projects, and these  weaknesses had not yet been fully corrected. Specifically, by not  managing requirements to ensure that they addressed all needed  functionality and not fully documenting and executing key testing  activities, CMS did not have reasonable assurance that Healthcare.gov  and its supporting systems would perform as intended. In addition,  because it did not develop reliable project schedules, measure levels of  effort, effectively manage project data, and conduct progress and  milestone reviews, CMS had diminished visibility into the project’s status  and may have missed opportunities to take corrective actions and avoid  problems that occurred upon launch.

With the issuance of a new development contract for the FFM system,  CMS has taken the opportunity to make improvements in several of these  areas. However, until it ensures that it is fully implementing these best  practices for managing the development of Healthcare.gov and its  supporting systems, it increases the risk that future development will  experience additional problems.

Further, opportunities exist for HHS to strengthen the involvement of the  department’s CIO in conducting oversight of the management of  Healthcare.gov and its supporting systems. Until HHS does so it cannot  be assured that the implementation and ongoing operation of this high- risk IT investment will continue to provide adequate and sufficient support  to millions of Americans seeking to enroll in health care plans through the  federally facilitated marketplace.

While we previously made recommendations to OMB addressing the use  of dashboard ratings for overseeing IT projects’ performance, we found  that OMB had a limited role in overseeing the management of the  Healthcare.gov IT investment, along with investments in the website’s  supporting systems.

section_title: Recommendations for Executive Action
paragraphs: To improve requirements management for future development covering  systems supporting Healthcare.gov, we recommend that the Secretary of  Health and Human Services direct the Administrator of the Centers for  Medicare & Medicaid Services to direct the Chief Information Officer to  take the following two actions:  1.  Document the approval process for functional and technical design  requirements documentation.  2.  Implement the CMS procedure to obtain signatures from the three key  stakeholders—the CMS business owner, the CMS approval authority,  and the contractor organization approving authority—to ensure that  stakeholders have a shared understanding of all business, functional,  and technical requirements for systems supporting Healthcare.gov  prior to developing them.

To improve systems testing processes for future development covering  systems supporting Healthcare.gov, we recommend that the Secretary of  Health and Human Services direct the Administrator of the Centers for  Medicare & Medicaid Services to direct the Chief Information Officer to  take the following three actions:  3.  Document and approve systems testing policy and procedures,  including (1) the use of the system testing tool designed to integrate  systems development and systems testing and (2) requirements for  stakeholder review of systems test documentation that is intended to  ensure proper test coverage and to validate the results.  4.  Require key information in system test plans, as recommended by  best practices, including the means by which the quality of testing  processes will be assured, and the identification of responsibilities for  individuals or groups carrying out testing.  5.  Require and ensure key information is included in test cases, as  recommended by best practices, such as all outputs and exact values;  test case dependencies; inputs required to execute each test case;  and information about whether each test item has passed or failed  testing.

To improve oversight processes for systems development activities  related to systems supporting Healthcare.gov, we recommend that the  Secretary of Health and Human Services direct the Administrator of the  Centers for Medicare & Medicaid Services to direct the Chief Information  Officer to take the following two actions:  6.  Ensure schedules for the Healthcare.gov effort are well constructed  by, among other things, (1) logically sequencing activities,  (2) confirming the critical paths are valid, and (3) identifying  reasonable total float.  7.  Develop and implement policy and procedures for estimating level of  effort to ensure effort is estimated at the appropriate level  (requirements or program area), and define how levels of effort will be  used to monitor system development progress.

To improve oversight for Healthcare.gov and its supporting systems, we  recommend that the Secretary of Health and Human Services direct the  HHS Chief Information Officer to carry out authorized oversight  responsibilities. Specifically, the Chief Information Officer should ensure  the department-wide investment review board is active and carrying out  responsibilities for overseeing the performance of high-risk IT investments  such as those related to Healthcare.gov.

section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of our report (reprinted in appendix II),  HHS stated that it concurred with all of the recommendations and  identified actions being taken or planned to implement them. Among  others, these actions include instituting a process to ensure functional  and technical requirements are approved, developing and implementing a  unified standard set of approved system testing documents and policies,  and providing oversight for Healthcare.gov and its supporting systems  through the department-wide investment review board. If the department  ensures that these and other actions it identified are effectively  implemented, then CMS should be better positioned to more effectively  manage current and future systems development efforts for  Healthcare.gov and its supporting systems.

In addition, the HHS audit liaison provided technical comments from CMS  via e-mail. In the comments, CMS disagreed with our characterization of  the 11,000 FFM critical code violations that were identified by the IV&V  assessment team in July 2014. CMS stated that these code violations  were identified very early on in the development phase of building the  eligibility and enrollment module and that most of the risk represented by  these code violations is to the cost of maintaining the code over time,  rather than to its successful functionality. The agency added that any  defects which could cause problems with the functionality of the  Healthcare.gov system would have been identified and addressed during  subsequent testing. However, the IV&V assessment stated that the  review was based on a “snapshot of the production code” and not code  that was in development. In addition, while the assessment team noted  that 328 of the code violations may result in maintainability issues, the  team stated that the remaining violations could cause issues in production  if not corrected. Other technical comments provided by HHS were  incorporated as appropriate.

The Chief of Policy, Budget, and Communications within OMB’s Office of  E-Government & Information Technology also provided technical  comments via e-mail. In the comments, OMB took issue with our  statement that it did not conduct a TechStat review when the IT  Dashboard indicated problems 7 months prior to the initial launch of  Healthcare.gov and its supporting systems. According to the OMB official,  a brief dip in the risk rating, such as the one experienced in March 2013,  did not necessitate a formal TechStat. The official further stated that the  tech surge that OMB instituted shortly after the launch of the system,  which included an assessment of its problems, effectively represented a  large-scale and comprehensive TechStat session and replaced the need  for a separate OMB- or agency-led review. Nevertheless, had such an  assessment or a TechStat been conducted earlier in the system  development process, the results could have been used to identify and  correct deficiencies prior to system launch.

We are sending copies of this report to the Secretary of Health and  Human Services, the Director of the Office of Management and Budget,  and other interested parties. In addition, the report is available at no  charge on the GAO website at http://www.gao.gov.

Should you or your staffs have questions on matters discussed in this  report, please contact me at (202) 512-6304. I can also be reached by e- mail at melvinv@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  GAO staff who made major contributions to this report are listed in  appendix II.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this study were to (1) describe the problems  encountered in developing and deploying Healthcare.gov and its  supporting systems, and determine the status in addressing these  deficiencies; and (2) determine the extent to which the Centers for  Medicare & Medicaid Services (CMS) oversaw the development effort  and applied disciplined systems development practices to manage  requirements and conduct systems testing, as well as the extent to which  the Department of Health and Human Services (HHS) and the Office of  Management and Budget (OMB) provided oversight of the effort.

Technical direction letters provide supplementary guidance to contractors regarding tasks  contained in their statements of work or change requests.

Software Engineering Institute, CMMI for Development, Version 1.3, CMU/SEI-2010-TR- 033 (November 2010, Hanscom AFB, MA). The Software Engineering Institute is a  federally funded research and development center operated by Carnegie Mellon  University. Its mission is to advance software engineering and related disciplines to  ensure the development and operation of systems with predictable and improved cost,  schedule, and quality.

CALT is CMS’s project management system and requirements repository.

The Requirements Management Plan states that requirements should be approved by an  official within the Office of Information Services, but that this function can be delegated to  other CMS responsible officials.

To determine whether requirements maintained bidirectional  traceability, we analyzed data extracts of all DSH and eligibility and  enrollment module functional requirements from CALT and  interdependencies between higher-level and lower-level requirements.  We also analyzed requirements documentation developed under the  new systems development contract to identify CMS’s current process  for maintaining bidirectional traceability. In addition, we interviewed  CMS officials as well as DSH system development contractors to  obtain an understanding of the requirements management processes,  including a live demonstration.

With respect to systems testing, we reviewed the CMS testing framework,  contract statements of work for the DSH and FFM systems, independent  verification and validation reports from September 2012 to July 2014, and  system test documentation for these systems. We focused our review on  the extent to which CMS applied selected key best practices for software  and system (1) test plans and (2) test cases.

We assessed all 14 FFM and 19 DSH system test plans documented  prior to system launch in October 2013 against best practices  identified by the Institute of Electrical and Electronics Engineers that  describe key elements that should be included in test plans. In  addition, we assessed the 11 FFM system test plans CMS had  documented after the new development contract to determine the  extent to which these test plans included the key elements identified  in best practices.

We also assessed DSH and FFM system test cases against best  practices identified by the Institute of Electrical and Electronics  Engineers that describe key elements that should be included in test  cases. In doing so, we analyzed and evaluated all DSH system test  cases provided from CMS and documented prior to system launch in  October 2013. In addition, we reviewed a non-generalizable random  sample of 83 test cases for the FFM system from a population of 585  test cases provided from CMS and documented prior to system  launch in October 2013. To determine the extent to which CMS  included key elements in test cases developed after the new FFM  systems development contract, we reviewed a non-generalizable  random sample of 83 test cases from a population of 388. Lastly, we  interviewed CMS officials, as well as DSH and FFM system testing  contractors, to obtain an understanding of the system testing process.

To determine the extent to which CMS, HHS, and OMB oversaw the  systems development effort, we obtained and analyzed documentation,  such as project schedules, the CMS eXpedited Life Cycle policy, the HHS  Enterprise Performance Life Cycle, as well as technical review board  presentations and summary letters. We also reviewed project  management documentation in CMS’s CALT system. Lastly, we reviewed  pertinent oversight laws such as the Clinger-Cohen Act of 1996 and key  practices for providing investment oversight that are outlined in GAO’s IT  investment management framework.

In evaluating the effectiveness of oversight, we focused on (1) project  schedules, (2) level-of-effort estimates, (3) data management, and  (4) progress and milestone reviews.

To determine whether reliable schedules were available to assist with  project oversight, we reviewed and analyzed four key subproject  schedules for the FFM system, since these subprojects were a major  focus of 2014 systems development efforts. Three of the schedules  relate to the Plan Management, Small Business Health Options  Program, and Financial Management modules of the FFM system,  which were planned for initial open enrollment, but had been  postponed in August 2013. The fourth schedule related to the  eligibility and enrollment module of the FFM system, which is for  enrolling individuals for health care coverage. We evaluated the  extent to which these schedules were well-constructed as defined in  our Schedule Assessment Guide. Our methodology to determine the  extent to which project schedules were well-constructed included five  levels of compliance. “Fully met” means the program office provided  complete evidence that satisfied the elements of the best practice.  “Substantially met” means the program office provided evidence that  satisfied a large portion of the elements of the best practice. “Partially  met” means the program office provided evidence that satisfied about  half of the elements of the best practice. “Minimally met” means the  program office provided evidence that satisfied a small portion of the  elements of the best practice. “Not met” means the program office  provided no evidence that satisfied any of the elements of the best  practice.

To determine the extent to which CMS monitored the project against  levels of effort, we reviewed the CMS Requirements Management  Plan dated August 2012, and analyzed and evaluated, against the  plan, levels of effort documented in the CALT system for all DSH and  FFM eligibility and enrollment module functional requirements. For  functional requirements developed after the new FFM contract was  awarded, we interviewed CMS officials and obtained documentation  regarding their efforts in estimating levels of effort for new  development.

To determine the extent to which CMS monitored data management  activities, we reviewed CMS plans and procedures, such as Project  Management Plans and the Requirements Management Plan, for  managing key project files and functional requirements, and evaluated  the extent to which they adhered to CMS plans and procedures within  the CALT system. In addition, we reviewed all DSH and FFM eligibility  and enrollment module functional requirements contained in CALT to  determine the extent to which CMS and its contractor documented  key information used for overseeing development progress, such as  requirements status fields.

To determine whether progress and milestone reviews were  conducted in accordance with CMS and HHS policy, we reviewed the  eXpedited Life Cycle Process and available project process  agreements, and analyzed and evaluated all documentation  pertaining to CMS’s progress and milestone reviews for its DSH, FFM,  and Enterprise Identity Management systems prior to the October  2013 enrollment. In addition, we reviewed and analyzed progress and  milestone reviews held for FFM software releases that were in  production as of July 2014 and conducted after the new FFM systems  development contract was awarded.

Finally, to determine the extent to which CMS, HHS, and OMB provided  oversight in the development and implementation of Healthcare.gov and  its supporting systems, we interviewed knowledgeable officials, including  the CMS Deputy Chief Information Officer, the HHS Chief Information  Officer, and officials from OMB’s Office of e-Government and Information  Technology.

We also obtained documentation and interviewed officials at the  Department of Defense, the Department of Homeland Security, the  Internal Revenue Service, the Office of Personnel Management, the  Peace Corps, the Social Security Administration, and the Department of  Veterans Affairs to determine the extent of their role in developing and  implementing Healthcare.gov and its supporting systems.

To determine the reliability of the data provided from CMS information  systems, we performed basic steps to ensure the data provided were  valid, and we reviewed relevant information describing these systems.  Specifically, we interviewed knowledgeable agency officials within the  CMS Office of Information Services about these systems and asked  specific questions to understand the controls in place for ensuring the  integrity and reliability of the data contained within them. We did not  assess the reliability of the systems used to maintain these data or the  processes used in extracting the data for our engagement purposed.  Based on the results of these efforts, we found the data to be sufficiently  reliable for our work.

We conducted this performance audit from December 2013 to March  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

section_title: Appendix II: Comments from the Department of Health and Human Services

section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Christie Motley (assistant  director), Teresa Tucker (assistant director), James Ashley, Christopher  Businsky, Juana Collymore, Nicole Jarvis, Kendrick Johnson, Jason Lee,  Jennifer Leotta, Lee McCracken, Thomas Murphy, Constantine  Papanastasiou, Andrew Stavisky, and Christy Tyson made key  contributions to this report.


Question-summary annotations: questions: question: How has CMS made improvements?
answer: As it has undertaken further development, CMS has made improvements in some of these areas, by, for example, establishing new requirements management processes and improving test documentation.
child_questions: question: What weaknesses remain despite such improvements?
answer: However, weaknesses remain in its application of requirements, testing, and oversight practices.

question: How has the HHS's role negatively impacted Healthcare.gov?
answer: In addition, the Department of Health and Human Services (HHS) has not provided adequate oversight of the Healthcare.gov initiative through its Office of the Chief Information Officer.

question: Why did GAO recommend OMB improve oversight of IT projects?
answer: The Office of Management and Budget's (OMB) oversight role was limited, and GAO has previously recommended that it improve oversight of IT projects' performance.

questions: question: What did the Patient Protection and Affordable Care Act require?
answer: The Patient Protection and Affordable Care Act required the establishment of health insurance marketplaces to assist individuals in obtaining health insurance coverage.

question: What responsibility did CMS have?
answer: CMS, a component of HHS, was responsible for establishing a federally facilitated marketplace for states that elected not to establish their own.
child_questions: question: How was this marketplace to function?
answer: This marketplace is supported by an array of IT systems, which are to facilitate enrollment in qualifying health plans.
child_questions: question: How did Healthcare.gov connect to this marketplace?
answer: These include Healthcare.gov, the website that serves as the consumer portal to the marketplace, as well as systems for establishing user accounts, verifying eligibility, and facilitating enrollment.

questions: question: What was GAO asked to do in regards to CMS?
answer: GAO was asked to review CMS's management of the development of IT systems supporting the federal marketplace.
child_questions: question: What were GAO's objectives in reviewing CMS?
answer: Its objectives were to (1) describe problems encountered in developing and deploying systems supporting Healthcare.gov and determine the status of efforts to address deficiencies and (2) determine the extent to which CMS applied disciplined practices for managing and overseeing the development effort, and the extent to which HHS and OMB provided oversight.
child_questions: question: How did GAO fulfill these objectives?
answer: To do this, GAO reviewed program documentation and interviewed relevant CMS and other officials.