Financial Management Systems: Lack of Disciplined Processes Puts Implementation of HHS' Financial System at Risk
Published: 2004-09-23T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In June 2001, the Secretary of HHS directed the department to establish a unified accounting system that, when fully implemented, would replace five outdated accounting systems. GAO was asked to review HHS' ongoing effort to develop and implement the Unified Financial Management System (UFMS) and to focus on whether the agency has (1) effectively implemented disciplined processes; (2) implemented effective information technology (IT) investment management, enterprise architecture, and information security management; and (3) taken actions to ensure that the agency has the human capital needed to successfully design, implement, and operate UFMS.
section_title: What GAO Found
paragraphs: HHS has not followed key disciplined processes necessary to reduce the risks associated with implementing UFMS to acceptable levels. While development of a core financial system can never be risk free, effective implementation of disciplined processes can reduce those risks to acceptable levels. The problems that have been identified in such key areas as requirements management, including developing a concept of operations, testing, data conversion, systems interfaces, and risk management, compounded by incomplete IT management practices, information security weaknesses, and problematic human capital practices, significantly increase the risks that UFMS will not fully meet one or more of its cost, schedule, and performance objectives. With initial deployment of UFMS at the Centers for Disease Control and Prevention (CDC) scheduled for October 2004, HHS has not developed sufficient quantitative measures for determining the impact of the many process weaknesses identified by GAO and others to evaluate its project efforts. Without well-defined requirements that are traceable from origin to implementation, HHS cannot be assured that the system will provide the functionality needed and that testing will identify significant defects in a timely manner prior to rollout when they are less costly to correct. The agency has not developed the necessary framework for testing requirements, and its schedule leaves little time for correcting process weaknesses and identified defects. HHS has focused on meeting its predetermined milestones in the project schedule to the detriment of disciplined processes. If HHS continues on this path, it risks not achieving its goal of a common accounting system that produces data for management decision making and financial reporting and risks perpetuating its long-standing accounting system weaknesses with substantial workarounds to address needed capabilities that have not been built into the system. Accordingly, GAO believes these issues need to be addressed prior to deployment at CDC. Beyond the risks associated with this specific system development, HHS has departmental weaknesses in IT investment management, enterprise architecture, and information security. Because of the risks related to operating UFMS in an environment with flawed information security controls, HHS needs to take action to ensure that UFMS benefits from strong information security controls. HHS is modifying its IT investment management policies, developing an enterprise architecture, and responding to security weaknesses with several ongoing activities, but substantial progress in these areas is needed to prevent increased risks to cost, schedule, and performance objectives for UFMS. In human capital, many positions were not filled as planned and strategic workforce planning was not timely. HHS has taken the first steps to address these issues; however, ongoing staff shortages have played a role in several key deliverables being significantly behind schedule.

section_title: Letter
paragraphs: The ability to produce the information needed to efficiently and effectively  manage the day-to-day operations of the federal government and provide  accountability to taxpayers and the Congress has been a long-standing  challenge for federal agencies. To address some of these problems, many  agencies are in the process of replacing their core financial systems as part  of their financial management system improvement efforts. Although the  implementation of any major system is not a risk-free proposition,  organizations that follow and effectively implement accepted best  practices in systems development and implementation (commonly referred  to as disciplined processes) can reduce these risks to acceptable levels.   The use of the term acceptable levels acknowledges the fact that any  systems acquisition has risks and will suffer the adverse consequences  associated with defects.  However, effective implementation of the  disciplined processes reduces the potential for risks to occur and helps  prevent those that do occur from having any significant adverse impact on  the cost, timeliness, and performance of the project.
Because of the importance of these financial management system  improvement efforts and your question as to whether agencies are  employing disciplined processes in implementing new systems, you asked  us to evaluate the current plans for implementing financial management  systems at the Chief Financial Officer Act (CFO) agencies. As agreed with  your offices, we initiated our review at the Department of Health and  Human Services (HHS). HHS has undertaken a multiyear effort to  implement its Unified Financial Management System (UFMS), a new core  financial system, to help HHS management monitor budgets, conduct  operations, evaluate program performance, and make financial and  programmatic decisions.  As a core financial system, UFMS will interface  with an estimated 110 other HHS information systems.  HHS envisions the  eventual UFMS as a departmentwide system that will include core financial  systems currently under development at the National Institutes of Health  (NIH), the Centers for Medicare and Medicaid Services (CMS), along with  an integrated system for the Centers for Disease Control and Prevention  (CDC), the Food and Drug Administration (FDA), and the Program Support  Center (PSC), which provides accounting support for the remaining HHS  organizations.
This report provides our assessment of HHS’ ongoing effort to develop and  implement the integrated UFMS at CDC, FDA, and PSC, and focuses on  whether the agency has (1) effectively implemented key disciplined  processes in the development of UFMS to provide reasonable assurance  that UFMS meets its cost, schedule, and performance goals;   (2) implemented effective investment management, enterprise  architecture, and security management to support UFMS efforts; and   (3) taken actions to ensure that HHS has the human capital needed to  successfully design, implement, and operate UFMS.
To achieve these objectives, we reviewed documentation related to the  project and interviewed HHS officials and contractors used by HHS to  assist with implementation. We used relevant government and industry  standards, such as those from the Software Engineering Institute (SEI) and  the Institute of Electrical and Electronics Engineers (IEEE), along with key  best practice guides such as our Executive Guide: Creating Value Through  World-class Financial Management, to assess the status of HHS’  implementation of disciplined processes. This report does not assess HHS’  other financial management improvement efforts at NIH and CMS.  We  conducted our work in Washington, D.C., Rockville, Maryland, and Atlanta,  Georgia, from September 2003 through May 2004 in accordance with U.S.  generally accepted government auditing standards.  More details on our  scope and methodology can be found in appendix I.
subsections: section_title: Background
paragraphs: HHS is the federal government's principal agency for protecting the health  of Americans and provides essential human services, such as ensuring food  and drug safety and assisting needy families. HHS disburses almost a  quarter of all federal outlays and administers more grant dollars than all  other federal agencies combined, providing more than $200 billion of over  $350 billion in federal funds awarded to states and other entities in fiscal  year 2002, the most recent year for which these data are available. For  fiscal year 2004, HHS had a budget of $548 billion and over 66,000  employees. HHS comprises 11 agencies led by the Office of the Secretary  covering a wide range of activities including conducting and sponsoring  medical and social science research, guarding against the outbreak of  infectious diseases, assuring the safety of food and drugs, and providing  health care services and insurance.
HHS is required by the CFO Act of 1990 to modernize its financial  management systems and by the Federal Financial Management  Improvement Act (FFMIA) of 1996 to have auditors—as part of an audit  report on the agency’s annual financial statements—determine whether the  agency’s financial management systems comply substantially with three  requirements: (1) federal financial management systems requirements,   (2) applicable federal accounting standards, and (3) the U.S. Government  Standard General Ledger (SGL) at the transaction level.
While HHS has received unqualified opinions on its financial statements at  the consolidated departmental level since fiscal year 1999, the underlying  financial systems that assist in the preparation of financial statements have  not met all applicable requirements. For fiscal years 1997 through 2003,  HHS auditors reported that the department’s systems did not substantially  comply with federal financial management systems requirements, and for  fiscal year 2003, they reported that the systems also lacked compliance  with the SGL requirement. In describing the financial management  problems in the fiscal year 2003 financial statement audit report, the HHS  Inspector General (IG) stated that the department’s lack of an integrated  financial system and internal control weaknesses made it difficult for HHS  to prepare timely and reliable financial statements. The IG also noted that  preparation of HHS financial statements required substantial “work  arounds,” cumbersome reconciliations and consolidation processes, and  significant adjustments to reconcile subsidiary records to reported  balances on the financial statements.
subsections: section_title: HHS’ Financial System Implementation Effort
paragraphs: In June 2001, the Secretary of HHS directed the department to establish a  unified accounting system that, when fully implemented, would replace  five outdated accounting systems. HHS considers the UFMS program a  business transformation effort with IT, business process improvement, and  operations consolidation components. According to HHS, the program  supports the Office of Management and Budget’s (OMB) requirements for  each agency to implement and operate a single, integrated financial  management system (required by OMB Circular No. A-127). HHS asserts  that its approach will require it to institute a common set of business rules,  data standards, and accounting policies and procedures, thereby  significantly furthering the Secretary’s management objectives. Table 1  depicts the current accounting systems that will be replaced and the  organizations currently served.
In response to the Secretary’s direction, HHS began a project to improve its  financial management operations.  CMS and NIH had already initiated  projects to replace their financial systems.  Figure 1 illustrates the systems  being replaced, the new configuration, and the approximate known  implementation costs.
As shown in figure 1, HHS plans to pursue a phased approach to achieving  the Secretary’s vision. The first phase is to implement the system at CDC  and, as of May 2004, CDC was expected to begin using the system for its  operations starting in fiscal year 2005 (October 2004).  FDA was expected  to implement UFMS in May 2005, and the entities served by PSC were to be  phased in from July 2005 through April 2007. After all of the individual  component agency implementations have been completed, UFMS and HHS  consolidated reporting will be deployed.  This effort involves automating  the department’s financial reporting capabilities and is expected to  integrate the NIH Business and Research Support System (NBRSS) and  CMS’ Healthcare Integrated General Ledger Accounting System (HIGLAS)  into UFMS, which are scheduled to be fully implemented in 2006 and 2007,  respectively.  The focus of our review was on the system implementation  efforts associated with the HHS entities not covered by the NBRSS and  HIGLAS efforts.
As shown in figure 1, the costs for this financial management system  improvement effort can be broken down into four broad areas:  NIH, CMS,  all other HHS entities, and a system to consolidate the results of HHS’  financial management operations.  HHS estimates that it will spend about  $713 million as follows:  $110 million for its NIH efforts (NBRSS),  $393 million to implement HIGLAS, and   $210 million for remaining HHS organizations.
HHS has not yet developed an estimate of the costs associated with  integrating these efforts into the HHS unified financial management system  envisioned in Secretary Thompson’s June 2001 directive.
HHS selected a commercial-off-the-shelf (COTS) product, Oracle U.S.  Federal Financials software (certified by the Program Management Office  of the Joint Financial Management Improvement Program (JFMIP) for  federal agencies’ use), as the system it would use to design and implement  UFMS. The department has hired two primary contractors to help  implement UFMS. In November 2001, HHS awarded KPMG Consulting  (now BearingPoint) a contract as system integrator for assistance in  planning, designing, and implementing UFMS. As the systems integrator,  BearingPoint is expected to provide team members, who are experienced  in the enterprise resource planning (ERP) software and its installation,  configuration, and customization, with expertise in software, hardware,  business systems architecture, and business process and transformation.  HHS selected Titan Corporation to act as the project’s independent  verification and validation (IV&V) contractor, tasked with determining the  programmatic, management, and technical status of the UFMS project and  recommending actions to mitigate any identified risks to project success.
When fully implemented, UFMS is expected to permit the consolidation of  financial data across all HHS component agencies to support timely and  reliable departmentwide financial reporting. In addition, it is intended to  integrate financial information from the department’s administrative  systems, including travel management systems, property systems, logistics  systems, acquisition and contracting systems, and grant management  systems. The department’s goals in the development and implementation of  this integrated system are to achieve greater economies of scale; eliminate  duplication; provide better service delivery; and help management monitor  budgets, conduct operations, evaluate program performance, and make  financial and programmatic decisions.
section_title: HHS Has Not Effectively Implemented the Disciplined Processes Necessary to Reduce UFMS Program Risks to Acceptable Levels
paragraphs: Experience has shown that organizations that adopt and effectively  implement best practices, referred to in systems development and  implementation efforts as the disciplined processes, can reduce the risks  associated with these projects to acceptable levels.  Although HHS has  adopted some of the best practices associated with managing projects such  as UFMS, it has adopted other practices that significantly increase the risk  to the project.  Also, HHS has not yet effectively implemented several of the  disciplined processes—requirements management, testing, project  management and oversight, and risk management—necessary to reduce its  risks to acceptable levels and has exposed the project to unnecessary risk  that it will not achieve its cost, schedule, and performance objectives.
The project has been able to obtain high-level sponsorship at HHS with  senior financial management and HHS personnel routinely reviewing its  progress. HHS officials maintain that the project is on schedule and that the  functionality expected to be available for its first deployment, at CDC in  October 2004, is well known and acceptable to its users.  However, the  IV&V contractor identified a number of serious deficiencies that are likely  to affect HHS’ ability to successfully implement UFMS within its current  budget and schedule while providing the functionality needed to achieve its  goals. HHS management has been slow to take the recommended  corrective actions necessary to address the findings and recommendations  of its IV&V contractor. Further, it is not clear that the decision to proceed  from one project milestone to the next is based on quantitative data that  indicate tasks have been effectively completed.  Rather, decisions to  progress have been driven by the project’s schedule.  With a focus on  meeting schedule milestones and without quantitative data, HHS faces  significant risk that UFMS will suffer the adverse impacts on its cost,  schedule, and performance that have been experienced by projects with  similar problems.
subsections: section_title: Effective Implementation of the Disciplined Processes Are Key to Reducing Project Risks
paragraphs: Disciplined processes, which are fundamental to successful systems  development and implementation efforts, have been shown to reduce to  acceptable levels the risks associated with software development and  acquisition. A disciplined software development and acquisition process  can maximize the likelihood of achieving the intended results  (performance) within established resources (costs) on schedule. Although  there is no standard set of practices that will ever guarantee success,  several organizations, such as SEI and IEEE, as well as individual  experts, have identified and developed the types of policies, procedures,  and practices that have been demonstrated to reduce development time  and enhance effectiveness. The key to having a disciplined system  development effort is to have disciplined processes in multiple areas,  including project planning and management, requirements management,  configuration management, risk management, quality assurance, and  testing. Effective processes should be implemented in each of these areas  throughout the project life cycle because change is constant.  Effectively  implementing the disciplined processes necessary to reduce project risks  to acceptable levels is hard to achieve because a project must effectively  implement several best practices, and inadequate implementation of any  one may significantly reduce or even eliminate the positive benefits of the  others.
Acquiring and implementing a new financial management system requires a  methodology that starts with a clear definition of the organization's mission  and strategic objectives and ends with a system that meets specific  information needs.  We have seen many system efforts fail because  agencies started with a general need, such as improving financial  management, but did not define in precise terms (1) the specific problems  they were trying to solve, (2) what their operational needs were, and   (3) what specific information requirements flowed from these operational  needs.  Instead, they plunged into the acquisition and implementation  process in the belief that these specifics would somehow be defined along  the way.  The typical result was that systems were delivered well past  anticipated milestones; failed to perform as expected; and, accordingly,  were overbudget because of required costly modifications.
Figure 2 shows how organizations that do not effectively implement the  disciplined processes lose the productive benefits of their efforts as a  project continues through its development and implementation cycle.   Although undisciplined projects show a great deal of productive work at  the beginning of the project, the rework associated with defects begins to  consume more and more resources.  In response, processes are adopted in  the hopes of managing what later turns out, in reality, to have been  unproductive work.  Generally, these processes are “too little, too late” and  rework begins to consume more and more resources because sufficient  foundations for building the systems were not done or not done adequately.   Experience has shown that projects for which disciplined processes are  not implemented at the beginning are forced to implement them later when  it takes more time and they are less effective.
As shown in figure 2, a major consumer of project resources in  undisciplined efforts is rework (also known as thrashing).  Rework occurs  when the original work has defects or is no longer needed because of  changes in project direction.  Disciplined organizations focus their efforts  on reducing the amount of rework because it is expensive.  Fixing a defect  during the testing phase costs anywhere from 10 to 100 times the cost of  fixing it during the design or requirements phase.  As shown in figure 2,  projects that are unable to successfully address their rework will  eventually only be spending their efforts on rework and the associated  processes rather than on productive work.  In other words, the project will  continually find itself reworking items.  Appendix II provides additional  information on the disciplined processes.
section_title: HHS Has Not Effectively Implemented Key Processes Necessary to Reduce Risks to Acceptable Levels
paragraphs: We found that HHS has not implemented effective disciplined processes in  several key process areas that have been shown to form the foundation for  project success or failure including requirements management, testing,  project management and oversight, and risk management.  Problems with  HHS’ requirements management practices include the lack of  (1) a concept  of operations to guide the development of requirements, (2) traceability of  a requirement from the concept of operations through testing to ensure  requirements were adequately addressed in the system, and (3) specificity  in the requirements to minimize confusion in the implementation.  These  problems with requirements have resulted in a questionable foundation for  the systems’ testing process.  In addition, HHS has provided an extremely  limited amount of time to address defects identified from system testing,  which reflects an optimism not supported by other HHS testing efforts,  including those performed to test the conversion of data from CDC’s legacy  system to UFMS.  This type of short time frame generally indicates that a  project is being driven to meet predetermined milestones in the project  schedule.  While adherence to schedule goals is generally desirable, if  corners are cut and there is not adequate quantitative data to assess the  risks to the project of not implementing disciplined processes in these  areas, the risk of project rework or failure appreciably rises.  Ineffective  implementation of these processes exposes a project to the unnecessary  risk that costly rework will be required, which in turn will adversely affect  the project’s cost and schedule, and can adversely affect the ultimate  performance of the system.
An effective risk management process can be used by an agency to  understand the risks that it is undertaking when it does not implement an  effective requirements management process.  In contrast, HHS has  implemented risk management procedures that close risks before it is clear  that mitigating actions were effective.  HHS has agreed to change these  procedures so that the actions needed to address risks remain visible and  at the forefront.  While the executive sponsor for the UFMS project and  other senior HHS officials have demonstrated commitment to the project,  effective project management and oversight are needed to identify and  resolve problems as soon as possible, when it is the cheapest to fix them.   For example, HHS officials have struggled to address problems identified  by the IV&V contractor in a timely manner.  Moreover, HHS officials lack  the quantitative data or metrics to effectively oversee the project.  An  effective project management and oversight process uses such data to  understand matters such as (1) whether the project plan needs to be  adjusted and (2) oversight actions that may be needed to ensure that the  project meets its stated goals and complies with agency guidance.
Whereas, with ineffective project oversight, management can only respond  to problems as they arise.
subsections: section_title: HHS’ Requirements Management Process Is Ineffective
paragraphs: We found significant problems in HHS’ requirements management process.   (See appendix III for a more detailed discussion.)  We found that HHS had  not (1) developed a concept of operations that can be used to guide its  requirements development process, (2) maintained traceability between  the various requirements documents to ensure consistency, and   (3) developed requirements that were unambiguous.  Because of these  weaknesses, HHS does not have reasonable assurance that the UFMS  project is free of significant requirement defects that will cause significant  rework.
Requirements are the specifications that system developers and program  managers use to design, develop, and acquire a system. They need to be  unambiguous, consistent with one another, verifiable, and directly  traceable to higher-level business or functional requirements. It is critical  that requirements flow directly from the organization’s concept of  operations, which describes how the organization’s day-to-day operations  (1) are being carried out and (2) will be carried out to meet mission needs.   Examples of problems noted in our review include the following.
Requirements were not based on a concept of operations.  HHS has  prepared a number of documents that discuss various aspects of its  vision for UFMS.  However, these documents do not accomplish the  principal objective associated with developing a concept of  operations—specifying the high-level business processes that are  expected to form the basis for requirements definition. One such  document, issued April 30, 2004, discusses the use of shared service centers to perform financial management functions.  This document  was issued well after implementation efforts were under way and about  5 months before the expected deployment date of UFMS at CDC.  As  discussed in more detail in appendix III, the April 30 document does not  clearly explain who will perform these functions, and where and how  these functions will be performed.
Requirements were not traceable.  HHS developed a hierarchical  approach to defining its requirements.  HHS defined the high-level  requirements that were used to identify the requirements that could not  be satisfied by the COTS product.  Once these high-level requirements  were defined, a hierarchical requirements management process was  developed which included (1) reviewing and updating the requirements  through process design workshops,(2) establishing the initial baseline  requirements, (3) performing a fit/gap analysis, (4) developing gap  closure alternatives, and (5) creating the final baseline requirements.  The key in using such a hierarchy is that each step of the process builds  upon the previous step.   However, this traceability was not maintained  for the 74 requirements we reviewed.  Therefore, HHS has little  assurance that (1) requirements defined in the lower-level requirements  documents are consistent with and adequately cover the higher-level  requirements and (2) testing efforts based on lower-level requirements  documents will adequately assess whether UFMS can meet the high- level requirements used to define the overall functionality expected  from UFMS.  Appendix III provides more details on problems we  identified related to the traceability of requirements.
Requirements were not always specific.  Many requirements reviewed  were not sufficiently specific to reduce requirements-related defects to  acceptable levels.  For example, one inadequately defined requirement  stated that the system “shall track actual amounts and verify  commitments and obligations against the budget as revised, consistent  with each budget distribution level.” The “Define Budget Distributions”  process area was expected to provide the additional specificity needed  for this requirement. However, as of May 2004, this process document  stated that the functionality was “To Be Determined.” Until HHS  provides additional information concerning this requirement, it will not  be able to determine whether the system can meet the requirement.  Items that will need to be defined include the number of budget  distribution levels that must be supported and what it means to verify  the commitments and obligations against the revised budget.  Appendix  III includes more details on the problems related to the specificity of  HHS’ requirements.
HHS officials plan to use traditional testing approaches, including  demonstrations and validations, to show UFMS’ compliance with HHS  high-level requirements as well as the requirements contained in the  various other requirements documents. However, the effectiveness of the  testing process is directly related to the effectiveness of the requirements  management process.  HHS’ IV&V contractor reported that as of April 2004,  the UFMS test program had not been adequately planned to provide the  foundation for a comprehensive and coordinated process for validating  that UFMS has the functionality to meet the stated requirements.  For  example, the test planning documents reviewed by the IV&V contractor did  not have the detail typically found in test plans.  As of May 2004, the  information necessary for evaluating future testing efforts had not been  developed for the 44 requirements that we reviewed.  Because of the  weaknesses noted in the requirements management process, HHS does not  yet have a firm foundation on which to base an effective testing program.
section_title: Key Testing Processes Have Not Been Completed
paragraphs: Complete and thorough testing is essential to provide reasonable assurance  that new or modified systems will provide the capabilities in the  requirements.   Testing activities that can provide quantitative data on the  ability of UFMS to meet HHS’ needs are scheduled late in the  implementation cycle.  For example, system testing on the capabilities for  the CDC implementation was planned to start in August 2004 and to be  completed in a 6-week time frame before the system is expected to become  operational there.  This leaves HHS with little time to address any defects  identified during the system testing process and to ensure that the  corrective actions taken to address the defects do not introduce new  defects.  Because HHS has allotted little time for system testing and defect  correction, problems not corrected before system launch will in the worst  case result in system failure, or will have to be addressed during  operations, resulting in potentially costly and time-consuming rework.
Testing is even more challenging for this system development because HHS  had not fully developed its overall requirements traceability matrix before  testing to determine whether testing will address the requirements. HHS is  placing a great deal of reliance on system testing to provide reasonable  assurance of the functionality included in UFMS.  Also, with system testing  scheduled for August, HHS had not, as of May 2004, established an effective  management framework for testing.  For example, HHS had not (1) clearly  defined the roles and responsibilities of the developers and testers,   (2) developed acceptance criteria, and (3) strictly controlled the testing  environment. As the IV&V contractor noted, if testing is not properly  controlled and documented, there is no assurance that the system has been  adequately tested and will perform as expected. Accordingly, HHS will  need to develop such documents prior to conducting testing, such as  developing test cases and executing the actual tests.
Given the issues associated with HHS’ requirements management process,  even if HHS addresses these testing process weaknesses, evaluating UFMS  based solely on testing will not ensure that CDC’s and HHS’ needs will be  met. It is unlikely that the system testing phase will uncover all defects in  the UFMS system. In fact, testing, based on well-defined requirements,  performed through the system test phase, often catches less than 60  percent of a program’s defects.  In HHS’ case, problems with its poorly  defined requirements make creating test cases more challenging and  increase the likelihood that the systems test phase will identify significant  defects that are often identified by system testing. The remaining errors are  found through other quality assurance practices, such as code inspections,  or by end users after the software has been put into production.  Thus, it  will be important for HHS to implement a quality assurance program that is  both rigorous and well-structured.
subsections: section_title: Initial Data Conversion and System Interface Efforts Encountered Problems
paragraphs: The ability of HHS to effectively address its data conversion and system  interface challenges will also be critical to the ultimate success of UFMS.   In its white paper on financial system data conversion, JFMIP identified  data conversion as one of the critical tasks necessary to successfully  implement a new financial system.  Moreover, JFMIP stated that data  conversion is one of the most frequently underestimated tasks.  JFMIP also  noted that if data conversion is done right, the new system has a much  greater opportunity for success. On the other hand, converting data  incorrectly or entering unreliable data from a legacy system has lengthy  and long-term repercussions.  The adage “garbage in garbage out” best  describes the adverse impact. For example, the National Aeronautics and  Space Administration (NASA) cited data conversion problems as a major  reason that it was unable to prepare auditable financial statements from its  new financial management system.  HHS officials had initially expected to  perform only two data conversion testing efforts, but decided that two  additional data conversion testing efforts were needed after identifying 77  issues during the first data conversion test.  While there is no standard  number of data conversion tests that are needed, the key to successfully  converting data from a legacy system to a new system is that the data  conversion test is successfully executed with minimal errors.   In addition,  system interfaces had not been fully developed as expected for the  conference room pilots held in March and April 2004.  Proper  implementation of the interfaces between UFMS and the other systems it  receives data from and sends data to is essential for the successful  deployment of UFMS.
HHS had originally expected to perform two data conversion testing efforts  (commonly referred to as mock conversions) prior to the system being  implemented at CDC.  In discussions with HHS officials, we noted that  other agencies have found that many more mock conversions are required,  but HHS officials told us that the project schedule did not allow for many  more conversion efforts.  However, according to HHS, more than 8 months  of preparatory activities were completed before beginning the first mock  conversion. They also told us that at least some of these data-cleanup  efforts had started about 3 years ago.  As with other efforts on this project,  the quantitative data necessary to determine whether HHS’ expectations  were realistic, such as the number of issues identified during a mock  conversion, were not produced until late in the implementation cycle.  In  May 2004, HHS performed the first of its two planned mock conversions.   On the basis of the results of this effort, HHS has now decided that it will  need to perform two additional mock conversions before the October 2004  implementation at CDC.  As shown in the following examples of the  problems found in the first mock conversion, data cleanup was not  sufficient in at least some cases to support the data conversion efforts.
Employer identification numbers (EIN) assigned to customers caused  problems because adequate data cleanup efforts had not yet been  performed.  For example, multiple customers had the same EIN or an  EIN on the invoice did not have a corresponding customer.  In addition,  over 1,300 vendors lacked the necessary banking information.
Problems related to data quality and conversion logic were found in the  conversions related to general ledger account balances.  A primary  cause of the problems was that the legacy system performed its closing  activities by appropriation while UFMS does it by program.  On the basis  of a review of these problems by the project team, one of the team’s  recommendations was that a substantial data cleanup effort in the  legacy system be started to mitigate the problems identified in this mock  conversion.
Overall, HHS identified 77 issues that applied to 10 of the 11 business  activities covered by this mock conversion. Table 2 shows the types of  actions HHS identified as necessary to address these issues.
At the conclusion of the first mock conversion, the project team believed  that most of the major conversion issues had been identified and that  subsequent data conversion efforts would only identify issues that required  refinements to the solutions developed for the issues already identified.  On  the basis of the results of the first mock conversion, they also agreed to  perform two additional mock conversions.
We also noted similar problems in HHS’ efforts related to system interfaces.   For example, one purpose of the March/April 2004 conference room pilot  was to demonstrate several key system interfaces.  However, a key feature  of system interface efforts—error correction—was not available for  demonstration since it had not yet been developed.  At the conference  room pilot, a user asked about how the error correction process would  work for transactions that were not processed between two systems  correctly and the user was told that the project team had not yet worked  out how errors would be managed.  Until HHS defines and implements this  functionality, it will be unable to ensure that the processes being used for  exchanging data between UFMS and more than 30 CDC systems ensures  the necessary levels of data integrity.  Properly implementing the interfaces  will be critical to performing a realistic system test at CDC and ensuring  UFMS will properly operate when in production.  Also, HHS expects UFMS  to interface with about 110 systems when it is fully implemented.
section_title: HHS Risk Management Process Prematurely Closed Identified Risk As Being Resolved
paragraphs: In our view, a major value of a risk management system is the increased  visibility over the scope of work and resources needed to address the risks.   HHS officials have developed a risk assessment and mitigation strategy and  have implemented a process for managing UFMS risks that meets many of  the risk management best practices.  For example, they cited a program to  identify risks to the project, such as staffing shortages and training  deficiencies, and have HHS management focus on those risks.  Our review  confirmed that HHS does maintain a risk database and that these risks are  available for review and discussion during project oversight meetings.   However, we noted problems with the implementation of the risk  management system.
HHS routinely closed its identified risks on the premise that they had been  identified and were being addressed.  As of March 2004, 13 of the 44 project  risks identified by HHS were considered “closed,” even though it appeared  that actions taken to close the risks were still ongoing.  For example, HHS  had identified data conversion as a risk because the conversion might be  more complex, costly, and time consuming than previously estimated.   However, this risk was closed in February 2003 because a data conversion  strategy was in the project plan that UFMS officials considered as adequate  to mitigate the risk. HHS officials characterized this practice as intended to  streamline the number of risks for discussion at biweekly meetings.   Project officials defended this approach under the premise that if the  mitigating actions were not achieving their desired results, then the risk  would be “reopened.”  After we discussed this with HHS officials, they  agreed to revise their procedures to include a resolution column with more  information on why a risk was closed.  This change should improve  management’s ability to oversee the inventory of risks, their status, and the  effectiveness of the mitigating strategies.
section_title: Project Management Benefits from the Support of Senior Officials, but Corrective Actions Lag
paragraphs: According to HHS, the project has been able to obtain high-level  sponsorship from senior financial management officials who routinely  review its progress.  This sponsorship has enabled the project to gain  support from individuals critical to the implementation of UFMS at  organizational units such as CDC.  In addition, senior management officials  have received periodic reports from a contractor hired to perform  independent verification and validation that help identify issues needing  management attention.  Because of this strong support and oversight, HHS  officials said they believed that the risks associated with the project have  been reduced to acceptable levels and that the project can serve as a  management model.
While we agree that top management commitment and oversight together  comprise one critical factor in determining a project’s success, they are not  in themselves sufficient to provide reasonable assurance of the project’s  success. As noted in our discussion of disciplined processes, the  inadequate implementation of any one of the disciplined processes in  systems development can significantly reduce or overcome the positive  benefits of others.  In this case, it is important to act promptly to address  risks so as to minimize their impact.
In this regard, in February 2003, HHS obtained the services of the current  contractor to perform the IV&V function for the UFMS project.   As of May  2004, according to the contractor, its staff has participated in hundreds of  meetings at all levels within the project, provided written comments and  recommendations on over 120 project documents, and produced 55 project  status and assessment reports.  Twice a month it produces a report that is  sent directly to the Executive Sponsor of the UFMS project.  These reports  highlight the IV&V team’s view on the overall status of the UFMS project,  including a discussion of any impacts or potential impacts to the project  with respect to cost, schedule, and performance and a section on current  IV&V concerns and associated recommendations.  The IV&V contractor  reported several project management and oversight weaknesses that  increase the risks associated with this project that were not promptly  addressed.  Examples include the following.
Personnel.  Although the contractor hired by HHS to perform IV&V  services identified the lack of personnel as a major risk factor in June  2003, it took HHS and its system integrator over 6 months to  substantially address this weakness.   In February 2004, the IV&V  contractor reported this issue as closed.  In closing this issue, the IV&V  contractor noted that the availability of adequate resources was an  ongoing concern, and the issue may be reopened at a later date.  Related  human capital issues are discussed in a separate section of this report.
Critical path analysis. In August 2003, the IV&V contractor noted that  an effective critical path analysis had not been developed. A critical path  defines the series of tasks that must be finished on time for the entire  project to finish on schedule. Each task on the critical path is a critical  task.  As of April 2004, this weakness had not been effectively  addressed.  Until HHS can develop an effective critical path analysis for  this project, it does not have adequate assurance that it can understand  the impact of various project events, such as delays in project  deliverables.  HHS’ critical path report shows planned start and finish  dates for various activities, but does not show the actual progress so  that the impact of schedule slips can be analyzed.  The IV&V contractor  recommended that critical path analysis and discussion become a more  prominent feature of UFMS project management to monitor the  resources assigned to activities that are on the critical path.
Earned value management system. In August 2003, the IV&V  contractor also noted that an effective earned value management  system had not been implemented. Earned value management attempts  to compare the value of work accomplished during a given period with  the work scheduled for that period. By using the value of completed  work as a basis for estimating the cost and time needed to complete the  program, earned value can alert program managers to potential  problems early in the program.  For example, if a task is expected to  take 100 hours to complete and it is 50 percent complete, the earned  value management system would compare the number of hours actually  spent to complete the task to the number of hours expected for the  amount of work performed.  In this example, if the actual hours spent  equaled 50 percent of the hours expected, the earned value would show  that the project’s resources were consistent with the estimate. As of  April 2004, this weakness had not been effectively addressed.  Without  an effective earned value management system, HHS has little assurance  that it knows the status of the various project deliverables in the  context of progress and associated cost.  In other words, an effective  earned value management system would be able to provide quantitative  data on the status of a given project deliverable, such as a data  conversion program.  On the basis of this information, HHS  management would be able to determine whether the progress of a task  was within the expected parameters for completion.  Management  could then use this information to determine actions to take to mitigate  risk and manage cost and schedule performance.
The following additional significant issues were considered open by the  IV&V contractor as of April 2004.
Requirements management. The project had not produced an overall  requirements traceability matrix that identified all the requirements and  the manner in which each will be verified.  In addition, HHS had not  implemented a consistent approach to defining and maintaining a set of  “testable” requirements.
UFMS test program adequacy. The test program for UFMS had not been  adequately defined and the test documentation reviewed to date lacks  the detail typically found in test plans that are developed in accordance  with industry standards and best practices.
UFMS strategy documents. A number of key strategy documents that  provide the foundation for system development and operations had not  been completed as defined in the project schedule.  These documents  are used for guidance in developing documents for articulating the plans  and procedures used to implement UFMS.  Examples of the documents  that were 2 or more months late include the UFMS Business Continuity  Strategy, UFMS Lifecycle Test Strategy, Global Interface Strategy, and  Global Conversion Strategy.
In addition, the IV&V contractor has presented other issues, concerns, and  recommendations in its reports.  For example, a May 2004 report noted that  the IV&V contractor had expressed some concerns on the adequacy of the  project schedule and the status of some data conversion activities.  Our  review of the IV&V contractor’s concerns found that they are consistent  with those that we identified in our review of UFMS.
section_title: HHS Has Not Yet Developed the Quantitative Data Necessary for Assessing Whether the System Will Provide the Needed Functionality
paragraphs: The ability to understand the impact of the weaknesses we and the IV&V  contractor identified is limited because HHS has not effectively captured  the types of quantitative data or metrics that can be used to assess the  effectiveness of its management processes, such as identifying and  quantifying any weaknesses in its requirements management process.  This  information is necessary to understand the risk being assumed and  whether the UFMS project will provide the desired functionality.  HHS does  not have a metrics measurement process that allows it to fully understand  (1) its capability to manage the entire UFMS effort; (2) how its process  problems will affect the UFMS cost, schedule, and performance objectives;  and (3) the corrective actions needed to reduce the risks associated with  the problems identified.  Without such a process, HHS management can  only focus on the project schedule and whether activities have occurred as  planned, not whether the activities achieved their objectives.  Experience  has shown that such an approach leads to rework instead of making real  progress on the project.
SEI has found that metrics identifying important events and trends are  invaluable in guiding software organizations to informed decisions. Key  SEI findings relating to metrics include the following.
The success of any software organization depends on its ability to make  predictions and commitments relative to the products it produces.
Effective measurement processes help software groups succeed by  enabling them to understand their capabilities so that they can develop  achievable plans for producing and delivering products and services.
Measurements enable people to detect trends and to anticipate  problems, thus providing better control of costs, reducing risks,  improving quality, and ensuring that business objectives are achieved.
Defect tracking systems are one means of capturing quantitative data that  can be used to evaluate project efforts.  Although HHS has a system that  captures the defects that have been reported, we found that the agency has  not effectively implemented a process to ensure that defects are identified  and reported as soon as they have been identified.  For example, we noted  in the March/April 2004 conference room pilot that one of the users  identified a process weakness related to grant accounting as a  “showstopper.” However, this weakness did not appear in the defect  tracking system until about 1 month later.  As a result, during this interval,  the HHS defect tracking system did not accurately reflect the potential  problems identified by the users, and HHS management was unable to  determine (1) how well the system was working and (2) the amount of  work necessary to correct the defects.  Such information is critical when  assessing a project’s status.
According to HHS officials at of the end of our fieldwork, the UFMS project  is on schedule. However, while the planned activities may have been  performed, because there are not quantifiable criteria for assessing  progress, it is unclear whether they were performed successfully or  whether the activities have been accomplished substantively.  For example,  one major milestone was to conduct a conference room pilot in  March/April 2004.  HHS held the conference room pilot in March/April 2004,  and so it considered that the milestone had been met.  However, HHS did  not define what constituted success for this event, such as the users  identifying no significant defects in functionality.  A discussion of the  problems we identified with the March/April 2004 conference room pilot is  included in appendix III and clearly demonstrates that the objective of this  activity, to validate the prototype system and test interfaces, was not  achieved.  Therefore, by measuring progress based on the fact that this  conference room pilot was held, HHS has little assurance that the project is  in fact on schedule and can provide the desired functionality.  This  approach increases the risk that HHS will be surprised by a major  malfunction at a critical juncture in the project, such as when it conducts  system testing or attempts to implement the system at CDC.
Good metrics would enable HHS to assess the risk of moving forward on  UFMS with a much greater degree of certainty. HHS will be better able to  proactively manage UFMS through disciplined processes as opposed to  having to respond to problems as they arise.
section_title: Experience Has Shown the Effects of Not Effectively Implementing the Disciplined Processes
paragraphs: HHS’ inability to effectively implement the types of disciplined processes  necessary to reduce risks to acceptable levels does not mean that the  agency cannot put in place an effective process prior to the CDC  implementation. However, HHS has little time to (1) address long-standing  requirements management problems, (2) develop effective test cases from  requirements that have not yet been defined at the level necessary to  support effective testing efforts, and (3) develop and implement disciplined  test management processes before it can begin its testing efforts.   Furthermore, HHS will need to address its project management and  oversight weaknesses so that officials can understand (1) the impact that  the defects identified during system testing will have on the project’s  schedule and (2) the corrective actions needed to reduce the risks  associated with the problems identified.  Without effectively implementing  disciplined processes and the necessary metrics to understand the  effectiveness of the processes that it has implemented, HHS is incurring  unnecessary risks that the project will not meet its cost, schedule, and  performance objectives.
The kinds of problems we saw at HHS for the UFMS project have  historically not boded well for successful system development at other  federal agencies.  In 1999 we reported on a system at the Department of  the Interior’s Bureau of Indian Affairs (BIA) that had problems similar to  those discussed in this report. As is the case at HHS, Interior’s deficiencies  in requirements management and other disciplined processes meant that  Interior had no assurance that its newly acquired system would meet its  specific performance, security, and data management needs and that it  would be delivered on time and on schedule. To reduce these risks, we  recommended that Interior develop and implement an effective risk  management plan and that Interior ensure that all project decisions were  (1) based on objective data and demonstrated project accomplishments  and (2) driven by events, not the schedule.   In subsequent reviews we  noted that, like HHS, Interior planned to use testing to demonstrate that the  system could perform its intended functions.
However, as we reported in September 2000, BIA did not follow sound  practices in conducting its system and user acceptance tests for this  system.  Subsequently, in May 2004, the agency reported that only one  function had been successfully implemented and that it was in the process  of evaluating the capabilities and shortcomings of the system to determine  whether any other components could be salvaged for interim use while it  looked for a new system to provide the desired functionality.
In reports on other agencies, we have also identified weaknesses in  requirements management and testing that are similar to the problems we  identified at HHS. Examples of problems that have resulted from  undisciplined efforts include the following.
In April 2003, we reported that NASA had not implemented an effective  requirements management process and that these requirement  management problems adversely affected its testing activities. We also  noted that because of the testing inadequacies, significant defects later  surfaced in the production system.
In May 2004, we reported that NASA's new financial management  system, which was fully deployed in June 2003 as called for in the  project schedule, still did not address many of the agency's most  challenging external reporting issues, such as external reporting  problems related to property accounting and budgetary accounting.
In May 2004, we reported that for two major Department of Defense  (DOD) systems, the initial deployments for these systems did not  operate as intended and, therefore, did not meet component-level needs.  In large part, these operational problems were due to DOD not  effectively implementing the disciplined processes that are necessary to  manage the development and implementation of the systems in the  areas of requirements management and testing. DOD program officials  have acknowledged that the initial deployments of these systems  experienced problems that could be attributed to requirements and  testing.
The problems experienced by these other agencies are illustrative of the  types of problems that can result when disciplined processes are not  properly implemented.  Whether HHS will experience such problems  cannot be known until the agency obtains the quantitative data necessary  to indicate whether the system will meet its needs.  Accordingly, HHS will  need to ensure it adequately addresses the numerous weaknesses we and  the IV&V contractor identified and has reduced the risk to an acceptable  level before implementing UFMS at CDC.  As we will be discussing in the  next section, compounding the risk to UFMS from not properly  implementing disciplined processes, is the fact that HHS is introducing  UFMS into an environment with weaknesses in its departmentwide IT  management practices.
section_title: Weaknesses in HHS’ IT Management Practices and Information Security Controls Put UFMS at Risk
paragraphs: HHS has planned and developed UFMS using the agency’s existing IT  investment management processes. However, we have reported—and HHS  has acknowledged—weaknesses in IT investment management, enterprise  architecture, and information security. Such weaknesses increase the risk  that UFMS will not achieve planned results within the estimated budget  and schedule.
subsections: section_title: HHS’ Enterprise IT Management Processes Also Add Risk for UFMS
paragraphs: In addition to weaknesses in disciplined processes in the development of  UFMS, weaknesses in the HHS’ IT management processes also increase the  risks associated with UFMS. HHS is modifying its IT investment  management policies, developing an enterprise architecture, and  responding to security weaknesses with several ongoing activities, but  these changes may not be implemented in time to compensate for the  increased risks.
IT investment management provides for the continuous identification,  selection, control, life-cycle management, and evaluation of IT investments.  The Clinger-Cohen Act of 1996 lays out specific aspects of the process  that agency heads are to implement in order to maximize the value of the  agency’s IT investments. In addition, OMB and GAO have issued guidance  for agencies to use in implementing the Clinger-Cohen Act requirements for  IT investment management. Our Information Technology Investment  Management framework is a maturity model composed of five  progressive stages of maturity that an agency can achieve in its IT  investment management capabilities. These stages range from creating  investment awareness to developing a complete investment portfolio to  leveraging IT for strategic outcomes. The framework can be used both to  assess the maturity of an agency’s investment management processes and  as a tool for organizational improvement.
OMB Circular No. A-130, which implements the Clinger-Cohen Act,  requires agencies to use architectures. A well-defined enterprise  architecture provides a clear and comprehensive picture of the structure of  any enterprise by providing models that describe in business and  technology terms how the entity operates today and how it intends to  operate in the future. It also includes a plan for transitioning to this future  state. Enterprise architectures are integral to managing large-scale  programs such as UFMS. Managed properly, an enterprise architecture can  clarify and help optimize the interdependencies and relationships among  an organization’s business operations and the underlying IT infrastructure  and applications that support these operations. Employed in concert with  other important management controls, architectures can greatly increase  the chances that organizations’ operational and IT environments will be  configured to optimize mission performance. To aid agencies in assessing  and improving enterprise architecture management, we issued guidance  establishing an enterprise architecture management maturity framework.  That framework uses a five-stage maturity model outlining steps toward  achieving a stable and mature process for managing the development,  maintenance, and implementation of an enterprise architecture.
The reliability of operating environments, computerized data, and the  systems that process, maintain, and report these data is a major concern to  federal entities, such as HHS, that have distributed networks that enable  multiple computer processing units to communicate with each other. Such  a platform increases the risk of unauthorized access to computer resources  and possible data alteration. Effective departmentwide information  security controls will help reduce the risk of loss due to errors, fraud and  other illegal acts, disasters, or incidents that cause systems to be  unavailable. Inadequate security and controls can adversely affect the  reliability of the operating environments in which UFMS and its  applications operate. Without effective general controls, application  controls may be rendered ineffective by circumvention or modification.  For example, a control designed to preclude users from entering  unreasonably large dollar amounts in a payment processing system can be  an effective application control, but this control cannot be relied on if  general controls permit unauthorized program modifications to allow  certain payments to be exempted from it.
section_title: Key HHS IT Investment Management Policies Still under Development
paragraphs: UFMS is at increased risk because of previously reported weaknesses in  the process that HHS uses to select and control its IT investments.  In  January 2004, we reported that there were serious weaknesses in HHS IT  investment management. Notably, HHS had not (1) established procedures  for the development, documentation, and review of IT investments by its  review boards or (2) documented policies and procedures for aligning and  coordinating investment decision making among its investment  management boards. In addition, HHS had not yet established selection  criteria for project investments or a requirement that IT investments  support work processes that have been simplified or redesigned.
HHS is modifying several of its IT investment management policies,  including its capital planning and investment control guidance and its  governance policies; but as of May 12, 2004, these documents were not final  or available for review. Until HHS addresses weaknesses in its selection or  control processes, IT projects like UFMS will face an increased likelihood  that the projects will not be completed on schedule and within estimated  costs.
section_title: Risk to UFMS Are Heightened With the Absence of an Established Enterprise Architecture
paragraphs: In November 2003, we released a report noting the importance of  leadership to agency progress on enterprise architecture efforts. We  reported that federal agencies’ progress toward effective enterprise  architecture management was limited: In a schedule of five stages leading  to a highly effective enterprise architecture program, 97 percent of the  agencies surveyed were still in Stage 1—creating enterprise architecture  awareness. In that report, we noted that HHS had reached Stage 2— building the enterprise architecture management foundation—by  successfully satisfying all elements of that stage of the maturity framework.  In addition, HHS had successfully addressed three of six elements of the  Stage 3 maturity level—developing architecture products. HHS has laid  that foundation by (1) assigning enterprise architecture management roles  and responsibilities and (2) establishing plans for developing enterprise  architecture products and for measuring program progress and product  quality. Progressing through the next stage would involve defining the  scope of the architecture and developing products describing the  organization in terms of business, performance, information/data,  service/application, and technology. Once the scope is defined and  products developed, Stage 3 organizations track and measure progress  against plans; identify and address variances, as appropriate; and report on  their progress.
Although it has made progress, HHS has not yet established an enterprise  architecture to guide and constrain its IT projects. In January 2004, HHS’  acting chief architect told us that the department continues to work on  implementing an enterprise architecture to guide its decision making. He  also noted that HHS plans to make UFMS a critical component of the  enterprise architecture now under development. However, most of the  planning and development of the UFMS IT investment has occurred  without the guidance of an established enterprise architecture. Our  experience with other federal agencies has shown that projects developed  without the constraints of an established enterprise architecture are at risk  of being duplicative, not well integrated, unnecessarily costly to maintain  and interface, and ineffective in supporting missions.
section_title: HHS Information Security Weaknesses Are Unresolved, and Needed Information for UFMS Is Not Shared
paragraphs: HHS has recognized the need to improve information security throughout  the department, including in key operating divisions, and has various  initiatives under way; however, it has not yet fully implemented the key  elements of a comprehensive security management program. Unresolved  general control weaknesses at headquarters and in HHS’ operating  divisions include almost all areas of information system controls described  in our Federal Information System Controls Audit Manual (FISCAM).  These weaknesses are in entitywide security, access controls, system  software, application software, and service continuity and they are  significant and pervasive.
According to a recent IG report, the underlying cause for most of the  weaknesses was that the department did not have an effective management  structure in place to ensure that sensitive data and critical operations  received adequate attention and that appropriate security controls were  implemented to protect them. HHS has not sufficiently controlled network  access, appropriately limited mainframe access, or fully implemented a  comprehensive program to monitor access. Weaknesses in other  information security controls, including physical security, further increased  the risk to HHS’ information systems. As a result, sensitive data—including  information related to the privacy of U.S. citizens, payroll and financial  transactions, proprietary information, and mission-critical data—were at  increased risk of unauthorized disclosure, modification, or loss, possibly  without being detected. Overall, the IG concluded that the weaknesses left  the department vulnerable to unauthorized access to and disclosure of  sensitive information, malicious changes that could interrupt data  processing or destroy data files, improper payments, or disruption of  critical operations.
Extensive information security planning for UFMS was based on  requirements and applicable guidance set forth in the Federal Information  Security Management Act, OMB Circular No. A-130 Appendix III (Security  of Federal Automated Information Resources), National Institute of  Standards and Technology guidance, and our FISCAM. However, that  planning was done without complete information from the department and  operating divisions. HHS has not conducted a comprehensive,  departmentwide assessment of information security general controls.  Further, information security general controls at four operating divisions  have not been recently assessed.  UFMS officials told us they did not know  which operating divisions had conducted or contracted for a review of their  individual information security environments. Without departmentwide  and operating-division-specific assessments, HHS increases its risk that  information security general control weaknesses will not be identified and  therefore will not be subject to departmentwide resolution or mitigation by  UFMS controls.
According to HHS officials, some operating divisions that have been  assessed recently have not provided UFMS with current information on the  status of the outstanding weaknesses in their operating environments.  UFMS officials told us that they do not have assurance of the reliability of  the control environment of these operating divisions. Without information  on control weaknesses in the operating divisions, UFMS management has  not been in a position to develop mitigating controls that could compensate  for departmentwide weaknesses. As a result, UFMS planning for security  cannot provide reasonable assurance that the system is protected from loss  due to errors, fraud and other illegal acts, disasters, and incidents that  cause systems to be unavailable.
section_title: Human Capital Issues Increase Risk Associated with the Implementation of UFMS
paragraphs: Serious understaffing and incomplete workforce planning have plagued the  UFMS project.  Human capital management for the UFMS project includes  organizational planning, staff acquisition, and team development. It is  essential that an agency take the necessary steps to ensure that it has the  human capital capacity to design, implement, and operate a financial  management system. However, the UFMS project has experienced staff  shortages as high as 40 percent of the federal positions that HHS believed  were needed to implement UFMS.  Although the staff shortage has been  alleviated to a great extent, the impact of such a significant shortfall  lingers.  Further, HHS has not yet fully developed key workforce planning  tools, such as the CDC skills gap analysis, to help transform its workforce  so that it can effectively use UFMS.  It is important that agencies  incorporate strategic workforce planning by (1) aligning an organization’s  human capital program with its current and emerging mission and  programmatic goals and (2) developing long-term strategies for acquiring,  developing, and retaining an organization’s total workforce to meet the  needs of the future. This incorporates a range of activities from identifying  and defining roles and responsibilities to identifying team members to  developing individual competencies that enhance performance. Human  capital planning should be considered for all stages of the system  implementation.
subsections: section_title: Positions Were Not Filled as Planned
paragraphs: According to JFMIP’s Building the Work Force Capacity to Successfully  Implement Financial Systems, the roles needed on an implementation  team are consistent across financial system implementation projects and  include a project manager, systems integrator, functional experts,  information technology manager, and IT analysts. Many of these project  roles require the dedication of full-time staff for one or more of the  project’s phases.
HHS has identified the lack of resources as a risk to the project and  acquired the staff to fill some of the roles needed for a systems  implementation project. The project has a project manager, systems  integrator, and some functional experts. However, on the basis of our  review of the HHS Organization and Staffing Plan and the most recent  program management office organization chart, many positions were not  filled as planned. For example, as reported in the IV&V contractor’s  September 2003 report, some key personnel filled multiple positions and  their actual available time was inadequate to perform the allocated tasks— commonly referred to as staff being overallocated on the project. As a  result, some personnel were overworked, which according to the IV&V  contractor, could lead to poor morale. The UFMS organization chart also  showed that the UFMS project team was understaffed and that several  integral positions were vacant or filled with part-time detailees. As of  January 2004, 19 of the 47 UFMS positions in the UFMS Program  Management Office identified as needed for the UFMS project were not  filled. The vacant positions included key positions such as the enterprise  architect, purchasing, testing, and configuration management leads. While  HHS and the systems integrator have taken measures to acquire additional  human resources for the implementation of UFMS, scarce resources could  significantly jeopardize the project’s success and have led to several key  deliverables being significantly behind schedule, as discussed in the  section on disciplined processes. Without adequate resources to staff the  project, the project schedule could be negatively affected, project controls  and accountability could be diminished, and the successful implementation  of UFMS could be compromised.
section_title: Strategic Workforce Planning Is Incomplete
paragraphs: Strategic workforce planning is essential for achieving the mission and  goals of the UFMS project. As we have reported, there are five key  principles that strategic workforce planning should address:  Involve top management, employees, and other stakeholders in  developing, communicating, and implementing the strategic workforce  plan.
Determine the critical skills and competencies that will be needed to  achieve current and future programmatic results.
Develop strategies that are tailored to address gaps in the number,  deployment, and alignment of human capital approaches for enabling  and sustaining the contributions of all critical skills and competencies.
Build the capability needed to address administrative, educational, and  other requirements important to support workforce planning strategies.
Monitor and evaluate the agency’s progress toward its human capital  goals and the contribution that human capital results have made toward  achieving programmatic results.
HHS has taken first steps to address three of the five key principles  identified in our report on strategic workforce planning. To address the  first key principle, HHS’ top management first communicated the agency’s  goal to implement a unified financial management system in June 2001 and  has continued to communicate the agency’s vision. HHS has developed an  Organizational Change Management Plan and, according to the UFMS  project’s Statement of Work, HHS, in undertaking UFMS, will seek to  ensure that sufficient efforts are made to address communications, human  resources, and training requirements.
To meet the second principle of identifying the needed skills and  competencies, HHS developed a Global Organization Impact Analysis in  March 2003 and subsequently prepared an analysis for CDC that identified  workforce and training implications associated with the major changes that  will occur in its financial management business processes. However, more  work remains. Although a Global/CDC Pilot Competency Report was  prepared that focuses on preparing and equipping the workforce to  function effectively in the new environment, none of the other operating  divisions scheduled to implement UFMS had prepared a competency report  as of May 2004.
To effectively address the third principle of developing strategies to  address the gaps in human capital, HHS must first identify the skills and  competencies needed. HHS has plans to conduct a skills gap analysis on a  site-specific basis. However, as of May 2004, the CDC skills gap analysis  had not been completed. CDC officials maintain that they intend to wait  until after the system is implemented to assess the changes in individuals’  workloads and make decisions on staffing changes. In addition, HHS is  currently developing a global Workforce Transition Strategy, which the  other operating divisions will use as a model in developing their own  strategies. According to HHS officials, HHS has also prepared a global  training strategy.  Training plans are to be developed on a site-specific basis  using the global strategy as a model. Although CDC has a tentative  schedule for planned training, as of May 2004 the CDC training plan was  not complete.
As we have previously reported, having staff with the appropriate skills is  key to achieving financial management improvements, and managing an  organization’s employees is essential to achieving results. HHS already  faces challenges in implementing its financial management system due to  the lack of adequate resources. By not identifying staff with the requisite  skills to implement such a system and by not identifying gaps in needed  skills and filling them, HHS has reduced its chances of successfully  implementing and operating UFMS.
section_title: Conclusions
paragraphs: HHS has not followed key disciplined processes necessary to reduce the  risks associated with implementing UFMS to acceptable levels.  These  problems are similar to those encountered by other agencies that have  found themselves under strong pressure to skip steps in their haste to get  systems up and running and produce results.  If HHS continues on this  path, it runs a higher risk than necessary of finding, as many others have  already discovered, that the system may be more costly to operate, take  more time and effort to perform needed functions, be more disruptive to  the work of the agency, and may not achieve the intended improvement.
Ideally, HHS should not continue with its current approach for UFMS.   However, if HHS decides for operational reasons to continue its plan to  deploy UFMS at CDC in October 2004, then as a precursor to deployment at  CDC, there are several key steps that must be taken to mitigate the  significant risk related to this deployment.  To begin, HHS must determine  the system capabilities that are necessary for the CDC deployment and  identify the relevant requirements related to those capabilities.  The  associated requirements will have to be unambiguous and adequately  express how the system will work, be traceable from their origin through  implementation, and be sufficiently tested to confirm that the system meets  those functional needs.  Validating data conversion efforts and systems  interfaces will also be critical to the successful launch of UFMS.  HHS will  need to ensure that its desire to meet the October 2004 initial deployment  of UFMS is driven by successful completion of at least these key events  based on quantitative data rather than the schedule.  HHS should not  deploy UFMS at CDC until these critical steps are complete.
Before proceeding further with the UFMS implementation beyond CDC,  HHS should pause to assess whether an appropriate foundation is in place  so that UFMS will achieve its ultimate goals of a unified accounting system  that institutes common business rules, data standards, and accounting  policies and procedures.  From our perspective, HHS does not have a fully  developed view of how UFMS will operate because it moved forward with  the project before ensuring that certain key elements, such as a concept of  operations and an enterprise architecture, were completed.  Without  assurances that it is moving ahead with a solid foundation and a fully  developed and strongly administered plan for bringing the entire UFMS  project under the disciplined processes of requirements management,  testing, risk management, and the use of quantitative measures to manage  the project, HHS risks not achieving its goal of a common accounting  system that produces data for management decision making and financial  reporting and risks perpetuating its long-standing accounting system  weaknesses with substantial workarounds to address any needed  capabilities that have not been built into the system.
Because we have recently issued reports providing HHS with  recommendations to address weaknesses in IT investment management  processes, we are not making additional recommendations in this report  related to those two disciplines other than to reiterate the importance of  taking action on our prior recommendations.  It will be important that HHS  continue with its ongoing initiatives to strengthen these two areas.  Also,  HHS has not fully secured its information systems security environment to  offer an adequate basis for incorporating adequate security features into  UFMS as it is being developed.  Finally, addressing human capital and  staffing shortages that have also increased risks related to UFMS is  paramount to achieving the agency’s objectives for this project.
section_title: Recommendations for Executive Action
paragraphs: To help reduce risks associated with deployment of UFMS at CDC to  acceptable levels, we recommend that the Secretary of Health and Human  Services direct the Assistant Secretary for Budget, Technology, and  Finance to require that the UFMS program staff take the following nine  actions:  Determine the system capabilities that are necessary for the CDC  deployment.
Identify the relevant requirements related to the desired system  capabilities for the CDC deployment.
Clarify, where necessary, any requirements to ensure they (1) fully  describe the capability to be delivered, (2) include the source of the  requirement, and (3) are unambiguously stated to allow for quantitative  evaluation.
Maintain traceability of the CDC-related requirements from their origin  through implementation.
Use a testing process that employs effective requirements to obtain the  quantitative measures necessary to understand the assumed risks.
Validate that data conversion efforts produce reliable data for use in  UFMS.
Verify that systems interfaces function properly so that data exchanges  between systems are adequate to satisfy system needs.
Measure progress based on quantitative data rather than the occurrence  of events.
If these actions are not completed, delay deployment of UFMS at CDC.
Before proceeding with further implementation of UFMS after deployment  at CDC, we recommend that the Secretary of  Health and Human Services  direct the Assistant Secretary for Budget, Technology, and Finance to  require that the UFMS program staff take the following 14 actions:  Develop and effectively implement a plan on how HHS will implement  the disciplined processes necessary to reduce the risks associated with  this effort to acceptable levels. This plan should include the processes,  such as those identified by SEI and IEEE, that will be implemented and  the resources, such as staffing and funding, needed to implement the  necessary processes.
Develop a concept of operations in accordance with recognized industry  standards such as those promulgated by IEEE.  The concept of  operations should apply to all HHS entities that will be required to use  UFMS.  This concept of operations should contain a high-level  description of the operations that must be performed, who must  perform them, and where and how the operations will be carried out,  and be consistent with the current vision for the HHS information  system enterprise architecture.
Implement a requirements management process that develops  requirements that are consistent with the concept of operations and  calls for each of the resulting requirements to have the attributes  associated with good requirements: (1) fully describing the functionality  to be delivered, (2) including the source of the requirement, and   (3) stating the requirement in unambiguous terms that allows for  quantitative evaluation.
Maintain traceability of requirements among the various  implementation phases from origin through implementation.
Confirm that requirements are effectively used for (1) determining the functionality that will be available in UFMS at a  given location, (2) implementing the required functionality,  (3) supporting an effective testing process to evaluate whether UFMS is  ready for deployment,  (4) validating that data conversion efforts produce reliable data for use  in UFMS, and  (5) verifying that systems interfaces function properly so that data  exchanges between systems are adequate to satisfy each system’s  needs.
Develop and implement a testing process that uses adequate  requirements as a basis for testing a given system function.
Formalize risk management procedures to consider that  (1) all risks currently applicable to the UFMS project are identified and  (2) a risk is only closed after the risk is no longer applicable rather than  once management has developed a mitigation strategy.
Develop and implement a program that will identify the quantitative  metrics needed to evaluate project performance and risks.
Use quantitative measures to assess progress and compliance with  disciplined processes.
To help ensure that HHS reduces risks in the agencywide IT environment  associated with its implementation of UFMS, we recommend that the  Secretary of Health and Human Services direct the Assistant Secretary for  Budget, Technology, and Finance to require that the following seven  actions are taken by the IT program management staff, as appropriate:  Conduct assessments of operating divisions’ information security  general controls that have not been recently assessed.
Establish a comprehensive program to monitor access to the network,  including controls over access to the mainframe and the network.
Verify that the UFMS project management staff has all applicable  information needed to fully ensure a comprehensive security  management program for UFMS. Specifically, this would include  identifying and assessing the reported concerns for all HHS entities  regarding key general control areas of the information security  management process: (1) entitywide security planning,  (2) access controls,  (3) system software controls,  (4) segregation of duties, and  (5) application development and change controls.
To help improve the human capital initiatives associated with the UFMS  project, we recommend that the Secretary of Health and Human Services  direct the Assistant Secretary for Budget, Technology, and Finance to  require that the following four actions are taken by the UFMS program  management staff:  Assess the key positions needed for effective project management and  confirm that those positions have the human resources needed.  If  needed, solicit the assistance of the Assistant Secretary for Budget,  Technology, and Finance to fill key positions in a timely manner.
Finalize critical human capital strategies and plans related to UFMS  (1) skills gap analysis,  (2) workforce transition strategy, and  (3) training plans.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, HHS described the actions it  had taken to date to develop UFMS, including some actions related to our  recommendations, which if effectively implemented, should reduce project  risk.  HHS disagreed with our conclusion that a lack of disciplined  processes is placing the UFMS program at risk, stating that its processes  have been clear and rigorously executed.  HHS characterized the risk in its  approach as the result not of a lack of disciplined process but of an  aggressive project schedule.  HHS stated that it made a decision early in the  program to phase in the deployment of the system to obtain what it  referred to as incremental benefits, and said that a core set of requirements  will be available for the October 2004 release at CDC.  HHS added that if a  system functional capability becomes high risk for the pilot  implementation at CDC, it could be deferred to a subsequent release  without affecting the overall implementation.  HHS did not provide  examples of the functional capabilities that could be deferred under such a  scenario, but we understand that at least some functionality associated  with grant accounting being deployed at CDC is less than that originally  envisioned when we performed our review—less than 6 months before the  scheduled CDC implementation date.  HHS stated that it had reached every  major milestone to date within the planned timeframes and budget for  almost 3 years while managing to mitigate the cost, schedule, and technical  risks.  The agency considers this is a testament to UFMS management  disciplines, notwithstanding known needed improvements.
From our perspective, this project demonstrates the classic symptoms of a  schedule-driven effort for which key processes have been omitted or  shortcutted, thereby unnecessarily increasing risk.  This is a multiyear  project, and it is important that the project adhere to disciplined processes  that represent best practices.  We have no problem whatsoever with a  phased approach and view it as a sound decision for this project.  There is  no doubt that a phased approach can help reduce risks.  However, we do  not agree that a phased approach adequately mitigates risk in a project of  this magnitude, given the other problems we identified. As discussed in our  report and highlighted in the following sections that further evaluate HHS’  comments on our draft report, we identified a number of problems with  HHS’ methodology, including problems in requirements management,  testing, project management and oversight, and IT management, that are at  the heart of our concern.  Also, we are not saying that HHS is not following  any disciplined processes, and in this report we have recognized certain  HHS actions that we believe represent best practices that reduce risk.  We  are saying that HHS has not reduced its risk to an acceptable level because  a number of key disciplined processes were not yet in place or were not  effectively implemented.  We focused our 34 recommendations on tangible  actions that HHS can take to adequately mitigate risk.  Risk on a project  such as this can never be eliminated, but risk can be much better managed  than what we observed for this project.
With respect to HHS’ comment that all milestones have been met, as we  discussed in detail in this report, we caution that because HHS has  insufficient quantifiable criteria for assessing the quality of its progress and  the impact of identified defects, it does not have the information it needs to  determine whether the milestones have been substantively accomplished  and the nature and extent of resources needed to resolve remaining  defects.  A best practice is having quantitative metrics and a disciplined  process for continually measuring and monitoring results.
We stand firmly behind our findings that HHS had not reduced project risk  to an acceptable level because it had not adequately adhered to disciplined  processes called for in its stated implementation methodology.  We are  somewhat encouraged by the planned actions outlined in HHS’ comment  letter and the fact that it has now decided to delay initial implementation by  at least 2 weeks to address known problems and has indicated it may delay  the initial implementation further as needed.  Only time will tell how well  this project turns out, as the initial implementation at CDC represents just  the first phase.  Our hope is that the disciplined processes discussed in our  report and addressed in our recommendations will be followed and that  risks of a project of this magnitude and importance will be reduced to an  acceptable level.  If the past is prologue, taking the time to adhere to  disciplined processes will pay dividends in the long term.
subsections: section_title: Implementation Methodology
paragraphs: HHS stated that the underlying premise of our report is that there is one  correct way to perform an implementation for a project such as UFMS and  that this methodology, commonly referred to as the waterfall  methodology, is inappropriate for a COTS-based system.  Our report does  not call for the use of this or any other specific methodology.  Instead, we  have emphasized the importance of following disciplined processes in the  development and implementation of large and complex information  management systems, including financial management systems such as  UFMS.  As we have reiterated throughout this report, we view disciplined  processes as the key to successfully carrying out a system development  and implementation program whatever the methodology.
In the case of HHS’ COTS-based system development program, we did not  question the methodology, but have concerns about HHS’ ability to  successfully implement its methodology.  For example, as explained in our  report and reiterated in HHS’ comments, before a COTS software package  is selected for implementation, requirements need to be more flexible and  less specific than custom-developed software because no off-the-shelf  product is likely to satisfy all of the detailed requirements for a large,  complex organization such as HHS.  Once the product is selected, however,  a disciplined approach to COTS implementation demands that  requirements be defined at a level of specificity that allows the software to  be configured to fit the system under development and to be implemented  to meet the organization’s needs.  In discussing the HHS methodology, our  report is consistent with how HHS described its methodology in its  comments. As we noted in the report, the methodology selected by HHS  requires (1) reviewing and updating the requirements through process  design workshops, (2) establishing the initial baseline requirements,   (3) performing a fit/gap analysis, (4) developing gap closure alternatives,  and (5) creating the final baseline requirements.  However, as noted in our  report, HHS was unable to successfully implement its methodology for the  majority of the requirements we reviewed.  For example, one inadequately  defined requirement was linked to the budget distributions process.   However, this process, which should of provided additional specificity to  understand how the system needed to be configured, stated that the  process was “To Be Determined.”
section_title: Requirements Management
paragraphs: In its comments, HHS stated that in July 2002 it had developed a “target  business model” that is equivalent to a concept of operations for guiding its  development efforts.  The document HHS referenced, which we reviewed  during our audit, along with several other requirement-related documents  HHS had provided, did not have all the elements associated with a concept  of operations document as defined by IEEE.  For example, the document  did not address the modes of operation; user classes and how they should  interact; operational policies and constraints; costs of systems operations;  performance characteristics, such as speed, throughput, volume, or  frequency; quality attributes, such as availability, reliability, supportability,  and expandability; and provisions for safety, security, and privacy.  The  document does not address a number of other critical issues associated  with the project such as the use of shared services.  We also noted that  some HHS officials who had reviewed this document stated that it did not  resolve a number of issues that needed to be addressed.  For example, HHS  reviewers raised questions about who was responsible for several core  functions. When we performed our review, these types of questions  remained unanswered, although HHS said in its comments on our draft  report that it is taking steps to address these concerns and has now made  certain decisions regarding shared services.
In addition, HHS’ comment letter stated that it has developed a  requirements database that could be used to track the requirements and  that its requirements management process used two broad categories -  Program Management Office of JFMIP requirements and agency-specific  requirements.  HHS also stated that the requirements process has fully  defined and documented the expected behavior of UFMS and that the  agency-specific requirements it had identified had been developed in  accordance with industry best practices.  HHS noted that it has also  developed a requirements traceability verification matrix since our review.
The result, according to HHS, has been a requirements management  process that provides fully traceable requirements that are fully tested by  the implementation team.
Developing and effectively implementing the kinds of processes described  in HHS’ comments are positive steps that would reduce the risks associated  with requirements related defects.  However, since these key processes,  which were called for in our report and during meetings held with HHS  during our review, were developed and implemented after our work was  complete, we are unable to determine whether HHS has yet fully addressed  the weaknesses we observed.  As noted in our report, we found numerous  requirements that did not contain the necessary specificity to support a  good testing program. We also note that the HHS comments refer to these  processes being used for “testable” requirements but do not provide  information on how many of the 2,130 requirements contained in its  requirements database were considered testable and, therefore, subject to  this improved process.
section_title: Testing
paragraphs: While HHS stated in its comment letter that it has implemented a more  disciplined system testing process, its comments also raised concerns  about the thoroughness of the testing.  HHS noted that it has selected an  application certified by the Program Management Office of JFMIP and that  “80% of the requirements have been met  out of the box  functionality.”  Accordingly, HHS stated that it has, by design, tested these  requirements with less rigor than the agency specific requirements.  As  noted in HHS’ comments, its requirements management database contains  2,130 requirements that include requirements issued by the Program  Management Office of JFMIP.  However, according to the Program  Management Office of JFMIP, its testing efforts encompass about 331  requirements, or only about 16 percent of HHS’ stated requirements.
Compounding this limitation, while the Program Management Office of  JFMIP test results can be helpful, as the Program Management Office of  JFMIP has consistently made it clear to agencies, these tests are not  intended to take the place of agency-level tests. The Program Management  Office of JFMIP tests are in a controlled environment that is not intended to  represent the operating environment of a specific agency.  As the Project  Management Office of JFMIP points out on its Web site, agencies need to  (1) test the installed configured system to ensure continued compliance  with the governmentwide core requirements and any agency-specific  requirements, (2) assess the suitability of an application for the agency’s  operating environment, and (3) assess the COTS computing performance in  the agency’s environment for response time and transaction throughput  capacity.  For example, addressing this last point regarding transaction  throughput capacity has proven problematic to some agencies that  implemented a COTS package.  The system could have properly processed  a type of transaction, which is what the test requires in order to be  certified.  However, the system may require a number of separate  processing steps to accomplish this task.  Those steps may be acceptable at  an agency that has a relatively low volume of this type of transaction, but  may prove problematic for an agency with a high volume of this type of  transaction.
As noted in the HHS comments, it had not yet developed the test scripts  and other documentation that would have enabled us to assess the  adequacy of its system testing activities at the time of our review.   Therefore, we cannot conclude on whether its system testing activities will  have a reasonable assurance of detecting the majority of the defects.  HHS  noted that it had conducted preliminary testing, referred to as conference  room pilots, in August 2003 and in March and April 2004 and that these  activities were attended by finance, business, and program staff members  from across HHS, who will be the ultimate users of the new system.  As  noted in our report, our review of the conference room pilot conducted in  March and April 2004 found significant weaknesses in the processes being  used. This was the last conference pilot scheduled before the pilot  deployment at CDC.  We found that some of the stated requirements in a  given conference room pilot test script were not tested and defects  identified were not promptly recorded.   This is consistent with  observations made by HHS’ IV&V contractor on the August 2003  conference room pilots.  Furthermore, we observed that when users asked  about needed functionality, they were told that the functionality would be  developed later.  Therefore, we are encouraged by the statement in HHS’  comment letter that it will implement a disciplined system testing process.
In our report, we also noted that the system testing activities were  scheduled late in the first phase of the UFMS implementation process,  leaving little time for HHS to address any defects identified during system  testing and to ensure that the corrective actions taken to address the  defects do not introduce new defects.  HHS agreed that system testing  would ideally come earlier in the process and noted that although the  testing process is being performed late due to an aggressive time schedule,  it believed, based on its level of scrutiny, its testing plan will identify the  majority of the defects in the system.  We view this as adding to project  risk.  However, we are encouraged that in its comments on our draft report,  HHS said it was analyzing system integration test results prior to deploying  the system at CDC, and that this assessment may result in revising the  current software release strategy.
section_title: Program Management Oversight
paragraphs: In its comments, HHS stated that its combined use of software tools,  including TeamPlay from Primavera, provides management with  information for monitoring the project’s critical path and the earned value  of completed work and that this action was taken in October 2003 after an  August 2003 report from its IV&V contractor.  As with other process areas,  the key to reducing risks to acceptable levels is not only the tool that is  used but, more importantly, the effective implementation of that tool.  In  other words, simply selecting an industry standard practice or tool does  not guarantee success. As noted in a May 2004 IV&V report, as of April  2004, the IV&V contractor was still raising concerns about HHS’ ability to  perform critical path and earned value analysis. HHS acknowledged in its  comments on our draft report that it continues to work on improving the  information provided in the critical path reports and is executing a plan to  implement the remainder of the IV&V suggestions.  As we discussed  previously in this report, without an effective critical path analysis and an  earned value management system, HHS does not have adequate assurance  that it can understand the impact of various project events, such as delays  in project deliverables, and that it knows the status of the various project  deliverables in the context of progress and associated cost.  We continue to  believe that management needs this information to determine actions to  take to mitigate risk and manage cost and schedule performance.
HHS also stated that all of the needed improvements in its project  execution were identified and documented prior to and during our review  by its IV&V contractor and that improvements continue to be implemented.   Our report clearly identifies areas of mutual concern by us and the IV&V  contractor as well as areas where our work uncovered additional issues.   Regardless of who identified the problems, we remain concerned that HHS  has been slow to act upon the weaknesses identified by the IV&V  contractor and has not yet clearly identified actions planned to address our  recommendations.  Our report provides examples where it has taken HHS  months to address the findings made by its IV&V contractor.
Regarding quantitative measures, HHS agreed that quantitative measures  are crucial to UFMS success and stated that it has struck an adequate  balance between the number of measures used to assess UFMS progress  and the effort and costs required to develop and maintain the measures.   HHS described several measures related to its defect-tracking processes  that are associated with its system testing efforts. We agree with HHS that  the measures listed in its comment letter are critical to assessing system  stability and readiness, but HHS’ comments did not indicate whether it is  also capturing metrics on items that can help it understand the risks  associated with the processes it is implementing, such as with its  requirements management process.  For example, HHS stated that system  testing had not identified any requirements problems, which indicated the  requirements were defined thoroughly.  However, system testing is  normally not designed to capture requirements problems since, as noted in  HHS’ comment letter, testing is structured to determine whether the system  is meeting requirements that have been documented.  Therefore, it is not  clear whether HHS has fully developed a metric process that will address  its needs throughout the phased deployments.
Regarding human capital, HHS said that it faces its share of challenges in  obtaining full-time federal staff due to the temporary nature of an  implementation project and the agency’s objective to staff a highly  competent program team and not a permanent federal bureaucracy.  We  recognize that HHS and the systems integrator it has under contract to  assist with the project have taken measures to acquire additional staff for  the implementation of UFMS.  We also recognize the challenge in finding  people with the needed skills.  Our concern is that the UFMS project has  experienced staff shortages as high as 40 percent of the federal positions  that HHS believed were needed to implement UFMS.  This shortage of staff  resources led to several key deliverables being significantly behind  schedule.  Also, while HHS said that CDC has the vast majority of its  required positions filled, we found that many of the positions for this  operating division were filled with staff from the program management  office for the project, which affects the work that should be done to  manage and oversee the project.  As stated in our report, without adequate  staff resources, the project schedule can be negatively affected, project  controls and accountability can be diminished, and the successful  implementation of UFMS may be compromised.
section_title: IT Management
paragraphs: With respect to IT management, including investment management,  enterprise architecture, and information security, HHS elaborated on  further activities taken to address weaknesses that we had pointed out in  our draft report. In its comments, HHS referenced a Web site that provides  its IT investment policy dated January 2001, which we had already  reviewed and which agency officials stated was in the process of being  updated. In January 2004, we recommended 10 actions the department  should take to improve its IT investment management process. One action  called for HHS to revise the department’s IT investment management policy  to include (1) how this process relates to other agency processes, (2) an  identification of external and environmental factors, (3) a description of  the relationship between the process and the department’s enterprise  architecture, and (4) the use of independent verification and validation  reviews, when appropriate. HHS concurred with our recommendations.   Further, although HHS’ comments indicated that we made a  recommendation related to enterprise architecture, as we stated in our  conclusions, we did not make recommendations about enterprise  architecture in this report.
We agree with HHS that progress has been made in its information security  management.  However, HHS did not address the potential impact that  outstanding departmentwide information security controls weaknesses  could have on the reliability and integrity of the new financial management  system.  HHS will need to ensure effective information security controls  departmentwide for UFMS operations.
section_title: GAO’s Review Process
paragraphs: In its response to a draft of this report, HHS stated that the timing of our  review of the UFMS was not optimal and required significant staff time for  meetings and preparation, document requests, and communications.  In  HHS’ opinion, GAO involvement was in itself a significant contributor to  project schedule risk.  In our view, we conducted this engagement in a  professional, constructive manner in which we worked proactively with  HHS to provide timely observations on the implementation of UFMS.  The  timing of our review was aimed at providing input early in the process so  that HHS can act to address weaknesses and reduce the risk of  implementing a system that does not meet needs and expectations and  requires costly rework and work-arounds to operate.  We have found in our  reviews of other agencies’ system implementation efforts that effective  implementation of disciplined processes can reduce risks that have an  adverse impact on the cost, timeliness, and performance of a project.   Through early recognition and resolution of the weaknesses identified,  HHS can optimize its opportunities to reduce the risks that UFMS will not  fully meet one or more of its cost, schedule, and performance objectives.   Further, in performing our review, we made every effort to reduce  inconvenience to HHS.  For example, HHS asked us and we agreed to  postpone our initial meetings with HHS staff until after the completion of  HHS’ fiscal year 2003 financial statement audit.  We also followed HHS’  protocols in scheduling meetings and requested documentation that should  have been readily available, at this stage of the UFMS.  HHS’ adoption of  several of our recommendations evidences the added value of our review  and implementation of all 34 of our recommendations will add even greater  value to the project.
As agreed with your offices, unless you announce the contents of this  report earlier, we will not distribute it until 30 days after its date. At that  time, we will send copies to the Chairman and Ranking Minority Member,  Senate Committee on Governmental Affairs, and other interested  congressional committees. We are also sending copies to the Secretary of  Health and Human Services and the Director of the Office of Management  and Budget. Copies will also be made available to others upon request.  The  report will also be available at no charge on GAO’s Web site at  http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact Sally E. Thompson, Director, Financial Management and  Assurance, who may be reached at (202) 512-9450 or by e-mail at  thompsons@gao.gov, or Keith A. Rhodes, Chief Technologist, Applied  Research and Methods, who may be reached at (202) 512-6412 or by e-mail  at rhodesk@gao.gov. Staff contacts and other key contributors to this  report are listed in appendix V.
section_title: Scope and Methodology
paragraphs: Our review of the Department of Health and Human Services’ (HHS)  ongoing effort to develop and implement a unified accounting system  focused on one of the three concurrent but separate projects: the ongoing  implementation of the Unified Financial Management System (UFMS) at  the Centers for Disease Control and Prevention (CDC), the Food and Drug  Administration and HHS’ Program Support Center (PSC).   This project will  be carried out in a phased approach. HHS is currently implementing UFMS  at CDC, and it is scheduled to go live in October 2004.  The other two  projects are the Centers for Medicare and Medicaid Services’ (CMS)  implementation of the Healthcare Integrated General Ledger Accounting  System to replace the Financial Accounting Control System, and the  National Institutes of Health’s (NIH) implementation of the NIH Business  and Research Support System to replace the Central Accounting System.
To assess HHS’ implementation of disciplined processes, we reviewed  industry standards and best practices from the Institute of Electrical and  Electronics Engineers (IEEE), Software Engineering Institute (SEI),  Project Management Institute, Joint Financial Management Improvement  Program (JFMIP), GAO executive guides, and prior GAO reports. We  reviewed and analyzed UFMS planning documents related to project  management, testing, data conversion, requirements management, risk  management, and configuration management. We also reviewed minutes  from key meetings, such as the Information Technology Investment Review  Board meetings, Risk Management meetings, and Planning and  Development Committee meetings. In addition, we reviewed reports issued  by the independent verification and validation (IV&V) contractor and  interviewed the systems integrator to clarify the status of issues discussed  in the reports.
To assess whether HHS had established and implemented disciplined  processes related to requirements management, we  reviewed strategy and planning documents, including its Financial  Shared Services Study Concept of Operation, dated April 30, 2004;    reviewed HHS’ procedures for defining its requirements management  framework and compared these procedures to its current practices;  reviewed guidance published by IEEE and SEI and publications by  experts to determine the attributes that should be used in developing  good requirements and selected over 70 requirements and performed an  in-depth review and analysis to determine whether they could be  traced  between the various process documents;  attended the second conference room pilot (the session held in  Rockville, Maryland) to evaluate whether the test scripts demonstrated  the functionality of the listed requirements; and  reviewed IV&V contractor reports to obtain its perspective on HHS’  requirements management processes.
To assess the risk management process, we reviewed the 44 risks  documented in the PMOnline risk management tool to determine the  current status of the risk and to assess the risk mitigation plan. We  interviewed agency officials to obtain explanations for the status of the  risks. We analyzed the project schedule and IV&V status reports to assess  the probability of HHS meeting its projected completion dates for  development, implementation, and testing.
To assess information technology (IT) management practices, we reviewed  prior GAO reports on governmentwide investment management and  enterprise architecture. We also reviewed and analyzed relevant IT policies  and plans and HHS documentation on the IT investment management  processes. To assess information security practices, we relied on prior  years’ audit work performed in this area. We reviewed pertinent HHS  security policies and procedures, and reviewed HHS’ efforts to minimize  potential and actual risks and exposures.
To determine whether HHS had the human resources capacity to  successfully design, implement, and operate the financial management  system, we reviewed JFMIP’s Core Competencies for Project Managers  Implementing Financial Systems in the Federal Government, Building  the Work Force Capacity to Successfully Implement Financial Systems,  and Core Competencies in Financial Management for Information  Technology Personnel Implementing Financial Systems in the Federal  Government and prior GAO reports related to strategic workforce  planning. We analyzed the UFMS program management office organization  chart and obtained related information on project staffing. We also  interviewed HHS officials and the IV&V contractor to discuss staffing  resource issues.
For these areas, we interviewed HHS, UFMS, IV&V, and systems integrator  officials to discuss the status of the project and their roles in the project.
On April 26, 2004, and May 12, 2004, we briefed HHS management on our  findings so that action could be taken to reduce risks associated with the  UFMS project.  We performed our work at HHS headquarters in  Washington, D.C.; at the UFMS site in Rockville, Maryland; and at CDC  offices in Atlanta, Georgia. Our work was performed from September 2003  through May 2004 in accordance with U.S. generally accepted government  auditing standards. We did not review the prior implementation of Oracle at  NIH or the ongoing implementation of Oracle at CMS. We requested  comments on a draft of this report from the Secretary of Health and Human  Services or his designee.  Written comments from the Department of Health  and Human Services are reprinted in appendix IV and evaluated in the  “Agency Comments and Our Evaluation” section.
section_title: Disciplined Processes Are Key to Successful System Development and Implementation Efforts
paragraphs: Disciplined processes have been shown to reduce the risks associated with  software development and acquisition efforts to acceptable levels and are  fundamental to successful systems acquisition. A disciplined software  development and acquisition process can maximize the likelihood of  achieving the intended results (performance) within established resources  (costs) on schedule. Although a standard set of practices that will  guarantee success does not exist, several organizations, such as SEI and  IEEE, and individual experts have identified and developed the types of  policies, procedures, and practices that have been demonstrated to reduce  development time and enhance effectiveness. The key to having a  disciplined system development effort is to have disciplined processes in  multiple areas, including requirements management, testing, project  planning and oversight, and risk management.
subsections: section_title: Requirements Management
paragraphs: Requirements are the specifications that system developers and program  managers use to design, develop, and acquire a system. They need to be  carefully defined, consistent with one another, verifiable, and directly  traceable to higher-level business or functional requirements. It is critical  that they flow directly from the organization’s concept of operations (how  the organization’s day-to-day operations are or will be carried out to meet  mission needs).
According to IEEE, a leader in defining the best practices for such efforts,  good requirements have several characteristics, including the following: The requirements fully describe the software functionality to be  delivered. Functionality is a defined objective or characteristic action of  a system or component. For example, for grants management, a key  functionality includes knowing (1) the funds obligated to a grantee for a  specific purpose, (2) the cost incurred by the grantee, and (3) the funds  provided in accordance with federal accounting standards.
The requirements are stated in clear terms that allow for quantitative  evaluation. Specifically, all readers of a requirement should arrive at a  single, consistent interpretation of it.
Traceability among various requirement documents is maintained.
Requirements for projects can be expressed at various levels depending  on user needs. They range from agencywide business requirements to  increasingly detailed functional requirements that eventually permit the  software project managers and other technicians to design and build the  required functionality in the new system. Adequate traceability ensures  that a requirement in one document is consistent with and linked to  applicable requirements in another document.
The requirements document contains all of the requirements identified  by the customer, as well as those needed for the definition of the system.
Studies have shown that problems associated with requirements definition  are key factors in software projects that do not meet their cost, schedule,  and performance goals.  Examples include the following:   A 1988 study found that getting a requirement right in the first place  costs 50 to 200 times less than waiting until after the system is  implemented to get it right.
A 1994 survey of more than 8,000 software projects found that the top  three reasons that projects were delivered late, over budget, and with  less functionality than desired all had to do with requirements  management.
A 1994 study found that the average project experiences about a 25  percent increase in requirements over its lifetime, which translates into  at least a 25 percent increase in the schedule.
A 1997 study noted that between 40 and 60 percent of all defects found  in a software project could be traced back to errors made during the  requirements development stage.
section_title: Testing
paragraphs: Testing is the process of executing a program with the intent of finding  errors.  Because requirements provide the foundation for system testing,  specificity and traceability defects in system requirements preclude an  entity from implementing a disciplined testing process. That is,  requirements must be complete, clear, and well documented to design and  implement an effective testing program. Absent this, an organization is  taking a significant risk that substantial defects will not be detected until  after the system is implemented. As shown in figure 3, there is a direct  relationship between requirements and testing.
Although the actual testing occurs late in the development cycle, test  planning can help disciplined activities reduce requirements-related  defects. For example, developing conceptual test cases based on the  requirements derived from the concept of operations and functional  requirements stages can identify errors, omissions, and ambiguities long  before any code is written or a system is configured. Disciplined  organizations also recognize that planning the testing activities in  coordination with the requirements development process has major  benefits.
Although well-defined requirements are critical for implementing a  successful testing program, disciplined testing efforts for projects such as  UFMS have several characteristics, which include the following:  Testers who assume that the program has errors.  Such testers are likely  to find a greater percentage of the defects present in the system.  This is  commonly called the “testing mindset.”
Test plans and scripts that clearly define what the expected results  should be when the test case is properly executed and the program does  not have a defect that would be detected by the test case.  This helps to  ensure that defects are not mistakenly accepted.
Processes that ensure test results are thoroughly inspected.
Test cases that include exposing the system to invalid and unexpected  conditions as well as the valid and expected conditions.  This is  commonly referred to as boundary condition testing.
Testing processes that determine if a program has unwanted side  effects.  For example, a process should update the proper records  correctly but should not delete other records.
Systematic gathering, tracking, and analyzing statistics on the defects  identified during testing.
Although these processes may appear obvious, they are often overlooked  in testing activities.
section_title: Project Planning and Oversight
paragraphs: Project planning is the process used to establish reasonable plans for  carrying out and managing the software project.  This includes   (1) developing estimates of the resources needed for the work to be  performed, (2) establishing the necessary commitments, and (3) defining  the plan necessary to perform the work.  Effective planning is needed to  identify and resolve problems as soon as possible, when it is the cheapest  to fix them.  According to one author, the average project spends about  80 percent of its time on unplanned rework—fixing mistakes that were  made earlier in the project.  Recognizing that mistakes will be made in a  project is an important part of planning.  According to this author,  successful system development activities are designed so that the project  team makes a carefully planned series of small mistakes to avoid making  large, unplanned mistakes.  For example, spending the time to adequately  analyze three design alternatives before selecting one results in time spent  analyzing two alternatives that were not selected.  However, discovering  that a design is inadequate after development can result in code that must  be rewritten two times, at a cost greater than analyzing the three  alternatives in the first place.  This same author notes that a good rule of  thumb is that each hour a developer spends reviewing project requirements  and architecture saves 3 to 10 hours later in the project.
Project oversight can also be a valuable contributor to successful projects.   Agency management can perform oversight functions, such as project  reviews and participating in key meetings, to help ensure that the project  will meet the agency needs.  Management can also use IV&V reviews to  provide it with assessments of the project’s software deliverables and  processes.  Although independent of the developer, IV&V is an integral part  of the overall development program and helps management mitigate risks.
section_title: Risk Management
paragraphs: Risk and opportunity are inextricably related.  Although developing  software is a risky endeavor, risk management processes should be used to  manage the project’s risks to acceptable levels by taking the actions  necessary to mitigate the adverse effects of significant risks before they  threaten the project’s success.  If a project does not effectively manage its  risks, then the risks will manage the project.
Risk management is a set of activities for identifying, analyzing, planning,  tracking, and controlling risks.  Risk management starts with identifying  the risks before they can become problems.  If this step is not performed  well, then the entire risk management process may become a useless  exercise since one cannot manage something that one does not know  anything about.  As with the other disciplined processes, risk management  is designed to eliminate the effects of undesirable events at the earliest  possible stage to avoid the costly consequences of rework.
After the risks are identified, they need to be analyzed so that they can be  better understood and decisions can be made about what actions, if any,  will be taken to address them.  Basically, this step includes activities such  as evaluating the impact on the project if the risk does occur, determining  the probability of the event occurring, and prioritizing the risk against the  other risks.  Once the risks are analyzed, a risk management plan is  developed that outlines the information known about the risks and the  actions, if any, which will be taken to mitigate those risks.  Risk monitoring  is a continuous process because both the risks and actions planned to  address identified risks need to be monitored, to ensure that the risks are  being properly controlled and that new risks are identified as early as  possible.  If the actions envisioned in the plan are not adequate, then  additional controls are needed to correct the deficiencies identified.
section_title: An Effective Requirements Management Process and the UFMS Functionality for CDC Had Not Been Fully Developed
paragraphs: HHS has not implemented an effective requirements management process  to reduce requirements-related defects to acceptable levels or to support  an effective testing process.  In reviewing HHS’ requirements management  process, we found (1) the requirements were not based on a concept of  operations that should provide the framework for the requirements  development process, (2) traceability was not maintained between various  requirements documents, and (3) the requirements contained in the  documents do not provide the necessary specificity.  Because of these  weaknesses, HHS does not have reasonable assurance that it has reduced  its requirements-related defects to acceptable levels.  Furthermore, the  requirements management problems we noted also prevent HHS from  developing an effective testing process until they are adequately addressed.   Although HHS has performed some functions that are similar to testing,  commonly referred to as conference room pilots, to help it determine  whether the system will meet its needs, these efforts have not provided the  quantitative data needed to provide reasonable assurance that the system  can provide the needed capability.  Therefore, HHS is depending on system  testing, which is not expected to start until less than 2 months before  system implementation, to provide it with the quantitative data needed to  determine whether the system will meet its needs.
subsections: section_title: Requirements Were Not Based on a Complete Concept of Operations
paragraphs: Requirements for UFMS were not based on a concept of operations. The  concept of operations—which contains a high-level description of the  operations that must be performed, who must perform them, and where  and how the operations will be carried out—provides the foundation on  which requirements definitions and the rest of the systems planning  process are built. Normally, a concept of operations is one of the first  documents to be produced during a disciplined development effort.   According to the IEEE Standards, a concept of operations is a user- oriented document that describes the characteristics of a proposed system  from the users’ viewpoint. Its development is a particularly critical step at  HHS because of the organizational complexity of its financial management  activities and the estimated 110 other systems HHS expects to interface  with UFMS.
In response to our requests for a UFMS concept of operations, HHS  officials provided its Financial Shared Services Study Concept of  Operation, dated April 30, 2004, that studied several approaches for HHS  management to consider for implementing shared services. While making  a decision on whether to operate in a shared services environment is  important because it will dictate such items as hardware, network, and  software needs, this study lacks many of the essential elements needed for  a concept of operations document that can be used to fully inform users  about the business processes that will be used by UFMS.  Without this  information, the document cannot serve as the foundation for HHS’  requirements management processes.
HHS management has stated that it plans to establish centers of excellence  for UFMS and has identified four functions as candidates to begin shared  services.  These functions are UFMS operations and maintenance,  customer service (call center), vendor payments, and e-travel.  HHS  management also decided that establishing a center of excellence for  operations and maintenance should begin right away.  Basically, this center  of excellence will perform such UFMS operations and maintenance  functions as maintaining the data tables in the UFMS database, managing  various periodic closings, and performing various user maintenance  functions as well as some security functions.  While HHS officials advised  us that they had selected PSC to operate the operations and maintenance  center of excellence, there is limited time to establish the center before  UFMS’ planned deployment date at CDC.  In addition, HHS has still not  identified (1) who will operate the other centers of excellence and the  location(s) performing these functions and (2) how these functions will be  performed.  To address these open issues, HHS has asked several HHS  operating divisions to submit business plans for operating a center of  excellence.
We also analyzed various other strategy and planning documents that are  expected to be used in developing UFMS. Like the Financial Shared  Services Study Concept of Operation, none of these other documents  individually or in their totality addressed all of the key elements of a  concept of operations. For example, operational policies and constraints  have not been addressed. Moreover, profiles of user classes describing  each class of user, including responsibilities, education, background, skill  level, activities, and modes of interaction with the current system, have not  been developed.  In fact, as of May 2004, HHS has been unable to get  agreement on all the standard processes that it will use.  For example,  when HHS attempted to develop a standard way of recording grant-related  information, the project team members were unable to get agreement  between the various operating divisions on how to develop crosscutting  codes that would have to be maintained at the departmental level.  Part of  the process of developing a concept of operations for an organization  includes describing how its day-to-day operations will be carried out to  meet mission needs.  The project team tasked with developing and  implementing a UFMS common accounting system attempted to develop  standardized processes that would be used for the UFMS project. They  held meetings with several different operating divisions to reach agreement  on how the processes should be structured. Unfortunately, an agreement  between the various parties could not be reached, and the decision on how  these processes would be defined was deferred for further discussion for at  least 6 months.
Since standardized processes could not be agreed upon at the outset,  additional requirements definition and validation activities must be  conducted later in the development cycle when they are more costly to  implement.  In addition, process modifications will affect all users,  including those who have been trained in and perform financial  management functions using the original process. These users may have to  undergo additional training and modify their existing understanding of how  the system performs a given function.
Because HHS has not developed a complete concept of operations,  requirements definition efforts have not had the benefit of documentation  that fully depicts how HHS’ financial system will operate, and so HHS  cannot ensure that all requirements for the system’s operations have been  defined. Without well-defined requirements, HHS cannot be certain that the  level of functionality that will be provided by UFMS is understood by the  project team and users and that the resulting system will provide the  expected functionality.
section_title: Approach to Requirements Management Does Not Provide Traceability or the Necessary Specificity
paragraphs: HHS has adopted an approach to requirements development that its  officials believe is suited to the acquisition and development of commercial  off-the-shelf software (COTS).  HHS officials have stated that the  requirements management process that we reviewed was adopted based on  their belief that for COTS development, they do not need to fully define the  UFMS requirements because UFMS is not a traditional system development  effort. Therefore, they adopted the following approach.
Define high-level requirements that could be used to guide the selection  and implementation of the system.
Understand how the COTS-based system meets the high-level  requirements defined for UFMS and how HHS must (1) modify its  existing processes to match the COTS processes or (2) identify the areas  or gaps requiring custom solutions.
Develop specific requirements for the areas that require custom  solutions and document those requirements in the requirements  repository tool as derived requirements.
HHS used a hierarchical approach to develop the specific requirements  from the high-level requirements used to acquire the system.  These high- level requirements and the related supporting documentation were  expected to help HHS identify the requirements that could not be satisfied  by the COTS product.  This approach includes using the high-level  requirements to (1) update the requirements through process design  workshops, which generated business processes; (2) establish initial  baseline requirements; (3) perform a fit/gap analysis; (4) develop gap  closure alternatives; and (5) create the final baseline requirements. The key  advantage in using such a hierarchy is that each step of the process builds  upon the previous one.  However, unidentified defects in one step migrate  to the subsequent steps where they are more costly to fix and thereby  increase the risk that the project will experience adverse effects on its  schedule, cost, and performance objectives.
HHS recognized that the high-level requirements associated with the COTS  processes are “by definition, insufficient to adequately define the required  behavior of the COTS based system.” However, HHS has stated that UFMS  will be able to demonstrate compliance with these requirements as well as  the requirements derived from high-level requirements associated with its  custom development through traditional testing approaches including  demonstrations and validations.
We agree with HHS’ position that requirement statements for COTS  products need to be more flexible and less specific before a product is  selected because of the low probability that any off-the-shelf product will  satisfy the detailed requirements of an organization like HHS. As HHS has  noted, COTS products are designed to meet the needs of the marketplace  not a specific organization. However, once the product is selected,  requirements must be defined at a level that allows the software to be  configured to fit the system under development and implemented to meet  the organization’s needs.  As noted elsewhere, on the basis of the  requirements we reviewed, HHS had not accomplished this objective.   Furthermore, we identified numerous instances in which each documented  requirement used to design and test the system was not traceable forward  to the business processes and therefore could not build upon the next step  in moving through the hierarchy.  This is commonly referred to as  traceability. Furthermore, the requirements (1) lacked the specific  information necessary to understand the required functionality that was to  be provided and (2) did not describe how to determine quantitatively,  through testing or other analysis, whether the systems would meet HHS’  needs.
One example showing that HHS did not adequately define a requirement  and maintain traceability through the various documents is an HHS  requirement regarding general ledger entries that was inadequately  defined. The high-level requirement stated that the system “shall define,  generate, and post compound general ledger debit and credit entries for a  single transaction.” The system was also expected to “accommodate at  least 10 debit and credit pairs,” but this information was not included in the  process document for the Create Recurring Journals process, to which the  requirement was tied. Therefore, someone implementing this functionality  from this process document would not know the number of debit and  credit pairs that must be supported. Furthermore, in April 2004, HHS  conducted a demonstration for the users to validate that this functionality  had been implemented. Although the demonstration documentation stated  that this requirement would be covered, none of the steps in the test  scripts actually demonstrated (1) how the system would process a general  ledger entry that consisted of 10 debit and credit pairs or (2) examples of  transactions that would require such entries. Since HHS has neither  demonstrated the functionality nor defined what entries need to be  supported, HHS does not yet have reasonable assurance the system can  address this requirement.
HHS expects that UFMS will be able to demonstrate compliance with the  HHS high-level requirements as well as the derived requirements  associated with its custom development through traditional testing  approaches including demonstrations and validations. However, we found  that as of May 2004, the necessary information to evaluate future testing  efforts had not been developed for many of the requirements that we  reviewed.
section_title: Conference Room Pilots Provide Little Confidence in Functionality
paragraphs: HHS has conducted two conference room pilots that were to help  determine and validate that the UFMS design and configuration meets HHS  functional requirements.  Such demonstrations, properly implemented,  could be used to reduce the risks associated with the requirements  management process weaknesses we identified.  However, based on our  review of the conference room pilots, the pilots did not (1) significantly  reduce the risks associated with requirements management processes  discussed above and (2) provide HHS with reasonable assurance that the  functionality needed by its users had been implemented in UFMS.
The first conference room pilot, held in August 2003, was designed to   (1) demonstrate the functionality present in the COTS system that HHS  believed could be used without modification and (2) identify any gaps in  the functionality provided by the base system. The second conference  room pilot in March and April 2004 was conducted to demonstrate the  functionality present in the system that should be available for the October  2004 implementation at CDC. This demonstration was expected to show  that the gaps in functionality identified in the first conference room pilot  had been addressed. Problems with these demonstrations include the  following:  The IV&V contractor noted that some of the test scripts involved a  number of requirements that were only partially addressed or not  addressed at all.  The IV&V contractor expressed concern that HHS  would not be mapping these requirements designated as “fits” to test  cases until system testing.  According to the IV&V contractor, if some of  the “fits” turn out to be “gaps” as a result of system testing, HHS may not  have enough time to provide a solution without compromising the  project schedule.
In our observations of the second conference room pilot held in March  and April 2004, we noted several cases in which the users were told that  the system’s approach to address a given issue had not yet been defined  but that the issue would be resolved before the system was deployed.  One such issue was the process for handling erroneous transactions  received from other systems.  For example, procedures to correct errors  in the processing of voucher batches had not been fully defined as of the  demonstration.  HHS officials stated that this would be addressed after  this second conference room pilot.  Additionally, during the  demonstration it was unclear how five-digit object class codes used in  the system will migrate to interfacing systems.  We observed that four- digit object class codes from certain grant systems were cross-walked to  five-digit object class codes when interfaced with the Oracle system.   However, it was not clear how the data would be converted back to four- digit object class codes to flow back to the grant systems.
The scripts used for the second conference room pilot did not maintain  traceability to the associated requirements.
In discussing our observations on the March and April 2004 conference  room pilot, HHS officials stated that the conference room pilots were not a  phase of formal testing but rather a structured working session (first  conference room pilot) and a demonstration (second conference room  pilot). However, they stated that the system test in August 2004—less than  2 months before the system is implemented at CDC—would verify that  UFMS satisfies all requirements and design constraints.
section_title: Comments from the Department of Health and Human Services
section_title: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Acknowledgments
paragraphs: Staff members who made key contributions to this report were Linda  Elmore, Amanda Gill, Rosa Harris, Maxine Hattery, Lisa Knight, Michael  LaForge, W. Stephen Lowrey, Meg Mills, David Powner, Gina Ross, Norma  Samuel, Yvonne Sanchez, Sandra Silzer, and William Thompson.
section_title: GAO’s Mission
paragraphs: The Government Accountability Office, the audit, evaluation and  investigative arm of Congress, exists to support Congress in meeting its  constitutional responsibilities and to help improve the performance and  accountability of the federal government for the American people. GAO  examines the use of public funds; evaluates federal programs and policies;  and provides analyses, recommendations, and other assistance to help  Congress make informed oversight, policy, and funding decisions. GAO’s  commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost  is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts  newly released reports, testimony, and correspondence on its Web site. To  have GAO e-mail you a list of newly posted products every afternoon, go to  www.gao.gov and select “Subscribe to Updates.”
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Congressional Relations
section_title: Public Affairs