Financial Management Systems: HHS Faces Many Challenges in Implementing Its Unified Financial Management System
Released: 2004-09-30T00:00:00

Letter

Disciplined Processes Are Key to Successful System Implementation

The ability to produce the information needed to efficiently and effectively manage the day-to-day operations of the federal government and provide accountability to taxpayers and the Congress has been a long-standing challenge for federal agencies. To help address this challenge, many agencies are in the process of replacing their core financial systems as part of their financial management system improvement efforts. Although the implementation of any major system is not a risk-free proposition, organizations that follow and effectively implement disciplined processes can reduce these risks to acceptable levels. The use of the term acceptable levels acknowledges the fact that any systems acquisition has risks and will suffer the adverse consequences associated with defects. However, effective implementation of the disciplined processes reduces the potential for risks to occur and helps prevent those that do occur from having any significant adverse impact on the cost, timeliness, and performance of the project. A disciplined software development and acquisition process can maximize the likelihood of achieving the intended results (performance) within established resources (costs) on schedule.

Although there is no standard set of practices that will ever guarantee success, several organizations, such as the Software Engineering Institute (SEI) and the Institute of Electrical and Electronic Engineers (IEEE), as well as individual experts have identified and developed the types of policies, procedures, and practices that have been demonstrated to reduce development time and enhance effectiveness. The key to having a disciplined system development effort is to have disciplined processes in multiple areas, including project planning and management, requirements management, configuration management, risk management, quality assurance, and testing. Effective processes should be implemented in each of these areas throughout the project life cycle because change is constant. Effectively implementing the disciplined processes necessary to reduce project risks to acceptable levels is hard to achieve because a project must effectively implement several best practices, and inadequate implementation of any one practice may significantly reduce or even eliminate the positive benefits of the others.

Successfully acquiring and implementing a new financial management system requires a process that starts with a clear definition of the organization’s mission and strategic objectives and ends with a system that meets specific information needs. We have seen many system efforts fail because agencies started with a general need, such as improving financial management, but did not define in precise terms (1) the specific problems they were trying to solve, (2) what their operational needs were, and (3) what specific information requirements flowed from these operational needs. Instead, they plunged into the acquisition and implementation process in the belief that these specifics would somehow be defined along the way. The typical result was that systems were delivered well past anticipated milestones; failed to perform as expected; and, accordingly, were overbudget because of required costly modifications.

Undisciplined projects typically show a great deal of productive work at the beginning of the project, but the rework associated with defects begins to consume more and more resources. In response, processes are adopted in the hopes of managing what later turns out to have been unproductive work. Generally, these processes are “too little, too late” because sufficient foundations for building the systems were not established or not established adequately. Experience has shown that projects for which disciplined processes are not implemented at the beginning are forced to implement them later when it takes more time and the processes are less effective.

A major consumer of project resources in undisciplined efforts is rework (also known as thrashing). Rework occurs when the original work has defects or is no longer needed because of changes in project direction. Disciplined organizations focus their efforts on reducing the amount of rework because it is expensive. Experts have reported that fixing a defect during the testing phase costs anywhere from 10 to 100 times the cost of fixing it during the design or requirements phase. Projects that are unable to successfully address their rework will eventually only be spending their time on rework and the associated processes rather than on productive work. In other words, the project will continually find itself reworking items.

HHS Had Not Effectively Implemented Disciplined Processes, Information Technology Management Practices, and Human Capital Planning

We found that HHS had adopted some best practices in its development of UFMS. The project had support from senior officials and oversight by independent experts, commonly called independent verification and validation (IV&V) contractors. We also view HHS’ decision to follow a phased implementation to be a sound approach.

However, at the time of our review, HHS had not effectively implemented several disciplined processes essential to reducing risks to acceptable levels and therefore key to a project’s success, and had adopted other practices that put the project at unnecessary risk. HHS officials told us that they had carefully considered the risks associated with implementing UFMS and that they had put in place strategies to manage these risks and to allow the project to meet its schedule within budget. However, we found that HHS had focused on meeting its schedule to implement the first phase of the new system at the Centers for Disease Control and Prevention (CDC) in October 2004, to the detriment of disciplined processes and thus had introduced unnecessary risks that may compromise the system’s cost, schedule, and performance. We would now like to briefly highlight a few of the key disciplined processes that HHS had not fully implemented at the time of our review. These matters are discussed in detail in our report.

Requirements management. Requirements are the specifications that system developers and program managers use to design, develop, and acquire a system. Requirements management deficiencies have historically been a root cause of systems that do not meet their cost, schedule, and performance objectives. Effective requirements management practices are essential for ensuring the intended functionality will be included in the system and are the foundation for testing. We found significant problems in HHS’ requirements management process and that HHS had not developed requirements that were clear and unambiguous.

Testing. Testing is the process of executing a program with the intent of finding errors. Without adequate testing, an organization (1) is taking a significant risk that substantial defects will not be detected until after the system is implemented and (2) does not have reasonable assurance that new or modified systems will function as planned. We found that HHS faced challenges in implementing a disciplined testing program, because, first of all, it did not have an effective requirements management system that produced clear, unambiguous requirements upon which to build its testing efforts. In addition, HHS had scheduled its testing activities, including those for converting data from existing systems to UFMS, late in the implementation cycle leaving little time to correct defects identified before the scheduled deployment in October 2004.

Project management and oversight using quantitative measures. We found that HHS did not have quantitative metrics that allowed it to fully understand (1) its capability to manage the entire UFMS effort; (2) how problems in its management processes would affect the UFMS cost, schedule, and performance objectives; and (3) the corrective actions needed to reduce the risks associated with the problems identified with its processes. Such quantitative measures are essential for adequate project management oversight. Without such information, HHS management can only focus on the project schedule and whether activities have occurred as planned, not on whether the substance of the activities achieved their system development objectives. As we note in our report, this is not an effective practice.

Risk management. We noted that HHS routinely closed its identified risks on the premise that they were being addressed. Risk management is a continuous process to identify, monitor, and mitigate risks to ensure that the risks are being properly controlled and that new risks are identified and resolved as early as possible. An effective risk management process is designed to mitigate the effects of undesirable events at the earliest possible stage to avoid costly consequences.

In addition, HHS’ effectiveness in managing the processes associated with its data conversion and UFMS interfaces will be critical to the success of this project. For example, CDC’s ability to convert data from its existing systems to the new system will be crucial to helping ensure that UFMS will provide the kind of data needed to manage CDC’s programs and operations. The adage “garbage in garbage out” best describes the adverse impact. Furthermore, HHS expects that once UFMS is fully deployed, the system will need to interface with about 110 other systems, of which about 30 system interfaces are needed for the CDC deployment. Proper implementation of the interfaces between UFMS and the other systems it receives data from and sends data to is essential for providing data integrity and ensuring that UFMS will operate as it should and provide the information needed to help manage its programs and operations.

Compounding these UFMS-specific problems are departmentwide weaknesses we have previously reported in information technology (IT) investment management, enterprise architecture, and information security. Specifically, HHS had not established the IT management processes needed to provide UFMS with a solid foundation for development and operation. Such IT weaknesses increase the risk that UFMS will not achieve planned results within the estimated budget and schedule. We will now highlight the IT management weaknesses that HHS must overcome: Investment management. HHS had weaknesses in the processes it uses to select and control its IT investments. Among the weaknesses we previously identified, HHS had not (1) established procedures for the development, documentation, and review of IT investments by its review boards or (2) documented policies and procedures for aligning and coordinating investment decision making among its investment management boards. Until HHS addresses weaknesses in its selection or control processes, IT projects like UFMS will face an increased likelihood that the projects will not be completed on schedule and within estimated costs.

Enterprise architecture. While HHS is making progress in developing an enterprise architecture that incorporates UFMS as a central component, most of the planning and development of the UFMS IT investment had occurred without the guidance of an established enterprise architecture. An enterprise architecture is an organizational blueprint that defines how an entity operates today and how it intends to operate in the future and invest in technology to transition to this future state. Our experience with other federal agencies has shown that projects developed without the constraints of an established enterprise architecture are at risk of being duplicative, not well integrated, unnecessarily costly to maintain and interface, and ineffective in supporting missions.

Information security. HHS had not yet fully implemented the key elements of a comprehensive security management program and had significant and pervasive weaknesses in its information security controls. The primary objectives of information security controls are to safeguard data, protect computer application programs, prevent unauthorized access to system software, and ensure continued operations. Without adequate security controls, UFMS cannot provide reasonable assurance that the system is protected from loss due to errors, fraud and other illegal acts, disasters, and incidents that cause systems to be unavailable.

Finally, we believe it is essential that an agency take the necessary steps to ensure that it has the human capital capacity to design, implement, and operate a financial management system. We found that staff shortages and limited strategic workforce planning have resulted in the project not having the resources needed to effectively design, implement, and operate UFMS. We identified the following weaknesses: Staffing. HHS had not filled positions in the UFMS Program Management Office that were identified as needed. Proper human capital planning includes identifying the workforce size, skills mix, and deployment needed for mission accomplishment and to create strategies to fill the gaps. Scarce resources could significantly jeopardize the project’s success and have led to several key UFMS deliverables being significantly behind schedule.

Strategic workforce planning. HHS had not yet fully developed key workforce planning tools, such as the CDC skills gap analysis, to help transform its workforce so that it can effectively use UFMS. Strategic workforce planning focuses on developing long-term strategies for acquiring, developing, and retaining an organization’s total workforce (including full- and part-time federal staff and contractors) to meet the needs of the future. Strategic workforce planning is essential for achieving the mission and goals of the UFMS project. By not identifying staff with the requisite skills to operate such a system and by not identifying gaps in needed skills and filling them, HHS has not optimized its chances for the successful implementation and operation of UFMS.

Action Is Needed to Mitigate Risk

To address the range of problems we have just highlighted, our report includes 34 recommendations that focus on mitigating the risks associated with this project. We made 8 recommendations related to the initial deployment of UFMS at CDC that are specifically tied to implementing critical disciplined processes. In addition, we recommended that until these 8 recommendations are substantially addressed, HHS delay the initial deployment. The remaining 25 recommendations were centered on developing an appropriate foundation for moving forward and focused on (1) disciplined processes, (2) IT security controls, and (3) human capital issues.

In its September 7, 2004, response to a draft of our report, HHS disagreed regarding management of the project and whether disciplined processes were being followed. In its comments, HHS characterized the risk in its approach as the result, not of a lack of disciplined processes, but of an aggressive project schedule. From our perspective, this project demonstrated the classic symptoms of a schedule-driven effort for which key processes had been omitted or shortcutted, thereby unnecessarily increasing risk. As we mentioned at the outset of our testimony, this is a multiyear project with an estimated completion date in fiscal year 2007 and a total estimated cost of over $700 million. With a project of this magnitude and importance, we stand by our position that it is crucial for the project to adhere to disciplined processes that represent best practices. Therefore, in order to mitigate its risk to an acceptable level, we continue to believe it is essential for HHS to adopt and effectively implement our 34 recommendations.

In commenting on our draft report, HHS also indicated that actions had either been taken, were under way, or were planned that address a number of our recommendations. In addition, HHS subsequently contacted us on September 23, 2004, to let us know that it had decided to delay the implementation of a significant amount of functionality associated with the CDC deployment from October 2004 until April 2005 in order to address the issues that had been identified with the project. HHS also provided us with copies of IV&V reports and other documentation that had been developed since our review. Delaying implementation of significant functionality at CDC is a positive step forward given the risks associated with the project. This delay, by itself, will not reduce the risk to an acceptable level, but will give HHS a chance to implement the disciplined processes needed to do so.

HHS will face a number of challenges in the upcoming 6 months to address the weaknesses in its management of the project that were discussed in our report. At a high level, the key challenge will be to implement an event driven project based on effectively implemented disciplined processes, rather than a schedule-driven project. It will be critical as well to address the problems noted in the IV&V reports that were issued during and subsequent to our review. If the past is prologue, taking the time to adhere to disciplined processes will pay dividends in the long term.

Mr. Chairman, this concludes our statement. We would be pleased to answer any questions you or other members of the Subcommittee may have at this time.

Contacts and Acknowledgments

For further information about this statement, please contact Jeffrey C. Steinhoff, Managing Director, Financial Management and Assurance, who may be reached at (202) 512-2600 or by e-mail at steinhoffj@gao.gov, or Keith A. Rhodes, Chief Technologist, Applied Research and Methodology Center for Engineering and Technology, who may be reached at (202) 512- 6412 or by e-mail at rhodesk@gao.gov. Other key contributors to this testimony include Kay Daly, Michael LaForge, Chris Martin, and Mel Mench.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.