DOD Business Systems Modernization: Longstanding Management and Oversight Weaknesses Continue to Put Investments at Risk
Released: 2003-03-31T00:00:00

Letter

DOD Is Investing Billions of Dollars Annually to Operate, Maintain, and Modernize Its Amalgamation of Business Systems

As part of its ongoing business systems modernization program, and consistent with our past recommendation, DOD has created an inventory of its existing and new business system investments. As of October 2002, DOD reported that this inventory consisted of 1,731 systems and system acquisition projects across DOD’s functional areas. In particular, DOD reported that it had 374 separate systems to support its civilian and military personnel function, 335 systems to perform finance and accounting functions, and 221 systems that support inventory management. Table 1 presents the composition of DOD business systems by functional area.

As we have previously reported, this systems environment is not the result of a systematic and coordinated departmentwide strategy, but rather is the product of unrelated, stovepiped initiatives to support a set of business operations that are nonstandard and duplicative across DOD components. Consequently, DOD’s amalgamation of systems is characterized by (1) multiple systems performing the same tasks; (2) the same data stored in multiple systems; (3) manual data entry and reentry into multiple systems; and (4) extensive data translations and interfaces, each of which increases costs and limits data integrity. Further, as we have reported, these systems do not produce reliable financial data to support managerial decisionmaking and ensure accountability. To the department’s credit, it recognizes the need to eliminate as many systems as possible and integrate and standardize those that remain. In fact, three of the four Defense Finance and Accounting Service (DFAS) projects that are the subject of the report being released today were collectively intended to reduce or eliminate all or part of 17 different systems that perform similar functions. For example, the Defense Procurement Payment System (DPPS) was intended to consolidate eight contract and vendor pay systems; the Defense Departmental Reporting System (DDRS) is intended to reduce the number of departmental financial reporting systems from seven to one; and the Defense Standard Disbursing System (DSDS) is intended to eliminate four different disbursing systems.

The fourth system, the DFAS Corporate Database/Corporate Warehouse (DCD/DCW), is intended to serve as the single DFAS data store, meaning it would contain all DOD financial information required by DFAS and be the central point for all shared data within DFAS.

For fiscal year 2003, DOD has requested approximately $26 billion in IT funding to support a wide range of military operations and business functions. This $26 billion is spread across the military services and defense agencies—each receiving its own allocation of IT funding. The $26 billion supports three categories of IT—business systems, business systems infrastructure, and national security systems—the first two of which comprise the earlier cited 1,731 new and existing business systems projects.

At last year’s hearing, DOD was asked about the makeup of its $26 billion in IT funding, including what amounts relate to business systems and related infrastructure, at which time answers were unavailable. As we are providing in the report being released today and as shown in figure 1, approximately $18 billion—about $5.2 billion for business systems and $12.8 billion for business systems infrastructure—relates to the operation, maintenance, and modernization of the 1,731 business systems that DOD reported having in October 2002. Figure 2 provides the allocation of DOD’s business systems modernization budget for fiscal year 2003 budget by component.

However, recognizing the need to modernize and making funds available are not sufficient for improving DOD’s current systems environment. Our research of successful modernization programs in public and private- sector organizations, as well as our reviews of these programs in various federal agencies, has identified a number of IT disciplines that are necessary for successful modernization. These disciplines include having and implementing (1) an enterprise architecture to guide and constrain systems investments; (2) an investment management process to ensure that systems are invested in incrementally, are aligned with the enterprise architecture, and are justified on the basis of cost, benefits, and risks; and (3) a project oversight process to ensure that project commitments are being met and that needed corrective action is taken. These institutionalized disciplines have been long missing at DOD, and their absence is a primary reason for the system environment described above.

Key Modernization Management Weaknesses Continue, But DOD Plans to Correct Them

The future of DOD’s business systems modernization is fraught with risk, in part because of longstanding and pervasive modernization management weaknesses. As we have reported, these weaknesses include (1) lack of an enterprise architecture; (2) inadequate institutional and project-level investment management processes; and (3) limited oversight of projects’ delivery of promised system capabilities and benefits on time and within budget. To DOD’s credit, it recognizes the need to address each of these weaknesses and has committed to doing so.

DOD Is Developing, But Still Is Without, a Departmentwide Enterprise Architecture

Effectively managing a large and complex endeavor requires, among other things, a well-defined and enforced blueprint for operational and technological change, commonly referred to as an enterprise architecture. Developing, maintaining, and using architectures is a leading practice in engineering both individual systems and entire enterprises. Government- wide requirements for having and using architectures to guide and constrain IT investment decisionmaking are also addressed in federal law and guidance. Our experience has shown that attempting a major systems modernization program without a complete and enforceable enterprise architecture results in systems that are duplicative, are not well integrated, are unnecessarily costly to maintain and interface, do not ensure basic financial accountability, and do not effectively optimize mission performance.

In May 2001, we reported that DOD had neither an enterprise architecture for its financial and financial-related business operations nor the management structure, processes, and controls in place to effectively develop and implement one. Further, we stated that DOD’s plans to continue spending billions of dollars on new and modified systems independently from one another, and outside the context of a departmental modernization blueprint, would result in more systems that are duplicative, noninteroperable, and unnecessarily costly to maintain and interface; moreover, they would not address longstanding financial management problems. To assist the department, we provided a set of recommendations on how DOD should approach developing its enterprise architecture.

In September 2002, the Secretary of Defense designated improving financial management operations (including such business areas as logistics, acquisition, and personnel management) as one of the department’s top 10 priorities. In addition, the Secretary established a program to develop an enterprise architecture, and DOD plans to have the architecture developed by May 2003. Subsequently, the National Defense Authorization Act for Fiscal Year 2003 directed DOD to develop, by May 1, 2003, an enterprise architecture, including a transition plan for its implementation. The act also defined the scope and content of the enterprise architecture and directed us to submit to congressional defense committees an assessment of DOD’s actions to develop the architecture and transition plan no later than 60 days after their approval. Finally, the act prohibited DOD from obligating more than $1 million on any financial systems improvement until the DOD comptroller makes a determination regarding the necessity or suitability of such an investment.

In our February 2003 report on DOD enterprise architecture efforts, we stated our support for the Secretary’s decision to develop the architecture and recognized that DOD’s architecture plans were challenging and ambitious. However, we also stated that despite taking a number of positive steps toward its architecture goals, such as establishing a program office responsible for managing the enterprise architecture, the department had yet to implement several key recommendations and certain leading practices for developing and implementing architectures. For example, DOD had yet to (1) establish the requisite architecture development governance structure needed to ensure that ownership of and accountability for the architecture is vested with senior leaders across the department; (2) develop and implement a strategy to effectively communicate the purpose and scope, approach to, and roles and responsibilities of stakeholders in developing the enterprise architecture; and (3) fully define and implement an independent quality assurance process. We concluded that not implementing these recommendations and practices increased DOD’s risk of developing an architecture that would be limited in scope, would be resisted by those responsible for implementing it, and would not support effective systems modernization. To assist the department, we made additional recommendations with which DOD agreed. We plan to continue reviewing DOD’s efforts to develop and implement this architecture pursuant to our mandate under the fiscal year 2003 defense authorization act.

DOD Has Yet to Implement Effective Investment Management Processes

The Clinger-Cohen Act, federal guidance, and recognized best practices provide a framework for organizations to follow to effectively manage their IT investments. Collectively, this framework addresses IT investment management at the institutional or corporate level, as well as the individual project or system level. The former involves having a single, corporate approach governing how the organization’s portfolio of IT investments is selected, controlled, and evaluated across its various components, including assuring that each investment is aligned with the organization’s enterprise architecture. The latter involves having a system/project-specific investment approach that provides for making investment decisions incrementally and ensuring that these decisions are economically justified on the basis of current and credible analyses.

Corporate investment management approach: DOD has yet to establish and implement an effective departmentwide approach to managing its business systems investment portfolio. In May 2001, we reported that DOD did not have a departmentwide IT investment management process through which to assure that its enterprise architecture, once developed, could be effectively implemented. We therefore recommended that DOD establish a system investment selection and control process that treats compliance with the architecture as an explicit condition to meet at key decision points in the system’s life cycle and that can be waived only if justified by compelling written analysis.

Subsequently, in February 2003, we reported that DOD had not yet established the necessary departmental investment management structure and process controls needed to adequately align ongoing investments with its architectural goals and direction. Instead, the department continued to allow its component organizations to make their own parochial investment decisions, following different approaches and criteria. In particular, DOD had not established and applied common investment criteria to its ongoing IT system projects using a hierarchy of investment review and funding decisionmaking bodies, each composed of representatives from across the department. DOD also had not yet conducted a comprehensive review of its ongoing IT investments to ensure that they were consistent with its architecture development efforts. We concluded that until it takes these steps, DOD will likely continue to lack effective control over the billions of dollars it is currently spending on IT projects. To address this, we recommended that DOD create a departmentwide investment review board with the responsibility and authority to (1) select and control all DOD financial management investments and (2) ensure that its investment decisions treat compliance with the financial management enterprise architecture as an explicit condition for investment approval that can be waived only if justified by a compelling written analysis. DOD concurred with our recommendations and is taking steps to address them.

Project/system-specific investment management: DOD has yet to ensure that its investments in all individual systems or projects are economically justified and that it is investing in each incrementally. In particular, none of the four DFAS projects addressed in the report being issued today had current and reliable economic justifications to demonstrate that they would produce value commensurate with the costs and risks being incurred. For example, we found that although DCD was initiated to contain all DOD financial data required by DFAS systems, planned DCD capabilities had since been drastically reduced. Despite this, DFAS planned to continue investing in DCD/DCW without having an economic justification showing whether its revised plans were cost effective. Moreover, DOD planned to continue investing in the three other projects even though none had current economic analyses that reflected material changes to costs, schedules, and/or expected benefits since the projects’ inception. For example, the economic analysis for DSDS had not been updated to reflect material changes in the project, such as changing the date for full operational capability from February 2003 to December 2005—a schedule change of almost 3 years that affected delivery of promised benefits. Similarly, the DPPS economic analysis had not been updated to recognize an estimated cost increase of $274 million and schedule slip of almost 4 years. After recently reviewing this project’s change in circumstances, the DOD Comptroller terminated DPPS after 7 years of effort and an investment of over $126 million, citing poor program performance and increasing costs. Table 2 highlights the four projects’ estimated cost increases and schedule delays.

Our work on other DOD projects has shown a similar absence of current and reliable economic justification for further system investment. For example, we reported that DOD’s ongoing and planned investment in its Standard Procurement System (SPS) was based on an outdated and unreliable economic analysis, and even this flawed analysis did not show that the system was cost beneficial, as defined. As a result, we recommended that investment in future releases or major enhancements to the system be made conditional on the department’s first demonstrating that the system was producing benefits that exceeded costs and that future investment decisions be made on the basis of complete and reliable economic justifications. DOD is currently in the process of addressing this recommendation.

Beyond not having current and reliable economic analyses for its projects, DOD has yet to adopt an incremental approach to economically justifying and investing in all system projects. For example, we have reported that although DOD had divided its multiyear, billion-dollar SPS project into a series of incremental releases, it had not treated each of these increments as a separate investment decision. Such an incremental approach to system investment helps to prevent discovering too late that a given project is not cost beneficial. However, rather than adopt an incremental approach to SPS investment management, the department chose to treat investment in SPS as one, monolithic investment decision, justified by a single, all-or-nothing economic analysis. This approach to investing in large systems, like SPS, has proven ineffective in other federal agencies, resulting in huge sums being invested in systems that do not provide commensurate value, and thus has been abandoned by successful organizations.

We also recently reported that while DOD’s Composite Health Care System II had been structured into a series of seven increments (releases), the department had not treated the releases to date as separate investment decisions supported by incremental economic justification. In response to our recommendations, DOD committed to changing its strategy for future releases to include economically justifying each release before investing in and verifying each release’s benefits and costs after deployment.

Effective Oversight of IT Projects Remains an Unanswered Challenge

The Clinger-Cohen Act of 1996 and federal guidance emphasize the need to ensure that IT projects are being implemented at acceptable costs and within reasonable and expected timeframes and that they are contributing to tangible, observable improvements in mission performance (that is, that projects are meeting the cost, schedule, and performance commitments upon which their approval was justified). They also emphasize the need to regularly determine each project’s progress toward expectations and commitments and to take appropriate action to address deviations.

Our work on specific DOD projects has shown that such oversight does not always occur, a multi-example case in point being the four DFAS accounting system projects that are the subject of our report being released today. For these four projects, oversight responsibility was shared by the DOD comptroller, DFAS, and the DOD chief information officer (CIO). However, these oversight authorities have not ensured, in each case, that the requisite analytical basis for making informed investment decisions was prepared. Moreover, they have not regularly monitored system progress toward expectations so that timely action could have been taken to correct deviations, even though each case had experienced significant cost increases and schedule delays (see table 2). Their respective oversight activities are summarized below: DOD Comptroller—Oversight responsibility for DFAS activities, including system investments, rests with the DOD Comptroller. However, DOD Comptroller officials were not only unaware of cost increases and schedule delays on these four projects, they also told us that they do not review DFAS system investments to ensure that they are meeting cost, schedule, and performance commitments because this is DFAS’s responsibility.

DFAS—This DOD agency has established an investment committee to, among other things, oversee its system investments. However, the committee could not provide us with any evidence demonstrating meaningful oversight of these four projects, nor could it provide us with any guidance describing the committee’s role, responsibilities, and authorities, and how it oversees projects.

DOD CIO—Oversight of the department’s “major” IT projects, of which two of the four DFAS projects (DCD/DCW and DPPS) qualify, is the responsibility of DOD’s CIO. However, this organization did not adequately fulfill this responsibility on either project because, according to DOD CIO officials, they have little practical authority in influencing component agency-funded IT projects.

Thus, the bad news is that these three oversight authorities have jointly permitted approximately $316 million to be spent on the four accounting system projects without knowing if material changes to the projects’ scopes, costs, benefits, and risks warranted continued investment. The good news is that the DOD Comptroller recently terminated one of the four (DPPS), thereby avoiding throwing good money after bad, and DOD has agreed to implement the recommendations contained in our report released today, which calls for DOD to demonstrate that the remaining three projects will produce benefits that exceed costs before further investing in each.

Our work on other DOD projects has shown similar voids in oversight. For example, we reported that SPS’s full implementation date slipped by 3 ½ years, with further delays expected, and the system’s life-cycle costs grew by 23 percent, from $3 billion to $3.7 billion. However, none of the oversight authorities responsible for this project, including the DOD CIO, had required that the economic analysis be updated to reflect these changes and thereby provide a basis for informed decisionmaking on the project’s future. To address this issue, we recommended, among other things, that the lines of oversight responsibility and accountability of the project be clarified and that further investment in SPS be limited until such investment could be justified. DOD has taken steps to address some of our recommendations. For example, it has clarified organizational accountability and responsibility for the program. However, much remains to be done before the department will be able to make informed, data- driven decisions about whether further investment in the system is justified.

Our Recommendations Provide a Roadmap for Improving Management of Business Systems Modernization

We have made numerous recommendations to DOD that collectively provide a valuable roadmap for improvement as the department attempts to create the management infrastructure needed to effectively undertake a massive business systems modernization program. This collection of recommendations is not without precedent, as we have provided similar ones to other federal agencies, such as the Federal Aviation Administration, the Internal Revenue Service, and the former U.S. Customs Service, to aid them in building their respective capacities for managing modernization programs. In cases where these recommendations have been implemented properly, we have observed improved modernization management and accountability.

Our framework for DOD provides for developing a well-defined and enforceable DOD-wide enterprise architecture to guide and constrain the department’s business system investments, including specific recommendations for successfully accomplishing this, such as creating an enterprise architecture executive committee whose members are singularly and collectively responsible and accountable for delivery and approval of the architecture and a proactive enterprise architecture marketing and communication program to facilitate stakeholder understanding, buy-in, and commitment to the architecture.

Our recommendations also provide for establishing a DOD-wide investment decisionmaking structure that consists of a hierarchy of investment boards that are responsible for ensuring that projects meet defined threshold criteria and for reviewing and deciding on projects’ futures on the basis of a standard set of investment criteria, two of which are alignment with the enterprise architecture and return on investment.

In addition, our recommendations include ensuring that return on investment is analytically supported by current and reliable economic analyses showing that benefits are commensurate with costs and risks, and that these analyses and associated investment decisions cover incremental parts of each system investment, rather than treating the system as one, all-or-nothing, monolithic pursuit. Further, our recommendations provide clear and explicit lines of accountability for project oversight and continuous monitoring and reporting of progress against commitments to ensure that promised system capabilities and benefits are being delivered on time and within budget.