Defense IRM: Poor Implementation of Management Controls Has Put Migration Strategy at Risk
Published: 1997-10-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO provided information on the Department of Defense's (DOD) Corporate Information Management (CIM) initiative, focusing on: (1) the number and cost of systems designated for migration, the number of legacy systems already terminated or scheduled for termination, and savings resulting from terminations of legacy systems; and (2) whether DOD's management control and oversight processes for migration systems are ensuring that the investment in CIM-related technology is economically sound and in compliance with its technical and data standards.
section_title: What GAO Found
paragraphs: GAO noted that: (1) from fiscal years 1995 through 2000, DOD plans to spend at least $18 billion on migration; (2) DOD has selected 363 migration systems and has targeted 1,938 legacy systems for potential termination; (3) despite this substantial investment, DOD did not adhere to decisionmaking and oversight processes it established to ensure that the economical and technical risks associated with migration projects have been mitigated; (4) DOD's primary corporate-level control mechanism for ensuring that sound management and development practices were followed for each migration investment has not been effective; (5) DOD does not have assurance that the migration systems being developed will help achieve its technology goals and that sound business decisions were made in selecting the systems; (6) DOD's traditional acquisition oversight processes also did not support the migration effort because they have not fully ensured that economic analyses for migration projects are prepared and reviewed and that the systems comply with technical and data standards; (7) had there been more rigorous attention to established oversight procedures and sound business practices, DOD might well have avoided the migration problems GAO identified in previous reviews, which unnecessarily cost it hundreds of millions of dollars; (8) in implementing the migration strategy, DOD did not ensure that it had adequate departmentwide visibility over status, costs, and progress; (9) as a result, it could not provide accurate and reliable information, as requested, on the number and cost of systems designated for migration, and the numbers of systems terminated and scheduled to be terminated; (10) DOD has not been able to convincingly demonstrate whether the migration strategy has been successful, and it has not been able to provide the Congress with accurate and reliable information needed for decisionmaking purposes; (11) DOD has taken a positive step to turn around its information technology investment process as part of its implementation of the Clinger-Cohen Act of 1996; (12) the Secretary of Defense outlined his expectations for improvements in management processes and information resources related to information technology; and (13) these expectations incorporate some of the most important elements of the Clinger-Cohen Act and are an excellent starting point for bringing meaningful change to the current information technology management process.

section_title: Letter
paragraphs: As you know, in 1989, the Department of Defense started its Corporate Information Management (CIM) initiative in an effort to save billions of dollars by streamlining operations and deploying standard information systems to support common business operations. A key part of this initiative is Defense’s migration effort, which involves replacing its functionally duplicative and inefficient automated information systems with the Department’s best existing information systems. Concerned that the billions of dollars projected to be spent on CIM-related technology efforts are at high risk, you asked that we provide information on the status and progress of Defense’s migration effort and assess whether Defense has effective controls in place to manage and oversee the initiative. This report also highlights the implications of our findings for Defense as it responds to investment management requirements of the Clinger-Cohen Act of 1996.
We are sending copies of this report to the Chairman of the Senate Committee on Governmental Affairs and to the Chairmen and Ranking Minority Members of the Senate Committee on Armed Services; Subcommittee on Defense, Senate Committee on Appropriations; House Committee on National Security; Subcommittee on National Security, House Committee on Appropriations; and Senate and House Committees on the Budget; the Secretary of Defense; the Acting Assistant Secretary of Defense for Command, Control, Communications and Intelligence; the Acting Under Secretary of Defense (Comptroller); and the Director, Office of Management and Budget. Copies will also be made available to others upon request.
If you have any questions about this report, please call Mickey McDermott, Assistant Director, at (202) 512-6240. Other major contributors to this report are listed in appendix V.
subsections: section_title: Introduction
subsections: section_title: Background
paragraphs: The Department of Defense (DOD) began its Corporate Information Management initiative (CIM) in October 1989 to help meet the challenge of effectively managing its diverse operations as it downsized its forces and activities. At the time, the Department believed that the thousands of systems and numerous administrative and mission-related processessupporting DOD functions were redundant and inefficient and that they should be standardized and made more efficient. To address these problems, Defense planned to simplify and improve business processes; centralize responsibility and authority in functional areas, such as finance, personnel, communications, health affairs, logistics, command and control, and intelligence; and develop an integrated communications and data processing infrastructure based on departmentwide standards.
From the outset, DOD recognized that implementing CIM would be difficult because the basic tenet of the initiative—managing and implementing business improvements and building corporate systems along functional lines—represented a major shift in the way Defense traditionally did business. Whereas each military and Defense agency had historically managed its own business functions and information systems, CIM called on senior functional officials, known as Principal Staff Assistants (PSAs),together with their Defense component counterparts to be responsible for implementing improvements and information systems within the Department’s business functions across service and agency lines.
As CIM was implemented, Defense emphasized two ways of achieving process improvements and addressing problems associated with its disparate and stovepiped information technology environment. The first was to improve, or reengineer, business processes first and then apply technology to the new processes. The second involved selecting the best DOD information systems from pools of existing, or legacy, systems that provide similar automated support services and eventually replacing the duplicative systems with the best systems. A few years into the CIM effort, this became known as migration. Defense believed that if done properly, migration could cut costs associated with developing and maintaining disparate systems supporting the same functions. It also believed that migration would help to standardize business processes and allow Defense to achieve savings that could be put to better use in advancing warfighting capabilities. Figure 1.1 illustrates how migration would work in one functional area.
For its finance and accounting functions, Defense's goal is to reduce the number of systems from the 324 that the services and agencies operated in 1991 to 32 systems.
As of the end of fiscal year 1996, the Defense Finance and Accounting Service's strategic plan stated it had reduced the number of finance and accounting systems to 217.
The migration strategy for accounting and finance is to  (1) identify a single migration system for each of the Department's finance and accounting functions, such as civilian pay, (2) standardize the procedures and practices used to perform these functions, and (3) migrate from existing service-unique systems to the migration systems.
For some of these functions, such as general fund accounting, Defense established a two-pronged approach: (1) migrate initially to a single interim system or set of systems within (2) migrate from the service systems to a departmentwide system or set of systems.
section_title: Migration Has Become the Focal Point for CIM
paragraphs: While Defense began CIM emphasizing the need to improve, or reengineer, processes before applying technology, it ended up placing more priority on obtaining quick savings through an accelerated migration strategy, which began in October 1993. This strategy called for selection of migration systems in 6 months and departmentwide transition to selected systems in the following 3 years. Defense believed that setting tight time frames for migration with some potential slippage would allow it to “harvest the low-hanging fruit” of potential savings before reengineering. By default, this meant that the increased emphasis on migration would postpone the dramatic gains that could be achieved through reengineering.
The risks involved with postponing reengineering efforts were significant. Reengineering identifies, analyzes, and redesigns an organization’s core business processes, aiming to achieve dramatic improvements in critical areas of performance, such as cost, quality, service, and speed. It focuses on redesigning the business process as a whole in order to achieve the greatest possible benefits to an organization and its customers. Migration, on the other hand, focuses on standardizing existing processes and information systems. This can yield some benefits, such as reducing the need to maintain disparate systems that support the same functions, but seldom yields dramatic improvements. In addition, if the process is inefficient or outmoded, migration only serves to perpetuate a bad process. Finally, choosing migration before reengineering may cause future reengineering efforts to be more difficult by entrenching inefficient and ineffective work processes.
We raised these concerns both before and after Defense decided to embark on the accelerated migration strategy. We believed that the shift in emphasis toward migration seriously endangered CIM’s chances for success. For this reason, and because CIM-related technology investments, which are expected to total billions of dollars each year, are vulnerable to waste and mismanagement, we designated CIM as a high-risk government information technology initiative. Nevertheless, Defense proceeded to concentrate on migration even in areas where it had recognized that there was a significant need to change business processes. One such area was transportation. We reported in 1996 that even though Defense recognized that its military transportation processes were fragmented, outdated, inefficient, and costly, it focused on technology solutions rather than the need to identify and correct the root causes of its transportation problems.
Because reengineering took a backseat to migration in other business areas as well, CIM has never been able to achieve the level of cost savings and process improvements originally intended. A recent reportconducted by RAND, a consulting organization, notes that “The CIM effort is today widely viewed as a failure in most quarters of the DOD. It has not resulted in either significant process reengineering or visible savings in the hardware and software required to support all the varied information systems in the Defense infrastructure.”
Defense is still planning to invest at least $18 billion in its migration effort from fiscal years 1995 through 2000. Because this investment is significant, we were asked for information on the status and progress of migration and whether Defense has significant controls in place to manage and oversee the effort.
subsections: section_title: Status of the Migration Effort
paragraphs: Defense reported that it has selected 363 systems for migration. Forty-nine of the 363 migration systems are large-scale, or major, systems that are expected to cost at least $13 billion—or about 72 percent of the total $18 billion cost estimate—to develop, deploy, and maintain over fiscal years 1995 through 2000. Defense has identified 1,938 legacy systems for potential termination. As of April 1996, its Defense Integration Support Tools (DIST) database showed that 281 had already been terminated, 886 were scheduled for termination by the year 2000, and 771 more were identified for potential elimination after the year 2000.
Table 1.1 provides the number of systems selected by each functional area, the number of legacy systems identified for potential termination, and DOD’s estimated costs of developing, deploying, and maintaining the migration systems. We excluded costs for 27 systems for which DOD collected costs because the systems were classified. And costs for the remaining 121 of 363 systems were not included in the table because the Office of the ASD C3I had not collected the costs for these systems. Table 1.2 provides the name, status, and cost for each of the 49 major systems.
section_title: Objectives, Scope, and Methodology
paragraphs: We were asked (1) for information on the number and cost of systems designated for migration, the number of legacy systems already terminated or scheduled for termination, and savings resulting from terminations of legacy systems and (2) whether Defense’s management control and oversight processes for migration systems are ensuring that the investments are economically sound and in compliance with Defense’s technical and data standards.
To assess the status of Defense’s overall migration strategy and obtain available information on the number and cost of systems designated for migration, the number of legacy systems already terminated and scheduled for future termination, and savings resulting from terminations of legacy systems, we analyzed a Defense April 1996 report to the Congress containing information on the schedule, cost, and status of the migration systems. Defense prepared this report in response to a requirement in Section 366 of the National Defense Authorization Act for fiscal year 1996. We also obtained and analyzed a copy of the DIST database as of April 1996 that Defense used to develop the report to the Congress, as well as to provide its own senior managers information on migration system selections and legacy system terminations. Additionally, we reviewed—but did not independently verify—cost information that Defense reported to Congress as part of the April 1996 Section 366 report. We also interviewed senior DOD officials to determine if additional cost and performance information on migration systems existed at the Office of the Secretary of Defense (OSD) level.
We assessed the reliability of the data presented to the Congress in two ways. First, we identified missing and conflicting information in the April 1996 Section 366 report to the Congress and requested that the ASD C3I staff to clarify conflicting information. Second, we compared the data reported to the Congress for three functional activities (business areas)—Transportation, Civilian Personnel, and Clinical Health—against data provided to us by the functional area managers at these activities. We then established and analyzed a database containing schedule and available cost information for the migration and legacy systems and modified the database to reflect updated schedule, costs, and other descriptive data Defense provided to us during the course of the audit.
To assess Defense’s management control and oversight processes for migration systems to determine whether the investments are economically justified and comply with Defense technical and data standards, we reviewed the department-level management and oversight processes for approving the PSAs’ migration system selections and for overseeing major migration systems’ acquisitions by the Major Automated Information System Review Council (MAISRC). Our review did not focus on the process by which the PSAs select, develop, and manage their migration systems.
Nor did we examine acquisition review processes within the individual functional areas employed for nonmajor system projects.
To analyze Defense’s oversight processes for reviewing and approving the PSAs’ migration system selections, we reviewed department-level approval of the PSAs’ selections. We obtained a list of migration systems that PSAs had selected for their functional areas. We then visited the Office of the ASD C3I and the Defense Information Systems Agency (DISA) and reviewed the business case analysis documentation, technical analysis documentation, and other documentation provided to those offices for department-level approval of the PSA’s migration and interim system selections. We also interviewed officials from the Office of the ASD C3I, DISA, and the offices of selected PSAs regarding the documentation supporting the selections.
To review Defense’s acquisition oversight processes for major migration systems, we first identified the major migration systems, as defined by Defense regulations. We then obtained and analyzed the progress reports and other documentation provided to MAISRC for those systems. We also interviewed Defense representatives and officials responsible for information systems oversight in the Office of the ASD C3I and offices of DISA, the Defense Acquisition Board (DAB), MAISRC, and Program Analysis and Evaluation (PA&E). We also interviewed representatives of the program offices or oversight offices for selected systems for which oversight had been delegated by MAISRC. Through document reviews and interviews, we determined whether economic analyses for major migration systems had been independently reviewed and if so, whether the reviews had identified problems in the analyses. We also determined whether DOD guidance existed on preparing economic analyses for information systems. A GAO economist met with PA&E analysts, reviewed problems that were identified with economic analyses, and verified the validity of the problems reported.
Additionally, we determined whether MAISRC had information on alternative analyses performed for major migration system selections. Lastly, we determined whether MAISRC has sufficient oversight information to ensure that major migration systems are complying with applicable technical and data standards that are necessary to achieve an interoperable systems environment.
We reviewed Defense’s policies and guidance for migration systems to ensure that information technology is acquired, managed, and used in the most efficient and effective manner. Our assessment included analyzing the National Defense Authorization Acts, Committee reports, and Conference reports for fiscal years 1993 through 1997; Defense Office of Inspector General reports; our prior studies of CIM and the migration system strategy, other available evaluations of the CIM and the migration strategy, and existing legislation affecting these efforts. The major studies reviewed included studies by RAND and the Defense Science Board.Pertinent existing legislation includes the Clinger-Cohen Act, the Government Performance and Results Act, the Paperwork Reduction Act, and the Chief Financial Officers Act.
We conducted our review from August 1996 through July 1997 in accordance with generally accepted government auditing standards. We requested comments on a draft of this report from the Department of Defense. The Acting Secretary for Command, Control, Communications and Intelligence provided us with written comments. These comments are discussed in chapter 4 and reprinted in appendix I.
section_title: Management Control Processes Over Individual Migration Efforts Have Broken Down
paragraphs: Embarking on the migration strategy was an extremely risky endeavor for the Department of Defense. First, in developing standard systems within functional areas, the Department had to carefully consider complex technical issues, such as interfacing migration systems with other systems and contending with nonstandard data formats and definitions. For Defense, such complexities were compounded by its sheer size and the numbers of disparate systems. We believe an even tougher obstacle facing Defense, however, was the prevailing culture, which was based on decentralized department organizational structure, nonstandard processes and procedures. In general, each military service and Defense agency has historically managed its own business functions and information technology projects, whereas CIM and migration required business improvements and information systems to be managed on a Departmentwide, or corporate basis.
Therefore, to ensure the strategy’s success, it was vital for Defense to establish an effective decision-making and oversight environment for making migration investments. It would need controls and processes that ensured inefficient administrative and mission-related work processes were modernized before significant technology investments were made to support them. It would also need controls that ensured that the migration projects themselves were effectively managed as investments so that the Department could target resources and attention to priority areas and stop those projects that failed to meet their goals. Finally, Defense needed life cycle management controls that ensured, on a system-by-system basis, that sound management and development practices were followed.
However, Defense’s primary corporate-level control mechanism for ensuring that sound management and development practices were followed for each migration investment has not been effective. This control requires that all selections to be submitted for approval by the Assistant Secretary of Defense for Command, Control, Communications and Intelligence (ASD C3I), who is also the Department’s Chief Information Officer (CIO). In approving the systems, the Assistant Secretary is to review data, technical, and programmatic factors relating to the selection. We found, however, that most migration selections never came in for this approval and those that did were approved in the absence of critical technical and programmatic supporting documents. (A description of the migration decision process is provided in appendix II.)
Because this control was playing a marginal role in ensuring the success of migration, we assessed whether Defense’s acquisition oversight processes—which are designed to augment management oversight for the Department’s most expensive information systems—were helping to ensure that the major migration investments were economically and technically sound. These processes have also broken down where migration is concerned. Had there been more rigorous attention to oversight in both areas, Defense may well have avoided the migration problems we identified in previous reviews that cost the Department hundreds of millions of dollars.
In implementing the migration strategy, Defense did not ensure that it had adequate visibility over status, costs, and progress. As a result, it has not been able to (1) demonstrate whether the migration strategy has been successful, (2) provide the Congress with accurate and reliable information needed for oversight purposes, and (3) provide its own decisionmakers at the headquarters level with information needed to oversee the strategy.
subsections: section_title: Most Systems Not Approved by Defense’s Senior Technology Manager
paragraphs: After selecting a migration system, the Principal Staff Assistant (PSA)responsible for a functional area is to submit the system for approval by the Assistant Secretary of Defense for Command, Control, Communications and Intelligence—the Senior Information Management Official and Chief Information Officer in DOD. In his memorandum setting forth the requirements for selecting and reviewing migration systems, the ASD C3I stated that in approving the selections, he would consider data, technical, and programmatic factors. For example, the ASD C3I review would include such issues as whether the migration systems will lend themselves to data sharing and whether they conform to DOD’s technical standards—which act as a set of “building codes” for constructing systems to ensure they will be compatible with its information infrastructure and technically interoperable with each other. This approval is important because, as the Department’s Chief Information Officer, the Assistant Secretary needs to ensure that the migration systems help facilitate the sharing of information across service and agency lines.
However, both the PSAs and the ASD C3I did not adhere to this oversight process. First, 245 selections, or about 67 percent, were never submitted for this oversight even though all 363 system selections should have  been.4, 5 Of the 49 major, or large scale, migration systems, only 16 were submitted for this review. Thus, for the bulk of migration systems, the Assistant Secretary was unable to ensure that Defense technical standards would be met and that the best system development practices were being followed.
Nine of the systems that were submitted for approval were grandfathered into approval because development for them was well underway at the time the accelerated migration system strategy began in 1993.
Forty-two of the systems that were approved by the ASD C3I as migration systems did not follow DOD’s migration system approval process described in appendix II. Instead, the ASD C3I approved these 42 Command and Control systems based on reviews by the Military Communications Electronics Board. oversight have been approved—whether adequately supported by technical and business case justification or not—the ASD C3I oversight has essentially become a meaningless process.
section_title: Acquisition Oversight Process Did Not Ensure System Selections Were Economically and Technically Sound
paragraphs: Because the ASD C3I oversight control over the migration strategy played a marginal role in ensuring that sound management and development practices are followed, we assessed whether Defense’s traditional major information systems acquisition oversight processes were doing so for 43 of the major migration systems. These review processes are designed to assess whether projects are affordable and financial and operational risks have been minimized.
Once under acquisition oversight, systems are normally reviewed and approved at each of four milestones. The reviews involve assessing such matters as whether the proposed system is being developed in accordance with Defense policies, procedures, and regulations; whether the systems’ program managers took steps to minimize the cost of a new system by ensuring full and open competition; and whether the program managers will effectively use advanced system design and software engineering technology to minimize software and maintenance costs.
However, these reviews have not been effective for the migration effort. As discussed in the following section, MAISRC could not provide sufficient assurance that the major migration systems are economically justified and comply with Defense’s technical and data standards.
subsections: section_title: Ineffective Review and Preparation of Economic Analyses
paragraphs: A principal tool of acquisition oversight is the economic analysis because it helps to ensure that the system chosen for development is cost-effective. If done properly, it can enable reviewers to determine whether DOD has sufficient funds in its budget to pay for the system, that is, whether the system is affordable. It can also reveal potential conflicts between available funding and the planned schedule for deployments. To arrive at these conclusions, the economic analysis establishes baseline life-cycle costs and benefit estimates for the project and calculates the project’s return on investment.
The importance of developing complete and accurate economic analyses is underscored by several governmentwide requirements. For example, the Office of Management and Budget’s (OMB) Circular A-130, Management of Federal Information Resources, calls on agencies “to conduct benefit-cost analyses to support ongoing management oversight processes that maximize return on investment and minimize financial and operational risks for investments in major information systems on an agencywide basis.” Likewise, OMB’s Circular A-11, Part 3, Planning Budgeting and Acquisition of Fixed Assets, (July 16, 1996), and its Bulletin No. 95-03, Planning and Budgeting for the Acquisition of Fixed Assets, state that “the planning for fixed asset acquisitions should be based on a systematic analysis of expected benefits and costs.”
However, even though economic analyses play a critical role in assessing whether system development efforts will be cost-effective and beneficial, Defense has not yet established a set of minimum standards that an economic analysis must meet to be considered valid. In addition, it has not published official guidance for preparing an economic analysis. PA&E developed and published an unofficial economic analysis guide that recommended, but did not require, standard methods and formats for preparing an economic analysis for an information system development/modernization project. According to a PA&E representative, this unofficial guide is no longer adequate because it does not require the degree of standardization in methodology and analytic techniques needed to support a portfolio management approach for managing information technology investments, as required by the Clinger-Cohen Act of 1996. For example, the guide does not require that returns on investment for systems development/modernization projects be calculated in a standard manner using a standard definition. This lack of standardization results in economic analyses for different systems that are not comparable enough for DOD managers to have a good basis for deciding which systems offer the highest payoffs. The PA&E official stated that in conjunction with DOD’s implementation of a portfolio management approach, it plans to officially publish appropriate documents and provide minimum standards and guidance for the economic analyses process.
Our review also found that DOD decisionmakers do not view economic analyses as key tools for deciding whether to invest in an information system development or modernization project. As a result, DOD often lacks complete and accurate information on system development/modernization projects’ estimated costs and benefits at the time decisions are made to invest in the projects. Specifically, as the following examples show, we found that (1) economic analyses for many systems have not been submitted for independent review and (2) a significant portion of those that were submitted were inadequately prepared. Thus, many of the benefits that could be derived from this tool have not been realized.
Twelve of the 43 major migration systems have not yet submitted an economic analysis, or an update of a previously prepared economic analysis, for independent review. These systems were under development or modernization and DOD had invested hundreds of millions of dollars in them in total. These included 5 systems that were under direct MAISRC oversight and 7 systems for which MAISRC delegated oversight responsibility to a service or agency. These 12 systems, or components of them, are all in the second development phase or beyond. Delaying the preparation or updating of a previously prepared economic analysis to the later stages of development for these 12 systems defeats the purpose of the economic analysis, which is to demonstrate that a proposal to invest in a new system is valid before that investment is made.
Four systems that were under direct MAISRC oversight, were in the first development phase—concept exploration—and were, therefore, not yet required by DOD’s acquisition regulations to submit an economic analysis or an update of a previously prepared economic analysis for independent review. Although these four systems were technically in compliance with Defense’s acquisition regulations, DOD had already made major investments in them without the benefit of knowing whether returns on investment are going to be acceptable.
Even though Defense has not established standards for the required analyses, Program Analysis and Evaluation (PA&E) staff who review the economic analyses that are under direct MAISRC oversight told us that 10 of the 19 economic analyses reviewed had problems. These problems included the following:
Understating or omitting the costs of standardizing data, implementing the standard data in the systems, and developing system interfaces.
Failing to estimate the amount of savings expected for terminating duplicative legacy systems.
Relying on professional judgment to make unsupported assumptions rather than making objective analyses to estimate the value of benefits and costs. For example, in one case, the economic analysis estimated a cost avoidance associated with replacing an old system by assuming that both the old and the replacement systems’ software maintenance costs could be accurately estimated using two different rates per line of code, but the analysis provided no data supporting the validity of either rate.
After identifying problems with these 10 economic analyses, PA&E staff worked with MAISRC analysts and the systems’ program managers to address the problems. However, DOD continued to develop the systems in spite of the fact that the investments had not been justified by complete and accurate economic analyses.
PA&E analysts told us that there were other problems that impeded their review and verification of the economic analyses. For example, economic analyses are often not updated and provided to PA&E for review after major changes occur in the project, such as significant cost growth or redirection of the project. A second problem is that economic analyses are often not supported by analyses of alternatives that weigh the cost and benefits of various technical options, such as whether to buy commercial off-the-shelf software or develop a system in-house. This analysis would help Defense decisionmakers make sound decisions on whether the proposed alternatives offer sufficient military or economic benefits to be worth their cost, and to determine which alternative is the best approach. It would also identify alternatives that DOD may want to reconsider at a later time if the selected approach runs into difficulties.
section_title: DOD’s Acquisition Oversight Organizations Lack Assurance That Major Systems Comply With Technical and Data Standards
paragraphs: Defense has established several sets of standards that are designed to ensure that systems developed are compatible with its communications and computing infrastructure and that they are technically interoperable with each other. Some Defense leaders consider systems interoperability and the ability to exchange data across functional lines to be the most important consideration in migration system development, transcending economic benefits. These standards include the Technical Architecture Framework for Information Management (TAFIM), Defense Information Infrastructure Common Operating Environment (DII COE), and DOD standard data. DOD system acquisition directives call on MAISRC and DAB to ensure that program managers comply with the Department’s policies and procedures and use best practices in developing and modernizing individual information systems. These best practices include building systems and databases that comply with applicable technical standards and use DOD standard data.
However, MAISRC and DAB do not have adequate assurance that the major migration systems are complying with applicable technical standards and are using standard data. For example, out of 43 major systems under MAISRC oversight, program managers reported to MAISRC that (1) only 19 were in compliance with the TAFIM standards or had plans to comply with the TAFIM standards and (2) 9 systems were using DOD standard data or had plans to use standard data. We found similar results for compliance with DII COE standards. Specifically, program managers reported that only 25 systems were in compliance or had plans to be in compliance with DII COE. These self-reports by program managers are questionable because they are not independently verified.
In addition to obtaining information on technical and data standards directly from program managers, MAISRC and DAB also obtain such information from the Defense Information Systems Agency (DISA). DISA supports MAISRC and DAB oversight of major systems by performing technical reviews of system documentation to determine if it indicates that interoperability issues are being addressed or will be addressed as the system is developed. According to a DISA representative, DISA reviewed system documentation that had been prepared for 37 of the 43 MAISRC migration systems and all 6 of the DAB systems and found that the documentation indicated interoperability issues were planned to be addressed for each of them.
DISA also provides feedback, advice, and assistance on interoperability and related issues during integrated product team meetings with MAISRC, DAB, and system program managers. During these meetings and on other occasions when it is asked to do so, DISA provides MAISRC and DAB general information on compliance with standards, such as whether the major systems’ program managers have contacted DISA and appear to be making reasonable attempts to bring their systems into compliance with applicable standards. DISA also assists program managers in developing strategies and cost estimates for achieving compliance with standards.
However, DISA does not regularly report detailed information to MAISRC and DAB that would enable these oversight organizations to ensure that each major system is adequately complying with applicable technical and data standards. For example, DISA does not regularly report to MAISRC and DAB such detailed information as (1) each system’s current compliance with applicable standards—that is, each system’s current status relative to the eight levels that DOD has defined for the DII COE, and each system’s number and percentage of data elements that have been approved as DOD data standards in the Defense Data Dictionary System, (2) whether each program office prepared sound strategies, schedules, and cost estimates for achieving compliance with applicable standards, (3) whether each system’s compliance strategy and cost estimate were approved by DISA, (4) each system’s current schedule and cost status for achieving compliance compared with its baseline schedule and cost estimate, and (5) whether each system has been independently certified by DISA’s Joint Interoperability Testing Command to be in compliance with the technical and data standards. MAISRC analysts responsible for overseeing the major migration systems confirmed that they did not know many of the systems’ current status regarding compliance with applicable technical standards and use of DOD standard data, or whether the program managers had developed and were following sound strategies for bringing the systems into compliance.
The technical and data standards are supposed to help pave the way to an interoperable systems environment. However, without complete and accurate data on individual systems’ compliance with standards, MAISRC and DAB cannot assure Defense’s Chief Information Officer that the Department’s major information systems are complying with the standards. Without this information, as well as standards compliance information from the managers of DOD’s nonmajor systems, the CIO cannot gauge the Department’s progress toward achieving its goal of an interoperable systems environment. Additionally, this information is critical if the CIO is to successfully implement an integrated technical architecture for the Department as required by the Clinger-Cohen Act of 1996.
section_title: Better Oversight Could Have Helped to Prevent Problems Identified in Previous GAO Reviews
paragraphs: The lack of rigorous oversight for the migration strategy has increased the risk that Defense will pursue flawed system strategies. In fact, our previous reviews of migration systems identified a number of problems that could have been prevented had there been better oversight by MAISRC and the ASD C3I. For example, in several reviews, we found that functional areas did not account for various categories of significant costs when making their migration decisions. These findings are highlighted as follows.
In reviewing the depot maintenance standard system migration effort, we found that Defense did not address the full costs of developing interfaces needed to allow system components to exchange data with information systems currently used by the services to accomplish their missions. One official estimated that this represented $70 million in costs.In analyzing what it would cost to develop its Standard Accounting and Reporting System (STARS), we reported that Defense neglected to consider internal project management costs and costs to enhance all of the STARS components to bring them into compliance with DOD’s standard general ledger, key accounting requirements, and the standard budget and accounting classification code.
In reviewing the transportation migration effort, we found that Defense did not include all costs associated with its evaluation of in-house systems when analyzing costs and savings for its 28 migration systems. These included $16 million for its analysis of candidate migration systems and $2 million for maintaining migration system hardware. We also found that if these costs were included in its systems selection analyses, Defense would have found that the overall return on investment would have decreased and that it may actually lose money on its investment.
In previous reviews, we also found that Defense functional areas did not adequately consider other alternatives to developing systems in-house. For example, while the transportation area reviewed commercial off-the-shelf transportation software projects for some transportation business areas, this review was inadequate because it did not (1) analyze the degree to which unmodified software could meet unique Defense requirements, (2) identify the expected cost to make necessary software modifications, (3) determine the time required to make the modifications, and (4) provide for a hands-on view of the software in operation. In addition, Defense concluded that software packages that could provide some degree of transportation functionality would require modifications that were too costly. However, Defense could not provide documented analysis to support this conclusion. Further, Defense planned on making $13 million worth of software modifications to just five of its in-house selections. We believe better oversight by the CIO (then referred to as the Senior Information Management Official) may well have forced greater consideration of commercial packages in the transportation area.
Finally, our review of Defense’s effort to develop a standard suite of migration systems for materiel management showed that the Department spent hundreds of millions of dollars without achieving the expected benefits because it did not adequately anticipate and mitigate risks. From 1992 to late 1995, Defense spent about $714 million developing standard systems with minimal results. During that time, there were dramatic changes in the goals and expectations for the program and only one application was partially deployed. Because of changes in objectives and scheduling and problems in development, prospects for achieving the original objective of implementing a standard suite of integrated materiel management systems appeared dim.
In 1996, Defense finally abandoned its strategy to develop a standard suite of materiel management systems because of funding cuts, cost overruns, and schedule delays and embarked on a new strategy that involved individual deployment of nine system applications at selected sites as the applications were developed. However, the decision to drastically change the course of the strategy was initiated without first conducting critical economic and risk assessments that would estimate the costs, benefits, and risks of alternative strategies and having the analyses independently reviewed by MAISRC and PA&E. The need for department-level review of analyses supporting this decision was important, given the fact that the new strategy represented a departure from Defense’s goal of eliminating redundant legacy systems and varied business processes.
section_title: Defense Lacks Visibility Over Migration Effort
paragraphs: A successful information technology investment process cannot operate without accurate, reliable, and up-to-date data on project costs, benefits, and risks. It is the basis for informed decision-making. However, in implementing the migration strategy, Defense neglected to provide visibility over progress and costs. The absence of reliable and accurate performance, cost, and schedule information on the migration effort has been debilitating in several respects. First, it has impaired the Department’s ability to demonstrate whether the migration strategy has been successful. Second, it has kept Defense from providing the Congresswith accurate and reliable information needed for oversight purposes. Third, it has prevented Defense from providing its own senior managers with information needed to oversee the migration effort.
subsections: section_title: Key Performance Issues Have Not Been Tracked
paragraphs: Since it embarked on the migration strategy, Defense has not tracked overall departmentwide savings or validated improvements to operations resulting from the migration strategy. In particular, it has not been systematically tracking such key performance issues as (1) administrative and operational cost savings resulting from elimination of redundant systems, (2) cost reductions resulting from improved information systems support to functional areas, (3) management or staff productivity improvements, and (4) benefits accruing to mission effectiveness that are attributable to information technology support. Having this type of information is the only means of ensuring that the billions of dollars being spent on the migration are producing sufficient returns and achieving departmentwide progress. Without it, Defense cannot justify whether the migration strategy has been a worthwhile investment or support the need to continue pursuing migration.
At one point in the migration effort, as directed by the Congress, Defense developed performance measures. However, it did not regularly collect data on all the measures, and the accuracy of much of the information it collected was questionable because of the data integrity problems plaguing the database Defense relies on for migration inventory and schedule-related information. In 1995, the House Committee on National Security directed Defense to reevaluate its measures. In March of 1997, a Defense working group proposed a revised set of performance measures relating to migration systems and other information technology issues within the Department. However, senior Defense management did not approve the measures and instead tasked the working group to redo them to ensure that they are linked to the Department’s strategic information technology management goals. The working group has not established a schedule for completing this effort and finalizing the measures.
Having performance measures will help to validate individual reports of migration successes, such as the migration systems characterized as successful in Defense’s report to the Congress pursuant to Section 381 of the National Defense Authorization Act for Fiscal Year 1995. These are the Standard Procurement System, the Distribution Standard System, the Defense Civilian Personnel Data System, and the Defense Medical Logistics Standard Support System.
section_title: Cost Information Is Incomplete and May Be Significantly Understated
paragraphs: We could not determine the full cost of the migration effort because Defense’s migration cost information is incomplete and may be significantly understated. For example, when Defense provided migration cost data to us in December 1996, it could only provide costs for 242 of 363 migration systems. We did not include costs in this report for 27 of 242 systems for which DOD collected costs because the systems were classified. However, the remaining 121 systems were not included because the Office of the ASD C3I had not collected the costs for these systems. When we obtained cost information directly from three of the functional areas, we found that at least $1.6 billion in costs had been excluded from the approximately $18 billion total. This included about $1.4 billion for clinical health systems, $56 million for civilian personnel systems, and $111 million for transportation systems.
We also found that the $18 billion total cost estimate does not account for some very important costs related to developing, deploying, and maintaining migration systems both before and after a project has been initiated. For example, the transportation functional area has an office designated to oversee the development and deployment of its migration systems. But, when accounting for costs for the systems, DOD does not factor in the cost to maintain this oversight responsibility. In addition, our previous reviews of finance and logistics migration efforts have found that DOD did not account for costs for these projects relating to such activities as project management and developing interfaces with other systems. Our detailed analysis of DOD’s migration cost information is provided in appendix III.
section_title: Schedule Information Is Unreliable
paragraphs: We could not accurately determine how many legacy systems have been terminated and how many are scheduled for termination because the database Defense uses to track information systems is plagued with data integrity problems.
Defense’s own analyses of the database have shown that it cannot readily provide simple descriptive information for many systems. For example, a February 1997 DOD analysis of DIST showed that 55 percent of the migration systems in the database had incomplete information on interfaces with other systems and 77 percent had incomplete data on installations where the systems operated.
Our analysis of DIST found that 61 percent the migration system implementation dates and 32 percent of legacy system termination dates were questionable. In addition, when we compared DIST scheduling information to information maintained by three functional areas we reviewed, we found that most of the migration systems implementation dates and legacy system termination dates in DIST were incorrect. For example, DIST showed that 92 legacy systems were terminated by April 1996 in the clinical health, civilian personnel, and transportation areas, while functional area managers told us that only 43 had actually been terminated. Also, for the three functional areas, DIST showed that 53 legacy systems were scheduled for future termination while functional area managers told us that 91 were slated for future termination.
We also found that Defense has not ensured that the data definitions and formats used in DIST are fully compatible with data maintained in other Defense information systems that track and report on systems. While DOD is attempting to address this issue, DISA provided us information showing that only 66 of the 218 data elements contained in DIST have been approved as standard data elements in the Defense Data Dictionary System. Without standard definitions and data formats, data cannot be easily transferred to DIST from the other systems that may be used by functional area managers and other decisionmakers.
Defense officials acknowledge that DIST is incomplete and inaccurate, and the Department has begun efforts to make the database more accurate and more user friendly. However, Defense officials also stated that they still used DIST to generate reports to the Congress because it was the only departmentwide database containing schedule and other descriptive data on all Defense migration and legacy systems. A detailed analysis of DIST’s integrity problems is provided in appendix IV.
section_title: Reforming DOD’s Information Technology Investment Process Requires Effective Implementation of the Clinger-Cohen Act
paragraphs: Defense has begun implementing Division E of the Clinger-Cohen Act of 1996 (Public Law 104-106), which the Congress passed in an effort to put an end to problems associated with the development and implementation of government information systems—such as those evident in the migration effort. Under this legislation, Defense is required to establish additional controls to ensure that more attention is devoted to the selection, control, and evaluation of information technology projects and that the Department’s technology investments are managed from a portfolio perspective.
Implementing the Clinger-Cohen Act in DOD can bring meaningful reform to the management of migration and other information technology investments. However, the current structure of the Chief Information Officer (CIO) position in the Department will not permit the CIO to devote full attention to reforming information resources management within DOD. And it will continue to impose responsibilities on the CIO that conflict with the CIO’s obligation to provide independent oversight over the development and implementation of information systems.
subsections: section_title: The Clinger-Cohen Act
paragraphs: The Clinger-Cohen Act recognizes that the key to successfully managing information technology projects is ensuring that investment processes provide for the continued identification, selection, control, life-cycle management, and evaluation of information technology investments. Best practices on which the act is based have shown that to ensure an investment process is successful, top executives need to periodically assess all major projects—proposed, under development, and operational—then prioritize them and make funding decisions based on factors such as cost, risk, return on investment, and support of mission-related outcomes. Once projects are selected for funding, executives need to monitor them continually, taking quick actions to resolve development problems and mitigate risks. After a project is implemented, executives should evaluate actual versus expected results and revise their investment management process based on lessons learned.
As our guide, Assessing Risks and Returns: A Guide for Evaluating Federal Agencies’ Information Technology Investment Decision-making, points out, a key to success in this type of management is considering all the major technology investments that are vying for funding at a designated level (departmental, functional area, or service/agency level) as a total package, or portfolio, of possible technology investments. Once this perspective is adopted, an organization can focus scarce information technology resources on the projects with the greatest impact on mission and concentrate management attention on those high-impact projects that become troubled projects. It can also establish performance goals and stop or replace those systems or initiatives that fail to meet those goals.
This type of decision-making process can be applied to almost any organization, even one that is as highly decentralized as DOD. According to our investment guide, separate information technology investment decision-making processes can exist at various levels. In DOD, such processes could exist at the departmental level and the functional area or service/agency level, provided that the Department can identify which major projects and initiatives should be managed at each level. Criteria for determining projects that should be managed at the various levels may include the dollar amount of the investment, the degree of risk associated with the project, and whether the system is to be shared across functional area lines or service/agency lines.
The Clinger-Cohen Act contains the following additional requirements that are important for DOD to implement in order to address problems evident in its migration strategy.
Agencies are to determine whether their administrative and mission-related business processes should be improved before investing in major information systems to support them.
The investment process is to provide a means for senior management to obtain timely information regarding progress (at established milestones) in terms of cost, capability of the system to meet requirements, timeliness, and quality. It should also provide for the evaluation of the results of information technology investments.
Performance measures are to be prescribed for information technology used by or to be acquired for the agency.
The CIO is to monitor the performance of information technology programs; evaluate the performance of those programs on the basis of applicable performance measures; and advise the agency head regarding whether to continue, modify, or terminate the program or project.
The CIO is to be responsible for providing advice and other assistance to agency heads and senior managers to ensure that information technology is acquired and information resources are managed for the agency in a manner that implements the policies and procedures of the act and the priorities of the agency head.
The CIO is to develop, maintain, and facilitate the implementation of a sound and integrated information technology architecture for the agency. The architecture is an integrated framework for evolving or maintaining existing information technology and acquiring new information technology to achieve the agency’s strategic and information resources management (IRM) goals.
section_title: DOD Implementation of the Clinger-Cohen Act
paragraphs: As a first step in implementing the Clinger-Cohen Act, on June 2, 1997, the Secretary of Defense outlined his expectations for improvements in information technology-related management processes and information resources. We believe the expectations identified by the Secretary are an excellent starting point for implementing the Clinger-Cohen Act and bringing meaningful change to the current information technology management process. For example, the Secretary has called on the CIO to design and implement a process for maximizing the value and assessing and managing the risks of DOD information technology acquisitions. This process is to provide for the selection of information technology investments to be made by the Department, the management of such investments, and the evaluation of the results of such investments; be integrated with processes for making budget, financial, and program management decisions; include minimum criteria to be applied in considering whether to undertake a particular investment in information systems, including criteria related to the quantitatively expressed projected net, risk-adjusted return on investment, and specific quantitative and qualitative criteria for comparing and prioritizing alternative information system investment projects; identify, for each proposed investment, quantifiable measurements for determining the net benefits and risks of the investment; and provide the means for senior managers to be able to obtain timely information regarding the progress of an investment in an information system, including the milestones for measuring progress on an independently verifiable basis, in terms of cost, capability of the system to meet specified requirements, timeliness, and quality.
In addition, the Secretary has called on the CIO to institutionalize performance-based and results-based management for information technology. In doing so, the CIO is to work with the Chief Financial Officer, the Principal Staff Assistants (PSAs), and the Defense components. The CIO is to establish goals for improving the efficiency and effectiveness of DOD operations and issue instructions to functional areas on performance measurements. The CIO is also to monitor the performance of information technology programs, evaluate the performance of those programs on the basis of applicable performance measurements, and advise the Secretary of Defense regarding whether to continue, modify, or terminate programs or projects. Additionally, the Secretary established a Chief Information Officer Council for DOD to serve as the principal forum for discussing improvements in DOD practices for the management of information technology.
In outlining his expectations, the Secretary noted that the act poses questions that should be answered before investing in information technology, including:
What functions are we performing and are they consistent with the mission? If we should be performing particular functions, could they be performed more effectively and at lower cost by the private sector?
The Secretary further stated that if a function should indeed be performed by the Department, the law requires that the function be examined and redesigned or reengineered before applying new technology.
section_title: Challenges Confronting DOD in Implementing Clinger-Cohen
paragraphs: Proper implementation of the Clinger-Cohen Act would help to address a number of weaknesses that are currently standing in the way of Defense’s ability to provide a good decision-making and oversight environment for information technology projects. For example, with full implementation of the act, Defense could begin considering information technology investments as a total package of possible projects so that it can target resources to those projects having the greatest impact on mission and concentrate management attention on troubled areas. It could also strengthen visibility over project performance, costs, and schedules so that senior managers can begin comparing the results being achieved against projected, costs, benefits, and risks and to identify actual or potential managerial, organizational, or technical problems.
Moreover, in implementing the act, Defense has an opportunity to (1) begin enforcing compliance with data and technical standards to ensure that DOD’s goals for interoperability and the sharing of information are met and (2) increase oversight for functional area assessments of whether to reengineer, migrate, or undertake some other path toward improvement.
However, the current structure of the CIO position in Defense will not permit the CIO to effectively serve as a bridge between top management, line management, and information management support officials and identify opportunities to use information technology to enhance performance.
The Clinger-Cohen Act requires that information resources management be the CIO’s primary responsibility and that the CIO be involved in key decisions regarding the application of information technology in support of the agency’s missions. Currently, the Assistant Secretary of Defense for Command, Control, Communications and Intelligence (ASD C3I) also acts as the CIO. By asking the CIO to also shoulder a heavy load of programmatic responsibility, Defense has made it difficult, if not impossible, for the CIO to devote full attention to IRM issues. These issues go well beyond the development and modernization of information systems.
For example, Defense needs its CIO to be heavily involved with implementing a more aggressive and proactive computer security program. As we reported in May 1996, attackers have seized control of entire Defense systems, many of which support critical functions, such as weapons systems research and development, logistics, and finance. Attackers have also stolen, modified, and destroyed data and software. In addition, Defense needs its CIO to help ensure that Year 2000 corrections are made to all of DOD’s information systems. If systems are not corrected on time, the impact on Defense operations could be widespread, costly, and debilitating to important warfighting and administrative operations. While DOD has delegated year 2000 responsibility to its components, it still needs to ensure, at the department level, that sufficient priority and resources are being devoted to the problem and that all systems have been identified and corrected.
There is also a direct conflict of responsibilites between the oversight and programmatic obligations associated with the two positions. The ASD C3I serves as the Principal Staff Assistant for command, control, communications, and intelligence systems. These systems represent about 45 percent of the migration system investment—about $8.5 billion of the total $18 billion migration investment. This poses a conflict for both control mechanisms we assessed. For the first control—the ASD C3I approval process—the same individual is responsible for both selecting a system and approving the selection. For the second control—acquisition oversight—Defense’s Acquisition Executive is also responsible for developing 14 of the 43 major automated information systems under the Major Automated Information System Review Council (MAISRC).
A second challenge confronting DOD in implementing the Clinger-Cohen Act is the prevailing organizational structure and embedded culture found throughout the Department. Specifically, the three military services have clearly defined roles and responsibilities and separate budget authority, program execution, and functional authority for the enforcement of national defense policy and objectives. As we have reported throughout the CIM initiative, this environment has promoted stovepipe systems solutions in each component agency and has made it difficult to implement departmentwide oversight or visibility over information resources. This same condition has contributed to the difficulty that has limited the Department in modernizing business processes and implementing corporate information systems across service and agency lines. This is most evident in the perceived failure of the Corporate Information Management initiative (CIM), which was intended to reengineer business processes throughout the Department. In doing so, the Department expected to save billions by having more efficient, effective business processes running across service and component lines. However, these benefits have yet to be widely achieved after 8 years of effort. Without the Secretary’s strong and continued support for management processes and controls designed to improve information management initiatives, Clinger-Cohen Act implementation could well suffer similar results.
section_title: Conclusions, Recommendations, and Agency Comments
paragraphs: In taking its initial steps to implement the Clinger-Cohen Act, Defense has recognized that it needs a better information technology investment environment. However, implementing the act will not be easy given weaknesses pervading the current decision-making and oversight environment. Examples included the following:
DOD’s management and oversight processes over information technology projects are not effectively integrated to enable the Department to manage from a portfolio perspective. Considering investments as a total package of possible projects is important because it forces an agency to decide which projects are the most critical to meeting mission needs and thus should receive the most resources and attention. Life-cycle management controls that provide oversight on a system-by-system basis can supplement and enhance this process, but they cannot be a substitute for it.
DOD still has not established information technology performance measures. These need to be in place before a new investment process can begin so that senior managers can begin comparing results being achieved against projected costs, benefits, and risks.
As far as visibility over costs and schedule are concerned, DOD has fragments of a mechanism for collecting and maintaining all project information. However this mechanism—DIST—does not maintain cost information and its accuracy and reliability are questionable. Further, as we have recently reported to the Director of the Defense Information Systems Agency, efforts to improve the DIST have been slow-moving.
While the Clinger-Cohen Act requires agencies to conduct post-implementation reviews, DOD’s oversight processes have concentrated on projects prior to their implementation. Thus, in addition to focusing on the pre-implementation stages of system life-cycles, DOD will have to ensure that more attention is given to whether implemented systems are achieving the forecasted benefits and continuing to meet mission needs after they are implemented.
Currently, DOD does not have an effective mechanism in place to ensure that the CIO has complete and accurate information on DOD-wide compliance with technical and data standards. Such a mechanism is necessary to enable the CIO to achieve the Department’s goals for systems interoperability, effectively implement an integrated technical architecture for the entire Department, and ensure the efficient exchange of standard data among systems.
Moreover, our review of the migration strategy suggests that a real threat to successful implementation of the Clinger-Cohen Act is the Department’s consistent lack of adherence to sound decision-making and oversight processes. For example, at the oversight level, systems that clearly lacked key pieces of technical, programmatic, and economic justification were allowed to go forward. At the decision-making level, many functional areas did not even bother to submit their systems or analyses that supported their decisions to department-level oversight controls. If they were effectively followed, the life-cycle management controls would have helped ensure that sound business and development practices were followed for the migration systems.
Perhaps, the greatest challenge to successful operation, however, will be operating in an organizational environment that has resisted departmentwide oversight and visibility over information resources. For example, the Department’s CIM effort largely failed in meeting its original goals of bringing widespread efficiencies to its business processes. The Department was unable to create meaningful change across organizational lines and management support was insufficient to overcome initial resistance to new ways of doing business. Efforts to implement the Clinger-Cohen Act will require unwavering top-level commitment to overcome both organizational resistance and to institute meaningful controls.
subsections: section_title: Recommendations
paragraphs: To ensure that DOD’s continued investment in migration systems provides measurable improvements in mission-related and administrative processes, we recommend that the Secretary of Defense require the Defense components to rank development/modernization migration systems justifications and complete them on an expedited basis. Further, the Secretary should require the Chief Information Officer to review these justifications and certify that they include the following:
A business case of operational alternatives for each functional area that clearly demonstrates that continued development and deployment of the migration system is the best solution for improving performance and reducing costs in the functional area it serves, when compared to other available alternatives. The alternatives analyzed should include reengineering the functional area’s processes before making investments in information systems and using, when appropriate, the private sector to perform major functions now performed by government personnel and information systems.
An economic analysis showing a return on investment or other results-based benefits to the Department that justify further investment in the migration system.
Current compliance with applicable Defense technical standards and uses standard data, or a schedule and plan for bringing the system into compliance with these standards.
Lastly, the Secretary of Defense should require the Chief Information Officer to establish routine procedures for reporting on the status of reviews of migration system justifications to the Deputy Secretary of Defense so the information can be used in the Department’s planning, programming, and budgeting system. Any exception to the accomplishment of these reviews should be approved by the Deputy Secretary of Defense.
Further, we recommend that the Department’s Chief Information Officer revise Defense’s policies, practices, and procedures to institutionalize the management of information systems and technology expenditures as investments and ensure that these investments provide measurable improvements in mission performance. In the course of taking action on these matters, we recommend that the Secretary direct that the Chief Information Officer, in coordination with the Chief Financial Officer and other appropriate Defense officials take the following actions:
Ensure that Defense’s strategic information technology planning and investment control policies, practices, and procedures include requirements that Principal Staff Assistants document that they and the key stakeholders for their functional areas (including the services, agencies, and major commands) have (1) conducted thorough economic and risk analyses of alternative operational approaches (business case analyses) for accomplishing the mission of the functional area, (2) examined trade-offs among the competing proposals, and (3) prioritized the alternative proposals based on mission impact, risk, and return.
Finalize and issue guidance for developing and using analyses of alternatives and economic analyses for information system decision-making. Once this guidance is issued, the CIO should require that all Defense program managers and Defense managers, as appropriate, be trained in using the guidance. Also, the CIO should ensure that the guidance defines a standard return on investment definition for Defense information systems and require that this definition be used to calculate all returns on investment for information systems efforts. Additionally, the CIO should require that all major migration and other information systems under direct review by the Major Automated Information System Review Council (MAISRC), and those delegated by MAISRC to other oversight organizations for review, have their alternatives analyses and economic analyses independently verified by Defense’s Program Analysis and Evaluation (PA&E) office, or another qualified independent review organization, in accordance with this guidance before major investments are authorized for system development/modernization.
Require post-implementation reviews of migration and other information systems and ensure that these reviews are designed to compare actual systems’ costs, benefits, risks, and returns against the original baseline estimates/projections and determine the causes of any differences between planned and actual results.
Expedite the definition, coordination, testing, and implementation of information management performance measures in the Department and establish milestones for evaluating progress in implementing these performance measures.
Modify the DIST system or acquire/develop a new Defense-wide management information system or systems for tracking and reporting key schedule, progress, and performance information on migration systems and other Defense information systems and ensure the system or systems contain complete, current, and accurate schedule data necessary to track the progress of each migration system’s development/deployment and each legacy system’s termination; budgeted and actual cost data on each system for which the Department maintains such data (an alternative to putting budget and cost data in DIST is to establish the capability to directly interface with other systems in the Under Secretary of Defense (Comptroller’s) Office or other Defense organizations containing systems’ budget and/or cost data); data necessary to track the progress of each migration system in complying with applicable Defense technical and data standards, including whether each system has been independently certified to be in compliance with applicable technical and data standards; data for tracking progress in accomplishing the mission-based performance goals and information management performance goals for the functional areas supported by each system once these data have been identified; information determined to be needed for oversight by the Defense Acquisition Board (DAB), MAISRC, and PA&E; and other information determined to be needed for management and oversight by the Defense CIO, the CIO Council, other Defense senior managers, and the Congress.
Develop and implement management controls and a quality assurance program to ensure the DIST data’s accuracy and completeness since these data are used to track and report to senior Defense managers and the Congress on the overall status of the migration initiative and on other Defense information management initiatives.
section_title: Agency Comments and Our Evaluation
paragraphs: The Department of Defense provided written comments on a draft of this report, which are reprinted in appendix I. The Acting Assistant Secretary of Defense for Command, Control, Communications and Intelligence concurred with five of our recommendations and partially concurred with two recommendations. Defense did not concur with the remaining five recommendations and expressed concerns about certain aspects of our report. Defense’s concerns are highlighted and discussed below. Appendix I also provides detailed responses to DOD’s views on our recommendations and to other specific comments on our findings.
Generally, the Department concurred with our recommendations that the Department revise or develop internal policies and procedures to conform to the Clinger-Cohen Act and to improve the performance of information systems management. Defense also noted that the issues we reported are in the process of being or will be addressed as it implements the Clinger-Cohen Act, the Federal Acquisition Streamlining Act, the Government Performance and Results Act, and the Paperwork Reduction Act.
However, it did not agree to a recommendation in our draft that it limit further investments to expenditures that meet mission critical needs until the investments are economically, functionally, and technically justified and such justifications are independently reviewed. In DOD’s view, limiting migration system spending would adversely affect military readiness, system development, and government contract obligations and increase system obsolescence. We understand DOD’s concerns regarding readiness and contract obligations. That is why our recommendation in the draft report recognized that critical needs still need to be met. Our point is that greater management attention needs to be placed on the decision-making process for approving and funding the development and modernization of migration systems to achieve intended benefits and minimize unnecessary costs. In order to clarify this position, we modified our recommendation to focus on strengthening the controls for the migration system decision-making process so that these investments can be better considered in the Department’s budget process.
In addition, Defense did not concur with a proposal in the draft report that it consider separating the CIO and the ASD C3I positions so that the CIO can devote full attention to departmentwide information resource management issues and provide independent oversight. It noted, however, that all ASD C3I functions are being reviewed by the Department’s Task Force on Defense Reform as part of its review of the Office of the Secretary of Defense’s organizational structure. We withdrew the proposal because DOD is now considering this matter. Nevertheless, our concern that the current structure of the CIO position does not allow the CIO to devote full attention to critical IRM issues—such as computer security, the Year 2000 problem, and the need to develop and implement an integrated information technology architecture—remains valid.
Defense also stated that our report overlooked one of the key cost reduction aspects of the migration effort—that is, modernizing multiple systems within each service and Defense agency is more expensive than simply modernizing one or a few departmentwide migration systems. While these cost reductions may be possible, Defense has not yet demonstrated that it has achieved such savings. Further, our review determined that Defense does not know, or track, cost reductions that result from its migration efforts. Finally, the achievements that Defense refers to in its comment letter are mostly the results of process reengineering, not the migration of automated systems.
Defense also stated that our report negatively portrays the role and benefits of acquisition streamlining within the Department. Defense reported that it uses a “best-value” approach to modernizing its systems that balances the functional and technical capabilities while still considering cost. We disagree with Defense’s characterization of our report. We fully support acquisition streamlining efforts in DOD because they promote economy, efficiency, and effectiveness in the procurement of property and services. However, these three elements must be carefully balanced in order to achieve the results intended by the Division D of the Clinger-Cohen Act of 1996 which covers federal acquisition reform. Such a balance is particularly important to achieve prudent decisions for an investment as substantial as the multibillion dollar migration initiative, and it is contemplated by the Clinger-Cohen Act of 1996 to properly control these investments.
In our view, for the migration effort, it appears that expediency was achieved at the expense of economy and effectiveness. In conducting our review, we sought to examine whether the Department of Defense was following its own policy for controlling migration investments. When we found that these policies were not executed as intended, we turned to the Department’s acquisition review process, which should have supplemented oversight for the major migration systems. However, even here, we found that DOD for the most part, was not providing sufficient assurance that migration investments were worthwhile. Specifically, we found that controls that Defense built into this process would not ensure success.
In addition, the migration strategy is a high-risk endeavor because it requires Defense to carefully consider complex technical issues that are compounded by the sheer size of the Department, the number of disparate systems, and the prevailing culture, which promotes stovepiped systems solutions and hampers departmentwide oversight or visibility over information resources. As such, the success of the strategy hinges on whether practical and sound management practices—such as conducting economic, risk, and alternative analyses and providing rigorous oversight over the selection process—are followed. As we note in the report, these practices are required by executive branch policies; congressional reform initiatives, including the Clinger-Cohen Act, which took effect August 8, 1996; and/or Defense regulations. Under the Clinger-Cohen Act, for example, Defense is required to implement a process for selecting information technology investments using specific criteria for comparing and ranking alternative information system projects. The best practices on which the act is based have shown that to ensure the success of an investment process, top executives need to make funding decisions based on factors such as cost, risk, return on investment, and support of mission-related outcomes. Thus, unless Defense promptly embraces the need to develop and review economic, risk, and alternative analyses, it stands little chance of successfully implementing the Clinger-Cohen Act.
In its comments, Defense also expressed concern that the scope of our report was unnecessarily broader than it was when we began our review. We were originally asked for information on the status and cost of the migration effort. However, when we found that Defense could not provide accurate and reliable information on migration or convincingly demonstrate whether the strategy has been successful, our congressional requester asked that we review whether Defense’s management control and oversight processes for migration were ensuring that the investments are economically sound and complied with technical and data standards. We reviewed the reliability of the acquisition process controls over migration investments because they are the leading Defense-recognized controls over major investments. Our congressional requester also asked that we focus on DOD’s implementation of the Clinger-Cohen Act because it offered a chance for DOD to bring meaningful reform to the management of migration and other information technology investments. Many information technology investments for systems in the Department are, in fact, migration systems.
Finally, Defense’s comments on our report identify a number of other disagreements with our findings and conclusions. We reviewed these comments, incorporated them into our report where appropriate, and followed up with Defense on additional information it provided after submitting its response to us.