United States General Accounting Office

GAO Report to the Ranking Minority Member,

Committee on Governmental Affairs,

U.S. Senate

October 1997 DEFENSE IRM

Poor Implementation

of Management

Controls Has Put

Migration Strategy at

Risk

GAO/AIMD-98-5

GAO United States

General Accounting Office

Washington, D.C. 20548

Accounting and Information

Management Division

B-275670 

October 20, 1997

The Honorable John Glenn

Ranking Minority Member

Committee on Governmental Affairs

United States Senate

Dear Senator Glenn:

As you know, in 1989, the Department of Defense started its Corporate Information

Management (CIM) initiative in an effort to save billions of dollars by streamlining operations

and deploying standard information systems to support common business operations. A key

part of this initiative is Defense’s migration effort, which involves replacing its functionally

duplicative and inefficient automated information systems with the Department’s best existing

information systems. Concerned that the billions of dollars projected to be spent on CIM-related

technology efforts are at high risk, you asked that we provide information on the status and

progress of Defense’s migration effort and assess whether Defense has effective controls in

place to manage and oversee the initiative. This report also highlights the implications of our

findings for Defense as it responds to investment management requirements of the

Clinger-Cohen Act of 1996.

We are sending copies of this report to the Chairman of the Senate Committee on Governmental

Affairs and to the Chairmen and Ranking Minority Members of the Senate Committee on Armed

Services; Subcommittee on Defense, Senate Committee on Appropriations; House Committee

on National Security; Subcommittee on National Security, House Committee on Appropriations;

and Senate and House Committees on the Budget; the Secretary of Defense; the Acting

Assistant Secretary of Defense for Command, Control, Communications and Intelligence; the

Acting Under Secretary of Defense (Comptroller); and the Director, Office of Management and

Budget. Copies will also be made available to others upon request.

If you have any questions about this report, please call Mickey McDermott, Assistant Director,

at (202) 512-6240. Other major contributors to this report are listed in appendix V.

Sincerely yours,

Jack L. Brock, Jr.

Director, Defense Information and

 Financial Management Systems
Executive Summary

Purpose In 1989, the Department of Defense started its Corporate Information

Management (CIM) initiative in an effort to save billions of dollars by

streamlining operations and deploying standard information systems to

support common business operations. However, 8 years after beginning

CIM and making substantial investments, Defense has not met its savings

goal because it has not fully implemented sound management practices to

carry out this initiative. GAO placed the effort on its high-risk list in

February 1995, in part because it found that CIM-related technology

investments, which are expected to total billions of dollars each year, are

vulnerable to waste and mismanagement.

A key part of CIM is Defense’s migration effort, which involves replacing its

functionally duplicative and inefficient automated information systems

with the best existing systems. Defense believes that migration can cut

costs associated with developing and maintaining disparate systems

supporting the same functions. It also believes that if done properly,

migration can help standardize business processes. Concerned that

billions of dollars projected to be spent on CIM-related technology efforts

are at high risk, the Ranking Minority Member of the Senate Committee on

Governmental Affairs asked GAO to (1) provide information on the number

and cost of systems designated for migration, the number of legacy

systems already terminated or scheduled for termination, and savings

resulting from terminations of legacy systems and (2) determine whether

Defense’s management control and oversight processes for migration

systems are ensuring that the investments are economically sound and in

compliance with its technical and data standards.

Background When it initiated CIM, Defense believed that the thousands of automated

information systems supporting its business operations—which include

such functions as logistics, communications, personnel, health affairs, and

finance—were redundant and inefficient. These operations were

traditionally carried out in isolation by the individual military services and

Defense agencies. As such, the information systems supporting these

discrete operations were developed independently even though they may

well have served similar purposes.

DOD’s CIM initiative included several aspects: (1) corporate policy/planning,

(2) process and data modeling, (3) process improvement, (4) performance

measurement, (5) standard information systems, and (6) computing and

communications infrastructure. As CIM was implemented, Defense

emphasized two ways of achieving process improvements and addressing

Page 2 GAO/AIMD-98-5 Defense Migration Strategy
Executive Summary

problems associated with its disparate and stovepiped information

technology environment: (1) reengineering business processes first and

then applying technology to the new processes and (2) selecting the best

DOD information systems from pools of existing, or legacy, systems that

provide the same automated support services and eventually replacing the

duplicative systems with the best systems. The second approach is known

as migration.

In October 1993, Defense embarked on an “accelerated” migration

strategy, which placed more emphasis on the second improvement

approach. As part of this strategy, it asked its managers to select migration

systems in 6 months and develop and deploy them departmentwide in the

following 3 years. Defense believed that instilling pressure to select and

deploy migration systems would reap savings much quicker than

streamlining or reengineering the business processes and acquiring

systems after those processes were reengineered.

The consequence of the increased emphasis on migration was that the

dramatic gains that could be achieved through reengineering would be

postponed. This impaired the chances of CIM achieving its objectives for

dramatic improvements and cost savings in several respects. First, it kept

Defense from focusing on redesigning core business processes, which

promised order-of-magnitude improvements. Second, it increased the risk

that bad business processes would be perpetuated. Third, it would make

future reengineering efforts more difficult by entrenching inefficient and

ineffective work processes. As a result, reengineering never took hold in

many of DOD’s functional areas. The migration arm of CIM, however, is still

active.

Results in Brief From fiscal years 1995 through 2000, Defense plans to spend at least

$18 billion on migration. It has selected 363 migration systems and has

targeted 1,938 legacy systems for potential termination. Despite this

substantial investment, Defense did not adhere to decision-making and

oversight processes it established to ensure that the economical and

technical risks associated with migration projects have been mitigated.

Defense’s primary corporate-level control mechanism for ensuring that

sound management and development practices were followed for each

migration investment has not been effective. This control requires that all

migration system selections be approved by the Assistant Secretary of

Defense for Command, Control, Communications and Intelligence

Page 3 GAO/AIMD-98-5 Defense Migration Strategy
Executive Summary

(ASD C3I). Despite this requirement, about 67 percent of migration

selections were never submitted for approval, and many of those that were

submitted were approved in the absence of critical technical and

programmatic support. Consequently, Defense does not have assurance

that the migration systems being developed will help achieve its

technology goals and that sound business decisions were made in

selecting the systems.

Because this control mechanism did not support the migration effort, GAO

also assessed whether Defense’s traditional acquisition oversight

processes, which are designed to help ensure that individual major

information system investments are economically and technically sound,

were helping to ensure that the risks associated with migration were

mitigated. However, these processes also did not support the migration

effort because they have not fully ensured that economic analyses for

migration projects are prepared and reviewed and that the systems comply

with technical and data standards. For example, economic analyses for 12

of 43 major migration systems—for which Defense has invested hundreds

of millions of dollars in total—have not yet been submitted for

independent review. Delaying the preparation of an economic analysis to

the later stages of development defeats the purpose of the analysis—to

demonstrate that a proposal to invest in a new system is valid before that

investment is made.

Had there been more rigorous attention to established oversight

procedures and sound business practices, Defense might well have

avoided the migration problems GAO identified in previous reviews, which

unnecessarily cost the Department hundreds of millions of dollars. One

functional area, for example, embarked on and later abandoned a

substantially flawed effort to develop a standard suite of migration

systems for materiel management after spending over $700 million without

strong oversight. In addition, some functional areas did not account for

various categories of significant costs when making their migration

decisions or adequately consider alternatives to developing systems

in-house.

In implementing the migration strategy, Defense did not ensure that it had

adequate departmentwide visibility over status, costs, and progress. As a

result, it could not provide accurate and reliable information, as requested,

on the number and cost of systems designated for migration, and the

numbers of systems terminated and scheduled to be terminated.

Furthermore, the Department has not been able to convincingly

demonstrate whether the migration strategy has been successful or not,

Page 4 GAO/AIMD-98-5 Defense Migration Strategy
Executive Summary

and it has not been able to provide the Congress with accurate and reliable

information needed for decision-making purposes.

Defense has taken a positive step to turn around its information

technology investment process as part of its implementation of the

Clinger-Cohen Act of 1996 (Division E of Public Law 104-106). This

legislation requires federal agencies to have processes and information in

place to help ensure that information technology projects (1) are being

implemented at acceptable costs, within reasonable and expected time

frames, and (2) are contributing to tangible, observable improvements in

mission performance. In June 1997, the Secretary of Defense outlined his

expectations for improvements in management processes and information

resources related to information technology. These expectations

incorporate some of the most important elements of the Clinger-Cohen Act

and are an excellent starting point for bringing meaningful change to the

current information technology management process. Additionally, the

Department’s Task Force on Defense Reform is currently examining the

current structure of the Chief Information Officer (CIO) position to ensure

that the CIO can devote full attention to reforming information resources

management within the Department.

In implementing the Clinger-Cohen Act, the Department faces a formidable

challenge in successfully implementing real change across an

organizational structure that has clearly defined roles and responsibilities

for the three individual services to execute national defense policy and

objectives. The separation of budget authority, program execution, and

functional authority have all contributed to an environment that has

fostered stovepipe systems within each service and has made

departmentwide oversight difficult. For Clinger-Cohen implementation to

make a difference, any new processes or requirements must be

successfully accomplished within this environment.

Principal Findings

Management Control

Processes Over Individual

Migration Efforts Have

Broken Down

Defense’s primary corporate-level control mechanism for ensuring that

sound management and development practices are followed for each

migration investment has not been effective. This control requires the

functional area manager, known as the Principal Staff Assistant1

 (PSA), to

1

The PSA is the senior executive-level manager who is responsible for the management of a defined

function or functions within the DOD.

Page 5 GAO/AIMD-98-5 Defense Migration Strategy
Executive Summary

submit all systems selected for migration for approval by the Assistant

Secretary of Defense for Command, Control, Communications and

Intelligence. This approval is important because the Assistant Secretary, as

the Department’s Chief Information Officer, needs to ensure that DOD’s

technology-related goals are met.

However, this control mechanism has broken down in two respects. First,

even though all system selections should have been submitted for review

and approval by the Assistant Secretary, 245 of the 363 selections were

never submitted for this oversight. Second, most of the systems that were

submitted for this review were approved in the absence of critical

technical and programmatic support. Specifically, about 19 percent were

approved without supporting technical justification and about 54 percent

were approved without documents showing that the functional area had

evaluated different options for improving a business area, such as

reengineering. Because all of the migration selections submitted for this

oversight have been approved—whether adequately supported with

technical and programmatic justification or not—this control has

essentially become meaningless.

Because the oversight from the ASD C3I that was specifically established

for migration selections has been marginal in trying to ensure that sound

management and development practices are followed for migration

selection, GAO assessed whether the Department’s traditional major

information systems acquisition oversight processes were doing so for 43

of the major migration systems.2

 Major systems are Defense’s most

expensive and critical information systems and can cost up to hundreds of

millions of dollars to develop. These review processes are designed to

assess whether projects are affordable and financial and operational risks

have been minimized.

However, these processes did not provide sufficient assurance that the

systems (1) were economically justified or (2) complied with Defense’s

technical and data standards—which are intended to help pave the way

toward an interoperable systems environment. For example, economic

analyses or updates of previously prepared economic analyses for 12 of

the 43 systems had not been submitted for independent review. These 12

systems are under development or modernization and Defense has

invested hundreds of millions of dollars in them. In addition, even though

the Department recognizes that economic analyses play a critical role in

2

There are 49 major migration systems; however, GAO’s review focused on the 43 major migration

systems under the oversight of the Major Automated Information System Review Council. The

remaining six systems are under the oversight of the Defense Acquisition Board.

Page 6 GAO/AIMD-98-5 Defense Migration Strategy
Executive Summary

assessing whether system development efforts will be cost-effective and

beneficial, it has not yet published official guidance to standardize the

methodology and analytic techniques for preparing economic analyses.

GAO’s previous reviews of migration systems identified a number of

problems that could have been prevented had there been better oversight

by the Major Automated Information System Review Council (MAISRC) and

the ASD C3I. For example, GAO’s review of the materiel management

migration strategy showed that a functional area was able to embark and

spend over $700 million pursuing a substantially flawed effort—which was

later abandoned—without rigorous department-level oversight. In

addition, in previous reviews of migration efforts in depot maintenance,

transportation, and finance, GAO found that functional areas failed to

account for various categories of significant costs when making their

migration decisions—including costs related to interfacing migration

systems with other systems, project management, and the system

selection process. More rigorous oversight and guidance could have

helped ensure that these costs were included.

GAO also found a lack of good departmentwide visibility over the migration

effort in terms of project costs and progress. For example, DOD has not

been tracking key performance issues, such as cost savings resulting from

migration or management and staff productivity improvements. Further, it

does not have a complete picture of the costs of all migration projects and

its scheduling information is inaccurate and unreliable. As a result, the

Department has not provided its own department-level decisionmakers

with information needed to manage the effort, and it has not provided the

Congress with complete and accurate information on migration. Moreover,

in the absence of good performance measures for migration, the

Department has not been able to show whether the overall strategy has

been successful or not.

Reforming DOD’s

Information Technology

Investment Process

Requires Effective

Implementation of the

Clinger-Cohen Act

Defense recognizes the need for a better information technology

investment environment and has taken steps to implement the

Clinger-Cohen Act of 1996. The Congress passed the act in an effort to put

an end to poorly managed and wasteful information technology projects.

Among other things, it requires agencies to adopt an investment process

that provides for the continual identification, selection, control, life-cycle

management, and evaluation of information technology projects. As a first

step in implementing the act, the Secretary of Defense directed the ASD

C3I, as the Department’s Chief Information Officer, to take the lead in

Page 7 GAO/AIMD-98-5 Defense Migration Strategy
Executive Summary

implementing an investment process as well as a performance- and

results-based management strategy for information technology.

This is an important move toward bringing meaningful change to the

current decision-making and oversight environment for migration.

However, the current structure of the CIO position in the Department

seriously limits the CIO’s ability to effectively serve as a bridge between top

management, line management, and information management support

officials and to identify opportunities to use information technology to

enhance performance. At present, the ASD C3I also serves as the

Department’s CIO. Asking the same individual to serve in both capacities

prevents the CIO from devoting full attention to reforming information

resources management within the Department. It also means that the ASD

C3I will continue to be responsible for providing oversight for the same

systems that he is responsible for selecting, developing, and implementing

in his capacity as a PSA. This shortchanges much-needed independent

oversight for about 45 percent of the Department’s migration system

investment.

Finally, effective implementation of the Clinger-Cohen Act will not occur

unless Defense’s information and system investment control processes

successfully address the challenge posed by the prevailing organizational

structure and culture found throughout the Department. This condition

has promoted stovepipe systems solutions in each component agency and

has made it difficult to implement departmentwide oversight or visibility

over information resources. This same condition has contributed to the

difficulty that has limited the Department in modernizing business

processes and implementing corporate information systems across service

and agency lines. This is most evident in the perceived failure of the CIM

initiative, which was intended to reengineer business processes

throughout the Department. By doing so, the Department expected to save

billions by having more efficient, effective business processes running

across service and component lines. However, these benefits have yet to

be widely achieved after 8 years of effort. Without the Secretary’s strong

and continued support for management processes and controls designed

to improve information management initiatives, Clinger-Cohen

implementation could well suffer similar results.

Recommendations To ensure that continued investment in migration systems provides

measurable improvements in mission-related and administrative

processes, GAO is recommending that the Secretary of Defense require

Page 8 GAO/AIMD-98-5 Defense Migration Strategy
Executive Summary

Defense components to rank development/modernization systems

justifications and complete them on an expedited basis. GAO is further

recommending that the Chief Information Officer certify that these

justifications include the following:

• An analysis of operational alternatives that clearly demonstrate that

continuing with migration is the best solution for improving performance

and reducing costs in the functional area.

• An economic analysis showing a return on investment or other mission

benefits that justify further investment.

• Documentation showing that the system currently complies with

applicable Defense technical standards and uses standard data.

GAO’s recommendations are also aimed at (1) correcting weaknesses

within the current life-cycle management environment and (2) ensuring

the successful reengineering of processes and implementing of corporate

information systems for functional areas.

Agency Comments The Acting Assistant Secretary of Defense for Command, Control,

Communications and Intelligence provided written comments on a draft of

this report. Defense concurred with five recommendations and partially

concurred with two recommendations. Defense did not concur with the

remaining five recommendations and expressed concerns about certain

aspects of the report. Generally, the Department concurred with GAO’s

recommendations that it revise or develop internal policies and

procedures to conform to the Clinger-Cohen Act, develop and implement

performance measures, and improve the performance of internal tracking

systems.

The Department did not agree to a proposal in the draft report that it limit

further investments in ongoing migration systems to those that meet

critical needs until they are independently determined to be economically,

functionally, and technically justified. Defense believed that such a

limitation would not only adversely affect military readiness, system

development, and contract obligations, but also increase system

obsolescence. GAO understands DOD’s concerns regarding readiness and

contract obligations. That is why the recommendation in the draft report

recognized that critical needs still need to be met. GAO’s point is that

greater management attention needs to be placed on the migration

decision-making process for approving and funding the development and

modernization of migration systems to achieve intended benefits and

Page 9 GAO/AIMD-98-5 Defense Migration Strategy
Executive Summary

minimize unnecessary costs. In order to better clarify this position, GAO

reworded its recommendation to focus on strengthening the controls for

the decision-making process so that these investments can be better

considered in the Department’s budget process.

Defense also did not concur with a proposal in the draft report that it

consider separating the CIO and the ASD C3I positions so that CIO can devote

full attention to departmentwide information resource management issues

and provide independent oversight. It noted, however, that all ASD C3I

functions are being reviewed by the Department’s Task Force on Defense

Reform as part of its review of the Office of the Secretary of Defense’s

organizational structure. GAO withdrew the proposal because DOD is now

considering this matter. Nevertheless, the concern that the current

structure of the CIO position does not allow the CIO to devote full attention

to critical IRM issues—such as computer security, the Year 2000 problem,

and the need to develop and implement an integrated information

technology architecture—remains valid.

Defense also stated that the report negatively portrays Defense’s efforts to

streamline its acquisition processes. GAO disagrees. Acquisition

streamlining allows the Department to greatly simplify its acquisition

processes for information technology purchases. However, acquisition

streamlining was not intended to excuse DOD from exercising management

controls to ensure that those purchases and related system development

efforts made good business sense. Clinger-Cohen implementation should

allow the Department an opportunity to develop more appropriate

controls.

Defense’s comments are discussed in chapter 4 and reprinted in 

appendix I.

Page 10 GAO/AIMD-98-5 Defense Migration Strategy
Page 11 GAO/AIMD-98-5 Defense Migration Strategy
Contents

Executive Summary 2

Chapter 1 

Introduction

16

Background 16

Migration Has Become the Focal Point for CIM 18

Objectives, Scope, and Methodology 24

Chapter 2 

Management Control

Processes Over

Individual Migration

Efforts Have Broken

Down

28

Most Systems Not Approved by Defense’s Senior Technology

Manager

29

Acquisition Oversight Process Did Not Ensure System Selections

Were Economically and Technically Sound

31

Better Oversight Could Have Helped to Prevent Problems

Identified in Previous GAO Reviews

37

Defense Lacks Visibility Over Migration Effort 39

Chapter 3 

Reforming DOD’s

Information

Technology

Investment Process

Requires Effective

Implementation of the

Clinger-Cohen Act

43

The Clinger-Cohen Act 43

DOD Implementation of the Clinger-Cohen Act 45

Challenges Confronting DOD in Implementing Clinger-Cohen 46

Chapter 4 

Conclusions,

Recommendations,

and Agency

Comments

50

Recommendations 51

Agency Comments and Our Evaluation 54

Appendixes Appendix I: Comments From the Department of Defense 58

Appendix II: Description of the Migration System Decision

Process

77

Appendix III: Detailed Analysis of Migration Cost Information 81

Page 12 GAO/AIMD-98-5 Defense Migration Strategy
Contents

Appendix IV: Detailed Analysis of Database Used to Collect

Migration Information

84

Appendix V: Major Contributors to This Report 88

Related GAO Products 89

Tables Table 1.1: Status Reported to GAO on Defense Migration Systems

Selections and Costs

21

Table 1.2: Major Migration and Interim Systems/Applications

Selected for Each Functional Area

22

 Table III.1: Costs Reported to the Congress Compared to Costs

Reported to GAO for Clinical Health, Civilian Personnel, and

Transportation

82

Table IV.1: Levels of Incompleteness in DIST in April 1996 and

February 1997

84

Figures Figure 1.1: An Example of Migration in Practice 18

Figure II.1: Migration Selection and Oversight Processes 77

Figure IV.1: Number of Incorrect and Correct Migration and

Legacy System Dates for Clinical Health, Civilian Personnel, and

Transportation

85

Page 13 GAO/AIMD-98-5 Defense Migration Strategy
Contents

Abbreviations

ASD C3I Assistant Secretary of Defense for Command, Control,

Communications and Intelligence

CIO Chief Information Officer

CIM Corporate Information Management initiative

DOD Department of Defense

DAB Defense Acquisition Board

DII COE Defense Information Infrastructure Common Operating

Environment

DISA Defense Information Systems Agency

DIST Defense Integration Support Tools

FY fiscal year

IRM information resources management

JTA Joint Technical Architecture

MAISRC Major Automated Information System Review Council

OMB Office of Management and Budget

OSD Office of the Secretary of Defense

PSA Principal Staff Assistant

PA&E Program Analysis and Evaluation

SFFAS Statements of Federal Financial Accounting Standards

STARS Standard Accounting and Reporting System

TAFIM Technical Architecture Framework for Information

Management

Page 14 GAO/AIMD-98-5 Defense Migration Strategy
Page 15 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1 

Introduction

Background The Department of Defense (DOD) began its Corporate Information

Management initiative (CIM) in October 1989 to help meet the challenge of

effectively managing its diverse operations as it downsized its forces and

activities. At the time, the Department believed that the thousands of

systems and numerous administrative and mission-related processes1

supporting DOD functions were redundant and inefficient and that they

should be standardized and made more efficient. To address these

problems, Defense planned to

• simplify and improve business processes;

• centralize responsibility and authority in functional areas, such as finance,

personnel, communications, health affairs, logistics, command and

control, and intelligence; and

• develop an integrated communications and data processing infrastructure

based on departmentwide standards.

From the outset, DOD recognized that implementing CIM would be difficult

because the basic tenet of the initiative—managing and implementing

business improvements and building corporate systems along functional

lines—represented a major shift in the way Defense traditionally did

business. Whereas each military and Defense agency had historically

managed its own business functions and information systems, CIM called

on senior functional officials, known as Principal Staff Assistants (PSAs),2

together with their Defense component counterparts to be responsible for

implementing improvements and information systems within the

Department’s business functions across service and agency lines.

As CIM was implemented, Defense emphasized two ways of achieving

process improvements and addressing problems associated with its

disparate and stovepiped information technology environment.3

 The first

was to improve, or reengineer, business processes first and then apply

technology to the new processes. The second involved selecting the best

1

Also known as business processes.

2

PSAs are the Under Secretaries of Defense, the Director of Defense Research and Engineering, the

Assistant Secretaries of Defense, the Director of Operational Test and Evaluation, the General Counsel

of DOD, the Inspector General, the Assistants to the Secretary of Defense and the Office of the

Secretary of Defense (OSD) Directors or equivalents who report directly to the Secretary or the

Deputy Secretary of Defense. In essence, they are top executives at the department level who, together

with their Defense component counterparts, are charged with developing corporate systems and

reengineering business processes within their respective functional areas.

3

The CIM initiative included several aspects, such as (1) corporate policy/planning, (2) process and

data modeling, (3) process improvement, (4) performance measurement, (5) standard information

systems, and (6) computing and communications infrastructure. DOD considered all the aspects of

CIM to be important to effectively field information systems that support its mission.

Page 16 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1 

Introduction

DOD information systems from pools of existing, or legacy, systems that

provide similar automated support services and eventually replacing the

duplicative systems with the best systems. A few years into the CIM effort,

this became known as migration.4

 Defense believed that if done properly,

migration could cut costs associated with developing and maintaining

disparate systems supporting the same functions. It also believed that

migration would help to standardize business processes and allow Defense

to achieve savings that could be put to better use in advancing warfighting

capabilities. Figure 1.1 illustrates how migration would work in one

functional area.

4

In DOD, migration can also involve developing or acquiring a new system, rather than choosing from

existing systems. However, most migration efforts have involved choosing from existing systems.

Page 17 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1 

Introduction

Figure 1.1: An Example of Migration in

Practice

For its finance and accounting functions, Defense's goal is to reduce the

number of systems from the 324 that the services and agencies operated

in 1991 to 32 systems. 

As of the end of fiscal year 1996, the Defense Finance and Accounting

Service's strategic plan stated it had reduced the number of finance and

accounting systems to 217. 

The migration strategy for accounting and finance is to 

(1) identify a single migration system for each of the Department's

finance and accounting functions, such as civilian pay,

(2) standardize the procedures and practices used to perform these

functions, and

(3) migrate from existing service-unique systems to the migration

systems.

For some of these functions, such as general fund accounting, Defense

established a two-pronged approach:

(1) migrate initially to a single interim system or set of systems within

each service and

(2) migrate from the service systems to a departmentwide system or

set of systems.

Migration Has

Become the Focal

Point for CIM

While Defense began CIM emphasizing the need to improve, or reengineer,

processes before applying technology, it ended up placing more priority on

obtaining quick savings through an accelerated migration strategy, which

began in October 1993. This strategy called for selection of migration

systems in 6 months and departmentwide transition to selected systems in

the following 3 years. Defense believed that setting tight time frames for

migration with some potential slippage would allow it to “harvest the

Page 18 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1 

Introduction

low-hanging fruit” of potential savings before reengineering. By default,

this meant that the increased emphasis on migration would postpone the

dramatic gains that could be achieved through reengineering.5

The risks involved with postponing reengineering efforts were significant.

Reengineering identifies, analyzes, and redesigns an organization’s core

business processes, aiming to achieve dramatic improvements in critical

areas of performance, such as cost, quality, service, and speed. It focuses

on redesigning the business process as a whole in order to achieve the

greatest possible benefits to an organization and its customers. Migration,

on the other hand, focuses on standardizing existing processes and

information systems. This can yield some benefits, such as reducing the

need to maintain disparate systems that support the same functions, but

seldom yields dramatic improvements. In addition, if the process is

inefficient or outmoded, migration only serves to perpetuate a bad

process. Finally, choosing migration before reengineering may cause

future reengineering efforts to be more difficult by entrenching inefficient

and ineffective work processes.

We raised these concerns both before6

 and after7 Defense decided to

embark on the accelerated migration strategy. We believed that the shift in

emphasis toward migration seriously endangered CIM’s chances for

success. For this reason, and because CIM-related technology investments,

which are expected to total billions of dollars each year, are vulnerable to

waste and mismanagement, we designated CIM as a high-risk government

information technology initiative.8

 Nevertheless, Defense proceeded to

concentrate on migration even in areas where it had recognized that there

was a significant need to change business processes. One such area was

transportation. We reported in 1996 that even though Defense recognized

that its military transportation processes were fragmented, outdated,

5

In an October 1993 memorandum, the Deputy Secretary of Defense announced a near-term strategy

that focused on migration and data standardization. The memorandum stated that completion of other

initiatives were not to be prerequisites of implementation of migration systems and data standards.

While some functional areas may have continued process reengineering initiatives, our work indicates

that many others gave priority to the migration strategy.

6

Defense ADP: Corporate Information Management Must Overcome Major Problems

(GAO/IMTEC-92-77, September 14, 1992).

7

Defense Management: Stronger Support Needed for Corporate Information Management Initiative to

Succeed (GAO/AIMD/NSIAD-94-101, April 12, 1994) and Defense Management: Impediments

Jeopardize Logistics Corporate Information Management (GAO/NSIAD-95-28, October 21, 1994).

8

Our high-risk effort, which began in 1990, identifies those federal program areas we consider high risk

because they are especially vulnerable to waste, fraud, abuse, and mismanagement. See High-Risk

Series: Information Management and Technology (GAO/HR-97-9, February 1997) and High-Risk Series:

An Overview (GAO/HR-95-1, February 1995) for a discussion on CIM.

Page 19 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1 

Introduction

inefficient, and costly, it focused on technology solutions rather than the

need to identify and correct the root causes of its transportation problems.9

Because reengineering took a backseat to migration in other business

areas as well, CIM has never been able to achieve the level of cost savings

and process improvements originally intended. A recent report10

conducted by RAND, a consulting organization, notes that “The CIM effort

is today widely viewed as a failure in most quarters of the DOD. It has not

resulted in either significant process reengineering or visible savings in the

hardware and software required to support all the varied information

systems in the Defense infrastructure.”

Defense is still planning to invest at least $18 billion in its migration effort

from fiscal years 1995 through 2000. Because this investment is significant,

we were asked for information on the status and progress of migration and

whether Defense has significant controls in place to manage and oversee

the effort.

Status of the Migration

Effort

Defense reported that it has selected 363 systems for migration. Forty-nine

of the 36311 migration systems are large-scale, or major, systems12 that are

expected to cost at least $13 billion—or about 72 percent of the total

$18 billion cost estimate—to develop, deploy, and maintain over fiscal

years 1995 through 2000. Defense has identified 1,938 legacy systems for

potential termination. As of April 1996, its Defense Integration Support

Tools (DIST) database showed that 281 had already been terminated, 886

were scheduled for termination by the year 2000, and 771 more were

identified for potential elimination after the year 2000.

Table 1.1 provides the number of systems selected by each functional area,

the number of legacy systems identified for potential termination, and

DOD’s estimated costs of developing, deploying, and maintaining the

migration systems. We excluded costs for 27 systems for which DOD

9

Defense Transportation: Migration Systems Selected Without Adequate Analysis (GAO/AIMD-96-81,

August 29, 1996).

10Strategic Appraisal 1997: Strategy and Defense Planning for the 21st Century, February 1997, RAND’s

Project Air Force; edited by Zalmay M.Khalilzad and David A Ochmanek.

11The numbers of migration systems selected and legacy systems identified for potential termination

are subject to change as PSAs make additional selections and reevaluate previous ones. Also, the PSAs

classified 138 of the 363 migration systems as interim systems.

12In DOD, major information systems projects are those that (1) have estimated program costs in

excess of $30 million in any 1 year, (2) have estimated program costs of over $120 million in total,

(3) have total life-cycle costs of over $360 million, or (4) are designated as being of special interest.

DOD periodically revises these dollar thresholds.

Page 20 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1 

Introduction

collected costs because the systems were classified. And costs for the

remaining 121 of 363 systems were not included in the table because the

Office of the ASD C3I had not collected the costs for these systems. Table

1.2 provides the name, status, and cost for each of the 49 major systems.

Table 1.1: Status Reported to GAO on Defense Migration Systems Selections and Costs

Dollars in millions

Functional area

Number of legacy

systems identified

for potential

termination

Number of migration/

interim systems

selected

Number of migration/

interim systems

reporting costs

Total cost reported to

GAO for fiscal years

1995 through 2000

Command and Control 67 35 34 4,493.2

Intelligence 304 69 46 572.6a

Mission Support

Atomic Energy 7 8 2 12.7

Communications 17 2 2 3,483.2

Environmental Security 286 11 6 33.1

Financeb 267 52 31 1,417.1

Health 62 55 49 2,533.8

Human Resources 198 20 10 933.1

Information Management 21 5 0 0.0

Inspector General 8 2 1 15.8

Logistics 496 63 53 4,459.1

Meteorology and

Oceanography 33 33 0 0.0

Policy 3 7 7 73.9

Procurement 67 1 1 282.1

Science and Technology 102c 0c 0c c

Total 1,938 363 242 $18,309.7

Note: We did not independently verify information provided in this table.

aCosts were not included for 27 systems for which DOD collected cost data because the systems

were classified.

bThe information presented for finance is not consistent with information presented in figure 1.1 of

this chapter because the counts of systems occurred at different points in time.

cThe science and technology functional area has not yet selected migration systems and has not

specified which, if any, of its legacy systems may be terminated.

Source: Information obtained from the DIST, DOD’s April 1996 Report to the Congress,

discussions with Defense functional area managers, and the ASD C3I. Cost data were provided as

of December 1996.

Page 21 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1 

Introduction

Table 1.2: Major Migration and Interim Systems/Applications Selected for Each Functional Area

Dollars in millions

System name Acronym

Current

milestonea

Total cost data

reported to GAO for

fiscal years 1995

through 2000

Systems being reviewed by the Defense Acquisition Board (six systems):

 Command and Control systems

1 Advanced Field Artillery Tactical Data

System AFATDS 3 $ 320.6

2 Combat Service Support Control System CSSCS 2 104.9

3 Forward Area Air Defense C2

Intelligence System FAADC2I 3 216.4

4 Maneuver Control System MCS 3 253.7

 Intelligence systems

5 All Source Analysis System ASAS 3 347.9

6 Joint Service Imagery Processing System JSIPS 2 b

Systems being reviewed by the Major Automated Information System Review Council (43 systems):

 Command and Control systems

1 Air Force Mission Support System AFMSS 3 351.9

2 Army Global Command and Control

System AGCCS 3 254.6

3 Counter Narcotics Command

Management Control System CN/CMS 3 112.2

4 Global Command and Control System GCCS 3 422.1

5 Naval Aviation Logistics Command

Management Information System NALCOMIS 3 93.5

6 Naval Tactical Command System - Afloat NTCS-A 3 314.1

7 Operations Support System OSS 3 174.1

8 Shipboard Non-Tactical ADP Program III SNAP III 3 401.4

9 Strategic War Planning System SWPS 3 343.4

10 Tactical Support Center TSC 3 154.3

 Intelligence systems

11 Exploitation Support System ESS 0 b

12 High Performance Computing

Modernization Program HPCMP 2 b

 Mission support systems

 Communications

13 Defense Information System Network DISN 0 2,556.2

14 Defense Message System DMS 2 927.0

(continued)

Page 22 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1 

Introduction

Dollars in millions

System name Acronym

Current

milestonea

Total cost data

reported to GAO for

fiscal years 1995

through 2000

 Finance

15 Defense Joint Military Pay System DJMS 1/2 114.9

16 Defense Procurement Payment System DPPS 1/2 68.5

17 Standard Accounting And Reporting

System STARS 2/3 263.7

 Health

18 Ambulatory Data System ADS 3 90.3

19 Composite Health Care System CHCS 3 785.6

20 Composite Health Care System II CHCS II 0 1.4

21 Corporate Executive Information System CEIS 0/1/2 18.2

 Human resources

22 Defense Civilian Personnel Data System DCPDS 1 58.5

23 Defense Commissary Information System DCIS 2 120.8

24 Defense Commissary Point of Sales

System POS 3 262.4

25 Joint Recruiting Information Support

System JRISS 0 207.0

26 Navy Standard Integrated Personnel

System NSIPS 0 b

27 Reserve Component Automation System RCAS 2/3 b

28 Standard Installation/Division Personnel

System - 3 SIDPERS-3 2 b

 Logistics

29 Ammunition Management Standard

System AMSS 0 55.7

30 Command and Control Information

Processing System C2IPS 3 311.8

31 Defense Automatic Addressing System DAAS 3 12.7

32 Defense Medical Logistics Standard

Support System DMLSS 2 129.0

33 Department of Army Movements

Management System - Redesigned DAMMS-R 3 43.9

34 Depot Maintenance System DMS 0c 438.6

35 Distribution Standard System DSS 3 270.2

36 Global Transportation Network GTN 2 179.5

37 Joint Computer-Aided Acquisition and

Logistics System JCALS 2 789.0

38 Joint Engineering Data Management

Information and Control System JEDMICS 3 243.7

(continued)

Page 23 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1 

Introduction

Dollars in millions

System name Acronym

Current

milestonea

Total cost data

reported to GAO for

fiscal years 1995

through 2000

39 Materiel Management System MMS 0c 1,149.2

40 Transportation Coordinators’ Automated

Information for Movement System II TC-AIMS II 0 15.1

41 Transportation Operational Personal

Property System TOPS 3 95.4

 Meteorology & Oceanography

42 Primary Oceanographic Prediction

System POPS 3 b

 Procurement

43 DOD Standard Procurement System SPS 1 282.1

Total $13,355.5

Note: We did not verify the information provided in this table. As discussed in chapter 2, these

costs may be inaccurate.

aThere are four milestones in the major system acquisition review process. These are milestone

0—approval to conduct concept studies; milestone 1—approval to begin a new acquisition

program; milestone 2—approval to enter engineering and manufacturing development; and

milestone 3—approval to produce, field, or deploy the system. Multiple milestones may be shown

for some projects because the projects’ subsystems or applications are in different phases of the

acquisition process.

bThe Office of the ASD C3I did not collect costs for this system.

cBoth of these programs were redirected and are no longer targeted to be standard migration

systems.

Source: Information DOD reported to the Congress in April 1996 and to GAO in the course of this

audit.

Objectives, Scope,

and Methodology

We were asked (1) for information on the number and cost of systems

designated for migration, the number of legacy systems already terminated

or scheduled for termination, and savings resulting from terminations of

legacy systems and (2) whether Defense’s management control and

oversight processes for migration systems are ensuring that the

investments are economically sound and in compliance with Defense’s

technical and data standards.

To assess the status of Defense’s overall migration strategy and obtain

available information on the number and cost of systems designated for

migration, the number of legacy systems already terminated and scheduled

Page 24 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1 

Introduction

for future termination, and savings resulting from terminations of legacy

systems, we analyzed a Defense April 1996 report to the Congress

containing information on the schedule, cost, and status of the migration

systems. Defense prepared this report13 in response to a requirement in

Section 366 of the National Defense Authorization Act for fiscal year 1996.

We also obtained and analyzed a copy of the DIST database as of April 1996

that Defense used to develop the report to the Congress, as well as to

provide its own senior managers information on migration system

selections and legacy system terminations. Additionally, we reviewed—but

did not independently verify—cost information that Defense reported to

Congress as part of the April 1996 Section 366 report. We also interviewed

senior DOD officials to determine if additional cost and performance

information on migration systems existed at the Office of the Secretary of

Defense (OSD) level.

We assessed the reliability of the data presented to the Congress in two

ways. First, we identified missing and conflicting information in the

April 1996 Section 366 report to the Congress and requested that the ASD

C3I staff to clarify conflicting information. Second, we compared the data

reported to the Congress for three functional activities (business

areas)—Transportation, Civilian Personnel, and Clinical Health—against

data provided to us by the functional area managers at these activities. We

then established and analyzed a database containing schedule and

available cost information for the migration and legacy systems and

modified the database to reflect updated schedule, costs, and other

descriptive data Defense provided to us during the course of the audit.

To assess Defense’s management control and oversight processes for

migration systems to determine whether the investments are economically

justified and comply with Defense technical and data standards, we

reviewed the department-level management and oversight processes for

approving the PSAs’ migration system selections and for overseeing major

migration systems’ acquisitions by the Major Automated Information

System Review Council (MAISRC).14 Our review did not focus on the process

by which the PSAs select, develop, and manage their migration systems.

13As noted in this chapter, this report did not include costs for 121 information systems.

14MAISRC provides acquisition oversight for information systems that (1) are anticipated to cost

$30 million or more a year, (2) have estimated program costs in excess of $120 million, (3) have

estimated life-cycle costs of more than $360 million, or (4) are designated for review by the ASD C3I,

who also chairs the MAISRC. MAISRC reviews such matters as whether a proposed system is being

developed in accordance with Defense policies, procedures, and regulations and whether system

managers took steps to minimize the cost of a new system.

Page 25 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1 

Introduction

Nor did we examine acquisition review processes within the individual

functional areas employed for nonmajor system projects.

To analyze Defense’s oversight processes for reviewing and approving the

PSAs’ migration system selections, we reviewed department-level approval

of the PSAs’ selections. We obtained a list of migration systems that PSAs

had selected for their functional areas. We then visited the Office of the

ASD C3I and the Defense Information Systems Agency (DISA) and reviewed

the business case analysis documentation, technical analysis

documentation, and other documentation provided to those offices for

department-level approval of the PSA’s migration and interim system

selections. We also interviewed officials from the Office of the ASD C3I,

DISA, and the offices of selected PSAs regarding the documentation

supporting the selections.

To review Defense’s acquisition oversight processes for major migration

systems, we first identified the major migration systems, as defined by

Defense regulations. We then obtained and analyzed the progress reports

and other documentation provided to MAISRC for those systems. We also

interviewed Defense representatives and officials responsible for

information systems oversight in the Office of the ASD C3I and offices of

DISA, the Defense Acquisition Board (DAB), MAISRC, and Program Analysis

and Evaluation (PA&E). We also interviewed representatives of the program

offices or oversight offices for selected systems for which oversight had

been delegated by MAISRC. Through document reviews and interviews, we

determined whether economic analyses for major migration systems had

been independently reviewed and if so, whether the reviews had identified

problems in the analyses. We also determined whether DOD guidance

existed on preparing economic analyses for information systems. A GAO

economist met with PA&E analysts, reviewed problems that were identified

with economic analyses, and verified the validity of the problems reported.

Additionally, we determined whether MAISRC had information on

alternative analyses performed for major migration system selections.

Lastly, we determined whether MAISRC has sufficient oversight information

to ensure that major migration systems are complying with applicable

technical and data standards that are necessary to achieve an

interoperable systems environment.

We reviewed Defense’s policies and guidance for migration systems to

ensure that information technology is acquired, managed, and used in the

most efficient and effective manner. Our assessment included analyzing

Page 26 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 1 

Introduction

the National Defense Authorization Acts, Committee reports, and

Conference reports for fiscal years 1993 through 1997; Defense Office of

Inspector General reports; our prior studies of CIM and the migration

system strategy, other available evaluations of the CIM and the migration

strategy, and existing legislation affecting these efforts. The major studies

reviewed included studies by RAND and the Defense Science Board.15

Pertinent existing legislation includes the Clinger-Cohen Act, the

Government Performance and Results Act, the Paperwork Reduction Act,

and the Chief Financial Officers Act.

We conducted our review from August 1996 through July 1997 in

accordance with generally accepted government auditing standards. We

requested comments on a draft of this report from the Department of

Defense. The Acting Secretary for Command, Control, Communications

and Intelligence provided us with written comments. These comments are

discussed in chapter 4 and reprinted in appendix I.

15Achieving an Innovative Support Structure for 21st Century Military Superiority: Higher Performance

at Lower Costs, 1996 Summer Study, Defense Science Board, Department of Defense. The Defense

Science Board is a Federal Advisory Committee established to provide independent advice to the

Secretary of Defense. Statements, opinions, conclusions, and recommendations in their reports do not

necessarily represent the official position of DOD.

Page 27 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

Embarking on the migration strategy was an extremely risky endeavor for

the Department of Defense. First, in developing standard systems within

functional areas, the Department had to carefully consider complex

technical issues, such as interfacing migration systems with other systems

and contending with nonstandard data formats and definitions. For

Defense, such complexities were compounded by its sheer size and the

numbers of disparate systems. We believe an even tougher obstacle facing

Defense, however, was the prevailing culture, which was based on

decentralized department organizational structure, nonstandard processes

and procedures. In general, each military service and Defense agency has

historically managed its own business functions and information

technology projects, whereas CIM and migration required business

improvements and information systems to be managed on a

Departmentwide, or corporate basis.

Therefore, to ensure the strategy’s success, it was vital for Defense to

establish an effective decision-making and oversight environment for

making migration investments. It would need controls and processes that

ensured inefficient administrative and mission-related work processes

were modernized before significant technology investments were made to

support them. It would also need controls that ensured that the migration

projects themselves were effectively managed as investments so that the

Department could target resources and attention to priority areas and stop

those projects that failed to meet their goals. Finally, Defense needed life

cycle management controls that ensured, on a system-by-system basis,

that sound management and development practices were followed.

However, Defense’s primary corporate-level control mechanism for

ensuring that sound management and development practices were

followed for each migration investment has not been effective. This

control requires that all selections to be submitted for approval by the

Assistant Secretary of Defense for Command, Control, Communications

and Intelligence (ASD C3I), who is also the Department’s Chief Information

Officer (CIO). In approving the systems, the Assistant Secretary is to review

data, technical, and programmatic factors relating to the selection. We

found, however, that most migration selections never came in for this

approval and those that did were approved in the absence of critical

technical and programmatic supporting documents. (A description of the

migration decision process is provided in appendix II.)

Because this control was playing a marginal role in ensuring the success of

migration, we assessed whether Defense’s acquisition oversight

Page 28 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

processes—which are designed to augment management oversight for the

Department’s most expensive information systems—were helping to

ensure that the major migration investments were economically and

technically sound. These processes have also broken down where

migration is concerned. Had there been more rigorous attention to

oversight in both areas, Defense may well have avoided the migration

problems we identified in previous reviews that cost the Department

hundreds of millions of dollars.

In implementing the migration strategy, Defense did not ensure that it had

adequate visibility over status, costs, and progress. As a result, it has not

been able to (1) demonstrate whether the migration strategy has been

successful, (2) provide the Congress with accurate and reliable

information needed for oversight purposes, and (3) provide its own

decisionmakers at the headquarters level with information needed to

oversee the strategy.

Most Systems Not

Approved by

Defense’s Senior

Technology Manager

After selecting a migration system, the Principal Staff Assistant (PSA)1

responsible for a functional area is to submit the system for approval by

the Assistant Secretary of Defense for Command, Control,

Communications and Intelligence—the Senior Information Management

Official and Chief Information Officer in DOD. In his memorandum2

 setting

forth the requirements for selecting and reviewing migration systems, the

ASD C3I stated that in approving the selections, he would consider data,

technical, and programmatic factors. For example, the ASD C3I review

would include such issues as whether the migration systems will lend

themselves to data sharing and whether they conform to DOD’s technical

standards—which act as a set of “building codes” for constructing systems

to ensure they will be compatible with its information infrastructure and

1

The PSA is the senior executive-level manager who is responsible for the management of a defined

function or functions within DOD.

2

Requirements for selecting and reviewing migration systems under DOD’s accelerated migration

strategy are described in two memorandums. The first memorandum was issued by the Deputy

Secretary of Defense in October 1993, and the second was issued by the Assistant Secretary of Defense

for Command, Control, Communications and Intelligence in November 1993.

Page 29 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

technically interoperable with each other.3 This approval is important

because, as the Department’s Chief Information Officer, the Assistant

Secretary needs to ensure that the migration systems help facilitate the

sharing of information across service and agency lines.

However, both the PSAs and the ASD C3I did not adhere to this oversight

process. First, 245 selections, or about 67 percent, were never submitted

for this oversight even though all 363 system selections should have 

been.4, 5 Of the 49 major, or large scale, migration systems, only 16 were

submitted for this review. Thus, for the bulk of migration systems, the

Assistant Secretary was unable to ensure that Defense technical standards

would be met and that the best system development practices were being

followed.

Second, most of the systems that were submitted for this review were

approved in the absence of critical technical and programmatic business

case support. For example, 23 of the 118 selections that were submitted

for this oversight, about 19 percent, were approved without documents

stating that the functional area complied or planned to comply with

technical standards. In addition, 64 of the 118 selections, about 54 percent,

were approved without documents showing that the PSA responsible for

the functional area performed some type of functional economic analysis

or other similar business case analysis. This analysis is the means by

which Defense managers are supposed to evaluate different options for

improving business areas, such as outsourcing, reengineering, or

migration. Because all of the migration selections submitted for this

3

Defense’s technical and data standards are designed to enable systems to easily interoperate and

transfer information. Its standard definitions for data elements are intended to ensure that users of all

Defense systems define the same data in the same way and have a common understanding of their

meaning. Defense has developed or is in the process of defining technical standards in the Technical

Architecture Framework for Information Management (TAFIM), the Joint Technical Architecture

(JTA), and the Defense Information Infrastructure Common Operating Environment (DII COE). The

Defense Information Systems Agency (DISA) is responsible for developing, obtaining from commercial

sources, and maintaining the compilation of Defense Information Infrastructure technical standards,

and it is responsible for maintaining a Defense data dictionary system as a repository of data

requirements and for facilitating the cross-functional coordination and approval of standard formats,

definitions, etc. PSAs, the military services, Defense agencies, and Joint Chiefs of Staff are responsible

for reaching agreement on the standards and approving them as DOD standard data elements. DISA is

then responsible for disseminating the approved standard data elements for use throughout the

Department.

4

Nine of the systems that were submitted for approval were grandfathered into approval because

development for them was well underway at the time the accelerated migration system strategy began

in 1993.

5

Forty-two of the systems that were approved by the ASD C3I as migration systems did not follow

DOD’s migration system approval process described in appendix II. Instead, the ASD C3I approved

these 42 Command and Control systems based on reviews by the Military Communications Electronics

Board.

Page 30 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

oversight have been approved—whether adequately supported by

technical and business case justification or not—the ASD C3I oversight has

essentially become a meaningless process.

Acquisition Oversight

Process Did Not

Ensure System

Selections Were

Economically and

Technically Sound

Because the ASD C3I oversight control over the migration strategy played a

marginal role in ensuring that sound management and development

practices are followed, we assessed whether Defense’s traditional major

information systems acquisition oversight processes were doing so for 43

of the major migration systems.6

 These review processes are designed to

assess whether projects are affordable and financial and operational risks

have been minimized.

Once under acquisition oversight, systems are normally reviewed and

approved at each of four milestones.7

 The reviews involve assessing such

matters as whether the proposed system is being developed in accordance

with Defense policies, procedures, and regulations; whether the systems’

program managers took steps to minimize the cost of a new system by

ensuring full and open competition; and whether the program managers

will effectively use advanced system design and software engineering

technology to minimize software and maintenance costs.

However, these reviews have not been effective for the migration effort.

As discussed in the following section, MAISRC could not provide sufficient

assurance that the major migration systems are economically justified and

comply with Defense’s technical and data standards.

Ineffective Review and

Preparation of Economic

Analyses

A principal tool of acquisition oversight is the economic analysis because

it helps to ensure that the system chosen for development is cost-effective.

If done properly, it can enable reviewers to determine whether DOD has

6

As noted earlier, there are 49 major migration systems. Six of these are under the oversight of the

Defense Acquisition Board (DAB). This is Defense’s senior-level forum for advising the Under

Secretary of Defense (Acquisition and Technology) on critical decisions concerning major weapon

systems and large-scale information systems, principally in the communication, command, control,

and intelligence functional areas. Criteria for DAB oversight are (1) estimated expenditures for

research, development, test, and evaluation of more than $355 million or (2) estimated procurement

costs of more than $2.135 billion. One difference between the DAB and MAISRC oversight processes is

that systems reviewed by DAB are required to have independent cost estimates rather than economic

analyses. According to the DAB analyst responsible for coordinating oversight of these systems,

Program Analysis and Evaluation (PA&E) analysts performed independent cost estimates for all six

migration systems under DAB oversight.

7

The four milestones are: milestone 0, approval to conduct concept studies; milestone 1, approval to

begin a new acquisition program; milestone 2, approval to enter engineering and manufacturing

development; and milestone 3, approval to produce, field, or deploy the system.

Page 31 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

sufficient funds in its budget to pay for the system, that is, whether the

system is affordable. It can also reveal potential conflicts between

available funding and the planned schedule for deployments. To arrive at

these conclusions, the economic analysis establishes baseline life-cycle

costs and benefit estimates for the project and calculates the project’s

return on investment.

The importance of developing complete and accurate economic analyses

is underscored by several governmentwide requirements. For example, the

Office of Management and Budget’s (OMB) Circular A-130, Management of

Federal Information Resources, calls on agencies “to conduct benefit-cost

analyses to support ongoing management oversight processes that

maximize return on investment and minimize financial and operational

risks for investments in major information systems on an agencywide

basis.” Likewise, OMB’s Circular A-11, Part 3, Planning Budgeting and

Acquisition of Fixed Assets, (July 16, 1996), and its Bulletin No. 95-03,

Planning and Budgeting for the Acquisition of Fixed Assets, state that “the

planning for fixed asset acquisitions should be based on a systematic

analysis of expected benefits and costs.”

However, even though economic analyses play a critical role in assessing

whether system development efforts will be cost-effective and beneficial,

Defense has not yet established a set of minimum standards that an

economic analysis must meet to be considered valid. In addition, it has not

published official guidance for preparing an economic analysis. PA&E

developed and published an unofficial economic analysis guide that

recommended, but did not require, standard methods and formats for

preparing an economic analysis for an information system

development/modernization project. According to a PA&E representative,

this unofficial guide is no longer adequate because it does not require the

degree of standardization in methodology and analytic techniques needed

to support a portfolio management approach for managing information

technology investments, as required by the Clinger-Cohen Act of 1996. For

example, the guide does not require that returns on investment for systems

development/modernization projects be calculated in a standard manner

using a standard definition. This lack of standardization results in

economic analyses for different systems that are not comparable enough

for DOD managers to have a good basis for deciding which systems offer

the highest payoffs. The PA&E official stated that in conjunction with DOD’s

implementation of a portfolio management approach, it plans to officially

publish appropriate documents and provide minimum standards and

guidance for the economic analyses process.

Page 32 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

Our review also found that DOD decisionmakers do not view economic

analyses as key tools for deciding whether to invest in an information

system development or modernization project. As a result, DOD often lacks

complete and accurate information on system development/modernization

projects’ estimated costs and benefits at the time decisions are made to

invest in the projects. Specifically, as the following examples show, we

found that (1) economic analyses for many systems have not been

submitted for independent review and (2) a significant portion of those

that were submitted were inadequately prepared. Thus, many of the

benefits that could be derived from this tool have not been realized.

• Twelve of the 43 major migration systems have not yet submitted an

economic analysis, or an update of a previously prepared economic

analysis, for independent review.8

 These systems were under development

or modernization and DOD had invested hundreds of millions of dollars in

them in total. These included 5 systems that were under direct MAISRC

oversight and 7 systems for which MAISRC delegated oversight

responsibility to a service or agency. These 12 systems, or components of

them, are all in the second development phase or beyond.9

 Delaying the

preparation or updating of a previously prepared economic analysis to the

later stages of development for these 12 systems defeats the purpose of

the economic analysis, which is to demonstrate that a proposal to invest in

a new system is valid before that investment is made.

• Four systems that were under direct MAISRC oversight, were in the first

development phase—concept exploration—and were, therefore, not yet

required by DOD’s acquisition regulations to submit an economic analysis

or an update of a previously prepared economic analysis for independent

review. Although these four systems were technically in compliance with

Defense’s acquisition regulations, DOD had already made major

investments in them without the benefit of knowing whether returns on

investment are going to be acceptable.

• Even though Defense has not established standards for the required

analyses, Program Analysis and Evaluation (PA&E) staff who review the

economic analyses that are under direct MAISRC oversight told us that 10 of

8

Of the 43 major migration systems under MAISRC review, 27 were under direct review by MAISRC

and 16 were delegated to other Defense oversight organizations, with MAISRC retaining responsibility

for ensuring adequate oversight.

9

The responsibility for independently reviewing economic analyses for the five systems that were

under direct MAISRC oversight rests with DOD’s Program Analysis and Evaluation staff, while the

responsibility for reviewing economic analyses for the seven delegated systems rests with the various

Defense organizations to which MAISRC delegated oversight responsibility.

Page 33 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

the 19 economic analyses reviewed had problems.10 These problems

included the following:

• Understating or omitting the costs of standardizing data, implementing

the standard data in the systems, and developing system interfaces.

• Failing to estimate the amount of savings expected for terminating

duplicative legacy systems.

• Relying on professional judgment to make unsupported assumptions

rather than making objective analyses to estimate the value of benefits

and costs. For example, in one case, the economic analysis estimated a

cost avoidance associated with replacing an old system by assuming

that both the old and the replacement systems’ software maintenance

costs could be accurately estimated using two different rates per line of

code, but the analysis provided no data supporting the validity of either

rate.

After identifying problems with these 10 economic analyses, PA&E staff

worked with MAISRC analysts and the systems’ program managers to

address the problems. However, DOD continued to develop the systems in

spite of the fact that the investments had not been justified by complete

and accurate economic analyses.

PA&E analysts told us that there were other problems that impeded their

review and verification of the economic analyses. For example, economic

analyses are often not updated and provided to PA&E for review after major

changes occur in the project, such as significant cost growth or redirection

of the project. A second problem is that economic analyses are often not

supported by analyses of alternatives that weigh the cost and benefits of

various technical options, such as whether to buy commercial off-the-shelf

software or develop a system in-house.11 This analysis would help Defense

decisionmakers make sound decisions on whether the proposed

alternatives offer sufficient military or economic benefits to be worth their

cost, and to determine which alternative is the best approach. It would

also identify alternatives that DOD may want to reconsider at a later time if

the selected approach runs into difficulties.

10We did not identify concerns or problems with the economic analyses for the remaining 8 of the 43

major migration systems that were under direct MAISRC oversight for which MAISRC had delegated

oversight responsibility to another DOD organization.

11The analysis of technical alternatives would normally precede the economic analysis. In Defense, an

economic analysis weighs the costs, benefits, and risks associated with maintaining the status quo

versus the chosen technical solution.

Page 34 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

DOD’s Acquisition

Oversight Organizations

Lack Assurance That Major

Systems Comply With

Technical and Data

Standards

Defense has established several sets of standards that are designed to

ensure that systems developed are compatible with its communications

and computing infrastructure and that they are technically interoperable

with each other. Some Defense leaders consider systems interoperability

and the ability to exchange data across functional lines to be the most

important consideration in migration system development, transcending

economic benefits. These standards include the Technical Architecture

Framework for Information Management (TAFIM), Defense Information

Infrastructure Common Operating Environment (DII COE), and DOD

standard data.12 DOD system acquisition directives call on MAISRC and DAB to

ensure that program managers comply with the Department’s policies and

procedures and use best practices in developing and modernizing

individual information systems. These best practices include building

systems and databases that comply with applicable technical standards

and use DOD standard data.

However, MAISRC and DAB do not have adequate assurance that the major

migration systems are complying with applicable technical standards13 and

are using standard data. For example, out of 43 major systems under

MAISRC oversight, program managers reported to MAISRC that (1) only 19

were in compliance with the TAFIM standards or had plans to comply with

the TAFIM standards and (2) 9 systems were using DOD standard data or had

plans to use standard data.14 We found similar results for compliance with

DII COE standards. Specifically, program managers reported that only 25

systems were in compliance or had plans to be in compliance with DII COE.

These self-reports by program managers are questionable because they are

not independently verified.

In addition to obtaining information on technical and data standards

directly from program managers, MAISRC and DAB also obtain such

information from the Defense Information Systems Agency (DISA). DISA

supports MAISRC and DAB oversight of major systems by performing

technical reviews of system documentation to determine if it indicates that

interoperability issues are being addressed or will be addressed as the

12See footnote 3 for more information on technical and data standards.

13All DOD technical standards may not apply to all systems and initiatives. For example, those

standards that apply to application software and data would not apply to a system or initiative that

included only computer equipment or hardware. To illustrate, Defense’s High Performance Computing

Modernization Program initiative largely involves the purchase of computer equipment and not the

development of software. Therefore, application software-related standards may not apply.

14In addition, information available to DAB indicated that all six migration systems under DAB

program managers have already prepared plans to bring their systems into compliance with DOD

technical standards.

Page 35 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

system is developed. According to a DISA representative, DISA reviewed

system documentation that had been prepared for 37 of the 43 MAISRC

migration systems and all 6 of the DAB systems and found that the

documentation indicated interoperability issues were planned to be

addressed for each of them.

DISA also provides feedback, advice, and assistance on interoperability and

related issues during integrated product team meetings with MAISRC, DAB,

and system program managers. During these meetings and on other

occasions when it is asked to do so, DISA provides MAISRC and DAB general

information on compliance with standards, such as whether the major

systems’ program managers have contacted DISA and appear to be making

reasonable attempts to bring their systems into compliance with

applicable standards. DISA also assists program managers in developing

strategies and cost estimates for achieving compliance with standards.

However, DISA does not regularly report detailed information to MAISRC and

DAB that would enable these oversight organizations to ensure that each

major system is adequately complying with applicable technical and data

standards.15 For example, DISA does not regularly report to MAISRC and DAB

such detailed information as (1) each system’s current compliance with

applicable standards—that is, each system’s current status relative to the

eight levels that DOD has defined for the DII COE, and each system’s number

and percentage of data elements that have been approved as DOD data

standards in the Defense Data Dictionary System, (2) whether each

program office prepared sound strategies, schedules, and cost estimates

for achieving compliance with applicable standards, (3) whether each

system’s compliance strategy and cost estimate were approved by DISA,

(4) each system’s current schedule and cost status for achieving

compliance compared with its baseline schedule and cost estimate, and

(5) whether each system has been independently certified by DISA’s Joint

Interoperability Testing Command to be in compliance with the technical

and data standards. MAISRC analysts responsible for overseeing the major

migration systems confirmed that they did not know many of the systems’

current status regarding compliance with applicable technical standards

and use of DOD standard data, or whether the program managers had

15A DISA representative stated that program managers are supposed to input data into DOD’s DIST

database that could be used by MAISRC and DAB to track each individual system’s progress in

complying with DII COE. For example, the representative noted that program managers are supposed to

input compliance data into DIST, including (1) their systems’ current level of compliance with DII COE,

(2) information on their strategies for achieving compliance, and (3) the expected date and cost to

achieve compliance. However, as discussed in appendix IV, our analysis of the data in the DIST

database showed they are incomplete and inaccurate. Until DOD corrects these deficiencies, MAISRC

and DAB cannot rely on DIST for tracking compliance with standards for individual systems.

Page 36 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

developed and were following sound strategies for bringing the systems

into compliance.

The technical and data standards are supposed to help pave the way to an

interoperable systems environment. However, without complete and

accurate data on individual systems’ compliance with standards, MAISRC

and DAB cannot assure Defense’s Chief Information Officer that the

Department’s major information systems are complying with the

standards. Without this information, as well as standards compliance

information from the managers of DOD’s nonmajor systems, the CIO cannot

gauge the Department’s progress toward achieving its goal of an

interoperable systems environment. Additionally, this information is

critical if the CIO is to successfully implement an integrated technical

architecture for the Department as required by the Clinger-Cohen Act of

1996.

Better Oversight

Could Have Helped to

Prevent Problems

Identified in Previous

GAO Reviews

The lack of rigorous oversight for the migration strategy has increased the

risk that Defense will pursue flawed system strategies. In fact, our

previous reviews of migration systems identified a number of problems

that could have been prevented had there been better oversight by MAISRC

and the ASD C3I. For example, in several reviews, we found that functional

areas did not account for various categories of significant costs when

making their migration decisions. These findings are highlighted as

follows.

• In reviewing the depot maintenance standard system migration effort, we

found that Defense did not address the full costs of developing interfaces

needed to allow system components to exchange data with information

systems currently used by the services to accomplish their missions. One

official estimated that this represented $70 million in costs.16

• In analyzing what it would cost to develop its Standard Accounting and

Reporting System (STARS), we reported that Defense neglected to consider

internal project management costs and costs to enhance all of the STARS

components to bring them into compliance with DOD’s standard general

ledger, key accounting requirements, and the standard budget and

accounting classification code.17

16Defense Management: Selection of Depot Maintenance Standard System Not Based on Sufficient

Analyses (GAO/AIMD-95-110, July 13, 1995).

17DOD Accounting Systems: Efforts to Improve System for Navy Need Overall Structure

(GAO/AIMD-96-99, September 30, 1996).

Page 37 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

• In reviewing the transportation migration effort, we found that Defense

did not include all costs associated with its evaluation of in-house systems

when analyzing costs and savings for its 28 migration systems. These

included $16 million for its analysis of candidate migration systems and

$2 million for maintaining migration system hardware. We also found that

if these costs were included in its systems selection analyses, Defense

would have found that the overall return on investment would have

decreased and that it may actually lose money on its investment.18

In previous reviews, we also found that Defense functional areas did not

adequately consider other alternatives to developing systems in-house. For

example, while the transportation area reviewed commercial off-the-shelf

transportation software projects for some transportation business areas,

this review was inadequate because it did not (1) analyze the degree to

which unmodified software could meet unique Defense requirements,

(2) identify the expected cost to make necessary software modifications,

(3) determine the time required to make the modifications, and (4) provide

for a hands-on view of the software in operation. In addition, Defense

concluded that software packages that could provide some degree of

transportation functionality would require modifications that were too

costly. However, Defense could not provide documented analysis to

support this conclusion. Further, Defense planned on making $13 million

worth of software modifications to just five of its in-house selections. We

believe better oversight by the CIO (then referred to as the Senior

Information Management Official) may well have forced greater

consideration of commercial packages in the transportation area.

Finally, our review19 of Defense’s effort to develop a standard suite of

migration systems for materiel management showed that the Department

spent hundreds of millions of dollars without achieving the expected

benefits because it did not adequately anticipate and mitigate risks. From

1992 to late 1995, Defense spent about $714 million developing standard

systems with minimal results. During that time, there were dramatic

changes in the goals and expectations for the program and only one

application was partially deployed. Because of changes in objectives and

scheduling and problems in development, prospects for achieving the

original objective of implementing a standard suite of integrated materiel

management systems appeared dim.

18Defense Transportation: Migration Systems Selected Without Adequate Analysis (GAO/AIMD-96-81,

August 29, 1996).

19Defense IRM: Critical Risks Facing New Materiel Management Strategy (GAO/AIMD-96-109,

September 6, 1996).

Page 38 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

In 1996, Defense finally abandoned its strategy to develop a standard suite

of materiel management systems because of funding cuts, cost overruns,

and schedule delays and embarked on a new strategy that involved

individual deployment of nine system applications at selected sites as the

applications were developed. However, the decision to drastically change

the course of the strategy was initiated without first conducting critical

economic and risk assessments that would estimate the costs, benefits,

and risks of alternative strategies and having the analyses independently

reviewed by MAISRC and PA&E.

20 The need for department-level review of

analyses supporting this decision was important, given the fact that the

new strategy represented a departure from Defense’s goal of eliminating

redundant legacy systems and varied business processes.21

Defense Lacks

Visibility Over

Migration Effort

A successful information technology investment process cannot operate

without accurate, reliable, and up-to-date data on project costs, benefits,

and risks. It is the basis for informed decision-making. However, in

implementing the migration strategy, Defense neglected to provide

visibility over progress and costs. The absence of reliable and accurate

performance, cost, and schedule information on the migration effort has

been debilitating in several respects. First, it has impaired the

Department’s ability to demonstrate whether the migration strategy has

been successful. Second, it has kept Defense from providing the Congress22

with accurate and reliable information needed for oversight purposes.

Third, it has prevented Defense from providing its own senior managers

with information needed to oversee the migration effort.

Key Performance Issues

Have Not Been Tracked

Since it embarked on the migration strategy, Defense has not tracked

overall departmentwide savings or validated improvements to operations

resulting from the migration strategy. In particular, it has not been

systematically tracking such key performance issues as (1) administrative

and operational cost savings resulting from elimination of redundant

20The materiel management standard system was originally projected to cost $5.3 billion to develop

and deploy. The threshold for major information system projects is $360 million in total life-cycle

costs.

21Under the revised strategy, the military services would be allowed to keep their legacy systems

longer than anticipated, and some legacy systems would not be shut down.

22For example, Defense reported to the Congress that it had selected 363 migration and interim

systems, using DIST as the source of the data. Defense also provided cost data that were collected by

the Office of the ASD C3I based on a different list of 245 migration and interim systems. Since the ASD

C3I’s staff relied largely on an official list of 245 systems that ASD C3I issued in July 1995 to provide

cost information to the Congress, that information was incomplete. In addition, the ASD C3I’s staff did

not reconcile the two lists of systems in its report to the Congress.

Page 39 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

systems, (2) cost reductions resulting from improved information systems

support to functional areas, (3) management or staff productivity

improvements, and (4) benefits accruing to mission effectiveness that are

attributable to information technology support. Having this type of

information is the only means of ensuring that the billions of dollars being

spent on the migration are producing sufficient returns and achieving

departmentwide progress.23 Without it, Defense cannot justify whether the

migration strategy has been a worthwhile investment or support the need

to continue pursuing migration.

At one point in the migration effort, as directed by the Congress,24 Defense

developed performance measures. However, it did not regularly collect

data on all the measures, and the accuracy of much of the information it

collected was questionable because of the data integrity problems

plaguing the database Defense relies on for migration inventory and

schedule-related information. In 1995, the House Committee on National

Security directed Defense25 to reevaluate its measures. In March of 1997, a

Defense working group proposed a revised set of performance measures

relating to migration systems and other information technology issues

within the Department. However, senior Defense management did not

approve the measures and instead tasked the working group to redo them

to ensure that they are linked to the Department’s strategic information

technology management goals. The working group has not established a

schedule for completing this effort and finalizing the measures.

Having performance measures will help to validate individual reports of

migration successes, such as the migration systems characterized as

successful in Defense’s report to the Congress pursuant to Section 381 of

the National Defense Authorization Act for Fiscal Year 1995. These are the

Standard Procurement System, the Distribution Standard System, the

Defense Civilian Personnel Data System, and the Defense Medical

Logistics Standard Support System.

23According to a DOD representative, the Department performed intense reviews of migration systems

during the last three cycles of its planning, programming, and budgeting process. During these reviews,

DOD asked program managers and other senior managers to provide performance information on their

systems. However, the representative stated that the performance information provided was limited

because DOD does not routinely track this information for its information systems.

24National Defense Authorization Act for Fiscal Year 1995.

25H. Rep. No. 104-131, p. 161.

Page 40 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

Cost Information Is

Incomplete and May Be

Significantly Understated

We could not determine the full cost of the migration effort because

Defense’s migration cost information is incomplete and may be

significantly understated. For example, when Defense provided migration

cost data to us in December 1996, it could only provide costs for 242 of 363

migration systems. We did not include costs in this report for 27 of 242

systems for which DOD collected costs because the systems were

classified. However, the remaining 121 systems were not included because

the Office of the ASD C3I had not collected the costs for these systems.

When we obtained cost information directly from three of the functional

areas, we found that at least $1.6 billion in costs had been excluded from

the approximately $18 billion total. This included about $1.4 billion for

clinical health systems, $56 million for civilian personnel systems, and

$111 million for transportation systems.

We also found that the $18 billion total cost estimate does not account for

some very important costs related to developing, deploying, and

maintaining migration systems both before and after a project has been

initiated. For example, the transportation functional area has an office

designated to oversee the development and deployment of its migration

systems. But, when accounting for costs for the systems, DOD does not

factor in the cost to maintain this oversight responsibility. In addition, our

previous reviews of finance and logistics migration efforts have found that

DOD did not account for costs for these projects relating to such activities

as project management and developing interfaces with other systems. Our

detailed analysis of DOD’s migration cost information is provided in

appendix III.

Schedule Information Is

Unreliable

We could not accurately determine how many legacy systems have been

terminated and how many are scheduled for termination because the

database Defense uses to track information systems is plagued with data

integrity problems.

Defense’s own analyses of the database have shown that it cannot readily

provide simple descriptive information for many systems. For example, a

February 1997 DOD analysis of DIST showed that 55 percent of the migration

systems in the database had incomplete information on interfaces with

other systems and 77 percent had incomplete data on installations where

the systems operated.

Our analysis of DIST found that 61 percent the migration system

implementation dates and 32 percent of legacy system termination dates

Page 41 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 2 

Management Control Processes Over

Individual Migration Efforts Have Broken

Down

were questionable. In addition, when we compared DIST scheduling

information to information maintained by three functional areas we

reviewed, we found that most of the migration systems implementation

dates and legacy system termination dates in DIST were incorrect. For

example, DIST showed that 92 legacy systems were terminated by

April 1996 in the clinical health, civilian personnel, and transportation

areas, while functional area managers told us that only 43 had actually

been terminated. Also, for the three functional areas, DIST showed that 53

legacy systems were scheduled for future termination while functional

area managers told us that 91 were slated for future termination.

We also found that Defense has not ensured that the data definitions and

formats used in DIST are fully compatible with data maintained in other

Defense information systems that track and report on systems. While DOD

is attempting to address this issue, DISA provided us information showing

that only 66 of the 218 data elements contained in DIST have been approved

as standard data elements in the Defense Data Dictionary System. Without

standard definitions and data formats, data cannot be easily transferred to

DIST from the other systems that may be used by functional area managers

and other decisionmakers.

Defense officials acknowledge that DIST is incomplete and inaccurate, and

the Department has begun efforts to make the database more accurate and

more user friendly. However, Defense officials also stated that they still

used DIST to generate reports to the Congress because it was the only

departmentwide database containing schedule and other descriptive data

on all Defense migration and legacy systems. A detailed analysis of DIST’s

integrity problems is provided in appendix IV.

Page 42 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 3 

Reforming DOD’s Information Technology

Investment Process Requires Effective

Implementation of the Clinger-Cohen Act

Defense has begun implementing Division E of the Clinger-Cohen Act of

1996 (Public Law 104-106), which the Congress passed in an effort to put

an end to problems associated with the development and implementation

of government information systems—such as those evident in the

migration effort. Under this legislation, Defense is required to establish

additional controls to ensure that more attention is devoted to the

selection, control, and evaluation of information technology projects and

that the Department’s technology investments are managed from a

portfolio perspective.

Implementing the Clinger-Cohen Act in DOD can bring meaningful reform

to the management of migration and other information technology

investments. However, the current structure of the Chief Information

Officer (CIO) position in the Department will not permit the CIO to devote

full attention to reforming information resources management within DOD.

And it will continue to impose responsibilities on the CIO that conflict with

the CIO’s obligation to provide independent oversight over the development

and implementation of information systems.

The Clinger-Cohen

Act

The Clinger-Cohen Act recognizes that the key to successfully managing

information technology projects is ensuring that investment processes

provide for the continued identification, selection, control, life-cycle

management, and evaluation of information technology investments. Best

practices on which the act is based have shown that to ensure an

investment process is successful, top executives need to periodically

assess all major projects—proposed, under development, and

operational—then prioritize them and make funding decisions based on

factors such as cost, risk, return on investment, and support of

mission-related outcomes. Once projects are selected for funding,

executives need to monitor them continually, taking quick actions to

resolve development problems and mitigate risks. After a project is

implemented, executives should evaluate actual versus expected results

and revise their investment management process based on lessons

learned.

As our guide, Assessing Risks and Returns: A Guide for Evaluating Federal

Agencies’ Information Technology Investment Decision-making,

1

 points

out, a key to success in this type of management is considering all the

major technology investments that are vying for funding at a designated

level (departmental, functional area, or service/agency level) as a total

1

GAO/AIMD-10.1.13 February 1997, Version 1.

Page 43 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 3 

Reforming DOD’s Information Technology

Investment Process Requires Effective

Implementation of the Clinger-Cohen Act

package, or portfolio, of possible technology investments. Once this

perspective is adopted, an organization can focus scarce information

technology resources on the projects with the greatest impact on mission

and concentrate management attention on those high-impact projects that

become troubled projects. It can also establish performance goals and

stop or replace those systems or initiatives that fail to meet those goals.

This type of decision-making process can be applied to almost any

organization, even one that is as highly decentralized as DOD. According to

our investment guide, separate information technology investment

decision-making processes can exist at various levels. In DOD, such

processes could exist at the departmental level and the functional area or

service/agency level, provided that the Department can identify which

major projects and initiatives should be managed at each level. Criteria for

determining projects that should be managed at the various levels may

include the dollar amount of the investment, the degree of risk associated

with the project, and whether the system is to be shared across functional

area lines or service/agency lines.

The Clinger-Cohen Act contains the following additional requirements that

are important for DOD to implement in order to address problems evident

in its migration strategy.

• Agencies are to determine whether their administrative and

mission-related business processes should be improved before investing in

major information systems to support them.

• The investment process is to provide a means for senior management to

obtain timely information regarding progress (at established milestones) in

terms of cost, capability of the system to meet requirements, timeliness,

and quality. It should also provide for the evaluation of the results of

information technology investments.

• Performance measures are to be prescribed for information technology

used by or to be acquired for the agency.

• The CIO is to monitor the performance of information technology

programs; evaluate the performance of those programs on the basis of

applicable performance measures; and advise the agency head regarding

whether to continue, modify, or terminate the program or project.

• The CIO is to be responsible for providing advice and other assistance to

agency heads and senior managers to ensure that information technology

is acquired and information resources are managed for the agency in a

manner that implements the policies and procedures of the act and the

priorities of the agency head.

Page 44 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 3 

Reforming DOD’s Information Technology

Investment Process Requires Effective

Implementation of the Clinger-Cohen Act

• The CIO is to develop, maintain, and facilitate the implementation of a

sound and integrated information technology architecture for the agency.

The architecture is an integrated framework for evolving or maintaining

existing information technology and acquiring new information technology

to achieve the agency’s strategic and information resources management

(IRM) goals.

DOD Implementation

of the Clinger-Cohen

Act

As a first step in implementing the Clinger-Cohen Act, on June 2, 1997, the

Secretary of Defense outlined his expectations for improvements in

information technology-related management processes and information

resources.2

 We believe the expectations identified by the Secretary are an

excellent starting point for implementing the Clinger-Cohen Act and

bringing meaningful change to the current information technology

management process. For example, the Secretary has called on the CIO to

design and implement a process for maximizing the value and assessing

and managing the risks of DOD information technology acquisitions. This

process is to

• provide for the selection of information technology investments to be

made by the Department, the management of such investments, and the

evaluation of the results of such investments;

• be integrated with processes for making budget, financial, and program

management decisions;

• include minimum criteria to be applied in considering whether to

undertake a particular investment in information systems, including

criteria related to the quantitatively expressed projected net, risk-adjusted

return on investment, and specific quantitative and qualitative criteria for

comparing and prioritizing alternative information system investment

projects;

• identify, for each proposed investment, quantifiable measurements for

determining the net benefits and risks of the investment; and

• provide the means for senior managers to be able to obtain timely

information regarding the progress of an investment in an information

system, including the milestones for measuring progress on an

independently verifiable basis, in terms of cost, capability of the system to

meet specified requirements, timeliness, and quality.

In addition, the Secretary has called on the CIO to institutionalize

performance-based and results-based management for information

2

Memorandum from the Secretary of Defense to the Secretaries of the Military Departments, Chairman

of the Joint Chiefs of Staff, Under Secretaries of Defense, and others on the Implementation of

Subdivision E of the Clinger-Cohen Act of 1996, dated June 2, 1997.

Page 45 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 3 

Reforming DOD’s Information Technology

Investment Process Requires Effective

Implementation of the Clinger-Cohen Act

technology. In doing so, the CIO is to work with the Chief Financial Officer,

the Principal Staff Assistants (PSAs), and the Defense components. The CIO

is to establish goals for improving the efficiency and effectiveness of DOD

operations and issue instructions to functional areas on performance

measurements. The CIO is also to monitor the performance of information

technology programs, evaluate the performance of those programs on the

basis of applicable performance measurements, and advise the Secretary

of Defense regarding whether to continue, modify, or terminate programs

or projects. Additionally, the Secretary established a Chief Information

Officer Council for DOD to serve as the principal forum for discussing

improvements in DOD practices for the management of information

technology.

In outlining his expectations, the Secretary noted that the act poses

questions that should be answered before investing in information

technology, including:

• What functions are we performing and are they consistent with the

mission?

• If we should be performing particular functions, could they be performed

more effectively and at lower cost by the private sector?

The Secretary further stated that if a function should indeed be performed

by the Department, the law requires that the function be examined and

redesigned or reengineered before applying new technology.

Challenges

Confronting DOD in

Implementing

Clinger-Cohen

Proper implementation of the Clinger-Cohen Act would help to address a

number of weaknesses that are currently standing in the way of Defense’s

ability to provide a good decision-making and oversight environment for

information technology projects. For example, with full implementation of

the act, Defense could begin considering information technology

investments as a total package of possible projects so that it can target

resources to those projects having the greatest impact on mission and

concentrate management attention on troubled areas. It could also

strengthen visibility over project performance, costs, and schedules so

that senior managers can begin comparing the results being achieved

against projected, costs, benefits, and risks and to identify actual or

potential managerial, organizational, or technical problems.

Moreover, in implementing the act, Defense has an opportunity to

(1) begin enforcing compliance with data and technical standards to

Page 46 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 3 

Reforming DOD’s Information Technology

Investment Process Requires Effective

Implementation of the Clinger-Cohen Act

ensure that DOD’s goals for interoperability and the sharing of information

are met and (2) increase oversight for functional area assessments of

whether to reengineer, migrate, or undertake some other path toward

improvement.

However, the current structure of the CIO position in Defense will not

permit the CIO to effectively serve as a bridge between top management,

line management, and information management support officials and

identify opportunities to use information technology to enhance

performance.

The Clinger-Cohen Act requires that information resources management

be the CIO’s primary responsibility and that the CIO be involved in key

decisions regarding the application of information technology in support

of the agency’s missions. Currently, the Assistant Secretary of Defense for

Command, Control, Communications and Intelligence (ASD C3I) also acts

as the CIO. By asking the CIO to also shoulder a heavy load of programmatic

responsibility, Defense has made it difficult, if not impossible, for the CIO

to devote full attention to IRM issues. These issues go well beyond the

development and modernization of information systems.

For example, Defense needs its CIO to be heavily involved with

implementing a more aggressive and proactive computer security

program. As we reported in May 1996,3

 attackers have seized control of

entire Defense systems, many of which support critical functions, such as

weapons systems research and development, logistics, and finance.

Attackers have also stolen, modified, and destroyed data and software. In

addition, Defense needs its CIO to help ensure that Year 20004

 corrections

are made to all of DOD’s information systems. If systems are not corrected

on time, the impact on Defense operations could be widespread, costly,

and debilitating to important warfighting and administrative operations.

While DOD has delegated year 2000 responsibility to its components, it still

needs to ensure, at the department level, that sufficient priority and

3

Information Security: Computer Attacks at Department of Defense Pose Increasing Risks

(GAO/AIMD-96-84, May 22, 1996).

4

The Year 2000 problem is rooted in the way dates are recorded and computed in automated

information systems. For the past several decades, systems have typically used two digits to represent

the year in order to conserve on electronic data storage and reduce operating costs. With this two-digit

format, however, the year 2000 is indistinguishable from 1900, 2001 from 1901, etc. As a result of this

ambiguity, system or application programs that use dates to perform calculations, comparisons, or

sorting may generate incorrect results when working with years after 1999.

Page 47 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 3 

Reforming DOD’s Information Technology

Investment Process Requires Effective

Implementation of the Clinger-Cohen Act

resources are being devoted to the problem and that all systems have been

identified and corrected.5

There is also a direct conflict of responsibilites between the oversight and

programmatic obligations associated with the two positions. The ASD C3I

serves as the Principal Staff Assistant for command, control,

communications, and intelligence systems. These systems represent about

45 percent of the migration system investment—about $8.5 billion of the

total $18 billion migration investment. This poses a conflict for both

control mechanisms we assessed. For the first control—the ASD C3I

approval process—the same individual is responsible for both selecting a

system and approving the selection. For the second control—acquisition

oversight—Defense’s Acquisition Executive is also responsible for

developing 14 of the 43 major automated information systems under the

Major Automated Information System Review Council (MAISRC).6

A second challenge confronting DOD in implementing the Clinger-Cohen

Act is the prevailing organizational structure and embedded culture found

throughout the Department. Specifically, the three military services have

clearly defined roles and responsibilities and separate budget authority,

program execution, and functional authority for the enforcement of

national defense policy and objectives. As we have reported7

 throughout

the CIM initiative, this environment has promoted stovepipe systems

solutions in each component agency and has made it difficult to

implement departmentwide oversight or visibility over information

resources. This same condition has contributed to the difficulty that has

limited the Department in modernizing business processes and

implementing corporate information systems across service and agency

lines. This is most evident in the perceived failure of the Corporate

Information Management initiative (CIM), which was intended to

reengineer business processes throughout the Department. In doing so,

the Department expected to save billions by having more efficient,

effective business processes running across service and component lines.

However, these benefits have yet to be widely achieved after 8 years of

5

We have been assessing Year 2000 efforts at DOD and have issued reports on Year 2000 work at

individual DOD components. We also plan to report on DOD’s overall response to the Year 2000 issue.

6

As the Department’s CIO, the ASD C3I is the Acquisition Executive for DOD’s major information

systems. In this capacity, the staff who oversee major information systems acquisition for MAISRC

report to the ASD C3I.

7

Defense ADP: Corporate Information Management Initiative Faces Significant Challenges

(GAO/IMTEC-91-35, April 22, 1991); Defense ADP: Corporate Information Management Initiative Must

Overcome Major Problems (GAO/IMTEC-92-77 September 14, 1992); and Defense Management:

Stronger Support Needed for Corporate Information Management Initiative to Succeed

(GAO/AIMD/NSIAD-94-101, April 12, 1994).

Page 48 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 3 

Reforming DOD’s Information Technology

Investment Process Requires Effective

Implementation of the Clinger-Cohen Act

effort. Without the Secretary’s strong and continued support for

management processes and controls designed to improve information

management initiatives, Clinger-Cohen Act implementation could well

suffer similar results.

Page 49 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 4 

Conclusions, Recommendations, and

Agency Comments

In taking its initial steps to implement the Clinger-Cohen Act, Defense has

recognized that it needs a better information technology investment

environment. However, implementing the act will not be easy given

weaknesses pervading the current decision-making and oversight

environment. Examples included the following:

• DOD’s management and oversight processes over information technology

projects are not effectively integrated to enable the Department to manage

from a portfolio perspective. Considering investments as a total package

of possible projects is important because it forces an agency to decide

which projects are the most critical to meeting mission needs and thus

should receive the most resources and attention. Life-cycle management

controls that provide oversight on a system-by-system basis can

supplement and enhance this process, but they cannot be a substitute for

it.

• DOD still has not established information technology performance

measures. These need to be in place before a new investment process can

begin so that senior managers can begin comparing results being achieved

against projected costs, benefits, and risks.

• As far as visibility over costs and schedule are concerned, DOD has

fragments of a mechanism for collecting and maintaining all project

information. However this mechanism—DIST—does not maintain cost

information and its accuracy and reliability are questionable. Further, as

we have recently reported to the Director of the Defense Information

Systems Agency,1

 efforts to improve the DIST have been slow-moving.

• While the Clinger-Cohen Act requires agencies to conduct

post-implementation reviews, DOD’s oversight processes have concentrated

on projects prior to their implementation. Thus, in addition to focusing on

the pre-implementation stages of system life-cycles, DOD will have to

ensure that more attention is given to whether implemented systems are

achieving the forecasted benefits and continuing to meet mission needs

after they are implemented.

• Currently, DOD does not have an effective mechanism in place to ensure

that the CIO has complete and accurate information on DOD-wide

compliance with technical and data standards. Such a mechanism is

1

We recently reported in Defense Computers: DOD’s Inventory of Automated Information Systems

Needs to Be Improved to Successfully Address Year 2000 Problems (GAO/AIMD-97-112, August 13,

1997) that Defense is planning to increase the accuracy of the tool by developing a purging

methodology to validate the data in DIST. At the end of January 1997, DIST officials told us that it

would take 90 days to determine the methodology. DOD reported that the methodology was finally

developed and implemented by the end of August 1997. DOD further reported that it is now checking

DIST data on a system-by-system basis and is developing parametric and consistency checks for the

data elements to ensure that the required data elements are fully and accurately populated by the

users.

Page 50 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 4 

Conclusions, Recommendations, and

Agency Comments

necessary to enable the CIO to achieve the Department’s goals for systems

interoperability, effectively implement an integrated technical architecture

for the entire Department, and ensure the efficient exchange of standard

data among systems.

Moreover, our review of the migration strategy suggests that a real threat

to successful implementation of the Clinger-Cohen Act is the Department’s

consistent lack of adherence to sound decision-making and oversight

processes. For example, at the oversight level, systems that clearly lacked

key pieces of technical, programmatic, and economic justification were

allowed to go forward. At the decision-making level, many functional areas

did not even bother to submit their systems or analyses that supported

their decisions to department-level oversight controls. If they were

effectively followed, the life-cycle management controls would have

helped ensure that sound business and development practices were

followed for the migration systems.

Perhaps, the greatest challenge to successful operation, however, will be

operating in an organizational environment that has resisted

departmentwide oversight and visibility over information resources. For

example, the Department’s CIM effort largely failed in meeting its original

goals of bringing widespread efficiencies to its business processes. The

Department was unable to create meaningful change across organizational

lines and management support was insufficient to overcome initial

resistance to new ways of doing business. Efforts to implement the

Clinger-Cohen Act will require unwavering top-level commitment to

overcome both organizational resistance and to institute meaningful

controls.

Recommendations To ensure that DOD’s continued investment in migration systems provides

measurable improvements in mission-related and administrative

processes, we recommend that the Secretary of Defense require the

Defense components to rank development/modernization migration

systems justifications and complete them on an expedited basis. Further,

the Secretary should require the Chief Information Officer to review these

justifications and certify that they include the following:

• A business case of operational alternatives for each functional area that

clearly demonstrates that continued development and deployment of the

migration system is the best solution for improving performance and

reducing costs in the functional area it serves, when compared to other

Page 51 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 4 

Conclusions, Recommendations, and

Agency Comments

available alternatives. The alternatives analyzed should include

reengineering the functional area’s processes before making investments

in information systems and using, when appropriate, the private sector to

perform major functions now performed by government personnel and

information systems.

• An economic analysis showing a return on investment or other

results-based benefits to the Department that justify further investment in

the migration system.

• Current compliance with applicable Defense technical standards and uses

standard data, or a schedule and plan for bringing the system into

compliance with these standards.

Lastly, the Secretary of Defense should require the Chief Information

Officer to establish routine procedures for reporting on the status of

reviews of migration system justifications to the Deputy Secretary of

Defense so the information can be used in the Department’s planning,

programming, and budgeting system. Any exception to the

accomplishment of these reviews should be approved by the Deputy

Secretary of Defense.

Further, we recommend that the Department’s Chief Information Officer

revise Defense’s policies, practices, and procedures to institutionalize the

management of information systems and technology expenditures as

investments and ensure that these investments provide measurable

improvements in mission performance. In the course of taking action on

these matters, we recommend that the Secretary direct that the Chief

Information Officer, in coordination with the Chief Financial Officer and

other appropriate Defense officials take the following actions:

• Ensure that Defense’s strategic information technology planning and

investment control policies, practices, and procedures include

requirements that Principal Staff Assistants document that they and the

key stakeholders for their functional areas (including the services,

agencies, and major commands) have (1) conducted thorough economic

and risk analyses of alternative operational approaches (business case

analyses) for accomplishing the mission of the functional area,

(2) examined trade-offs among the competing proposals, and

(3) prioritized the alternative proposals based on mission impact, risk, and

return.

• Finalize and issue guidance for developing and using analyses of

alternatives and economic analyses for information system

decision-making. Once this guidance is issued, the CIO should require that

Page 52 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 4 

Conclusions, Recommendations, and

Agency Comments

all Defense program managers and Defense managers, as appropriate, be

trained in using the guidance. Also, the CIO should ensure that the guidance

defines a standard return on investment definition for Defense information

systems and require that this definition be used to calculate all returns on

investment for information systems efforts. Additionally, the CIO should

require that all major migration and other information systems under

direct review by the Major Automated Information System Review Council

(MAISRC), and those delegated by MAISRC to other oversight organizations

for review, have their alternatives analyses and economic analyses

independently verified by Defense’s Program Analysis and Evaluation

(PA&E) office, or another qualified independent review organization, in

accordance with this guidance before major investments are authorized

for system development/modernization.

• Require post-implementation reviews of migration and other information

systems and ensure that these reviews are designed to compare actual

systems’ costs, benefits, risks, and returns against the original baseline

estimates/projections and determine the causes of any differences

between planned and actual results.

• Expedite the definition, coordination, testing, and implementation of

information management performance measures in the Department and

establish milestones for evaluating progress in implementing these

performance measures.

• Modify the DIST system or acquire/develop a new Defense-wide

management information system or systems for tracking and reporting key

schedule, progress, and performance information on migration systems

and other Defense information systems and ensure the system or systems

contain complete, current, and accurate

• schedule data necessary to track the progress of each migration

system’s development/deployment and each legacy system’s

termination;

• budgeted and actual cost data on each system for which the Department

maintains such data (an alternative to putting budget and cost data in

DIST is to establish the capability to directly interface with other systems

in the Under Secretary of Defense (Comptroller’s) Office or other

Defense organizations containing systems’ budget and/or cost data);

• data necessary to track the progress of each migration system in

complying with applicable Defense technical and data standards,

including whether each system has been independently certified to be in

compliance with applicable technical and data standards;

• data for tracking progress in accomplishing the mission-based

performance goals and information management performance goals for

Page 53 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 4 

Conclusions, Recommendations, and

Agency Comments

the functional areas supported by each system once these data have

been identified;

• information determined to be needed for oversight by the Defense

Acquisition Board (DAB), MAISRC, and PA&E; and

• other information determined to be needed for management and

oversight by the Defense CIO, the CIO Council, other Defense senior

managers, and the Congress.

• Develop and implement management controls and a quality assurance

program to ensure the DIST data’s accuracy and completeness since these

data are used to track and report to senior Defense managers and the

Congress on the overall status of the migration initiative and on other

Defense information management initiatives.

Agency Comments

and Our Evaluation

The Department of Defense provided written comments on a draft of this

report, which are reprinted in appendix I. The Acting Assistant Secretary

of Defense for Command, Control, Communications and Intelligence

concurred with five of our recommendations and partially concurred with

two recommendations. Defense did not concur with the remaining five

recommendations and expressed concerns about certain aspects of our

report. Defense’s concerns are highlighted and discussed below. Appendix

I also provides detailed responses to DOD’s views on our recommendations

and to other specific comments on our findings.

Generally, the Department concurred with our recommendations that the

Department revise or develop internal policies and procedures to conform

to the Clinger-Cohen Act and to improve the performance of information

systems management. Defense also noted that the issues we reported are

in the process of being or will be addressed as it implements the

Clinger-Cohen Act, the Federal Acquisition Streamlining Act, the

Government Performance and Results Act, and the Paperwork Reduction

Act.

However, it did not agree to a recommendation in our draft that it limit

further investments to expenditures that meet mission critical needs until

the investments are economically, functionally, and technically justified

and such justifications are independently reviewed. In DOD’s view, limiting

migration system spending would adversely affect military readiness,

system development, and government contract obligations and increase

system obsolescence. We understand DOD’s concerns regarding readiness

and contract obligations. That is why our recommendation in the draft

report recognized that critical needs still need to be met. Our point is that

Page 54 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 4 

Conclusions, Recommendations, and

Agency Comments

greater management attention needs to be placed on the decision-making

process for approving and funding the development and modernization of

migration systems to achieve intended benefits and minimize unnecessary

costs. In order to clarify this position, we modified our recommendation to

focus on strengthening the controls for the migration system

decision-making process so that these investments can be better

considered in the Department’s budget process.

In addition, Defense did not concur with a proposal in the draft report that

it consider separating the CIO and the ASD C3I positions so that the CIO can

devote full attention to departmentwide information resource

management issues and provide independent oversight. It noted, however,

that all ASD C3I functions are being reviewed by the Department’s Task

Force on Defense Reform as part of its review of the Office of the

Secretary of Defense’s organizational structure. We withdrew the proposal

because DOD is now considering this matter. Nevertheless, our concern

that the current structure of the CIO position does not allow the CIO to

devote full attention to critical IRM issues—such as computer security, the

Year 2000 problem, and the need to develop and implement an integrated

information technology architecture—remains valid.

Defense also stated that our report overlooked one of the key cost

reduction aspects of the migration effort—that is, modernizing multiple

systems within each service and Defense agency is more expensive than

simply modernizing one or a few departmentwide migration systems.

While these cost reductions may be possible, Defense has not yet

demonstrated that it has achieved such savings. Further, our review

determined that Defense does not know, or track, cost reductions that

result from its migration efforts. Finally, the achievements that Defense

refers to in its comment letter are mostly the results of process

reengineering, not the migration of automated systems.

Defense also stated that our report negatively portrays the role and

benefits of acquisition streamlining within the Department. Defense

reported that it uses a “best-value” approach to modernizing its systems

that balances the functional and technical capabilities while still

considering cost. We disagree with Defense’s characterization of our

report. We fully support acquisition streamlining efforts in DOD because

they promote economy, efficiency, and effectiveness in the procurement

of property and services. However, these three elements must be carefully

balanced in order to achieve the results intended by the Division D of the

Clinger-Cohen Act of 1996 which covers federal acquisition reform. Such a

Page 55 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 4 

Conclusions, Recommendations, and

Agency Comments

balance is particularly important to achieve prudent decisions for an

investment as substantial as the multibillion dollar migration initiative, and

it is contemplated by the Clinger-Cohen Act of 1996 to properly control

these investments.

In our view, for the migration effort, it appears that expediency was

achieved at the expense of economy and effectiveness. In conducting our

review, we sought to examine whether the Department of Defense was

following its own policy for controlling migration investments. When we

found that these policies were not executed as intended, we turned to the

Department’s acquisition review process, which should have

supplemented oversight for the major migration systems. However, even

here, we found that DOD for the most part, was not providing sufficient

assurance that migration investments were worthwhile. Specifically, we

found that controls that Defense built into this process would not ensure

success.

In addition, the migration strategy is a high-risk endeavor because it

requires Defense to carefully consider complex technical issues that are

compounded by the sheer size of the Department, the number of disparate

systems, and the prevailing culture, which promotes stovepiped systems

solutions and hampers departmentwide oversight or visibility over

information resources. As such, the success of the strategy hinges on

whether practical and sound management practices—such as conducting

economic, risk, and alternative analyses and providing rigorous oversight

over the selection process—are followed. As we note in the report, these

practices are required by executive branch policies; congressional reform

initiatives, including the Clinger-Cohen Act, which took effect August 8,

1996; and/or Defense regulations. Under the Clinger-Cohen Act, for

example, Defense is required to implement a process for selecting

information technology investments using specific criteria for comparing

and ranking alternative information system projects. The best practices on

which the act is based have shown that to ensure the success of an

investment process, top executives need to make funding decisions based

on factors such as cost, risk, return on investment, and support of

mission-related outcomes. Thus, unless Defense promptly embraces the

need to develop and review economic, risk, and alternative analyses, it

stands little chance of successfully implementing the Clinger-Cohen Act.

In its comments, Defense also expressed concern that the scope of our

report was unnecessarily broader than it was when we began our review.

We were originally asked for information on the status and cost of the

Page 56 GAO/AIMD-98-5 Defense Migration Strategy
Chapter 4 

Conclusions, Recommendations, and

Agency Comments

migration effort. However, when we found that Defense could not provide

accurate and reliable information on migration or convincingly

demonstrate whether the strategy has been successful, our congressional

requester asked that we review whether Defense’s management control

and oversight processes for migration were ensuring that the investments

are economically sound and complied with technical and data standards.

We reviewed the reliability of the acquisition process controls over

migration investments because they are the leading Defense-recognized

controls over major investments. Our congressional requester also asked

that we focus on DOD’s implementation of the Clinger-Cohen Act because it

offered a chance for DOD to bring meaningful reform to the management of

migration and other information technology investments. Many

information technology investments for systems in the Department are, in

fact, migration systems.

Finally, Defense’s comments on our report identify a number of other

disagreements with our findings and conclusions. We reviewed these

comments, incorporated them into our report where appropriate, and

followed up with Defense on additional information it provided after

submitting its response to us.

Page 57 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.

See comment 1.

See comment 2.

Page 58 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

See comment 3.

See comment 4.

Page 59 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

See comment 5.

See comment 6.

Page 60 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

Now on p. 51.

See comment 7.

Now on pp. 51-52.

Page 61 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

See comment 7.

Now on p. 52.

See comment 7.

Now on p. 52.

Now on p. 52.

See comment 8.

Page 62 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

Now on p. 52.

See comment 9.

Page 63 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

Now on p. 53.

See comment 10.

Page 64 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

Now on p. 53.

Now on p. 53.

Now on p. 53.

Page 65 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

Now on p. 54.

Page 66 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

See comment 11.

Page 67 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

Now on p. 33.

Now on p. 32.

Page 68 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

Now on p. 33.

Page 69 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

The following are GAO’s comments on the Department of Defense’s letter

dated September 12, 1997.

GAO Comments 1. The scope of our work was directed by the Congress. Defense is correct

that the initial scope was expanded, at the direction of our congressional

requester because Defense was unable to provide complete and accurate

information to answer the initial questions. These questions were (1) What

are the reported costs for migration systems and the legacy systems

Defense expects to terminate for fiscal years 1995, 1996, and 1997?

(2) How many and what type of migration systems did Defense designate,

how many legacy systems did Defense terminate, and how many are

scheduled for future termination? and (3) What economic studies did

Defense prepare and use to justify its migration actions? Because Defense

was unable to provide satisfactory answers to these initial questions, our

congressional requester directed us to expand our work to evaluate

Defense’s management controls over decision-making for selecting and

implementing migration systems. When we found that Defense was not

following the Department-level management controls that had been

established for overseeing the selection and implementation of migration

systems, we looked for other Department-level management controls over

migration systems. We found that Defense’s oversight of major systems

acquisitions by MAISRC and DAB was one of the few Department-level

management controls over migration systems. However, because these

management controls were limited to the major migration systems, we

looked for Defense actions that would increase Department-level

management control over all migration systems. This led us to examine

Defense’s actions to implement the Clinger-Cohen Act of 1996. Throughout

our review, we kept our congressional requester and our Defense point of

contact informed about changes in the scope of our work.

2. We reviewed the accuracy of all data in the report, especially data that

Defense questioned in tables, figures, and the text regarding (1) the

number of migration systems selected by the functional managers and

approved at the Department level, (2) the costs Defense reported to us for

migration systems, and (3) our findings related to the preparation and

independent review of economic analyses for major systems under review

by MAISRC. We contacted the appropriate Defense officials and discussed

each instance in which Defense questioned the accuracy of data. We also

requested that these officials provide us additional documentation

pertaining to the data in question. Based on these discussions and our

review of the additional documentation, we made changes where

Page 70 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

appropriate. In the few cases in which changes were warranted, we

changed the table, figure, or text.

3. We agree that CIM consists of several aspects, and we stated this in a

footnote in the report. However, our work showed that as CIM was

implemented, Defense emphasized two ways of achieving process

improvements and addressing problems associated with its disparate and

stovepiped information technology environment: (1) business process

reengineering and (2) migration.

4. As we state in our report, the $18 billion dollar estimate for developing,

maintaining, and deploying migration systems for fiscal years 1995 through

2000 is based on cost information provided to us by Defense. Defense

provided essentially the same information to the Congress in April 1996.

We agree it is incomplete and inaccurate, and we stated so in our report.

In addition, we agree that migration selections should be based on a

balanced assessment of functional, technical, and cost factors. However,

we found that Defense’s selections placed very little emphasis on costs

and related economic factors. Our work showed that in Defense, the

preparation of an economic analysis is often a paper exercise to produce a

document that decisionmakers do not use when deciding whether to

invest in an information system development or modernization project.

Additionally, we found that in many cases, Defense placed little

importance on having complete and accurate information on system

development/modernization projects’ estimated costs and benefits at the

time decisions were made to invest in the projects.

Further, we agree with Defense’s contention that modernizing multiple

systems would normally be expected to be more expensive than

modernizing a smaller number of migration systems. In fact, our report

does not take issue with the migration strategy’s potential to reduce costs.

Rather, it shows that Defense’s poor management controls over migration

systems may have kept Defense from achieving expected cost reductions

or, worse yet, resulted in wasting hundreds of millions of dollars on failed

efforts such as the materiel management migration initiative. Also,

because Defense has not tracked the cost reductions and other

measurable improvements that may have been achieved through

migration, Defense cannot demonstrate the extent that savings are actually

occurring.

Page 71 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

5. We fully support acquisition streamlining initiatives and encourage

improvements. However, a balance between function, technology, and

cost is as necessary as a balance between acquisition streamlining and

effective management controls to reduce risk. We found that Defense’s

streamlined approach to acquisition oversight and its revisions of its

acquisition policy in March 1996 resulted in inadequate management

control over major systems acquisition projects. Specifically, acquisition

processes did not fully ensure that economic analyses for migration

projects were prepared and reviewed and that systems complied with

technical and data standards. This increased the risk that functional areas

would develop faulty cost estimates and risk assessments and pursue

flawed efforts without rigorous department-level oversight.

While we do not support a one-size-fits-all approach to acquisition

oversight, we believe that Defense’s existing acquisition oversight

processes do not require the degree of standardization in methodology and

analytic techniques to ensure the preparation of the economic analyses

necessary to support a portfolio management approach for information

technology investments, as required by the Clinger-Cohen Act of 1996.

Further, under OMB’s Capital Programming Guide, which was issued in

July 1997, Defense will be required to perform an economic analysis, along

with other extensive planning steps, before submitting a budget proposal

for major information technology projects.

If Defense is to effectively implement the Clinger-Cohen Act of 1996 and

integrate its requirements with those of the Government Performance and

Results Act, the Paperwork Reduction Act, and the Chief Financial

Officers Act, changes will be necessary not only in Defense’s acquisition

oversight processes but also in its other information technology

investment and management control processes. Further, to effectively

implement the requirements specified in these laws, the Chief Financial

Officer, Defense Acquisition Executive, and leaders of the services,

agencies, and major commands must work cooperatively with the CIO.

6. We recognize that DISA reviewed system documentation for many of the

major migration systems to determine if it indicated that interoperability

issues were being addressed or were intended to be addressed as the

system was developed. However, our work showed that DISA does not

regularly report detailed information to MAISRC and DAB that would enable

these oversight organizations to ensure that each major system is

adequately complying with applicable technical and data standards. For

Page 72 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

example, DISA does not regularly report to MAISRC and DAB such detailed

information as

• each system’s current compliance with applicable standards—that is, each

system’s current status relative to the eight levels that Defense has defined

for the Defense Information Infrastructure Common Operating

Environment and each system’s number and percentage of data elements

that have been approved as Defense data standards in the Defense Data

Dictionary System;

• whether each program office prepared sound strategies, schedules, and

cost estimates for achieving compliance with applicable standards;

• whether each system’s compliance strategy and cost estimate were

approved by DISA;

• each system’s current schedule and cost status for achieving compliance

compared against its baseline schedule and cost estimate; and

• whether each system has been independently certified by DISA’s Joint

Interoperability Testing Command to be in compliance with the technical

and data standards.

MAISRC analysts responsible for overseeing the major migration systems

confirmed that they did not know (1) many of the systems’ current status

regarding compliance with applicable technical standards and use of

Defense standard data, or (2) whether the program managers had

developed and were following sound strategies for bringing the systems

into compliance.

As noted in our report, Defense’s technical and data standards are

supposed to help pave the way to an interoperable systems environment.

However, without complete and accurate data on individual systems’

compliance with standards, MAISRC and DAB or any other DOD manager

cannot assure the CIO that Defense’s major information systems are

complying with the standards. Without this information, as well as

standards compliance information from the managers of Defense’s

nonmajor systems, Defense’s CIO cannot gauge the Department’s progress

toward achieving its goal of an interoperable systems environment.

7. As noted in the “Agency Comments and Our Evaluation” section of our

report, we modified our recommendation to focus on strengthening the

controls for the migration system decision-making process so that these

investments can be better considered in the Department’s budget process.

Nevertheless, we remain concerned that if Defense proceeds with its

decisions to develop/modernize major migration systems without ensuring

Page 73 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

that they are based on sound investment decision-making practices, it will

risk some portion of these investments not providing measurable

improvements in mission performance for the warfighter. Defense will

also risk the systems’ not operating as intended and not being

interoperable with other systems.

We have reported on numerous occasions that poorly planned and

managed systems development/modernization efforts often fail and must

be restarted, delaying the delivery of systems that can effectively meet the

warfighter’s needs. This wastes the limited funds Defense has available for

providing information technology support to the warfighter and

unnecessarily diverts funds from other needed investments. For example,

our review of Defense’s materiel management migration strategy showed

that Defense spent over $700 million pursuing a substantially flawed effort

that was later abandoned. We found that Defense’s planning and

decision-making for this strategy were not sound: Defense did not

complete an economic analysis until nearly 3 years after the effort began,

and the analysis was not updated even though Defense dramatically

changed the course of the systems development. We are concerned that

Defense may repeat this experience unless it ensures that the remaining

migration system investment decisions are based on sound planning.

8. Defense’s response does not clearly specify why it partially concurred

rather than fully concurred. Nevertheless, it is critical that in revising its

directives and instructions, DOD require that the PSAs document that they

and the key stakeholders for their functional areas (including the services,

agencies, and major commands)

• have conducted thorough economic and risk analyses of alternative

operational approaches (business case analyses) for accomplishing the

mission of the functional area and

• have examined trade-offs among the completing proposals and ranked the

alternative proposals based on mission impact, risk, and return.

Undertaking such reviews is a sound business practice that is consistent

with the planning and management practices called for in the

Clinger-Cohen Act of 1996.

9. We disagree with DOD’s position that DODI 7041.3 provides adequate

guidance for preparing of an economic analysis for an information system

development/modernization effort. The existing DODI 7041.3 regulation,

dated November 7, 1995 , is very general. Defense itself recognized that the

Page 74 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

regulation was insufficient guidance for preparing an economic analysis

because it developed and published an unofficial economic analysis guide

to supplement the regulation. That unofficial guide recommended, but did

not require, standard methods and formats for preparing an economic

analysis for an information system development/modernization project.

According to a PA&E representative, this unofficial guide is no longer

adequate because it does not require the degree of standardization in

methodology and analytic techniques needed to support a portfolio

management approach for information technology investments, as

required by the Clinger-Cohen Act of 1996. For example, the guide does

not require that returns on investment for systems

development/modernization projects be calculated in a standard manner

using a standard definition. This lack of standardization results in

economic analyses for different systems that are not comparable enough

for DOD managers to have a good basis for deciding which systems offer

the highest payoffs. Defense stated that in conjunction with its

implementation of a portfolio management approach, it plans to officially

publish appropriate documents and provide minimum standards and

guidance for the economic analysis process. However, Defense has not

established a time frame for completing and publishing the guidance. An

expedited time frame is needed so that this effort can be given adequate

priority for resources.

10. Although DOD’s existing acquisition regulations do not require

independent review and verification of alternatives analyses for major

information technology investments, conducting independent reviews of

key decision-making documents is a good business practice consistent

with the type of sound information technology investment planning

promoted by the Clinger-Cohen Act of 1996. An independent review of an

alternatives analysis helps ensure that the methodology used to conduct

the analysis is sound and that the information is accurate and complete

and can be relied on for choosing among competing investment options.

This helps to ensure that the alternative analysis will serve its purpose: to

provide a reliable, detailed, systematic evaluation of alternative solutions

in terms of costs and accompanying benefits so that decisionmakers can

judge whether the proposed alternatives offer sufficient military or

economic benefits to be worth their cost, and to determine which

alternative is the best approach. By not requiring independent review of

alternatives analyses, Defense increases the risk of making poor decisions

to pursue particular options for developing or acquiring migration

systems. It also risks sending a message throughout the Department that it

is acceptable to decided to invest in major information technology

Page 75 GAO/AIMD-98-5 Defense Migration Strategy
Appendix I 

Comments From the Department of Defense

projects without the benefit of complete and accurate information on the

estimated costs and benefits of alternative investments.

Where economic analyses are concerned, Defense is correct in stating that

its existing acquisition regulations do not require that economic analyses

for delegated systems be independently reviewed by PA&E. Our

recommendation has been revised to enable verification by other qualified

reviewers. However, our recommendation is still valid since our review

revealed that Defense does not ensure that economic analyses for

delegated systems are independently reviewed by any qualified reviewer.

We identified seven major delegated migration systems for which

economic analyses had not been prepared or updated and independently

reviewed even though substantial investments had already been made in

them. The failure to obtain independent review for these systems and to

fully address concerns raised by the reviewer is unacceptable since it

increases the chances that inaccurate, incomplete, and unreliable cost and

benefit information will be used to justify a costly system project.

Assuming this risk is hardly a wise choice given that as a result, Defense

may end up using scarce resources to build information systems that offer

less payback than available alternatives. In addition, we believe that the

failure to obtain an independent review of an economic analysis is certain

to decrease the value of the analysis as a management tool for such

activities as Defense’s formulation of the Program Objective

Memorandum, budget justifications, ranking of investment alternatives,

laying a baseline for tracking performance improvements, coordination of

project development schedules and costs, sensitivity (“what if”) analyses,

and identification of project components that are unfunded.

Furthermore, even though DOD’s acquisition regulations do not require that

PA&E be the economic analysis reviewer for delegated systems, because

PA&E has the necessary expertise, it makes sense that DOD use PA&E to

conduct these reviews. Also, the reviews should be more consistent if PA&E

performs all the economic analysis reviews for major systems. Regardless

of whether PA&E or another qualified independent organization performs

the independent reviews, it is critical that an independent review be made

of the economic analyses for major information technology investments

and that problems identified in the analyses are addressed before

investment decisions are made.

11. See “Agency Comments and Our Evaluation” section in chapter 4.

Page 76 GAO/AIMD-98-5 Defense Migration Strategy
Appendix II 

Description of the Migration System

Decision Process

As shown in figure II.1, Defense’s migration system selection process is

principally the responsibility of the managers of Defense functional areas

(such as logistics, finance, human resources, and health affairs) with

assistance by the technical staff who report to and advise the ASD C3I.

Figure II.1: Migration Selection and

Oversight Processes

PSA selects a 

migration system for a 

functional area, such 

as logistics, finance, 

human resources, and 

health affairs.

All system selections 

are to go to the ASD 

C3I for approval.

Major system 

selections also go to 

one of Defense's 

acquisition review 

boards for acquisition 

oversight. 

all system selections

ASD C3I is to 

approve selections 

after review of data, 

technical, and 

programmatic issues.

major system selections

PSA selects a 

migration 

system

ASD C3I approves 

selections

Defense Acquisition 

Board or Major 

Automated 

Information System 

Review Council 

oversees major 

system acquisitions

Defense 

Information 

Systems Agency

Program 

Analysis and 

Evaluation

DISA and PA&E 

provide advice and 

assistance.

These boards are to 

ensure compliance with 

Defense policies, 

procedures, and 

regulations.

Source: GAO review of Defense regulations, memorandums, related documents, and interviews

with Defense officials.

The specific steps involved with the process illustrated above are

explained in more detail in the following sections.

Principal Staff Assistant

(PSA) Selects the

Migration System

PSAs are the senior Defense managers for the functional areas. They

include Under Secretaries, Assistant Secretaries, the Inspector General,

the Director of Operational Test and Evaluation, the General Counsel, and

a number of other top DOD officials. PSAs are responsible for choosing

migration systems from their respective functional area legacy systems or

for developing new systems.

An important first step involved with deciding whether to migrate and

determining the migration system(s) that will best support the functional

Page 77 GAO/AIMD-98-5 Defense Migration Strategy
Appendix II 

Description of the Migration System

Decision Process

area is assessing the functional area’s business processes and deciding if

changes are needed. For example, a functional area may be better off

outsourcing a process rather than attempting to bring improvements

through internal information system modifications. Or, it could decide that

it should fundamentally change its business processes and acquire

different information systems or significantly modify existing systems to

better support the new processes. In DOD, the functional area goes about

making this broad evaluation of paths toward improvement and making a

“business case” for the selected path through the preparation of a

functional economic analysis. In most cases, functional areas have

concluded that migration is a viable strategy for improvement.

When it embarked on its accelerated migration strategy in 1993, Defense

allowed PSAs to tailor the functional economic analysis, as appropriate, to

expedite the migration system selections. At the same time, however, PSAs

and program managers who manage the individual acquisition projects

were told that they should still follow requirements already established for

developing and maintaining automated systems (life-cycle management

processes). For example, managers would still need to perform an

economic analysis for migration systems to establish a baseline life-cycle

cost and benefit estimate for the projects.1

 This analysis follows the

decision to migrate has been made and should be based on an analysis of

alternative life-cycle costs, benefits, and risks involved in feasible

approaches for the project.

1

Defense regulations require that an economic analysis and an alternatives analysis be prepared to

support all major information systems investments that meet the MAISRC review criteria. However,

for those major systems that meet criteria for oversight by the DAB, Defense regulations require an

independent cost estimate, rather than an economic analysis. For nonmajor information system

investments that do not meet either MAISRC or DAB review criteria, OMB and Defense guidance, as

well as best industry and government practices, recommend that an economic analysis and an

evaluation of alternatives be prepared to help decisionmakers estimate the costs and benefits for the

project and choose the most cost-effective approach among competing information technology

investments.

Page 78 GAO/AIMD-98-5 Defense Migration Strategy
Appendix II 

Description of the Migration System

Decision Process

The ASD C3I Approves the

Systems Selected for

Migration2

 After the PSAs select the migration systems for their functional activities,

the selections are to be sent to the ASD C3I for approval. In deciding

whether to approve the selections, the ASD C3I reviews data, technical, and

programmatic factors. These include:

• Whether the migration systems will lend themselves to data-sharing within

their design. A key step in the process of ensuring this ability to share data

is known as data standardization. This involves reaching agreement on

standard definitions that are to be used for the data elements’ names,

meanings, and other characteristics so that all users of the data have a

common, shared understanding of it.

• Whether the migration systems currently conform to or can evolve to

conform with Defense’s technical standards. These technical standards

are a set of “building codes” for constructing systems to ensure that they

will be compatible with Defense’s communications and computing

infrastructure and technically interoperable with each other.3

• Whether the system selections are supported by a full or abbreviated

functional economic analysis or other similar justification.

The Defense Information Systems Agency (DISA) assists in the review of

these factors and passes on any concerns it has to the ASD C3I.

Major Migration Systems

Also Undergo DOD’s Major

Systems Acquisition

Review Process

In addition to the management and oversight processes established

specifically for the accelerated migration strategy, the major migration

systems are also subject to Defense’s traditional major information

systems acquisition oversight processes by either the Defense Acquisition

Board (DAB) or the Major Automated Information System Review Council

(MAISRC). DAB generally provides oversight for weapon systems while

MAISRC provides oversight for management information systems.

2

To foster an Enterprise Integration approach and augment the approval process for initial migration

selections, Defense established a top management oversight structure to help resolve issues affecting

multiple functional areas and to promote cross-functional planning and information-sharing. This

structure consisted of two committees: (1) the Enterprise Integration Executive Board, which was

chaired by the Deputy Secretary of Defense and included the most senior executive members in the

Department, and (2) the Enterprise Integration Corporate Management Council, which served as a

supporting committee and was co-chaired by the Principal Deputy Under Secretary of Defense for

Acquisition and Technology and the ASD C3I. However, these committees were deactivated in the

summer of 1995—about 1 year after they were established—after the Deputy Secretary decided they

were not working as intended.

3

Defense’s technical standards for constructing information systems include, among others, the

Technical Architecture Framework for Information Management (TAFIM) and the Defense

Information Infrastructure Common Operating Environment (DII COE).

Page 79 GAO/AIMD-98-5 Defense Migration Strategy
Appendix II 

Description of the Migration System

Decision Process

Specifically, DAB reviews those systems that (1) are estimated to eventually

cost over $355 million to research, develop, test, and evaluate, (2) have

estimated procurement costs of over $2.135 billion, or (3) are designated

for DAB review by the Under Secretary of Defense for Acquisition and

Technology, who also chairs DAB. MAISRC reviews those systems that (1) are

anticipated to cost $30 million or more a year, (2) have estimated program

costs in excess of $120 million, (3) have estimated life-cycle cost of more

than $360 million, or (4) are designated for MAISRC review by the ASD C3I,

who also chairs the MAISRC.

4

DAB and MAISRC reviews involve assessing such matters as whether the

proposed system is being developed in accordance with Defense policies,

procedures, and regulations; the systems’ program managers took steps to

minimize the cost of a new system by ensuring full and open competition;

and the program managers will effectively use advanced system design

and software engineering technology to minimize software development

and maintenance costs.

When the DAB or MAISRC staffs review a migration system, they obtain

assistance from the Program Analysis and Evaluation (PA&E) staff to

review the system’s economic analysis or cost estimate. This staff is

responsible for reviewing economic analyses for major information

systems and determining whether cost and benefit estimates are accurate

and complete. PA&E analysts are recognized as DOD’s experts on economic

and cost analyses. Organizationally, PA&E reports to the Under Secretary of

Defense (Comptroller). MAISRC requests that DISA provide technical reviews

of the plans and other key documents prepared to support the system’s

development. MAISRC also obtains assistance from other Defense

organizations in reviewing the systems.

Program managers for the systems reviewed by MAISRC provide periodic

progress information to MAISRC regarding such issues as cost, schedule,

return on investment, and compliance with technical and data standards.

Generally, this information is provided in written quarterly reports, which

may be tailored to meet the needs of both the program manager and

MAISRC. In addition, MAISRC conducts in-process reviews of the systems by

establishing “integrated product teams” that provide advice and assistance

to the systems’ program managers on various subjects as needed to

progress through the acquisition process.

4

These dollar amounts are in fiscal year 1996 constant dollars (dollar amounts after they have been

adjusted for the effect of inflation).

Page 80 GAO/AIMD-98-5 Defense Migration Strategy
Appendix III 

Detailed Analysis of Migration Cost

Information

Having accurate and reliable cost and inventory information is critical for

decision-making purposes. It helps Defense managers to determine

whether migration system investments will be cost-effective and whether

enough funds will be available for developing and deploying the system.

However, Defense’s migration cost information is incomplete and may be

significantly understated.

Collecting cost information for migration systems in itself is a challenging

task because Defense does not maintain complete cost information on all

of the migration systems in the Defense Integration Support Tools (DIST)

database—its primary information system tracking tool. Thus, to report to

the Congress and others on migration costs, Defense must turn to

numerous sources. For example, in providing information on costs to the

Congress in April 1996, Defense asked individual functional area managers

to provide total estimated cost data for each of their migration systems for

fiscal years 1995 through 2000.

Defense officials stated that cost data are not maintained in DIST because

functional area managers expressed concern that controls were not

adequate to ensure that only authorized personnel were allowed access to

the data. They also noted that Defense does not have budget information

for all the migration systems because it does not separately budget or

collect information for systems with annual cost of less than $2 million. In

fiscal year 1996, DOD’s Office of the Under Secretary of Defense

(Comptroller) maintained budget information on the systems that cost

$2 million or more and is currently working on including all migration

systems and other information system costs (for systems with annual

costs of more than $2 million) within one database.

Cost Information Is

Incomplete

In December 1996, Defense provided us with cost information on the

migration effort, but these data provided costs for only 242 of the 363

migration systems. We did not include costs in this report for 27 of 242

systems for which DOD collected costs because the systems were

classified. Defense did not provide us costs for the remaining 121 systems

because the Office of the ASD C3I had not collected costs for these

systems.

When we reviewed the cost data reported for the clinical health, civilian

personnel, and transportation areas, as part of our review of migration

Page 81 GAO/AIMD-98-5 Defense Migration Strategy
Appendix III 

Detailed Analysis of Migration Cost

Information

systems’ cost reported to the Congress in April 1996,1 we identified three

reasons why the functional areas did not report costs for some systems.

First, for some of the systems, these costs were not available at the time

the report was made to the Congress. Second, for other systems, the

functional area managers were not aware that the report was to include

interim migration systems. Third, for some systems, functional area

approval as a migration system may not have been completed at the time

cost data were gathered for the report to Congress. As table III.1

illustrates, the costs not reported represented a substantial portion of the

total being spent on migration for the three functional areas.

Table III.1: Costs Reported to the

Congress Compared to Costs

Reported to GAO for Clinical Health,

Civilian Personnel, and Transportation

Dollars in millions

Functional area

Total costs

reported

to the

Congress

Total costs

reported

to GAO

Difference

(amount not

reported to the

Congress)

Clinical

Health $1,014.0 $2,451.3 $1,437.3

Civilian

Personnel 67.4 123.9 56.5

Transportation 1,005.2 1,116.6 111.4

Total $2,086.6 $3,691.8 $1,605.2

Source: GAO’s analysis of information obtained from the April 1996 Report to the Congress,

Defense functional area managers, and the ASD C3I.

Significant Costs Not

Tracked

We also found that Defense does not account for some very important

costs related to developing, deploying, and maintaining migration systems

both before and after a project has been initiated. It is extremely important

to have accurate estimates up front on direct, indirect, and support costs

associated with a system development or acquisition project and to be

able to track these costs throughout the life-cycle. At the beginning of a

system development project, a complete cost picture provides an

organization with a realistic picture of what it will take to develop, deploy,

and maintain a system over its life-cycle. It also enables an organization to

make sound trade-off decisions among competing investments.

Continually tracking direct, indirect, and support costs helps managers

account for their past activities, manage current operations, and assess

progress toward planned objectives.

1

As stated in chapter 1, this report was made pursuant to a requirement in Section 366 of the National

Defense Authorization Act for Fiscal Year 1996.

Page 82 GAO/AIMD-98-5 Defense Migration Strategy
Appendix III 

Detailed Analysis of Migration Cost

Information

As noted in this report, when we conducted detailed reviews of selected

finance and logistics migration efforts, we found that DOD excluded

significant categories of costs from its initial life-cycle cost estimates.

Furthermore, our work in the clinical health, civilian personnel, and

transportation functional areas showed that DOD also does not capture and

report the total actual costs for its systems as they are developed,

deployed, and maintained. Instead, DOD decisionmakers employ cost

accounting systems that omit relevant project costs, such as those

associated with project management and oversight. The transportation

functional area, for example, has an office designated to oversee the

development and deployment of its migration systems. But, when

accounting for costs for the systems, DOD does not factor in the cost to

maintain this oversight responsibility. As a result, DOD cannot reliably

report migration system projects’ actual costs and compare them against

established baselines, and it cannot reliably use information relating to

projects’ actual costs to improve future cost estimating efforts.

Beginning October 1, 1996, a new federal accounting standard requires the

Department of Defense and other federal agencies to begin accounting for

all identifiable direct, indirect, and support costs of its outputs. This would

provide a mechanism for developing total costs of capital items, such as

information systems. This requirement is defined in the Statement of

Federal Financial Accounting Standards, (SFFAS) No. 4, Managerial Cost

Accounting Concepts and Standards for the Federal Government.

Page 83 GAO/AIMD-98-5 Defense Migration Strategy
Appendix IV 

Detailed Analysis of Database Used to

Collect Migration Information

There are a number of data integrity problems associated with the DIST

database, which is used to keep information on Defense’s migration

systems. These problems directly affect Defense’s ability to accurately

determine how many legacy systems were terminated and how many are

scheduled for termination. Defense officials acknowledge that the DIST

database is incomplete and inaccurate, but stated they used it to generate

reports to the Congress because it was the only departmentwide database

containing schedule and other descriptive data on all of Defense’s

migration and legacy systems.

When we obtained a copy of the DIST database for analysis in April 1996,

Defense’s own analysis of the DIST database confirmed that it had

incomplete data in several categories of key information. A more recent

Defense analysis of migration systems showed that DIST continues to have

high levels of incomplete data. Table IV.1 illustrates how information was

incomplete in several categories of information.

Table IV.1: Levels of Incompleteness in

DIST in April 1996 and February 1997 Percent of systems

with incomplete data

Data category

April

1996

February

1997

Interfaces with other systems 88 55

Computer installations where the system operated 79 77

Computer hardware on which the system operated 75 68

System software 66 61

Organization responsible for the system 32 26

Source: GAO analysis of DIST Data Status Working Papers.

Our analysis supports Defense’s concerns as still relevant and identified

the following additional data integrity problems.

• Large numbers of migration system implementation dates (220, or

61 percent) of the migration systems’ implementation dates and legacy

system termination dates (629, or 32 percent) shown in DIST were in the

year 2010. According to system documentation and our discussions with

officials responsible to operate DIST, the presence of the year 2010 for

these dates indicates the system owners responsible for the migration

systems did not provide a date, resulting in the DIST system automatically

inserting the default date of 2010.

Page 84 GAO/AIMD-98-5 Defense Migration Strategy
Appendix IV 

Detailed Analysis of Database Used to

Collect Migration Information

• As figure IV.1 illustrates, when we compared the DIST schedule data for

clinical health, civilian personnel, and transportation to the actual

schedule maintained for these areas, we found that incorrect data for

migration implementation and legacy system termination outweighed

correct data.

Figure IV.1: Number of Incorrect and

Correct Migration and Legacy System

Dates for Clinical Health, Civilian

Personnel, and Transportation

Migration system 

implementation 

dates

Legacy system 

termination dates

0

20

40

60

80

100

120

23

33

30

101

Correct Incorrect

Sources: GAO’s analysis of information obtained from the DIST, April 1996 Report to the

Congress, Defense functional area managers, and ASD C3I.

Functional area managers in these three areas identified additional

inaccuracies. For example, managers in the transportation area stated that

Page 85 GAO/AIMD-98-5 Defense Migration Strategy
Appendix IV 

Detailed Analysis of Database Used to

Collect Migration Information

the area had implemented six migration systems rather than none as

indicated in DIST. For the three functional areas, DIST showed that 92 legacy

systems were terminated by April 1996, while functional managers told us

that only 43 had actually been terminated. Also, for the three functional

areas, DIST showed that 53 legacy systems were scheduled for future

termination, but functional managers told us that 91 were slated for future

termination.

We also learned that DIST has other problems that hamper its use. One

problem is that Defense has not ensured that the data definitions and

formats used in DIST are fully compatible with data maintained in other

Defense information systems that track and report on systems. While DOD

is attempting to address this issue, DISA provided us information showing

that only 66 of the 218 data elements contained in the DIST have been

approved as standard data elements in the Defense Data Dictionary

System. Without standard definitions and data formats, data cannot be

easily transferred to DIST from other systems that may be used by PSAs,

program managers, and other decisionmakers.

In addition, Defense assigns unique numbers to each system in DIST, but

these numbers are not universally used throughout the Department. The

different number schemes make it more difficult to facilitate the exchange

of information between these systems. Use of the same identifying number

for each system throughout DOD is also important because Defense often

changes the names of its information systems over their life. Without

consistent identification numbers, tracking an information system in the

face of such name changes is more arduous.

Defense recognizes that it needs better information to deal with migration

and other technology issues for which DIST is relied on, such as the Year

2000 problem.1

 In November 1996, the Under Secretary for Defense

(Comptroller) and the ASD C3I issued a joint memorandum to senior

Defense managers stating that DIST was the backbone tool for managing

the Department’s information technology investment and that to be

effective, it must contain accurate data on all the Department’s

information systems. The memorandum also stated that registration of

information systems in the DIST was mandatory. However, the minimum

set of data that is required under this joint memorandum focused on

information needed to address Year 2000 issues and did not include

1

The Year 2000 problem refers to the possibility that many computer systems will fail on New Year’s

morning in the year 2000 because they were not designed to accommodate the change of date to the

new millennium.

Page 86 GAO/AIMD-98-5 Defense Migration Strategy
Appendix IV 

Detailed Analysis of Database Used to

Collect Migration Information

critical information on migration implementation dates, cost, savings, and

performance measures.1

1

See chapter 4, footnote 1, for recent Defense developments regarding DIST.

Page 87 GAO/AIMD-98-5 Defense Migration Strategy
Appendix V 

Major Contributors to This Report

Accounting and

Information

Management Division,

Washington, D.C.

Cristina T. Chaplain, Communications Analyst

Danny R. Latta, Assistant Director, Policies and Issues

Harold J. Brumm, Jr., Economist

Kansas City Field

Office

George L. Jones, Evaluator-in-Charge

David R. Solenberger, Senior Evaluator

Karen S. Sifford, Staff Evaluator

John G. Snavely, Staff Evaluator

Michael W. Buell, Staff Evaluator

Karl G. Neybert, Staff Evaluator

Page 88 GAO/AIMD-98-5 Defense Migration Strategy
Related GAO Products

High-Risk Series: Information Management and Technology (GAO/HR-97-9,

February 1997).

Air Traffic Control: Improved Cost Information Needed to Make Billion

Dollar Modernization Investment Decisions (GAO/AIMD-97-20, January 22,

1997).

Defense Communications: Performance Measures Needed to Ensure DISN

Program Success (GAO/AIMD-97-9, November 27, 1996).

Defense IRM: Strategy Needed for Logistics Information Technology

Improvement Efforts (GAO/AIMD-97-6, November 14, 1996).

Information Technology Investment: Agencies Can Improve Performance,

Reduce Costs, and Minimize Risks (GAO/AIMD-96-64, September 30, 1996).

DOD Accounting Systems: Efforts to Improve System for Navy Need Overall

Structure (GAO/AIMD-96-99, September 30, 1996).

Defense IRM: Critical Risks Facing New Materiel Management Strategy

(GAO/AIMD-96-109, September 6, 1996).

Defense Transportation: Migration Systems Selected Without Adequate

Analysis (GAO/AIMD-96-81, August 29, 1996).

Information Management Reform: Effective Implementation Is Essential

for Improving Federal Performance (GAO/T-AIMD-96-132, July 17, 1996).

Acquisition Reform: Regulatory Implementation of the Federal Acquisition

Streamlining Act of 1994 (GAO/NSIAD-96-139, June 28, 1996).

Medical ADP Systems: Defense Achieves Worldwide Deployment of

Composite Health Care System (GAO/AIMD-96-39, April 5, 1996).

Best Management Practices: Reengineering the Air Force’s Logistics

System Can Yield Substantial Savings (GAO/NSIAD-96-5, February 21, 1996).

Inventory Management: DOD Can Build on Progress in Using Best Practices

to Achieve Substantial Savings (GAO/NSIAD-95-142, August 4, 1995).

Information Technology Investment: A Governmentwide Overview

(GAO/AIMD-95-208, July 31, 1995).

Page 89 GAO/AIMD-98-5 Defense Migration Strategy
Related GAO Products

Defense Management: Selection of Depot Maintenance Standard System

Not Based on Sufficient Analyses (GAO/AIMD-95-110, July 13, 1995).

Defense Communications: Management Problems Jeopardize DISN

Implementation (GAO/AIMD-95-136, July 13, 1995).

Best Practices Methodology: A New Approach for Improving Government

Operations (GAO/NSIAD-95-154, May 1995).

Government Reform: Goal-Setting and Performance (GAO/AIMD/GGD-95-130R,

March 27, 1995).

Defense Infrastructure: Enhancing Performance Through Better Business

Practices (GAO/T-NSIAD/AIMD-95-126, March 23, 1995).

Paperwork Reduction Act: Reauthorization Can Strengthen Government’s

Management of Information and Technology (GAO/T-AIMD/GGD-95-80,

February 7, 1995).

Government Reform: Using Reengineering and Technology to Improve

Government Performance (GAO/T-OCG-95-2, February 2, 1995).

High-Risk Series: An Overview (GAO/HR-95-1, February 1995).

Reengineering: Opportunities to Improve (GAO/AIMD-95-67R, January 6, 1995).

Reengineering Organizations: Results of a GAO Symposium (GAO/NSIAD-95-34,

December 13, 1994).

Defense Management: Impediments Jeopardize Logistics Corporate

Information Management (GAO/NSIAD-95-28, October 21, 1994).

DOD’s CALS Initiative (GAO/AIMD-94-197R, September 30, 1994).

Defense Management Initiatives: Limited Progress in Implementing

Management Initiatives (GAO/T-AIMD-94-105, April 14, 1994).

Financial Management: Financial Control and System Weaknesses

Continue to Waste DOD Resources and Undermine Operations

(GAO/T-AIMD/NSIAD-94-154, April 12, 1994).

Page 90 GAO/AIMD-98-5 Defense Migration Strategy
Related GAO Products

Defense Management: Stronger Support Needed for Corporate

Information Management Initiative to Succeed (GAO/AIMD/NSIAD-94-101,

April 12, 1994).

Defense IRM: Management Commitment Needed to Achieve Defense Data

Administration Goals (GAO/AIMD-94-14, January 21, 1994).

Defense IRM: Business Strategy Needed for Electronic Data Interchange

Program (GAO/AIMD-94-17, December 9, 1993).

Financial Management: Reliability of Weapon System Cost Reports Is

Highly Questionable (GAO/AIMD-94-10, October 28, 1993).

Defense: Health Care (GAO/IMTEC-93-29R, June 3, 1993).

Mission-Critical Systems: Defense Attempting to Address Major Software

Challenges (GAO/IMTEC-93-13, December 24, 1992).

Defense ADP: Corporate Information Management Must Overcome Major

Problems (GAO/IMTEC-92-77, September 14, 1992).

Air Force ADP: Status of Logistics Modernization Projects and CIM Impacts

(GAO/IMTEC-92-66, July 30, 1992).

Defense Logistics: DOD Initiatives Related to Cutting Costs

(GAO/T-NSIAD-92-24, March 26, 1992).

Defense ADP: Lessons Learned From Development of Defense Distribution

System (GAO/IMTEC-92-25, March 20, 1992).

Automated Information Systems: Defense Should Stop Further

Development of Duplicative Recruiting Systems (GAO/IMTEC-92-15,

February 27, 1992).

Defense ADP: A Coordinated Strategy Is Needed to Implement the CALS

Initiative (GAO/IMTEC-91-54, September 13, 1991).

Defense Distribution System Illustrates Problems Facing Corporate

Information Management Interim Standard Systems (GAO/T-IMTEC-91-12,

April 23, 1991).

Page 91 GAO/AIMD-98-5 Defense Migration Strategy
Related GAO Products

Defense ADP: Corporate Information Management Initiative Faces

Significant Challenges (GAO/IMTEC-91-35, April 22, 1991).

Defense ADP: Corporate Information Management Savings Estimates Are

Not Supported (GAO/IMTEC-91-18, February 22, 1991).

Army Automation: Decisions Needed on SIDPERS-3 Before Further

Development (GAO/IMTEC-90-66, September 5, 1990).

Automated Information Systems: Defense’s Oversight Process Should Be

Improved (GAO/IMTEC-90-36, April 16, 1990).

Schedule Delays and Cost Overruns Plague DOD Automated Information

Systems (GAO/T-IMTEC-89-8, May 18, 1989).

Automated Information Systems: Schedule Delays and Cost Overruns

Plague DOD Systems (GAO/IMTEC-89-36, May 10, 1989).

(511355) Page 92 GAO/AIMD-98-5 Defense Migration Strategy
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100