United States Government Accountability Office

GAO Report to the Subcommittee on 

Readiness and Management Support, 

Committee on Armed Services, 

U.S. Senate 

DOD BUSINESS 

SYSTEMS 

MODERNIZATION 

Important 

Management Controls 

Being Implemented on 

Major Navy Program, 

but Improvements 

Needed in Key Areas 

September 2008 

 

GAO-08-896 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

September 2008

DOD BUSINESS SYSTEMS MODERNIZATION

Important Management Controls Being Implemented 

on Major Navy Program, but Improvements Needed in 

Highlights of GAO-08-896, a report to the Key Areas Subcommittee on Readiness and

Management Support, Committee on 

Armed Services, U.S. Senate 

The Department of Defense (DOD)

has long been challenged in 

implementing key information 

technology (IT) management 

controls on its thousands of

business system investments. For 

this and other reasons, GAO has 

designated DOD’s business systems 

modernization efforts as high-risk. 

One of the larger business system 

investments is the Department of

the Navy’s Enterprise Resource 

Planning (Navy ERP) program. 

Initiated in 2003, the program is to 

standardize the Navy’s business 

processes, such as acquisition and 

financial management. It is being 

delivered in increments, the first of 

which is to cost about $2.4 billion 

over its useful life and be fully

deployed in fiscal year 2013. GAO

was asked to determine whether

key IT management controls are 

being implemented on the program. 

To do this, GAO analyzed, for 

example, requirements 

management, economic 

justification, earned value 

management, and risk 

management.

What GAO Recommends

GAO is making recommendations 

to the Secretary of Defense aimed 

at improving cost and schedule 

estimating, earned value 

management, and risk management

weaknesses. DOD largely agreed

with GAO’s recommendations and 

described actions planned or under

way to address them.

DOD has implemented key IT management controls on its Navy ERP program 

to varying degrees of effectiveness. To its credit, the control associated with 

managing system requirements is being effectively implemented. In addition, 

important aspects of other controls have at least been partially implemented, 

including those associated with economically justifying investment in the 

program and proactively managing program risks. Nevertheless, other aspects 

of these controls, as well as the bulk of what is needed to effectively 

implement earned value management, which is a recognized means for 

measuring program progress, have not been effectively implemented. Among 

other things, these control weaknesses have contributed to the more than 2-

year schedule delay and the almost $600 million cost overruns already

experienced on the program since it began, and they will likely contribute to 

future delays and overruns if they are not corrected. Examples of the 

weaknesses are provided below. 

• Investment in the program has been economically justified on the basis of 

expected benefits that far exceed estimated costs ($8.6 billion versus $2.4

billion over a 20-year life cycle). However, important estimating practices, 

such as using historical cost data from comparable programs and basing 

the cost estimate on a reliable schedule baseline were not employed. 

While these weaknesses are important because they limit the reliability of 

the estimates, GAO’s analysis shows that they would not have altered the 

estimates to the point of not producing a positive return on investment. 

• Earned value management has not been effectively implemented. To its 

credit, the program office has elected to implement program-level earned 

value management. In doing so, however, basic prerequisites for 

effectively managing earned value have not been executed. In particular, 

the integrated master schedule was not derived in accordance with key 

estimating practices, and an integrated baseline review has not been 

performed on any of the first increment’s releases. 

• A defined process for proactively avoiding problems, referred to as risk 

management, has been established, but risk mitigation strategies have not 

been effectively implemented for all significant risks, such as those 

associated with data conversion and organizational change management, 

as well the risks associated with the above-cited weaknesses. 

The reasons that program management and oversight officials cited for these 

practices not being executed range from the complexity and challenges of 

managing and implementing a program of this size to limitations in the 

program office’s scope and authority. Notwithstanding the effectiveness with

which important aspects of several controls have been implemented, the 

above-cited weaknesses put DOD at risk of investing in a system solution that 

does not optimally support corporate mission needs and mission

performance, and meet schedule and cost commitments. 

To view the full product, including the scope 

and methodology, click on GAO-08-896. 

For more information, contact Randolph C. 

Hite at (202) 512-3439 or hiter@gao.gov. 
Contents 

Letter 1

Results in Brief 2

Background 7

Implementation of Key DOD and Related IT Acquisition 

Management Controls on Navy ERP Varies 21 

Conclusions 47

Recommendations for Executive Action 48 

Agency Comments and Our Evaluation 49 

Appendix I Objective, Scope, and Methodology 52 

Appendix II Comments from the Department of Defense 56 

Appendix III GAO Contact and Staff Acknowledgments 62 

Tables 

Table 1: Description and Status of the Navy ERP Pilots 8 

Table 2: Navy ERP Template 1 Releases 9 

Table 3: Organizations Responsible for Navy ERP Oversight and 

Management 10

Table 4: Description of Business System IT Acquisition 

Management Controls 18 

Table 5: Summary of Cost Estimating Characteristics That the Cost 

Estimate Satisfies 31 

Table 6: Satisfaction of Schedule Estimating Key Practices 38 

Figures 

Figure 1: Navy ERP Time Line 15 

Figure 2: Navy ERP Milestones for Beginning Deployment 16 

Figure 3: Navy ERP Life Cycle Cost Estimates in Fiscal Years 2003, 

2004, and 2007 16 

Figure 4: Cumulative Cost Variance for Navy ERP over a 17-Month 

Period 42

Figure 5: Cumulative Schedule Variance of the Navy ERP Program 

over a 17-Month Period 43 

Page i GAO-08-896 DOD Business Systems Modernization 
Abbreviations 

BEA business enterprise architecture 

BTA Business Transformation Agency 

CIO Chief Information Officer 

DBSMC Defense Business Systems Management Committee 

DOD Department of Defense 

DON Department of the Navy 

EVM earned value management 

FOC full operational capability 

GCSS-MC Global Combat Support System–Marine Corps 

IOC initial operational capability 

IRB investment review board

IT information technology 

MDA milestone decision authority 

NAVAIR Naval Air Systems Command 

NAVSEA Naval Sea Systems Command 

NAVSUP Naval Supply Systems Command 

Navy ERP Navy Enterprise Resource Planning 

NTCSS Naval Tactical Command Support System 

OMB Office of Management and Budget 

PMB performance measurement baseline

SAP Systems Applications and Products 

SPAWAR Space and Naval Warfare Systems Command 

TC-AIMS II Transportation Coordinators’ Automated Information for 

Movements System 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately. 

Page ii GAO-08-896 DOD Business Systems Modernization 
United States Government Accountability Office

Washington, DC 20548 

 

September 8, 2008 

The Honorable Daniel K. Akaka 

Chairman 

The Honorable John Thune 

Ranking Member 

Subcommittee on Readiness and Management Support 

Committee on Armed Services 

United States Senate 

The Honorable John Ensign 

United States Senate 

For decades, the Department of Defense (DOD) has been challenged in 

modernizing its timeworn business systems.1 In 1995, we designated DOD’s 

business systems modernization program as high risk, and continue to do 

so today.2

 Our reasons include the modernization’s large size, complexity, 

and its critical role in addressing other long-standing transformation and 

financial management challenges. Other reasons are that DOD has yet to 

institutionalize key system modernization management controls, and it has 

not demonstrated the ability to consistently deliver promised system 

capabilities and benefits on time and within budget. 

Nevertheless, DOD continues to invest billions of dollars in thousands of 

business systems, including about a hundred that the department has 

labeled as business transformational programs, 12 of which account for 

about 50 percent of these programs’ costs. The Navy Enterprise Resource 

Planning (Navy ERP) program is one such program. Initiated in 2003, Navy

ERP is to standardize the Navy’s acquisition, financial, program 

management, maintenance, plant and wholesale supply, and workforce 

management business processes across its dispersed organizational 

environment. As envisioned, the program consists of a series of major 

increments, the first of which includes three releases and is expected to 

cost approximately $2.4 billion over its 20-year life cycle and to be fully 

operational in fiscal year 2013. 

 1

Business systems are information systems, including financial and nonfinancial systems,

that support DOD business operations, such as civilian personnel, finance, health, logistics, 

military personnel, procurement, and transportation.

2

GAO, High-Risk Series: An Update, GAO-07-310 (Washington, D.C.: January 2007). 

Page 1 GAO-08-896 DOD Business Systems Modernization 
As agreed, our objective was to determine whether the Department of the 

Navy (DON)3

 is effectively implementing information technology (IT) 

management controls on Navy ERP. To accomplish this, we focused on 

the program’s first increment by analyzing a range of program 

documentation and interviewing cognizant officials relative to the 

following management areas: architectural alignment, economic 

justification, earned value management (EVM), requirements management, 

and risk management. We conducted this performance audit from June 

2007 to September 2008, in accordance with generally accepted 

government auditing standards. Those standards require that we plan and

perform the audit to obtain sufficient, appropriate evidence to provide a 

reasonable basis for our findings and conclusions based on our audit 

objective. We believe that the evidence obtained provides a reasonable 

basis for our findings and conclusions based on our audit objective. 

Additional details on our objective, scope, and methodology are in 

appendix I. 

DOD has implemented key IT management controls on its Navy ERP 

program to varying degrees of effectiveness. To its credit, one of the 

controls has been fully implemented; important aspects of other controls 

have not. Collectively, these management controls are to ensure that a 

given system investment represents the right solution to filling a mission 

need, meaning that the system is defined to (1) minimize overlap and 

duplication and maximize interoperability with related systems and 

(2) produce mission benefits commensurate with costs over its useful life. 

The controls are also to ensure that the system is acquired and deployed

the right way, meaning that it is done in a way to maximize the chances of 

delivering defined system capabilities and benefits on time and within 

budget. Given that deployment of Navy ERP is more than 2 years behind 

schedule and is to cost about $570 million4

 more than was originally 

envisioned, these goals have already not been fully met, in part because 

DOD program management and oversight entities have not fully 

implemented several key IT management controls. As a result, the 

department has yet to adequately demonstrate that the program’s first 

increment, as it has been defined, is the right solution, and it is likely that 

Results in Brief 

 3

The Department of the Navy is a major component of DOD, consisting of two uniformed

services: the Navy and the Marine Corps. 

4

This amount is the difference between the September 2007 estimated life cycle cost of 

$2,445.0 million and the September 2003 estimated life cycle cost of $1,873.1 million. 

Page 2 GAO-08-896 DOD Business Systems Modernization 
the department will continue to add to the program’s cost overruns and 

schedule delays that the program has already experienced to date. The

strengths and weaknesses associated with each of the IT management 

controls that we evaluated are described here: 

• Navy ERP compliance with DOD’s federated business enterprise 

architecture (BEA) has not been sufficiently demonstrated. To its credit, 

the program office followed DOD’s architecture compliance assessment 

guidance and used the related assessment tool. However, this guidance

and tool do not adequately provide for addressing all relevant aspects of 

architectural compliance. Specifically, the program’s compliance 

assessment (1) did not include all relevant architecture products, such as

those that describe technical and system elements; (2) was not used to 

identify potential areas of duplication across programs; and (3) did not 

address compliance with the DON enterprise architecture. These 

important steps were not performed because of policy, guidance, and tool 

limitations, and because aspects of the corporate BEA and the DON 

enterprise architecture, which are both major components of DOD’s 

federated BEA, have yet to be sufficiently defined to permit thorough 

compliance determinations in these areas. In addition, program oversight 

and approval authorities did not validate the program office’s compliance 

assessments. As a result, the department does not have a sufficient basis 

for knowing if Navy ERP has been defined to minimize overlap with, and 

duplication of other programs’ functions, and is being defined and 

designed to maximize interoperability among related programs. 

• Investment in Navy ERP has been economically justified on the basis of 

expected life cycle benefits that far exceed estimated life cycle costs 

($8.6 billion versus $2.4 billion over a 20-year life cycle). However, these 

benefit estimates have not been subject to analysis of how uncertainty in

assumptions and data could impact them, as prescribed in relevant 

guidance. According to program officials, such uncertainty analysis is not 

warranted because they have taken and continue to take steps to validate 

the assumptions and the data, such as using the latest budget data 

associated with retiring legacy systems, and monitoring changes to the 

systems’ retirement dates. While these steps are positive, they do not 

eliminate the need for uncertainty analysis. Accordingly, we assessed key 

uncertainty variables and found that while the inherent uncertainty in the 

estimates would reduce expected savings, the reduction would be small 

relative to a total benefit estimate of $8.6 billion. 

With respect to the cost estimate, our analysis showed that it was not 

derived using all key estimating practices contained in relevant guidance.

For example, the estimate was not grounded in a historical record of 

Page 3 GAO-08-896 DOD Business Systems Modernization 
comparable data from similar programs and was not based on a reliable 

schedule baseline, which are both necessary to having a cost estimate that 

can be considered credible and accurate. These practices were not 

employed for various reasons, including DOD’s lack of historical data from 

similar programs and the lack of an integrated master schedule for the 

program’s first increment that includes all three releases. Notwithstanding 

the fact that these limitations could materially increase the $2.4 billion 

cost estimate, it is nevertheless unlikely that they would increase the cost 

estimate to a level close to the uncertainty-adjusted benefit expectations.

Therefore, we have no reason to believe that Navy ERP will not produce a 

positive return on investment. 

• EVM, which is a means for determining and disclosing actual program 

performance in comparison with estimates, is not being effectively 

implemented in Navy ERP. To its credit, the program office has elected to

implement program-level EVM.5

 In doing so, however, basic EVM activities 

have not been executed, which has produced, and will likely continue to 

produce, actual program costs and schedules that do not track close to 

estimates. For example, an integrated baseline review, which is to verify 

that the program’s costs and schedule are reasonable given the program’s 

scope of work and associated risks, has not been performed on any of the 

first increment’s releases. According to program officials, this is because 

of the time it took to establish program-level EVM capabilities and because 

of their focus on deploying and stabilizing the first release. However, they 

recently stated that one has been tentatively scheduled for August 2008. By

not having an integrated master schedule that has been subject to a 

baseline review, as well as not employing other industry standards as 

discussed in this report, Navy ERP will be challenged in implementing 

EVM effectively, and cost overruns and lengthy schedule delays beyond

those already experienced by the program will likely occur. Our analysis 

of the latest estimate for completing just the budgeted development work 

for all three releases, which is about $844 million, shows that this estimate 

will most likely have an overrun of about $152 million. 

• An important requirements management activity has been effectively 

implemented in Navy ERP. Specifically, the program office has ensured 

that system requirements for the first release are traceable, as evidenced 

by our analysis of 60 randomly selected system-level requirements in 

 5

Program-level means that EVM covers all aspects of the program, regardless of whether 

they are performed by the government or the contractor. In contrast, EVM has historically 

been implemented on a contract-by-contract basis, and has not included government

performed work. 

Page 4 GAO-08-896 DOD Business Systems Modernization 
which we confirmed that 58 are traceable backward to operational 

requirements and forward to design specifications and test results. Such 

traceability is important because it increases the chances of delivering a 

system that performs as intended. Our analysis of requirements in this 

sample also confirmed that system requirements that had been reallocated 

from the first release to the other releases were traceable, thus

demonstrating traceability among product releases. 

• The program office has defined a process for proactively managing risks 

that reflects key practices governing how this IT management control 

should be performed. However, it has not effectively implemented this 

process for all identified risks. In particular, steps taken to mitigate the 

risks associated with converting data from existing systems to the new 

system and positioning user organizations for the operational changes 

associated with the new system were not effective. According to program 

officials, these mitigation strategies could not be effectively implemented 

because the program office does not have the authority to compel the user 

organizations to execute their part of the mitigation strategy. As a result, 

the first user organization to receive the Navy ERP experienced significant 

problems during recent operational testing, and these problems will take 

additional time and resources to correct. In addition, not all known risks 

have been captured in the risk inventory. For example, the risks 

associated with the two above discussed control weaknesses (not having 

adequately demonstrated the program’s architectural alignment and not 

having implemented program-level EVM according to industry practices) 

are not included in the risk inventory and are thus not being disclosed and 

addressed. This is important because not having effectively addressed 

such risks has contributed to schedule delays and will likely contribute to

more cost and schedule shortfalls. 

Notwithstanding the effectiveness with which several program 

management controls have been implemented on Navy ERP, the abovecited weaknesses in implementing other management controls put DOD at 

risk of investing in a system solution that does not optimally support 

corporate mission needs and mission performance and continuing to fall 

short of program schedule and cost commitments. Accordingly, we are 

making recommendations to the Secretary of Defense aimed at addressing 

the cost and schedule estimating, EVM, and risk management weaknesses, 

thereby better ensuring that the program is managed to deliver the right 

solution, the right way. We are not making recommendations in this report 

for addressing the architecture compliance weaknesses because we 

recently completed other work that is more broadly focused on this 

management control across multiple programs. 

Page 5 GAO-08-896 DOD Business Systems Modernization 
In written comments on a draft of this report, signed by the Deputy Under

Secretary of Defense (Business Transformation) and reprinted in appendix

II, the department stated that it concurred with two, and partially 

concurred with two, of our four recommendations. Further, it stated that it

has taken steps to address some of our recommendations, adding that it is

committed to implementing recommendations that contribute to the 

program’s success. 

In partially concurring with two of the recommendations, DOD concurred 

with most aspects of both. Nevertheless, for our recommendation aimed at

improving the program’s cost estimates, it stated that it did not concur 

with one aspect—ensuring that future cost estimates reflect the risk of not 

having cost data for comparable programs. In doing so, DOD stated that 

while the program had limited comparable data on which to base the 

program’s cost estimate, it had accounted for this limitation in the cost 

estimate’s uncertainty analysis. We do not agree that this risk was 

reflected in the uncertainty analysis. We examined this analysis as part of 

our review and found that it did not recognize this risk. Moreover, DOD’s 

comments did not include any evidence to the contrary. 

In addition, for our recommendation aimed at improving the program’s 

integrated master schedule, the department partially concurred with one 

of the five components of the recommendation—defining a critical path

that incorporates all three releases of the system. In doing so, DOD stated 

what our report already recognized, namely that the schedule reflects a 

separate critical path for each release and that this is due to the size and 

complexity of the releases. However, DOD offers no new information in its 

comments. Further, our report also recognizes that to be successful, large 

and complex programs that involve thousands of activities need to ensure 

that their schedules integrate these activities. In this regard, we support 

the department’s commitment to explore the feasibility of implementing 

this part of our recommendation. 

While concurring with all components of our other two recommendations, 

the department nevertheless provided comments relative to the program’s 

use of EVM to explain why Release 1.0 of the system was not subject to an

integrated baseline review. For several reasons discussed in the agency

comments section of this report, we do not agree with these additional 

comments. In addition, the department’s comments described actions 

planned or under way to address our recommendations. If fully and 

properly implemented, DOD’s described actions should go a long way in 

addressing the weaknesses that we identify in the report. 

Page 6 GAO-08-896 DOD Business Systems Modernization 
DON’s primary mission is to organize, train, maintain, and equip combatready naval forces capable of winning the global war on terrorism and any 

other armed conflict, deterring aggression by would-be foes, preserving 

freedom of the seas, and promoting peace and security. To support this 

mission, DON performs a variety of interrelated and interdependent 

business functions (e.g., acquisition and financial management), relying 

heavily on IT systems. In fiscal year 2008, DON’s budget for business 

systems and associated infrastructure was about $2.7 billion, of which $2.2 

billion was allocated to operations and maintenance of existing systems 

and the remaining $500 million to systems in development and 

modernization. Of the approximately 3,000 business systems that DOD 

reports in its current inventory, DON accounts for 904, or about 30 

percent, of the total. Navy ERP is one such system investment. 

In July 2003, the Assistant Secretary of the Navy for Research, 

Development, and Acquisition established Navy ERP to “converge” four 

separate pilot programs that were under way at four separate Navy 

commands.6

 This program is to leverage a commercial, off-the-shelf 

software known as an enterprise resource planning product. Such 

products consist of multiple, integrated functional modules that perform a 

variety of business-related tasks, such as acquisition and financial 

management. Table 1 provides a brief description and status of each of the

pilots. 

Background 

Navy ERP: A Brief 

Description 

 6

These commands are Naval Air, Naval Sea, Space and Naval Warfare, and Naval Supply 

Systems. 

Page 7 GAO-08-896 DOD Business Systems Modernization 
Table 1: Description and Status of the Navy ERP Pilots 

Pilot Description and status

SIGMA Deployed at Naval Air Systems Command (NAVAIR)a to support and link

such business functions as program management, contracting, financial, 

and human resources management. It was retired in the first quarter of 

fiscal year 2008. 

CABRILLO Deployed at Space and Naval Warfare Systems Command (SPAWAR)b to 

support Navy Working Capital Fund financial management. It is to be 

retired in fiscal year 2010. 

NEMAIS Deployed at Naval Sea Systems Command (NAVSEA)c to support 

regional maintenance, including intermediate-level maintenanced

management and human resources. It is to be retired in fiscal year 2011. 

SMART Deployed at Naval Supply Systems Command (NAVSUP)eand NAVAIR to

support national and local supply management, intermediate-level 

maintenance management and to interface with aviation depots. It was 

retired in fiscal year 2005. 

Source: GAO analysis of DON data. 

a

NAVAIR is responsible for developing, delivering, and supporting aircraft and weapons used by

sailors and marines. 

b

SPAWAR is responsible for developing, delivering, and supporting specialized command and control 

technologies, business information technology, and space capabilities.

c

NAVSEA is responsible for acquiring and maintaining the Navy’s ships and submarines. 

d

Intermediate-level maintenance is for repair or maintenance of items that do not have to go to the 

depot level for major work but cannot be maintained or repaired at the organizational level. 

e

NAVSUP is responsible for supply, fuel, and transportation, as well as other logistics programs. 

According to DOD, Navy ERP is to address the Navy’s long-standing 

problems related to financial transparency and asset visibility. Specifically, 

the program is intended to standardize the Navy’s acquisition, financial, 

program management, maintenance, plant and wholesale supply, and

workforce management business processes across its dispersed 

organizational components. When the program is fully implemented, it is 

to support over 86,000 users. 

Navy ERP is being developed in a series of increments using the Systems 

Applications and Products (SAP) commercial software package, 

augmented as needed by customized software. SAP consists of multiple, 

integrated functional modules that perform a variety of business related 

tasks, such as finance and acquisition. The first increment, called 

Template 1, is currently the only funded portion of the program and 

consists of three releases: 1.0 Financial and Acquisition, 1.1 Wholesale and 

Retail Supply, and 1.2 Intermediate-Level Maintenance. Release 1.0 is the 

largest of the three releases in terms of the functional requirements being 

Page 8 GAO-08-896 DOD Business Systems Modernization 
addressed. Specifically, it is to provide about 56 percent of Template 1 

requirements. See table 2 for a description of these releases.

Table 2: Navy ERP Template 1 Releases 

Release Functionality

1.0 Financial and Acquisition General Fund and Navy Working Capital Fund 

finance applications, such as billing, budgeting, and 

cost planning. 

Acquisition applications, such as activity based 

costing, contract awards, and budget exhibits.

Workforce management applications, such as

personnel administration, and training and events 

management. 

1.1 Wholesale and Retail Supply Wholesale applications, such as supply and demand 

planning, order fulfillment, and supply forecasting. 

Retail supply applications, such as inventory

management, supply and demand processing, and 

warehouse management. 

1.2 Intermediate-Level 

Maintenance 

Maintenance applications, such as maintenance 

management, quality management, and calibration 

management. 

Source: GAO analysis of DON data. 

DON estimates the life cycle cost for the program’s first increment to be 

about $2.4 billion, including about $1 billion for acquisition, and $1.4 

billion for operations and maintenance. The life cycle cost of the entire 

program has not yet been determined because future increments have not

been defined. The program office reported that approximately $400 million 

was spent from fiscal year 2004 through fiscal year 2007 on the first 

increment. For fiscal year 2008, about $200 million is planned to be spent.

To manage the acquisition and deployment of Navy ERP, DON established 

a program management office within the Program Executive Office for 

Executive Information Systems. The program office manages the 

program’s scope and funding and is responsible for ensuring that the 

program meets its objectives. To accomplish this, the program office is 

responsible for key program management areas, such as architectural 

alignment, economic justification, earned value management, 

requirements management, and risk management. In addition, various 

DOD and DON organizations share program oversight and review 

activities. A listing of key entities and their roles and responsibilities is in 

table 3. 

Program Oversight and 

Management Roles and 

Responsibilities 

Page 9 GAO-08-896 DOD Business Systems Modernization 
Table 3: Organizations Responsible for Navy ERP Oversight and Management 

Entity Roles and responsibilities 

Under Secretary of Defense for Acquisition, 

Technology, and Logistics 

Serves as the milestone decision authority (MDA), which according to DOD, has 

overall responsibility for the program, to include approving the program to proceed 

through its acquisition cycle on the basis of, for example, the acquisition plan, an 

independently evaluated economic analysis, and the Acquisition Program Baseline. 

Assistant Secretary of the Navy, Research, 

Development, and Acquisition

Serves as DON’s oversight organization for the program, to include enforcement of 

Under Secretary of Defense for Acquisition, Technology, and Logistics policies and 

procedures. 

Department of the Navy, Program Executive 

Office for Executive Information Systems 

Oversees a portfolio of large-scale projects and programs designed to enable common 

business processes and provide standard capabilities, to include reviewing the 

acquisition plan, economic analysis, and the Acquisition Program Baseline prior to 

approval by the MDA. 

Department of the Navy Chief Information 

Officer (CIO) 

Supports the department’s planning, programming, budgeting, and execution 

processes by ensuring that the program has achievable and executable goals and 

conforms to financial management regulations, and DON, DOD, and federal IT policies 

in several areas (e.g., security, architecture, and investment management); works 

closely with the program office during milestone review assessments. 

Office of the Secretary of Defense, Office of 

the Director for Program Analysis and 

Evaluation 

Verifies and validates the reliability of cost and benefit estimates found in economic 

analyses and provides its results to the MDA. 

Naval Center for Cost Analyses Performs independent costs estimates. 

Defense Business Systems Management 

Committee (DBSMC) 

Serves as the highest ranking governance body for business systems modernization 

activities and approves investments costing more than $1 million, as, for example, 

being compliant with the BEA. 

Investment Review Board (IRB) Reviews business system investments and has responsibility for recommending 

certification for all business system investments costing more than $1 million that are 

asserted as compliant with the BEA. 

Business Transformation Agency (BTA) Coordinates business transformation efforts across DOD and supports the IRBs and 

DBSMC. 

Navy ERP Program Management Office Performs day-to-day program management and, as such, is the single point of 

accountability for managing the program’s objectives through development, 

deployment, and sustainment. 

Source: DOD. 

Overview of Navy ERP’s 

Status 

The first increment of Navy ERP is currently in the production and 

deployment phase of the defense acquisition system.7

 The system consists 

of five key program life cycle phases and three related milestone decision 

points. These five phases and related milestones, along with a summary of 

 7

The defense acquisition system is a framework-based approach that is intended to 

translate mission needs and requirements into stable, affordable, and well-managed 

acquisition programs. 

Page 10 GAO-08-896 DOD Business Systems Modernization 
key program activities completed during or planned for each phase, are as 

follows: 

1. Concept Refinement: The purpose of this phase is to refine the initial 

system solution (concept) and create a strategy for acquiring the 

solution. This phase began in July 2003, at which time DON began to 

converge the four pilot programs into Navy ERP and developed its first 

cost estimate in September 2003. This phase of the program was 

combined with the next phase, thus creating a combined Milestone A/B 

decision point. 

2. Technology Development: The purpose of this phase is to determine 

the appropriate set of technologies to be integrated into the investment 

solution by iteratively assessing the viability of the various 

technologies while simultaneously refining user requirements. During 

the combined Concept Refinement and Technology Development 

phase, the program office prepared a concept of operations and 

operational requirements document; performed an analysis of 

alternatives, business case analysis, and economic analysis; and 

established its first Acquisition Program Baseline. It also selected SAP 

as the commercial off-the-shelf ERP software. The combined phase 

was completed in August 2004, when the MDA approved Milestone A/B 

to allow the program to move to the next phase.

3. System Development and Demonstration: The purpose of this phase is 

to develop a system and demonstrate through developer testing that 

the system can function in its target environment. This phase was 

completed in September 2007, when Release 1.0 passed development 

testing and its deployment to NAVAIR began. This was 17 months later 

than the program’s original schedule set in August 2004 but on time 

according to the revised schedule set in December 2006. 

In September 2004, the program office awarded a $176 million system 

integration contract to BearingPoint for full system design, 

development, and delivery using SAP’s off-the-shelf product and 

related customized software. In January 2006, the program office (1) 

reduced the contractor’s scope of work from development and 

integration of the first increment to only development of the first 

release and (2) assumed responsibility and accountability for overall 

system integration. According to the program office, reasons for this 

change included the need to change the development plan to reflect 

improvements in the latest SAP product released and the lack of 

authority by the contractor to adjudicate and reconcile differences 

among the various Navy user organizations (i.e., Navy commands). 

Page 11 GAO-08-896 DOD Business Systems Modernization 
In December 2006, the program office revised its Acquisition Program 

Baseline to reflect an increase of about $461 million in the life cycle 

cost estimate due, in part, to restructuring the program (e.g., changing

the order of the releases, changing the role of system integrator from 

contractor to the program office) and resolving problems related to, 

among other things, converting data from legacy systems to run on 

Navy ERP and establishing interfaces between legacy systems and 

Navy ERP. In addition, the program office awarded a $151 million 

contract for Release 1.1 and 1.2 configuration and development to IBM 

in June 2007. In September 2007, prior to entering the next phase, the 

program revised its Acquisition Program Baseline again to reflect a $9 

million decrease in the life cycle cost estimate and a 5-month increase 

in its program schedule. Soon after, the MDA approved Milestone C to 

move to the next phase.

4. Production and Deployment: The purpose of this phase is to achieve 

an operational capability that satisfies the mission needs, as verified 

through independent operational test and evaluation, and to 

implement the system at all applicable locations. This phase began in

September 2007, focusing first on achieving initial operational 

capability (IOC) of Release 1.0 at NAVAIR by May 2008. This date is 22 

months later than the baseline established for Milestone A/B in August 

2004, and 4 months later than the new baseline established in 

September 2007. According to program documentation, these delays 

were due, in part, to challenges experienced at NAVAIR in converting 

data from legacy systems to run on the new system and implementing 

new business procedures associated with the system. 

In light of the delays at NAVAIR in achieving IOC, the deployment 

schedules for the other commands were also revised. Specifically, 

Release 1.0 is still to be deployed at NAVSUP on October 2008, but 

Release 1.0 deployment at SPAWAR is now scheduled 18 months later 

than planned (October 2009), and deployment at NAVSEA general fund 

and Navy Working Capital Fund is now scheduled to be 12 months 

later than planned (October 2010 and 2011, respectively). Because of 

the Release 1.0 delays, Release 1.1 is now planned for deployment at 

NAVSUP 7 months later than planned (February 2010). Release 1.2 is 

still scheduled to be released at Regional Maintenance Centers in 

October 2010. 

Page 12 GAO-08-896 DOD Business Systems Modernization 
The program office is currently in the process of again re-baselining 

the program, and DON plans to address any cost overruns through 

reprogramming of fiscal year 2008 DON funds.8

 It estimates that this 

phase will be completed with full operational capability (FOC)9

 by 

August 2013 (26 months later than the baseline established in 2004, and 

5 months later than the re-baseline established in September 2007). 

5. Operations and Support: The purpose of this phase is to operationally 

sustain the system in the most cost-effective manner over its life cycle. 

In this phase, the program plans to provide centralized support to its 

users across all system commands. Each deployment site is expected 

to perform complementary support functions, such as data 

maintenance. 

Overall, Increment 1 was originally planned to reach FOC in fiscal year 

2011, and its estimated life cycle cost was about $1.87 billion.10 The 

estimate was later baselined11 in August 2004 at about $2.0 billion.12 In 

December 2006 and again in September 2007, the program was rebaselined. FOC is now planned for fiscal year 2013, and the estimated life 

cycle cost is about $2.4 billion (31 percent increase over the original 

 8

Congressional defense committees have established reprogramming guidelines, including 

setting dollar thresholds that direct DOD to seek the prior approval of the committees 

before executing the reprogramming of appropriated funds. In accordance with these 

guidelines, DOD’s financial management regulation provides that DOD does not need

congressional committee approval when the amount to be reprogrammed falls below 

certain thresholds (referred to as a "below-threshold reprogramming"). As relevant here, as

of fiscal year 2005, the threshold is $10 million or 20 percent of the program's 

appropriation, whichever is less. To fund shortfalls in Navy ERP, DON plans to reprogram 

amounts below this threshold from other programs. 

9

FOC means that the system has been successfully deployed in all intended locations.

10This 2003 estimate, which was prepared to assist in budget development and support the

Milestone A/B approval, was for development, deployment, and sustainment costs in fiscal

years 2003 through 2021. 

11According to DOD’s acquisition guidebook, an Acquisition Program Baseline is a program

manager’s estimated cost, schedule, and performance goals. Goals consist of objective 

values, which represent what the user desires and expects, and threshold values, which

represent acceptable limits. When the program manager determines a current cost,

schedule or performance threshold value will not be achieved, the MDA must be notified,

and a new baseline developed, reviewed by decision makers, and, if the program is to 

continue, approved by the MDA. 

12According to the August 2004 Acquisition Program Baseline, this estimate is for 

acquisition, operations, and support for fiscal years 2004 through 2021. 

Page 13 GAO-08-896 DOD Business Systems Modernization 
estimate).13 Key activities for each phase are depicted in figure 1, changes 

in the deployment schedule are depicted in figure 2, and cost estimates are 

depicted in figure 3. 

 13According to the September 2007 Acquisition Program Baseline, this estimate is for 

acquisition, operations, and support for fiscal years 2004 through 2023. 

Page 14 GAO-08-896 DOD Business Systems Modernization 
Figure 1: Navy ERP Time Line

Activity prior to the 2004 plan

2004 plan

2007 plan

Source: GAO analysis of DON data. 

Fiscal year

Concept refinement and 

technology development

Phase

System development 

and demonstration

Production and 

deployment

Program 

established

Template 1 contract awarded

Template 1 contract rescoped to Release 1.0

Re-baselined

Re-baselined

Releases 1.1 and 1.2 contract awarded

2003 2004 2005 2006 2008 2009 2010 2007 201 2011 2012 3

FOC

IOC FOC

IOC

Page 15 GAO-08-896 DOD Business Systems Modernization 
Figure 2: Navy ERP Milestones for Beginning Deployment

2007 plan

2008 plan

NAVSEA for General Fund

NAVSEA for Working Capital Fund

1.0 Financial and Acquisition 

1.1 Wholesale and Retail Supply 

1.2 Intermediate-Level Maintenance 

NAVSUP

Regional Maintenance Centers

NAVAIR

NAVSUP

SPAWAR

Source: GAO analysis of DON data. 

Fiscal year

Release

2007 2012 2008 2009 2010 2011

Figure 3: Navy ERP Life Cycle Cost Estimates in Fiscal Years 2003, 2004, and 2007 

0.0

0.5

1.0

1.5

2.0

2.5

3.0

2003 2004 2007

Dollars in billions

Fiscal year

Source: GAO analysis of DON data.

$1.87

$1.99

$2.44

Page 16 GAO-08-896 DOD Business Systems Modernization 
IT acquisition management controls are tried and proven methods, 

processes, techniques, and activities that organizations define and use to

minimize program risks and maximize the chances of a program’s success. 

Using these controls can result in better outcomes, including cost savings,

improved service and product quality, and a better return on investment. 

For example, two software engineering analyses of nearly 200 systems 

acquisition projects indicate that teams using systems acquisition controls

that reflected best practices produced cost savings of at least 11 percent 

over similar projects conducted by teams that did not employ the kind of 

rigor and discipline embedded in these practices.14 In addition, our 

research shows that these controls are a significant factor in successful 

acquisition outcomes, including increasing the likelihood that programs 

and projects will be executed within cost and schedule estimates.15

Use of IT Acquisition 

Management Controls 

Maximizes Chances for 

Success 

We and others have identified and promoted the use of a number of IT 

acquisition management controls associated with acquiring IT systems.16

See table 4 for a description of several of these activities. 

 14Donald E. Harter, Mayuram S. Krishnan, and Sandra A. Slaughter, “Effects of Process

Maturity on Quality, Cycle Time, and Effort in Software Product Development,” 

Management Science, 46, no. 4 (2000); and Bradford K. Clark, “Quantifying the Effects of

Process Improvement on Effort,” IEEE Software (November/December 2000). 

15GAO, Information Technology: DOD’s Acquisition Policies and Guidance Need to 

Incorporate Additional Best Practices and Controls, GAO-04-722 (Washington, D.C.: 

July 30, 2004). 

16GAO-04-722. 

Page 17 GAO-08-896 DOD Business Systems Modernization 
Table 4: Description of Business System IT Acquisition Management Controls 

Business system acquisition practice Description

Architectural alignment

To ensure that the acquisition is consistent 

with the organization’s enterprise 

architecture. 

Architectural alignment is the process for analyzing and verifying that the proposed 

architecture of the system being acquired is consistent with the enterprise architecture

for the organization acquiring the system. Such alignment is needed to ensure that 

acquired systems can interoperate and are not unnecessarily duplicative of one another.

Economic justification

To ensure that system investments are 

economically justified. 

Economic justification is the process for ensuring that acquisition decisions are based 

on reliable analyses of the proposed investment’s likely costs versus benefits over its 

useful life, as well as an analysis of the risks associated with actually realizing the 

acquisition’s forecasted benefits for its estimated costs. Economic justification is not a 

one-time event but rather is performed throughout an acquisition’s life cycle in order to 

permit informed investment decision making.

Earned value management

To ensure that actual progress against cost 

and schedule expectations is being 

measured. 

EVM is a tool that integrates the technical, cost, and schedule parameters of a contract 

and measures progress against them. During the planning phase, an integrated 

program baseline is developed by time phasing budget resources for defined work. As 

work is performed and measured against the baseline, the corresponding budget value 

is “earned.” Using this earned value metric, cost and schedule variances, as well as cost 

and time to complete estimates, can be determined and analyzed. 

Requirements management 

To ensure that requirements are traceable, 

verifiable, and controlled. 

Requirements management is the process for ensuring that the requirements are 

traceable, verifiable, and controlled. Traceability refers to the ability to follow a 

requirement from origin to implementation and is critical to understanding the 

interconnections and dependencies among the individual requirements and the impact 

when a requirement is changed. Requirements management begins when the 

solicitation’s requirements are documented and ends when system responsibility is 

transferred to the support organization. 

Risk management

To ensure that risks are identified and 

systematically mitigated. 

Risk management is the process for identifying potential acquisition problems and taking

appropriate steps to avoid their becoming actual problems. Risk management occurs 

early and continuously in the acquisition life cycle. 

Source: GAO.

Page 18 GAO-08-896 DOD Business Systems Modernization 
We have previously reported17 that DOD has not effectively managed a 

number of business system investments. Among other things, our reviews 

of individual system investments have identified weaknesses in such 

things as architectural alignment and informed investment decision 

making, which are also the focus areas of the Fiscal Year 2005 Defense 

Authorization Act business system provisions.18 Our reviews have also 

identified weaknesses in other system acquisition and investment 

management areas—such as EVM, economic justification, requirements 

management, and risk management.

Prior GAO Reviews Have 

Identified IT Acquisition 

Management Control 

Weaknesses on DOD 

Business System 

Investments 

In July 2007, we reported that the Army’s approach for investing about $5 

billion over the next several years in its General Fund Enterprise Business 

System, Global Combat Support System-Army Field/Tactical,

19 and Logistics 

Modernization Program did not include alignment with the Army enterprise 

architecture or use of a portfolio-based business system investment review 

process.20 Moreover, we reported that the Army did not have reliable analyses, 

such as economic analyses, to support its management of these programs. We 

concluded that, until the Army adopts a business system investment 

management approach that provides for reviewing groups of systems and 

making enterprise decisions on how these groups will collectively interoperate

to provide a desired capability, it runs the risk of investing significant resources 

in business systems that do not provide the desired functionality and efficiency. 

Accordingly, we made recommendations aimed at improving the department’s

efforts to achieve total asset visibility and enhancing its efforts to improve its 

control and accountability over business system investments. The department 

agreed with our recommendations. 

 17See, for example, GAO, DOD Business Transformation: Lack of an Integrated Strategy 

Puts the Army’s Asset Visibility System Investments at Risk, GAO-07-860 (Washington,

D.C.: July 27, 2007); GAO, Information Technology: DOD Needs to Ensure That Navy 

Marine Corps Intranet Program Is Meeting Goals and Satisfying Customers, GAO-07-51 

(Washington, D.C.: Dec. 8, 2006); GAO, Defense Travel System: Reported Savings 

Questionable and Implementation Challenges Remain, GAO-06-980 (Washington, D.C.: 

Sept. 26, 2006); GAO, DOD Systems Modernization: Uncertain Joint Use and Marginal

Expected Value of Military Asset Deployment System Warrant Reassessment of Planned

Investment, GAO-06-171 (Washington, D.C.: Dec. 15, 2005); and GAO, DOD Systems

Modernization: Planned Investment in the Navy Tactical Command Support System 

Needs to Be Reassessed, GAO-06-215 (Washington, D.C.: Dec. 5, 2005). 

18Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005, Pub. L. 

No. 108-375, § 332 (2004) (codified at 10 U.S.C. §§ 186 and 2222). 

19Field/tactical refers to Army units that are deployable to locations around the world, such

as Iraq or Afghanistan. 

20GAO-07-860. 

Page 19 GAO-08-896 DOD Business Systems Modernization 
We also reported that DON had not, among other things, economically 

justified its ongoing and planned investment in the Naval Tactical 

Command Support System (NTCSS)

21 and had not invested in NTCSS 

within the context of a well-defined DOD or DON enterprise architecture. 

In addition, we reported that DON had not effectively performed key 

measurement, reporting, budgeting, and oversight activities and had not

adequately conducted requirements management and testing activities. We

concluded that, without this information, DON could not determine 

whether NTCSS, as defined, and as being developed, is the right solution 

to meet its strategic business and technological needs. Accordingly, we 

recommended that the department develop the analytical basis to 

determine if continued investment in NTCSS represents prudent use of 

limited resources and to strengthen management of the program, 

conditional upon a decision to proceed with further investment in the 

program. The department largely agreed with our recommendations. 

In addition, we reported that the Army had not defined and developed its

Transportation Coordinators’ Automated Information for Movements System 

II (TC-AIMS II)—a joint services system with the goal of helping to manage 

the movement of forces and equipment within the United States and abroad—

in the context of a DOD enterprise architecture.

22 In addition, we reported 

that the Army had not economically justified the program on the basis of

reliable estimates of life cycle costs and benefits and had not effectively 

implemented risk management. As a result, we concluded that the Army did 

not know if its investment in TC-AIMS II, as planned, was warranted or 

represented a prudent use of limited DOD resources. Accordingly, we 

recommended that the department, among other things, develop the 

analytical basis needed to determine if continued investment in TC-AIMS II

represents prudent use of limited defense resources. In response, the 

department agreed with our recommendations and has since reduced the 

program’s scope by canceling future investments. 

Furthermore, in 2005, we reported that DON had invested approximately 

$1 billion in the four previously cited ERP pilots without marked 

improvement in its day-to-day operations.23 More specifically, we reported 

 21GAO-06-215. 

22GAO-06-171. 

23GAO, DOD Business Systems Modernization: Navy ERP Adherence to Best Business

Practices Critical to Avoid Past Failures, GAO-05-858 (Washington, D.C.: Sept. 29, 2005). 

Page 20 GAO-08-896 DOD Business Systems Modernization 
that the program office had not implemented an EVM system. We also 

identified significant challenges and risks as the project moved forward, 

such as developing and implementing system interfaces, converting data

from legacy systems into the ERP system, meeting its estimated 

completion date of 2011 at an estimated cost of $800 million, and achieving

alignment with DOD’s BEA. To address these areas, we made 

recommendations that DOD improve oversight of Navy ERP, including 

developing quantitative metrics to evaluate the program. DOD generally

agreed with our recommendations. 

DOD IT-related acquisition policies and guidance, along with other 

relevant guidance, provide an acquisition management control framework 

within which to manage business system programs like Navy ERP. 

Effective implementation of this framework can minimize program risks 

and better ensure that system investments are defined in a way to 

optimally support mission operations and performance, as well as deliver

promised system capabilities and benefits on time and within budget. To 

varying degrees of effectiveness, Navy ERP has been managed in 

accordance with aspects of this framework. However, implementation of 

key management controls has not been effective. Specifically,

Implementation of 

Key DOD and Related 

IT Acquisition 

Management Controls 

on Navy ERP Varies 

• compliance with DOD’s federated BEA has not been sufficiently 

demonstrated; 

• investment in the program has been economically justified on the basis of

expected life cycle benefits that will likely exceed estimated life cycle 

costs, although some estimating limitations nevertheless exist; 

• earned value management has not been effectively implemented; 

• an important requirements management activity has been effectively 

implemented; and 

• a risk management process has been defined, but not effectively 

implemented for all risks. 

The reasons that program management and oversight officials cited for 

why these key practices have not been sufficiently executed range from 

limitations in the applicable DOD guidance and tools to the complexity 

and challenges of managing and implementing a program of this size. Each 

of these reasons is described in the applicable sections of this report. By 

not effectively implementing all the above key IT acquisition management 

Page 21 GAO-08-896 DOD Business Systems Modernization 
functions, the program is at increased risk of (1) not being defined in a 

way that best meets corporate mission needs and enhances performance 

and (2) adding to the more than 2 years in program schedule delays and

about $570 million in program cost increases experienced to date. 

DOD and other guidance,24 recognize the importance of investing in 

business systems within the context of an enterprise architecture.25

Moreover, the Fiscal Year 2005 Defense Authorization Act requires that 

defense business systems be compliant with DOD’s federated BEA.26 Our 

research and experience in reviewing federal agencies show that not 

making investments within the context of a well-defined enterprise 

architecture often results in systems that are duplicative, are not well 

Navy ERP Compliance 

with DOD’s Federated BEA 

Has Not Been Sufficiently 

Demonstrated 

 24Department of Defense Architecture Framework, Version 1.0, Volume 1 (February 2004);

GAO, Information Technology: A Framework for Assessing and Improving Enterprise 

Architecture Management (Version 1.1), GAO-03-584G (Washington, D.C.: Apr. 1, 2003); 

Chief Information Officer Council, A Practical Guide to Federal Enterprise Architecture, 

Version 1.0 (February 2001); and Institute of Electrical and Electronics Engineers (IEEE), 

Standard for Recommended Practice for Architectural Description of Software-Intensive

Systems 1471-2000 (Sept. 21, 2000). 

25A well-defined enterprise architecture provides a clear and comprehensive picture of an

entity, whether it is an organization (e.g., a federal department) or a functional or mission 

area that cuts across more than one organization (e.g., personnel management). This

picture consists of snapshots of both the enterprise’s current or “as is” environment and its

target or “to be” environment, as well as a capital investment road map for transitioning 

from the current to the target environment. These snapshots consist of integrated “views,” 

which are one or more architecture products that describe, for example, the enterprise’s

business processes and rules; information needs and flows among functions, supporting 

systems, services, and applications; and data and technical standards and structures. 

26DOD has adopted a federated approach for developing its business mission area

enterprise architecture, which includes the corporate BEA representing the thin layer of 

DOD-wide corporate architectural policies, capabilities, rules, and standards; component 

architectures (e.g., Navy enterprise architecture); and program architectures (e.g., Navy 

ERP architecture). 

Page 22 GAO-08-896 DOD Business Systems Modernization 
integrated, are unnecessarily costly to interface and maintain, and do not 

optimally support mission outcomes.27

To its credit, the program office has followed DOD’s BEA compliance 

guidance.

28 However, this guidance does not adequately provide for 

addressing all relevant aspects of BEA compliance. Moreover, DON’s 

enterprise architecture, which is a major component of DOD’s federated 

BEA, as well as key aspects of DOD’s corporate BEA, have yet to be 

sufficiently defined to permit thorough compliance determinations. In 

addition, current policies and guidance do not require DON investments to 

comply with its enterprise architecture. This means that the department 

does not have a sufficient basis for knowing if Navy ERP has been defined 

to minimize overlap with and duplication of other programs’ functionality 

and maximize interoperability among related programs. Each of these 

architecture alignment limitations is discussed here: 

• The program’s compliance assessments did not include all relevant

architecture products. In particular, the program did not assess 

compliance with the BEA’s technical standards profile, which outlines, for 

example, the standards governing how systems physically communicate 

with other systems and how they secure data from unauthorized access.

This is particularly important because systems like Navy ERP need to 

share information with other systems and, for these systems to accomplish 

this effectively and efficiently, they need to employ common standards. A 

case in point is the relationship between Navy ERP and the Global Combat 

 27See, for example, GAO, Information Technology: FBI Is Taking Steps to Develop an

Enterprise Architecture, but much Remains to Be Accomplished, GAO-05-363 

(Washington, D.C.: Sept. 9, 2005); GAO, Homeland Security: Efforts Under Way to Develop 

Enterprise Architecture, but Much Work Remains, GAO-04-777 (Washington, D.C.: Aug. 6,

2004); GAO, Information Technology: Architecture Needed to Guide NASA’s Financial

Management Modernization, GAO-04-43 (Washington, D.C.: Nov. 21, 2003); GAO, DOD 

Business Systems Modernization: Important Progress Made to Develop Business 

Enterprise Architecture, but Much Work Remains, GAO-03-1018 (Washington, D.C.: 

Sept. 19, 2003); GAO, Information Technology: DLA Should Strengthen Business Systems

Modernization Architecture and Investment Activities, GAO-01-631 (Washington, D.C.: 

June 29, 2001); and GAO, Information Technology: INS Needs to Better Manage the 

Development of Its Enterprise Architecture, GAO/AIMD-00-212 (Washington, D.C.:

Aug. 1, 2000). 

28DOD, Business Enterprise Architecture Compliance Guidance (Apr. 10, 2006). 

Page 23 GAO-08-896 DOD Business Systems Modernization 
Support System—Marine Corps (GCSS-MC) program.29 Specifically, Navy 

ERP has identified 25 technical standards that are not in the BEA technical 

standards profile, and GCSS-MC has identified 13 technical standards that

are not in the profile. Among these non-BEA standards are program-unique 

information sharing protocols, which could limit information sharing 

between Navy ERP and GCSS-MC, and with other systems. 

In addition, the program office did not assess compliance with the BEA 

products that describe system-level characteristics. This is important 

because doing so would create a body of information about programs that 

could be used to identify common system components and services that 

could potentially be shared by the programs, thus avoiding wasteful 

duplication. For example, our analysis of Navy ERP program 

documentation shows that it contains system functions related to 

receiving goods, taking physical inventories, and returning goods, which 

are also system functions cited by the GCSS-MC program. However, 

because compliance with the BEA system products was not assessed, the 

extent to which these functions are potentially duplicative was not 

considered.

Furthermore, the program office did not assess compliance with BEA 

system products that describe data exchanges among systems. As we 

previously reported, establishing and using standard system interfaces is a 

critical enabler to sharing data.30 For example, Navy ERP program 

documentation indicates that it is to exchange inventory order and status 

data with other systems. System interfaces are important for 

understanding how information is to be exchanged between systems. 

However, since the program was not assessed for compliance with these 

products, it does not have the basis for understanding how its approach to

exchanging information differs from that of other systems that it is to 

interface with. Compliance against each of these BEA products was not 

assessed because DOD’s compliance guidance does not provide for doing 

 29Initiated in 2003, GCSS-MC is to modernize the Marine Corps logistics systems and 

thereby provide the decision makers with timely and complete logistics information to 

support the warfighter. Moreover, according to program officials, both GCSS-MC and Navy 

ERP are under the leadership of Navy’s Program Executive Office for Enterprise

Information Systems, which is responsible for developing, acquiring, and deploying 

seamless enterprise-wide information technology systems. 

30GAO, DOD Business Systems Modernization: Progress in Establishing Corporate 

Management Controls Needs to be Replicated Within Military Departments, GAO-08-705 

(Washington, D.C.: May 15, 2008). 

Page 24 GAO-08-896 DOD Business Systems Modernization 
so and, according to BTA officials, because some BEA system products 

are not sufficiently defined. According to these officials, BTA plans to 

continue to define these products as the BEA evolves. 

• The compliance assessment was not used to identify potential areas of 

duplication across programs, which DOD has stated is an explicit goal of 

its federated BEA and associated investment review and decision-making 

processes. More specifically, even though the compliance guidance 

provides for assessing programs’ compliance with the BEA product that 

defines DOD operational activities, and Navy ERP was assessed for 

compliance with this product, the results were not used to identify 

programs that support the same operational activities and related business 

processes. Given that the federated BEA is intended to identify and avoid 

not only duplications within DOD components, but also between 

components, it is important that such commonality be addressed. For 

example, BEA compliance assessments for Navy ERP and GCSS-MC, as 

well as two Air Force programs (Defense Enterprise Accounting and 

Management System—Air Force and the Air Force Expeditionary Combat 

Support System) show that each program supports at least six of the same 

BEA operational activities (e.g., conducting physical inventory, delivering 

property and services) and three of these four programs support at least 

18 additional operational activities (e.g., performing budgeting, managing 

receipt and acceptance). However, since the potential overlap among 

these and other programs was not assessed, these programs may be 

investing in duplicative functionality. Reasons for this were that the 

compliance guidance does not provide for such analyses to be conducted 

and programs have not been granted access rights to use this functionality 

in the compliance tool. 

• The program’s compliance assessment did not address compliance against 

DON’s enterprise architecture, which is one of the biggest members of the 

federated BEA. This is particularly important given that DOD’s approach 

to fully satisfying the architecture requirements of the Fiscal Year 2005 

Defense Authorization Act is to develop and use a federated architecture 

in which component architectures are to provide the additional details 

needed to supplement the thin layer of corporate policies, rules, and 

Page 25 GAO-08-896 DOD Business Systems Modernization 
standards included in the corporate BEA.31 As we recently reported,32

DON’s enterprise architecture is not mature because, among other things, 

it is missing a sufficient description of its current and future environments 

in terms of business and information/data. However, certain aspects of an 

architecture nevertheless exist and, according to DON CIO officials, these 

aspects will be leveraged in its efforts to develop a complete enterprise 

architecture. For example, the FORCEnet architecture is intended to 

document Navy’s technical infrastructure. Therefore, opportunities exist 

for DON to assess its programs in relation to these architecture products, 

and to understand where its programs are exposed to risks because 

products do not exist, are not mature, or at odds with other Navy 

programs. According to DOD officials, compliance with the DON 

architecture was not assessed because DOD compliance policy is limited 

to compliance with the corporate BEA, and a number of aspects of the 

DON enterprise architecture have yet to be sufficiently developed. 

• The program’s compliance assessment was not validated by DON or DOD

investment oversight and decision-making authorities. More specifically,

neither the DOD IRBs nor the DBSMC, nor the BTA in supporting both of 

these investment oversight and decision-making authorities, reviewed the 

program’s assessments. According to BTA officials, under DOD’s tiered 

approach to investment accountability, these entities are not responsible 

for validating programs’ compliance assessments. Rather, this is a 

component responsibility, and thus they rely on the military departments 

and defense agencies to validate the assessments. 

However, DON Office of the CIO, which is responsible for precertifying 

investments as compliant before they are reviewed by the IRB, did not 

validate any of the program’s compliance assessments. According to Office 

of the CIO officials, they rely on Functional Area Managers to validate a 

program’s compliance assessments. However, no DON policy or guidance 

exists that describes how the Functional Area Managers should conduct 

such validations. CIO officials stated that this is because these authorities 

do not have the resources that they need to validate the assessments, and 

 31As we recently reported, while the corporate BEA includes corporate policies, 

capabilities, rules, and standards, it is still evolving and will continue to add additional 

details. See GAO-08-705. 

32GAO, DOD Business Systems Modernization: Military Departments Need to Strengthen

Management of Enterprise Architecture Programs, GAO-08-519 (Washington, D.C.:

May 12, 2008). 

Page 26 GAO-08-896 DOD Business Systems Modernization 
because a number of aspects of the DON architecture are not yet 

sufficiently developed. 

Validation of program assessments is further complicated by the absence 

of information captured in the assessment tool about what program 

documentation or other source materials were used by the program office 

in making its compliance determinations. Specifically, the tool is only 

configured, and thus was only used, to capture the results of a program’s 

comparison of program architecture products to BEA products. Thus, it 

was not used to capture the system products used in making these 

determinations. 

The limitations in existing BEA compliance-related policy and guidance, 

the supporting compliance assessment tool, and the federated BEA, put 

programs like Navy ERP at increased risk of being defined and

implemented in a way that does not sufficiently ensure interoperability 

and avoid duplication and overlap. We recently completed a review 

examining multiple programs’ compliance with the federated BEA, 

including Navy ERP, for the Senate Armed Services Committee, 

Subcommittee on Readiness and Management Support. We addressed the 

architectural compliance guidance, tool, and validation limitations as part 

of this review.

33

The investment in Navy ERP has been economically justified on the basis

of expected life cycle benefits that far exceed estimated life cycle costs. 

According to the program’s benefit/cost analysis, Navy ERP will produce 

about $8.6 billion in estimated benefits for an estimated cost of about $2.4

billion over its 20-year life cycle. While these benefit estimates were not 

subject to any analysis of how uncertainty in assumptions and data could 

impact the estimates, as called for by relevant guidance, our examination 

of key uncertainty variables, such as the timing of legacy systems’ 

retirement, showed that the savings impact would be relatively minor. 

However, the reliability of the cost estimate is limited because it was 

derived using several, but not all, key estimating practices. For example, 

the estimate was not grounded in a historical record of comparable data

from similar programs and was not based on a reliable schedule baseline, 

Investment in Navy ERP 

Has Been Economically 

Justified, but Important 

Estimating Practices Were 

Not Implemented 

 33GAO, DOD Business Systems Modernization: Key Navy Programs’ Compliance with

DOD’s Federated Business Enterprise Architecture Needs to Be Adequately 

Demonstrated, GAO-08-972 (Washington, D.C.: Aug. 7, 2008). 

Page 27 GAO-08-896 DOD Business Systems Modernization 
which are both necessary to having a cost estimate that can be considered

credible and accurate. These practices were not employed for various 

reasons, including DOD’s lack of historical data from similar programs and 

the lack of an integrated master schedule for the program that includes all

releases. 

Notwithstanding the fact that these limitations could materially increase 

the $2.4 billion cost estimate, it is nevertheless unlikely that these factors 

would increase the estimate to a level approaching the program’s benefit 

expectations. Therefore, we have no reason to believe that Navy ERP will 

not produce a positive return on investment. 

Forecasting expected benefits over the life of a program is a key aspect of

economically justifying an investment. The Office of Management and 

Budget (OMB) guidance34 advocates economically justifying investments 

on the basis of expected benefits, costs, and risks. Since estimates of 

benefits can be uncertain because of the imprecision in both the 

underlying data and modeling assumptions used, the guidance also 

provides for analyzing and reporting the effects of this uncertainty. By 

doing this, informed investment decision making can occur through the 

life of the program, and a baseline can be established against which to 

compare the accrual of actual benefits from deployed system capabilities. 

The most recent economic analysis, dated August 2007, includes 

monetized benefit estimates for fiscal years 2004–2023, in three key 

areas—about $2.7 billion in legacy system cost savings, $3.3 billion in cost

savings from inventory reductions, and $2.7 billion in cost savings from 

labor productivity improvements. Collectively, these benefits total about

$8.6 billion.35

The program office calculated expected benefits in terms of cost savings, 

which is consistent with established practices and guidance. For example, 

the program is to result in the retirement of 138 legacy systems (including 

the 4 pilot systems) between fiscal years 2005 and 2015, and the yearly 

maintenance costs for a single system are expected to be as high as about 

Benefit Estimates Are 

Sufficiently Reliable Despite 

Absence of Uncertainty 

Analysis 

 34Office of Management and Budget, Circular No. A-94: Guidelines and Discount Rates for

Benefit-Cost Analysis of Federal Programs (Oct. 29, 1992). 

35The benefits estimates for the areas are rounded; therefore, they add to more than 

$8.6 billion.

Page 28 GAO-08-896 DOD Business Systems Modernization 
$39 million.36 According to relevant guidance, cost saving estimates should 

also be analyzed in terms of how uncertainty in assumptions and data

could impact them. However, the program office did not perform such 

uncertainty analysis. According to program officials, uncertainty analysis 

is not warranted because they have taken and continue to take steps to 

validate the assumptions and the data, such as using the latest budget data 

associated with the legacy systems, and monitoring changes to the 

systems’ retirement dates. While these steps are positive, they do not 

eliminate the need for uncertainty analysis. Accordingly, we assessed key 

uncertainty variables, such as the timing of the legacy systems’ retirement, 

and found that the retirement dates of some of these systems have 

changed since the estimate was prepared, due to, among other things, 

schedule delays in the program. While the inherent uncertainty in these 

dates would reduce expected savings (e.g., only $11 million based on the 

134 legacy systems that we examined),37 the reduction would be small 

relative to a total benefit estimate of $8.6 billion. 

A reliable cost estimate is a key variable in calculating return on 

investment, and it provides the basis for informed investment decision 

making, realistic budget formulation and program resourcing, meaningful 

progress measurement, proactive course correction, and accountability for 

results. According to OMB,38 programs must maintain current and welldocumented cost estimates, and these estimates must encompass the full 

life cycle of the program. OMB states that generating reliable cost 

estimates is a critical function necessary to support OMB’s capital 

programming process. Without reliable estimates, programs are at 

increased risk of experiencing cost overruns, missed deadlines, and 

performance shortfalls. 

Our research has identified a number of practices that are the basis of 

effective program cost estimating. We have issued guidance that 

The Cost Estimate Was Not

Reliably Derived 

 36For example, the Uniform Accounting Data Process System-Inventory Control Points has

yearly maintenance costs of $39.3 million. 

37These reductions in expected savings represent the costs of maintaining the legacy 

systems during the period in which the systems’ retirement dates have been delayed.

38OMB, Circular No. A-11, Preparation, Submission, and Execution of the Budget 

(Washington, D.C.: Executive Office of the President, June 2006); Circular No. A-130 

Revised, Management of Federal Information Resources (Washington, D.C.: Executive

Office of the President, Nov. 28, 2000); and Capital Programming Guide: Supplement to 

Circular A-11, Part 7, Preparation, Submission, and Execution of the Budget 

(Washington, D.C.: Executive Office of the President, June 2006). 

Page 29 GAO-08-896 DOD Business Systems Modernization 
associates these practices with four characteristics of a reliable cost 

estimate.39 Specifically, these four characteristics are as follows: 

• Comprehensive: The cost estimates should include both government and 

contractor costs over the program’s full life cycle, from the inception of 

the program through design, development, deployment, and operation and 

maintenance to retirement. They should also provide an appropriate level 

of detail to ensure that cost elements are neither omitted nor double 

counted and include documentation of all cost-influencing ground rules 

and assumptions. 

• Well-documented: The cost estimates should have clearly defined 

purposes and be supported by documented descriptions of key program or 

system characteristics (e.g., relationships with other systems, performance 

parameters). Additionally, they should capture in writing such things as 

the source data used and their significance, the calculations performed 

and their results, and the rationale for choosing a particular estimating 

method or reference. Moreover, this information should be captured in 

such a way that the data used to derive the estimate can be traced back to, 

and verified against, their sources. The final cost estimate should be 

reviewed and accepted by management on the basis of confidence in the 

estimating process and the estimate produced by the process.

• Accurate: The cost estimates should provide for results that are unbiased 

and should not be overly conservative or optimistic (i.e., they should 

represent most likely costs). In addition, the estimates should be updated 

regularly to reflect material changes in the program, and steps should be 

taken to minimize mathematical mistakes and their significance. Among 

other things, the estimate should be grounded in a historical record of cost

estimating and actual experiences on comparable programs. 

• Credible: The cost estimates should discuss any limitations in the analysis 

performed due to uncertainty or biases surrounding data or assumptions. 

Further, the estimates’ derivation should provide for varying any major 

assumptions and recalculating outcomes based on sensitivity analyses, 

and their associated risks and inherent uncertainty should be disclosed. 

Also, the estimates should be verified based on cross-checks using other 

estimating methods and by comparing the results with independent cost

estimates. 

 39GAO, Cost Assessment Guide: Best Practices for Estimating and Managing Program

Costs, Exposure Draft, GAO-07-1134SP (Washington, D.C.: July 2, 2007). 

Page 30 GAO-08-896 DOD Business Systems Modernization 
The $2.4 billion life cycle cost estimate for Navy ERP reflects many of the 

practices associated with a reliable cost estimate, including all practices 

associated with being comprehensive and well-documented, and several

related to being accurate and credible (see table 5). However, several 

important practices related to accuracy and credibility were not 

performed. To be reliable, a cost estimate should be comprehensive, welldocumented, accurate, and credible.

Table 5: Summary of Cost Estimating Characteristics That the Cost Estimate 

Satisfies 

Characteristic of reliable estimates Satisfied?a

Comprehensive Yes

Well-documented Yes

Accurate Partially

Credible Partially 

Source: GAO analysis of DON data. 

a

“Yes” means that the program office provided documentation demonstrating satisfaction of the 

criterion. “Partially” means that the program office provided documentation demonstrating satisfaction

of part of the criterion. “No” means that the program office has yet to provide documentation 

demonstrating satisfaction of the criterion. 

The cost estimate is comprehensive because it includes both the 

government and contractor costs specific to development, acquisition 

(nondevelopment), implementation, and operations and support over the 

program’s 20-year life cycle. Moreover, the estimate clearly describes how 

the various subelements are aggregated to produce amounts for each cost 

category, thereby ensuring that all pertinent costs are included, and no 

costs are double counted. Finally, cost-influencing ground rules and 

assumptions, such as the program’s schedule, labor rates, and inflation 

rates are documented. 

The cost estimate is also well-documented in that the purpose of the cost 

estimate is clearly defined, and the technical baseline includes, among 

other things, the hardware components and planned performance 

parameters. Furthermore, the calculations and results used to derive the 

estimate are documented, including descriptions of the methodologies 

used and evidence of traceability back to source data (e.g., vendor quotes,

salary tables). Also, the cost estimate was reviewed by the Naval Center 

for Cost Analysis and the Office of the Secretary of Defense, Director for 

Page 31 GAO-08-896 DOD Business Systems Modernization 
Program Analysis and Evaluation, which adds a level of confidence in the

estimating process and the estimate produced. 

However, the estimate lacks accuracy because not all important practices

related to this characteristic were performed. Specifically, while the 

estimate is grounded in documented assumptions (e.g., hardware 

refreshment every 5 years) and periodically updated to reflect changes to 

the program, it is not adequately grounded in historical experience with 

comparable programs. While the program office did leverage historical 

cost data from the Navy ERP pilot programs, program officials told us that 

the level of cost accounting on these programs did not provide sufficient 

data. As stated in our guide, estimates should be based on historical 

records of cost and schedule estimates from comparable programs, and 

such historical data should be maintained and used for evaluation 

purposes and future estimates on comparable programs. The importance 

of doing so is evident by the fact that Navy ERP’s cost estimate has 

increased by about $570 million since fiscal year 2003, which program 

officials attributed to, among other things, site implementation costs (e.g.,

training and converting legacy system data) not included in the original 

cost estimate, schedule delays, and the lack of historical data from similar 

ERP programs. 

This lack of cost data for large-scale ERP programs is, in part, due to DOD 

not having a standardized cost element structure for these programs that 

can be used for capturing actual cost data, which is a prerequisite to 

capturing and maintaining the kind of historical data that can inform cost 

estimates on similar programs. This means that programs like Navy ERP 

will not be able to ground their cost estimates in actual costs from 

comparable programs. According to officials with the Defense Cost and 

Resource Center,40 such cost element structures are needed, along with a 

requirement for programs to report on their costs, but approval and 

resources have yet to be gained for either these structures or the reporting 

of their costs. We recently completed work that addressed standardization 

 40The Defense Cost and Resource Center is responsible for collecting current and historical 

major defense acquisition program cost and software resource data in a joint service 

environment and making those data available for use by authorized government analysts

when estimating the cost of ongoing and future government programs. 

Page 32 GAO-08-896 DOD Business Systems Modernization 
of DOD’s ERP cost element structure and maintenance of a database for 

historical ERP cost data for use on ERP programs.41

Compounding the estimate’s limited accuracy are limitations in its 

credibility. Specifically, while the estimate satisfies some of the key 

practices for a credible cost estimate (e.g., confirming key cost drivers, 

performing sensitivity analyses,42 and having an independent cost estimate 

prepared by the Naval Center for Cost Analysis that was within 11 percent 

of the program’s estimate), the program lacks a reliable schedule baseline, 

which is a key component of a reliable cost estimate because it serves as 

the basis for future work to be performed. Other factors that limit 

confidence in the cost estimate’s accuracy are (1) past increases in the 

program’s cost estimate (as discussed earlier) and (2) trends in EVM data 

(as discussed later). Taken together, the program’s cost estimate is not 

sufficiently credible and accurate and thus not reliable. 

While important cost estimating practices were not implemented, it is 

nevertheless unlikely that these limitations would materially increase the 

$2.4 billion cost estimate to a level approaching the program’s $8.6 billion 

benefit expectations. 

Measuring and reporting progress against cost and schedule commitments 

(i.e., baselines) is a vital element of effective program management. EVM 

provides a proven means for measuring such progress and thereby 

identifying potential cost overruns and schedule delays early, when their 

impact can be minimized. 

To its credit, the program has elected to implement program-level EVM, 

which is a best practice that has rarely been implemented in the federal

government. In doing so, however, basic EVM activities have not been 

executed. In particular, an integrated baseline review, which is to verify 

that the program’s cost and schedule are reasonable given the program’s 

scope of work and associated risks, has not been performed. Moreover, 

other accepted industry standards have not been sufficiently implemented, 

Earned Value Management 

Has Not Been Effectively 

Implemented 

 41GAO, DOD Business Systems Modernization: Key Marine Corps System Acquisition 

Needs to Be Better Justified, Defined, and Managed, GAO-08-822 (Washington, D.C.: 

July 28, 2008). 

42Sensitivity analysis reveals how the cost estimate is affected by a change in a single 

assumption or cost driver while holding all other variables constant. 

Page 33 GAO-08-896 DOD Business Systems Modernization 
and surveillance of EVM implementation by an entity independent of the 

program office has not occurred. Not performing these important 

practices has contributed to the cost overruns and lengthy schedule delays 

already experienced on Release 1.0, and they will likely result in more. In 

fact, our analysis of the latest estimate to complete just the budgeted 

development work for all three releases, which is about $844 million, 

shows that this estimate will most likely be exceeded by about $152 

million. 

As we previously reported,43 EVM offers many benefits when done 

properly. In particular, it allows performance to be measured, and it serves 

as an early warning system for deviations from plans. It, therefore, enables 

a program office to mitigate the risks of cost and schedule overruns. OMB 

policy recognizes the use of EVM as an important part of program 

management and decision making.44

Implementing EVM at the program level rather than just the contract level 

is considered a best practice, and OMB recently began requiring it to 

measure how well a program’s approved cost, schedule, and performance 

goals are being met. According to OMB, integrating government and 

contractor cost, schedule, and performance status should result in better 

program execution through more effective management. In addition, 

integrated EVM data can be used to better justify budget requests. 

To minimize the risk associated with its decision to transition 

responsibility for Navy ERP system integration from the contractor to the 

government and to improve cost and schedule performance, the program 

office elected in October 2006 to perform EVM at the program level. We 

support the use of program-level EVM. However, if not implemented 

effectively, this program-level approach will be of little value.

A fundamental aspect of effective EVM is the development of a 

performance measurement baseline (PMB), which represents the 

cumulative value of planned work and serves as the baseline against which 

The Program Has Elected to 

Implement Program-Level EVM 

An Integrated Baseline Review 

Has Not Been Performed 

 43GAO, Missile Defense: Additional Knowledge Needed in Developing System for

Intercepting Long-Range Missiles, GAO-03-600 (Washington, D.C.: Aug. 21, 2003).

44OMB, Preparation, Submission, and Execution of the Budget, Circular A-11

(Washington, D.C.: Executive Office of the President, June 2006), part 7, Planning,

Budgeting, Acquisition, and Management of Capital Assets, sec. 300. 

http://www.whitehouse.gov/omb/circulars/index.html. 

Page 34 GAO-08-896 DOD Business Systems Modernization 
variances are calculated. According to relevant best practice guidance, a 

PMB consists of 

• a complete work breakdown structure, 

• a complete integrated master schedule, and 

• accurate budgets for all planned work.45

To validate the PMB, an integrated baseline review is performed to obtain 

stakeholder agreement on the baseline. According to DOD guidance and 

best practices, such a review should be held within 6 months of a contract

award and conducted on an as needed basis throughout the life of a 

program to ensure that the baseline reflects (1) all tasks in the statement 

of work, (2) adequate resources (staff and materials) to complete the 

tasks, and (3) integration of the tasks into a well-defined schedule. 

Further, the contract performance reports that are to be used to monitor 

performance against the PMB should be validated during the integrated 

baseline review.46

The program office has satisfied some of the prerequisites for having a 

reliable PMB, such as developing a work breakdown structure and 

specifying the contract performance reports that are to be used to monitor 

performance. However, it has not conducted an integrated baseline 

review. Specifically, a review was not conducted for Release 1.0, even 

though the contract was finalized about 30 months ago (January 2006). 

Also, while the review for Release 1.1 was recently scheduled for August 

2008, this is 8 months later than when such a review should be held, 

according to DOD guidance and best practices. This means that the 

reasonableness of the program’s scope and schedule relative to the 

program risks has not been assured, and has likely been, and will likely 

continue to be a primary contributor to future cost increases and schedule 

delays. 

According to program officials, a review was not performed on the first 

release because development of this release was largely complete by the 

 45GAO-07-1134SP. 

46Defense Contract Management Agency, Department of Defense Earned Value 

Management Implementation Guide (Washington, D.C.: October 2006); and 

GAO-07-1134SP. 

Page 35 GAO-08-896 DOD Business Systems Modernization 
time the program office established the underlying capabilities needed to 

perform program-level EVM. In addition, program officials stated that an

integrated baseline review has yet to be performed on the other two 

releases because their priority has been on deploying and stabilizing the 

first release. In our view, not assuring the validity of the PMB precludes 

effective implementation of EVM. Until a review is conducted, DOD will 

not have reasonable assurance that the program’s scope and schedule are 

achievable, and thus, additional cost and schedule overruns are likely. 

In 1996, DOD adopted industry EVM guidance47 that identifies 32 essential 

practices organized into five categories: (1) organization; (2) planning, 

scheduling and budgeting; (3) accounting; (4) analysis and management 

reports; and (5) revisions and data maintenance. DOD requires that all 

programs’ implementation of EVM undergo a compliance audit against the 

32 industry practices. In addition, DOD policy and guidance48 state that 

independent surveillance of EVM should occur over the life of the program 

to guarantee the validity of the performance data and ensure that EVM is 

being used effectively to manage cost, schedule, and technical

performance. 

On Navy ERP, compliance with the 32 accepted industry practices has not 

been verified, and surveillance of EVM by an independent entity has not 

occurred. Therefore, the program does not have the required basis for 

ensuring that EVM is being effectively implemented on Navy ERP. 

According to program officials, surveillance was performed by NAVAIR 

for Release 1.0. However, NAVAIR officials said that they did not perform 

such surveillance because they did not receive the Release 1.0 cost 

performance data needed to do so. Program officials also stated that 

DON’s Center for Earned Value Management49 has conducted an initial 

assessment of their EVM management system, and that they intend to have

the Center perform surveillance. However, they did not have a plan for 

Industry EVM Standards Have 

Not Been Fully Implemented 

and Independently Surveilled 

 47American National Standards Institute (ANSI)/Electronic Industries Alliance (EIA) 

Standard, EVM Systems Standard, ANSI/EIA-748-A-1998 (R2002), approved May 19, 1998; 

revised January 2002. 

48Defense Contract Management Agency, Department of Defense Earned Value 

Management Implementation Guide (Washington, D.C.: October 2006). See also DOD 

Memorandum: Revision to DOD Earned Value Management Policy (Washington, D.C.: 

Mar. 7, 2005). 

49DON established the Center for Earned Value Management in April 2007 to, among other

things, work with program offices to improve the accuracy of EVM data and to provide

independent assistance to program managers when requested. 

Page 36 GAO-08-896 DOD Business Systems Modernization 
accomplishing this. Until compliance with the standards is verified and 

continuous surveillance occurs, and deviations are addressed, the program 

will likely continue to experience cost overruns and schedule delays. 

The success of any program depends in part on having a reliable schedule 

of when the program’s work activities will occur, how long they will take, 

and how they are related to one another. As such, the schedule not only 

provides a road map for the systematic execution of a program but also 

provides the means by which to estimate costs, gauge progress, identify 

and address potential problems, and promote accountability. Our research 

has identified nine practices associated with effective schedule 

estimating.50 These practices are (1) capturing key activities, (2) 

sequencing key activities, (3) establishing the duration of key activities, (4) 

assigning resources to key activities, (5) integrating key activities 

horizontally and vertically, (6) establishing the critical path for key 

activities, (7) identifying “float time”51 between key activities, (8) 

distributing reserves to high-risk activities, and (9) performing a schedule 

risk analysis. 

The program’s estimated schedule was developed using some of these 

practices, but several key practices were not fully employed that are 

fundamental to having a schedule that provides a sufficiently reliable basis 

for estimating costs, measuring progress and forecasting slippages. On the

positive side, the schedule for the first two releases captures key activities

and their durations and is integrated horizontally and vertically, meaning 

that multiple teams executing different aspects of the program can 

effectively work to the same master schedule. Moreover, for these two 

releases, the program has established float time between key activities and 

distributed schedule reserve to high-risk activities. However, the program 

has not adequately sequenced and assigned resources to key program 

activities. Moreover, the estimated schedule for the first increment is not 

grounded in an integrated master schedule of all the releases, and thus the 

schedule for this increment does not reflect the program’s critical path of 

work that must be performed to achieve the target completion date. Also,

it does not reflect the results of a schedule-risk analysis across all three 

releases with schedule reserve allocated to high-risk activities because 

such risks were not examined. See table 6 for the results of our analyses

relative to each of the nine practices. 

Estimated Schedule Baseline 

Was Not Derived in 

Accordance with All Key 

Schedule Estimating Practices 

 50GAO-07-1134SP. 

51Float time is the amount of time an activity can slip before affecting the critical path. 

Page 37 GAO-08-896 DOD Business Systems Modernization 
Table 6: Satisfaction of Schedule Estimating Key Practices 

Practice Explanation Satisfied?aGAO analysis 

Capturing 

key 

activities 

The schedule should reflect all key activities (e.g., steps, 

events, outcomes) as defined in the program’s work 

breakdown structure, to include activities to be performed 

by both the government and its contractors. 

Yes The program’s estimated schedules for the 

first two releases reflect both government 

and contractor activities, such as 

development and testing of the software 

components, as well as key milestones for 

measuring progress. 

Sequencing 

key 

activities 

The schedule should be planned so that it can meet critical 

program dates. To meet this objective, key activities need 

to be logically sequenced in the order that they are to be 

carried out. In particular, activities that must be finished 

prior to the start of other activities (i.e., predecessor 

activities), as well as activities that cannot begin until other 

activities are completed (i.e., successor activities) should 

be identified. By doing so, interdependencies among 

activities that collectively lead to the accomplishment of 

events or milestones can be established and used as a 

basis for guiding work and measuring progress. 

Partially The schedules for the first two releases 

include the logical sequencing of most, but 

not all, activities. For example, 234 of 2,445 

activities in the Release 1.1 schedule were 

not sequenced properly. By not identifying 

the correct interdependencies and properly 

sequencing key activities, the schedule may 

not facilitate the meaningful tracking of 

progress. 

Establishing 

the duration 

of key 

activities 

The schedule should realistically reflect how long each 

activity will take to execute. In determining the duration of 

each activity, the same rationale, historical data, and 

assumptions used for cost estimating should be used for 

schedule estimating. Further, these durations should be as 

short as possible, and they should have specific start and 

end dates. Excessively long periods needed to execute an

activity should prompt further decomposition of the activity 

so that shorter execution durations will result. 

Yes The schedules for the first two releases 

establish realistic durations of key activities. 

For example, the schedule for Release 1.1 is

based on historical data from Release 1.0, 

which provides a level of confidence that the 

durations are reasonable. 

Assigning 

resources 

to key 

activities 

The schedule should reflect what resources (i.e., labor, 

material, and overhead) are needed to do the work, 

whether all required resources will be available when they 

are needed, and whether any funding or time constraints 

exist. 

Partially The schedules for the first two releases 

include the allocation of resources, such as 

personnel, to activities, but it does not reflect 

whether all resources will be available when

they are needed because the identified 

resources are shared across the three 

releases. Restated, personnel are assigned 

to activities across multiple releases, each of 

which is managed according to a separate 

schedule. Therefore, if one of the schedules 

were to be delayed, the other schedules that

required the same resources would likely 

also be delayed. 

Page 38 GAO-08-896 DOD Business Systems Modernization 
Practice Explanation Satisfied?a

GAO analysis 

Integrating 

key 

activities 

horizontally 

and 

vertically 

The schedule should be horizontally integrated, meaning 

that it should link the products and outcomes associated

with already sequenced activities. These links are 

commonly referred to as “handoffs” and serve to verify that 

activities are arranged in the right order to achieve 

aggregated products or outcomes. The schedule should 

also be vertically integrated, meaning that traceability 

exists among varying levels of activities and supporting 

tasks and subtasks. Such mapping or alignment among 

levels enables different groups to work to the same master 

schedule. 

Yes The schedules for the first two releases are 

both horizontally and vertically integrated, 

meaning that the activities across the 

multiple teams are arranged in the right 

order to achieve aggregated products or 

outcomes. In addition, traceability exists 

among varying levels of activities, which 

allows multiple teams (i.e., development, 

testing) to work to the same master 

schedule. 

Establishing 

the critical 

path for key 

activities 

Using scheduling software, the critical path—the longest 

duration path through the sequenced list of key activities—

should be identified. The establishment of a program’s 

critical path is necessary for examining the effects of any 

activity slipping along this path. Potential problems that 

might occur along or near the critical path should also be

identified and reflected in the scheduling of the time for 

high-risk activities (see next practice). 

Partially While the program has established the 

critical path for the first two releases 

separately, it has not done so for the entire 

first increment. This is because the program 

maintains separate schedules for each 

release, and in doing so, cannot identify the 

longest duration of tasks through the entire 

first increment. Without doing so, the effects 

of any slippage along the critical path on

future releases cannot be determined. 

Identifying

“float time” 

between 

key 

activities 

The schedule should identify float time—the time that a 

predecessor activity can slip before the delay affects 

successor activities—so that schedule flexibility can be

determined. As a general rule, activities along the critical 

path typically have the least amount of float time. 

Yes The schedules for the first two releases 

identify float time between key activities. 

Distributing 

reserves to 

high-risk 

activities 

The baseline schedule should include a buffer or a reserve 

of extra time. Schedule reserve for contingencies should 

be calculated by performing a schedule risk analysis (see 

next practice). As a general rule, the reserve should be 

applied to high-risk activities, which are typically found 

along the critical path. 

Partially The schedule allocates reserve for high-risk

activities on the critical path for Release 1.0.

However, because the program has not 

established a critical path for the first 

increment, it cannot allocate reserve for the 

high-risk activities on the entire program’s 

critical path. 

Schedule 

risk 

analysis 

should be 

performed

A schedule risk analysis should be performed using 

statistical techniques to predict the level of confidence in

meeting a program’s completion date. This analysis 

focuses not only on critical path activities but also on

activities near the critical path, since they can potentially 

affect program status. 

Partially A schedule risk analysis on the entire 

program was not performed. A schedule risk

analysis has been done on Release 1.0, and 

the program office plans to do one for 

Release 1.1. However, without analyzing the 

risks associated with the program’s entire 

schedule, the program cannot determine the 

level of confidence in meeting the program’s 

completion date. 

Source: GAO analysis of DON data. 

a

“Yes” means that the program provided documentation demonstrating satisfaction of the practice. 

“Partially” means that the program provided documentation demonstrating satisfaction of part of the 

practice. “No” means that the program has yet to provide documentation demonstrating satisfaction of 

the practice. 

Page 39 GAO-08-896 DOD Business Systems Modernization 
According to program documentation, they have plans to address the 

logical sequencing of activities (to ensure that it reflects how work is to be 

performed), but program officials stated that they do not plan to combine 

all three releases into a single integrated master schedule for the entire 

first increment of the program because doing so would produce an overly 

complex and nonexecutable schedule involving as many as 15,000 

activities. However, our research of and experience in evaluating major 

programs’ use of EVM and integrated master schedules show that while 

large, complex programs necessitate schedules involving thousands of 

activities, successful programs ensure that their schedules integrate these 

activities. In our view, not adequately performing these practices does not 

allow the program to effectively assign resources, identify the critical path, 

and perform a schedule risk analysis that would allow it to understand, 

disclose, and compensate for its schedule risks. This means that the 

program is not well-positioned to understand progress and forecast its 

impact. To illustrate, the program recently experienced delays in 

deploying its first release at NAVAIR, which according to a recent 

operational test and evaluation report

52 has significantly affected the 

schedule’s critical path. These schedule impacts are because resources 

supporting the deployment at NAVAIR began to shift to the next scheduled 

deployment site and thus are no longer available to resolve critical issues 

at NAVAIR. Since the schedule baseline is not integrated across all 

releases, the impact of this delay on other releases, and thus the program 

as a whole, cannot be readily determined. 

Program data show a pattern of actual cost overruns and schedule delays 

between January 2007 and May 2008. Moreover, our analysis of the data

supports a most likely program cost growth of about $152 million to 

complete all three releases. 

Trends in EVM Data Show 

Pattern of Cost Overruns and 

Schedule Slippages 

 52Department of the Navy, Commander, Operational Test and Evaluation Force, Navy

Enterprise Resource Planning System Initial Operational Test and Evaluation, OT-C1

Final Report to the Chief of Naval Operations (Norfolk, VA: June 13, 2008).

Page 40 GAO-08-896 DOD Business Systems Modernization 
Differences from the PMB are measured in both cost and schedule 

variances.53 Positive variances indicate that activities are costing less or 

are completed ahead of schedule. Negative variances indicate that 

activities are costing more or are falling behind schedule. These cost and 

schedule variances can then be used in forecasting the cost and time 

needed to complete the program. 

Based on program-provided data for the first increment over a 17-month 

period ending May 2008, the program has experienced negative cost 

variances. Specifically, while these cost variances have fluctuated during

this period, they have consistently been negative. (See fig. 4.) Moreover, 

our analysis of the cost to complete just the budgeted development work 

(also known as the PMB) for all three releases, which is about $844 

million,54 will be exceeded by between about $102 million and $316 million, 

with a most likely overrun of about $152 million.55 In contrast, the program 

office reports that the overrun at completion will be $55 million but has 

yet to provide us with documentation supporting this calculation. 

Moreover, our calculation does not reflect the recent problems discovered 

during the operational test and evaluation at NAVAIR and thus the overrun 

is likely to be higher. 

 53Cost variances compare the earned value of the completed work with the actual cost of

the work performed. For example, if a contractor completed $5 million worth of work, and 

the work actually cost $6.7 million, there would be a negative $1.7 million cost variance.

Schedule variances are also measured in dollars, but they compare the earned value of the 

work actually completed as of a point in time to the value of work that was expected to be 

completed. For example, if a contractor completed $5 million worth of work at the end of 

the month but was budgeted to complete $10 million worth of work, there would be a 

negative $5 million schedule variance. 

54This figure is the total estimated budget for the program. It represents the cumulative 

value of the budgeted cost of work scheduled over the life of the program.

55To make these assessments, we applied earned value analysis techniques to data from the

program’s contract performance reports. These techniques compare budget versus actual 

costs versus project status in dollar amounts. For our analysis, we used standard earned 

value formulas to calculate cost and schedule variance and forecast the range of cost 

overrun at completion. 

Page 41 GAO-08-896 DOD Business Systems Modernization 
Figure 4: Cumulative Cost Variance for Navy ERP over a 17-Month Period

-30

-25

-20

-15

-10

-5

0

5

Jan. Feb. Mar. Apr. May June July Aug. Sept. Oct. Nov. Dec. Jan. Feb. Mar. Apr. May

Dollars in millions

2007 2008

Source: GAO analysis based on Navy ERP data.

Fiscal year

During this same 17-month period, the program has experienced negative 

schedule variances and, since January 2008, they have almost doubled 

each month. Further, as of May 2008, the program had not completed 

about $24 million in scheduled work. (See fig. 5.) An inability to meet 

schedule performance is a frequent indication of future cost increases, as 

more spending is often necessary to resolve schedule delays.

Page 42 GAO-08-896 DOD Business Systems Modernization 
Figure 5: Cumulative Schedule Variance of the Navy ERP Program over a 17-Month Period

-30

-25

-20

-15

-10

-5

0

Jan. Feb. Mar. Apr. May June July Aug. Sept. Oct. Nov. Dec. Jan. Feb. Mar. Apr. May

Dollars in millions

2007 2008

Source: GAO analysis based on Navy ERP data.

Fiscal year

Because the program office has not performed important reliability 

checks, such as EVM validation and integrated baseline reviews, as 

discussed above, we cannot be certain that the PMB is reliable (i.e., 

reflects all the work to be done and has identified all the risks). As a result, 

the overrun that we are forecasting could be higher. 

By not executing basic EVM practices, the program has and will likely 

continue to experience cost and schedule shortfalls. Until the program 

office implements these important EVM practices, it will likely not be able

to track actual program costs and schedules close to estimates. 

Page 43 GAO-08-896 DOD Business Systems Modernization 
Well-defined and managed requirements are recognized by DOD guidance 

as essential, and they can be viewed as a cornerstone of effective system 

acquisition. One aspect of effective requirements management is 

requirements traceability.56 By tracing requirements both backward from 

system requirements to higher level business or operational requirements 

and forward to system design specifications and test plans, the chances of

the deployed product satisfying requirements is increased, and the ability 

to understand the impact of any requirement changes and thus make 

informed decisions about such changes, is enhanced. 

The program office is effectively implementing requirements traceability 

for its 1,733 Release 1.0 system requirements. To verify this traceability, 

we randomly selected and analyzed 60 of the 1,733 system requirements 

and confirmed that 58 of the 60 were traceable both backward to higher 

level requirements and forward to design specifications and test results. 

The remaining 2 had been allocated to the other releases, and thus we also 

confirmed the program’s ability to maintain traceability between product 

releases. In doing so, the program utilized a tool called DOORS, which if 

implemented properly, allows each requirement to be linked from its most 

conceptual definition to its most detailed definition, as well as to design 

specifications and test cases. In effect, the tool maintains the linkages 

among requirement documents, design documents, and test cases even if 

requirements change. 

If DON continues to effectively implement requirements traceability, it will 

increase the chances that system requirements will be met by the deployed 

system. 

Proactively managing program risks is a key acquisition management 

control and, if defined and implemented properly, it can increase the 

chances of programs delivering promised capabilities and benefits on time 

and within budget. To the program office’s credit, it has defined a risk 

management process that meets relevant guidance. However, it has not

effectively implemented the process for all identified risks. As a result, 

these risks have not been proactively mitigated and either have

Important Requirements 

Management Activity Has 

Been Effectively 

Implemented 

A Risk Management 

Process Has Been Defined, 

but All Risks Have Not 

Been Effectively Mitigated 

 56DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004). Software Engineering

Institute, Software Acquisition Capability Maturity Model®

 version 1.03, CMU/SEI-2002- 

TR-010 (Pittsburgh, PA: March 2002). 

Page 44 GAO-08-896 DOD Business Systems Modernization 
contributed to cost and schedule shortfalls, or could potentially contribute 

to such shortfalls. 

DOD acquisition management guidance, as well as other relevant guidance 

advocates identifying facts and circumstances that can increase the 

probability of an acquisition’s failing to meet cost, schedule, and 

performance commitments and then taking steps to reduce the probability 

of their occurrence and impact.57 In brief, effective risk management 

consists of: (1) establishing a written plan for managing risks; (2) 

designating responsibility for risk management activities; (3) encouraging 

program-wide participation in the identification and mitigation of risks; (4) 

defining and implementing a process that provides for the identification, 

analysis, and mitigation of risks; and (5) examining the status of identified

risks in program milestone reviews. 

The program office has developed a written plan for managing risks and 

established a process that together provide for the above-cited risk 

management practices. Moreover, it has largely followed its plan and 

process as per the following examples: 

• The program manager has been assigned overall responsibility for 

managing risks and serves as the chair of the risk management board.58

Also, a functional team lead (i.e., subject matter expert) is assigned 

responsibility for analyzing and mitigating each identified risk.

• Program-wide participation in the identification, analysis, and mitigation 

of risks is encouraged. Specifically, a manager for each release is 

responsible for providing risk management guidance to the staff, which 

includes staff identification and analysis of risks. Also, according to the 

program office’s risk management plan, all program personnel can submit

a risk for approval. In addition, stakeholders participate in risk 

management activities during acquisition milestone reviews. 

 57Department of Defense, Risk Management Guide for DOD Acquisition, 6th Edition,

Version 1.0., http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-version.pdf

(accessed March 13, 2008) and Software Engineering Institute, CMMI for Acquisition,

Version 1.2, CMU/SEI-2007-TR-017 (Pittsburgh, PA: November 2007). 

58The risk management board oversees risk management activities by assigning risk 

owners, approving and directing resources to facilitate risk handling strategies, and 

reviewing risk handling status. 

Page 45 GAO-08-896 DOD Business Systems Modernization 
• The program office has identified and categorized individual risks. As of 

June 2008, the risk database contained 15 active risks—3 high, 8 medium, 

and 4 low.59

• Program risks are considered during program milestone reviews. For 

example, during the program’s critical design review, which is a key event 

of the system development and demonstration phase, key risks regarding

implementing new business processes and legacy system changes were 

discussed. Furthermore, the program manager receives a monthly risk 

report that describes the status of program risks. 

However, the program office has not consistently followed other aspects of its 

process. In particular, it has not effectively implemented steps for mitigating the 

risks associated with (1) converting data from NAVAIR’s legacy systems to run 

on Navy ERP and (2) positioning NAVAIR for adopting the new business 

processes embedded in Navy ERP. As we have previously reported, it is 

important for organizations that are to operate and use commercial off-the-shelf 

software products, such as Navy ERP, to proactively manage and position

themselves for the organizational impact of introducing functionality 

embedded in the commercial products. If they do not, the organization’s 

performance will suffer.60

To the program office’s credit, it identified numerous risks associated with

data conversion and organizational change management and developed 

and implemented strategies that were intended to mitigate these risks. 

However, it closed these risks even though they were never effectively 

mitigated, as evidenced by the results of recently completed DON 

operational test and evaluation. According to the June 2008 operational 

test and evaluation report for NAVAIR, significant problems relating to 

both legacy system data conversion and adoption of new business 

processes were experienced. The report states that these problems have 

contributed to increases in the costs to operate the system, including 

 59Risk levels of high, medium or low are assigned using quantitative measurements of the 

probability of the risk occurring and the potential impact to the program’s cost, schedule, 

and performance. Based on that assessment, a risk level is assigned to represent the risk’s 

significance. 

60See, for example, GAO, Office of Personnel Management: Retirement Systems 

Modernization Program Faces Numerous Challenges, GAO-05-237 (Washington, D.C.: 

Feb. 28, 2005); and GAO, Information Technology: Customs Automated Commercial 

Environment Program Progressing, but Need for Management Improvements Continues, 

GAO-05-267 (Washington, D.C.: Mar. 14, 2005). 

Page 46 GAO-08-896 DOD Business Systems Modernization 
unexpected manual effort. It further states that these problems have 

rendered the deployed version not operationally effective and that 

deployment of the system to other sites should not occur until the change 

management process has been analyzed and improved. It also attributed 

the realization of the problems to the program office and NAVAIR not 

having adequately engaged and communicated early with each other to 

coordinate and resolve differences in organizational perspectives and 

priorities and provide intensive pre-deployment preparation and training. 

Program officials acknowledged these shortcomings and attributed them 

to their limited authority over the commands. In this regard, they have 

previously surfaced these risks with department oversight and approval 

authorities, but actions were not taken by these authorities that ensured 

that the risks were being effectively mitigated. 

Beyond not effectively mitigating these risks, the program office has not

ensured that all risks are captured in the risk inventory. For example, the 

inventory does not include the risks described in this report that are 

associated with not having adequately demonstrated the program’s 

alignment to the federated BEA and not having implemented programlevel EVM in a manner that reflects industry practices. This means that 

these risks are not being disclosed or mitigated. 

By not effectively addressing all risks associated with the program, these 

risks can and have become problems that contribute to cost and schedule 

shortfalls. Until all significant risks are proactively addressed, to include 

ensuring that all associated mitigation steps are implemented and that they

accomplished their intended purpose, the program will likely experience 

further problems at subsequent deployment sites. 

DOD’s success in delivering large-scale business systems, such as Navy 

ERP, is in large part determined by the extent to which it employs the kind 

of rigorous and disciplined IT management controls that are reflected in 

department policies and related guidance. While implementing these 

controls does not guarantee a successful program, it does minimize a 

program’s exposure to risk and thus the likelihood that it will fall short of 

expectations. In the case of Navy ERP, living up to expectations is 

important because the program is large, complex, and critical to 

addressing the department’s long-standing problems related to financial

transparency and asset visibility. 

The effectiveness to which key IT management controls have been 

implemented in Navy ERP varies, with one control and several aspects of

Conclusions 

Page 47 GAO-08-896 DOD Business Systems Modernization 
others being effectively implemented, and others less so. Moreover, those 

controls that have not been effectively implemented have, in part, 

contributed to the sizable cost and schedule shortfalls experienced to date 

on the program. Unless this changes, more shortfalls can be expected. 

While the program office is primarily responsible for ensuring that effective IT 

management controls are implemented, other oversight and stakeholder 

organizations share responsibility. For example, even though the program has 

not demonstrated its alignment with the federated BEA, it nevertheless 

followed established DOD architecture compliance guidance and used the 

related compliance assessment tool in assessing and asserting its compliance. 

The root cause for not demonstrating compliance thus is not traceable to the 

program office but rather is due to, among other things, the limitations of the 

compliance guidance and tool, and the program’s oversight entities not 

validating the compliance assessment and assertion. Also, the reason that the 

program’s cost estimate was not informed by the cost experiences of other 

programs of the same size and scope is because DOD does not have a standard

ERP cost element structure and has not maintained a historical database of 

costs for like programs to use. In contrast, effective implementation of other 

management controls, such as implementing EVM, requirements traceability, 

and risk management is the responsibility of the program office. 

All told, addressing the management control weaknesses requires the 

combined efforts of the various organizations that share responsibility for 

managing and overseeing the program. By doing so, the department can 

better assure itself that Navy ERP will optimally support its performance 

goals and will deliver promised capabilities and benefits on time and 

within budget. 

Because we recently completed work that more broadly addresses the 

above cited architectural alignment and comparable program cost data 

limitations, we are not making recommendations in this report for 

addressing them. 

To strengthen Navy ERP management control and better provide for the 

program’s success, we are making the following recommendations: 

• To improve the reliability of Navy ERP benefit estimates and cost 

estimates, we recommend that the Secretary of Defense direct the 

Secretary of the Navy, through the appropriate chain of command, to 

ensure that future Navy ERP estimates include uncertainty analyses of 

estimated benefits, reflect the risks associated with not having cost data 

Recommendations for 

Executive Action 

Page 48 GAO-08-896 DOD Business Systems Modernization 
for comparable ERP programs, and are otherwise derived in full 

accordance with the other key estimating practices, and economic analysis

practices discussed in this report. 

• To enhance Navy ERP’s use of EVM, we recommend that the Secretary of 

Defense direct the Secretary of the Navy, through the appropriate chain of

command, to ensure that (1) an integrated baseline review on the last two 

releases of the first increment is conducted, (2) compliance against the 32 

accepted industry EVM practices is verified, and (3) a plan to have an 

independent organization perform surveillance of the program’s EVM 

system is developed and implemented. 

• To increase the quality of the program’s integrated master schedule, we 

recommend that the Secretary of Defense direct the Secretary of the Navy,

through the appropriate chain of command, to ensure that the schedule (1) 

includes the logical sequencing of all activities, (2) reflects whether all 

required resources will be available when needed, (3) defines a critical 

path that integrates all three releases, (4) allocates reserve for the highrisk activities on the entire program’s critical path, and (5) incorporates 

the results of a schedule risk analysis for all three releases and 

recalculates program cost and schedule variances to more accurately 

determine a most likely cost and schedule overrun. 

• To improve Navy ERP’s management of program risks, we recommend 

that the Secretary of Defense direct the Secretary of the Navy, through the

appropriate chain of command, to ensure that (1) the plans for mitigating 

the risks associated with converting data from legacy systems to Navy 

ERP and positioning the commands for adopting the new business 

processes embedded in the Navy ERP are re-evaluated in light of the 

recent experience with NAVAIR and adjusted accordingly, (2) the status 

and results of these and other mitigation plans’ implementation are 

periodically reported to program oversight and approval authorities, (3) 

these authorities ensure that those entities responsible for implementing 

these strategies are held accountable for doing so, and (4) each of the risks 

discussed in this report are included in the program’s inventory of active 

risks and managed accordingly. 

In written comments on a draft of this report, signed by the Deputy Under

Secretary of Defense (Business Transformation) and reprinted in appendix

II, DOD stated that it concurred with two of our four recommendations 

and partially concurred with the remaining two. Further, it stated that it

has taken steps to address some of our recommendations, adding that it is

committed to implementing recommendations that contribute to the 

Agency Comments 

and Our Evaluation 

Page 49 GAO-08-896 DOD Business Systems Modernization 
program’s success. The department’s comments relative to both of the 

recommendations that it partially concurred with, as well as additional 

comments, are discussed below. 

For our recommendation associated with improving the program’s benefit 

and cost estimates, DOD concurred with two of the recommendation’s three 

parts, but it did not concur with one part—ensuring that future cost estimates

reflect the risk of not having cost data for comparable programs. While 

acknowledging that the program had limited cost data from comparable

programs on which to base its cost estimate, DOD stated that an uncertainty

analysis had been applied to the estimate to account for the risk associated 

with not having such data. The department further stated that actual

experience on the program will continue to be used to refine the program’s

cost estimating methodology. While we support DOD’s stated commitment to

using actual program cost experience in deriving future estimates, we do not

agree that the latest estimate accounted for the risk of not having cost data

from comparable programs. We examined the uncertainty analysis as part of 

our review, and found that it did not recognize this risk. Moreover, DOD’s

comments offered no new evidence to the contrary. 

For our recommendation associated with improving the program’s 

schedule estimating, DOD concurred with four of the recommendation’s 

five parts, and partially concurred with one part—ensuring that the 

schedule defines a critical path that integrates all releases. In taking this 

position, the department stated that a critical path has been established for 

each release rather than across all three releases, and it attributes this to 

the size and complexity of the program. We do not take issue with either 

of these statements, as they are already recognized in our report. However, 

DOD offers no new information in its comments. Further, our report also 

recognizes that to be successful, large and complex programs that involve 

thousands of activities need to ensure that their schedules integrate these 

activities. In this regard, we support the department’s commitment to 

explore the feasibility of implementing this part of our recommendation. 

In addition, while stating that it concurred with all parts of our 

recommendation associated with improving the program’s use of EVM, 

DOD nevertheless provided additional comments as justification for 

having not conducted an integrated baseline review on Release 1.0. 

Specifically, it stated that when it rebaselined this release in December 

2006, the release’s development activities were essentially complete and 

the release was in the latter stages of testing. Further, it stated that the 

risks associated with the Release 1.0 schedule were assessed 3 months 

after this rebaselining, and these risks were successfully mitigated. To 

Page 50 GAO-08-896 DOD Business Systems Modernization 
support this statement, it said that Release 1.0 achieved its “Go-Live” as 

scheduled at NAVAIR. We do not agree with these comments for several 

reasons. First, at the time of the rebaselining, about 9 months of scheduled 

Release 1.0 development remained, and thus the release was far from 

complete. Moreover, the significance of the amount of work that 

remained, and still remains today on Release 1.0 is acknowledged in 

DOD’s own comment that the scheduled integrated baseline review for 

Release 1.1 will also include remaining Release 1.0 work. Second, the 

Release 1.0 contract was awarded in January 2006, and DOD’s own 

guidance requires that an integrated baseline review be conducted within 

6 months of a contract’s award. Third, although DOD states that the 

program achieved “Go-Live” as scheduled on October 1, 2007, the program 

achieved initial operational capability 7 months later than established in 

the December 2006 baseline. 

In addition to these comments, the department also described actions 

under way or planned to address our recommendations. We support the 

actions described, as they are consistent with the intent of our

recommendations. If fully and properly implemented, these actions will go 

a long way in addressing the management control weaknesses that our 

recommendations are aimed at correcting. 

We are sending copies of this report to interested congressional 

committees; the Director, Office of Management and Budget; the 

Congressional Budget Office; the Secretary of Defense; and the 

Department of Defense Office of the Inspector General. We also will make

copies available to others upon request. In addition, the report will be 

available at no charge on the GAO Web site at http://www.gao.gov. 

If you or your staff have any questions about this report, please contact me 

at (202) 512-3439 or hiter@gao.gov. Contact points for our Offices of 

Congressional Relations and Public Affairs may be found on the last page 

of this report. GAO staff who made major contributions to this report are 

listed in appendix III. 

Randolph C. Hite 

Director, Information Technology Architecture 

 and Systems Issues 

Page 51 GAO-08-896 DOD Business Systems Modernization 
Appendix I: Objective, Scope, and 

Methodology 

Page 52 GAO-08-896 DOD Business Systems Modernization 

Appendix I: Objective, Scope, and 

Methodology 

Our objective was to determine whether the Department of the Navy is 

effectively implementing information technology management controls on 

the Navy Enterprise Resource Planning (Navy ERP) program. To 

accomplish this, we focused on the first increment of Navy ERP and the 

following management areas (1) architectural alignment, (2) economic 

justification, (3) earned value management (EVM), (4) requirements 

management, and (5) risk management. 

• To determine whether Navy ERP was aligned with the Department of 

Defense’s (DOD) federated business enterprise architecture (BEA), we 

reviewed the program’s BEA compliance assessments and system 

architecture products, as well as Versions 4.0, 4.1, and 5.0 of the BEA, and 

compared them with the BEA compliance requirements described in the 

Fiscal Year 2005 Defense Authorization Act1

 and DOD’s BEA compliance 

guidance, and we evaluated the extent to which the compliance 

assessments addressed all relevant BEA products. We also determined the 

extent to which the program-level architecture documentation supported 

the BEA compliance assessments. We obtained documentation, such as 

the BEA compliance assessments from the Navy ERP and Global Combat 

Support System—Marine Corps programs, as well as the Air Force’s 

Defense Enterprise Accounting and Management System and Air Force 

Expeditionary Combat Support System programs. We then compared 

these assessments to identify potential redundancies or opportunities for 

reuse and determined if the compliance assessments examined duplication 

across programs, and if the tool that supports these assessments is being 

used to identify such duplication. In doing so, we interviewed program 

officials and officials from the Department of the Navy, Office of the Chief

Information Officer and reviewed recent GAO reports to determine the 

extent to which the programs were assessed for compliance against the 

Department of the Navy enterprise architecture. We also interviewed 

program officials and officials from the Business Transformation Agency 

and the Department of the Navy, including the logistics Functional Area 

Manager, and obtained guidance documentation from these officials to 

determine the extent to which the compliance assessments were subject 

to oversight or validation.

• To determine whether the program had economically justified its 

investment in Navy ERP, we reviewed the latest economic analysis to 

determine the basis for the cost and benefit estimates. This included 

 1

Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005, Pub. L. 

No. 108-375, § 332 (2004) (codified at 10 U.S.C. §§ 186 and 2222). 
Appendix I: Objective, Scope, and 

Methodology 

evaluating the analysis against Office of Management and Budget guidance 

and GAO’s Cost Assessment Guide.2

 In doing so, we interviewed cognizant 

program officials, including the program manager and cost analysis team, 

regarding their respective roles, responsibilities, and actual efforts in 

developing and/or reviewing the economic analysis. We also interviewed 

officials at the Office of Program Analysis and Evaluation and Naval 

Center for Cost Analysis as to their respective roles, responsibilities, and 

actual efforts in developing and/or reviewing the economic analysis. We 

did not verify the validity of the source data used to calculate estimated 

benefits, such as those data used to determine the yearly costs associated 

with legacy systems planned for retirement. 

• To determine the extent to which the program had effectively 

implemented EVM, we reviewed relevant documentation, such as contract 

performance reports, acquisition program baselines, performance 

measurement baseline, and schedule estimates and compared them with 

DOD policies and guidance.3

 To identify trends that could affect the 

program baseline in the future, we assessed cost and schedule 

performance and, in doing so, we applied earned value analysis 

techniques4

 to data from contract performance reports. We compared the 

cost of work completed with the budgeted costs for scheduled work over 

a 17-month period, from January 2007 to May 2008, to show trends in cost 

and schedule performance. We also used data from the reports to estimate 

the likely costs at completion of the program through established earned 

value formulas. This resulted in three different values, with the middle 

value being the most likely. We checked EVM data to see if there were any 

mathematical errors or inconsistencies that would lead to the data being 

unreliable. We interviewed cognizant officials from the Naval Air Systems 

Command and program officials to determine whether the program had 

conducted an integrated baseline review, whether the EVM system had 

 2

Office of Management and Budget, Circular No. A-94: Guidelines and Discount Rates for

Benefit-Cost Analysis of Federal Program (Oct. 29, 1992), GAO-07-1134SP. 

3

Defense Contract Management Agency, Department of Defense Earned Value 

Management Implementation Guide (Washington, D.C.: Apr. 7, 2005). See also DOD 

Memorandum: Revision to DOD Earned Value Management Policy (Washington, D.C.: 

Mar. 7, 2005). 

4

The earned value concept is applied as a means of placing a dollar value on project status. 

The techniques we used compared the program’s budget to actual costs and project status 

in dollar amounts. We used standard earned value formulas to calculate cost and schedule

variance and forecast the range of cost overrun at program completion. 

Page 53 GAO-08-896 DOD Business Systems Modernization 
Appendix I: Objective, Scope, and 

Methodology 

been validated against industry guidelines,5 and to better understand the 

anomalies in the EVM data and determine what outside surveillance was 

being done to ensure that the industry standards are being met. We also 

reviewed the program’s schedule estimates and compared them with 

relevant best practices6

 to determine the extent to which they reflect key 

estimating practices that are fundamental to having a reliable schedule. In

doing so, we interviewed cognizant program officials to discuss their use 

of best practices in creating the program’s current schedule. 

• To determine the extent to which the program has effectively implemented 

requirements management, we reviewed relevant program documentation, 

such as the program management plan and baseline list of requirements. 

To determine the extent to which the program has maintained traceability 

backward to high-level business operation requirements and system 

requirements, and forward to system design specifications, and test plans, 

we randomly selected 60 program requirements and traced them both 

backward and forward. This sample was designed with a 5 percent 

tolerable error rate at the 95 percent level of confidence so that, if we 

found 0 problems in our sample, we could conclude statistically that the 

error rate was less than 5 percent. In addition, we interviewed program 

officials involved in the requirements management process to discuss their 

roles and responsibilities for managing requirements. 

• To determine the extent to which the program implemented risk 

management, we reviewed relevant risk management documentation, such 

as the program’s risk management plan and risk database reports 

demonstrating the status of the program’s major risks and compared the 

program office’s activities with DOD acquisition management guidance

and related industry practices.7

 We also reviewed the program’s mitigation 

process with respect to key risks to determine the extent to which these 

risks were effectively managed. In doing so, we interviewed cognizant 

program officials, such as the program manager and risk manager, to 

discuss their roles and responsibilities and obtain clarification on the 

 5

American National Standards Institute (ANSI) /Electronic Industries Alliance (EIA) 

EVM Systems Standard, ANSI/EIA-748-A-1998 (R2002), approved May 19, 1998; revised 

January 2002. 

6

GAO-07-1134SP. 

7

Department of Defense, Risk Management Guide for DOD Acquisition, 6th Edition,

Version 1.0., http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-version.pdf

(accessed March 13, 2008) and Software Engineering Institute, CMMI for Acquisition, 

Version 1.2, CMU/SEI-2007-TR-017 (Pittsburgh, PA: November 2007). 

Page 54 GAO-08-896 DOD Business Systems Modernization 
Appendix I: Objective, Scope, and 

Methodology 

program’s approach to managing risks associated with acquiring and 

implementing Navy ERP.

We conducted this performance audit at DOD offices in the Washington, 

D.C., metropolitan area and Annapolis, Md., from June 2007 to September 

2008, in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objective. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objective. 

Page 55 GAO-08-896 DOD Business Systems Modernization 
Appendix II: Comments from the Department 

of Defense 

Page 56 GAO-08-896 DOD Business Systems Modernization 

Appendix II: Comments from the Department 

of Defense 
Appendix II: Comments from the Department 

of Defense 

Page 57 GAO-08-896 DOD Business Systems Modernization 
Appendix II: Comments from the Department 

of Defense 

Page 58 GAO-08-896 DOD Business Systems Modernization 
Appendix II: Comments from the Department 

of Defense 

Page 59 GAO-08-896 DOD Business Systems Modernization 
Appendix II: Comments from the Department 

of Defense 

Page 60 GAO-08-896 DOD Business Systems Modernization 
Appendix II: Comments from the Department 

of Defense 

Page 61 GAO-08-896 DOD Business Systems Modernization 
Appendix III: GAO

St

 Contact and 

aff Acknowledgments

Page 62 GAO-08-896 DOD Business Systems Modernization 

Appendix III: GAO Contact and Staff 

Acknowledgments 

Randolph C. Hite, (202) 512-3439, or hiter@gao.gov

In addition to the individual named above, key contributors to this report 

were Neelaxi Lakhmani, Assistant Director; Monica Anatalio; Harold 

Brumm; Neil Doherty; Cheryl Dottermusch; Nancy Glover; Mustafa 

Hassan; Michael Holland; Ethan Iczkovitz; Anh Le; Josh Leiling; Emily 

Longcore; Lee McCracken; Madhav Panwar; Karen Richey; Melissa 

Schermerhorn; Karl Seifert; Sushmita Srikanth; Jonathan Ticehurst; and 

Adam Vodraska. 

GAO Contact 

Staff 

Acknowledgments 

(310659) 
GAO’s Mission The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To

have GAO e-mail you a list of newly posted products every afternoon, go 

to www.gao.gov and select “E-mail Updates.” 

The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, DC 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, DC 20548 

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548 

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Mail or Phone 

To Report Fraud,

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Public Affairs 

PRINTED ON RECYCLED PAPER