WEAPON SYSTEM 

REQUIREMENTS

Detailed Systems 

Engineering Prior to 

Product Development 

Positions Programs 

for Success

Report to Congressional Committees

November 2016

GAO-17-77

United States Government Accountability Office
United States Government Accountability Office

Highlights of GAO-17-77, a report to 

congressional committees

November 2016

WEAPON SYSTEM REQUIREMENTS

Detailed Systems Engineering Prior to Product 

Development Positions Programs for Success

Why GAO Did This Study

Cost and schedule growth in DOD 

major defense acquisition programs 

persist, and some acquisition reform 

proponents believe such growth is due 

to unplanned changes in program 

requirements (commonly referred to as 

"requirements creep"). GAO found in 

June 2015 that cost and schedule 

growth are often more directly related 

to a lack of systems engineering, 

which, if done, would reduce risk by

introducing discipline and rigor into the 

process of defining and understanding 

a program's initial requirements.

House Armed Services Committee 

Report 114-102 contained a provision 

for GAO to review the DOD 

requirements process. This report (1) 

identifies a framework for assessing 

the challenge posed by weapon 

system requirements and the extent of 

systems engineering done before 

product development begins; (2)

illustrates the relationship between

systems engineering and program

outcomes; and (3) assesses

implications for program oversight.

GAO analyzed a non-generalizable 

sample of nine case studies. GAO

assessed the extent to which systems 

engineering was conducted before 

development by reviewing program 

requirements and analyzing cost and 

schedule documentation for each case 

study. GAO also reviewed prior GAO 

work and interviewed DOD officials.

What GAO Recommends

To support oversight and inform 

budget decisions, Congress should 

consider requiring DOD to report on 

systems engineering status of each 

major acquisition program in the 

department’s annual budget request.

What GAO Found

GAO’s analysis of nine case studies identified four factors that frame the 

challenge posed by a given weapon system’s requirements: acquisition 

approach, technology status, design maturity, and system interdependency. 

Systems engineering is the primary means for determining whether and how that 

challenge can be met. It is a disciplined learning process that translates 

requirements into specific design features and thus identifies key risks to be 

resolved. GAO’s prior best practices work has found that if detailed systems 

engineering is done early, a program can resolve such risks through trade-offs 

and additional investments before a program starts. A key point in systems 

engineering where this match can be assessed is the preliminary design. As

shown below, establishing a preliminary design through early detailed systems 

engineering portends better program outcomes than doing so after program start.

Timing of Systems Engineering for Problematic and Successful Programs 

GAO’s analysis of selected Department of Defense (DOD) programs illustrates

the relationship among the four factors, systems engineering, and program 

outcomes. Programs with modest requirements and early detailed systems 

engineering had better outcomes. For example, the Small Diameter Bomb 

Increment I program, which delivered within cost and schedule estimates, had an 

incremental approach, mature technologies, a derivative design, and detailed 

systems engineering before development began. Programs that began with more 

challenging requirements and insufficient systems engineering reported worse 

outcomes. For example, the F-35 Lightning II, which has encountered significant 

cost and schedule problems, began development with a single-step approach, a 

highly complex design, immature technologies, and little systems engineering. 

Understanding the dynamic between a program’s requirements, risks, and the 

requisite systems engineering effort has important implications for oversight. A 

particular challenge is that Congress often must consider requests to authorize 

and fund a new program in advance of the start of product development, when 

the business case would be better established. DOD policy requires that DOD 

decision makers have information about a proposed program’s risk factors and 

systems engineering status, in a systems engineering plan, at the start of a new 

program. However, it is not clear whether Congress also has this information at 

that time. A systems engineering plan could provide more robust information to 

Congress when considering a budget request to start a new program. In 

commenting on a draft of this report DOD disagreed.

View GAO-17-77. For more information, 

contact Michael J. Sullivan at (202) 512-4841 

or sullivanm@gao.gov.
Page i GAO-17-77 DOD Systems Engineering

Letter 1

Background 3

Four Factors Frame the Challenge Posed by Requirements, and 

Detailed Systems Engineering Is Key to Knowing Whether and 

How the Challenge Can Be Met 7

Case Studies Illustrate the Relationship between Requirements 

Challenge, the Timing of Systems Engineering, and Program 

Outcomes 12

Knowledge of Systems Engineering Can Enhance Oversight 

before a Program Starts Product Development 25

Conclusions 29

Matter for Congressional Consideration 30

Agency Comments and Our Evaluation 30

Appendix I Objectives, Scope and Methodology 34

Appendix II Comments from the Department of Defense 37

Appendix III GAO Contact and Staff Acknowledgments 39

Tables

Table 1: Case Study Programs 2

Table 2: Development Cost and Schedule Outcomes of Nine Case 

Study Programs (then-year dollars in millions) 12

Table 3: Case Study Programs 34

Figures

Figure 1: Requirements Decomposition 5

Figure 2: Requirements and Systems Engineering 6

Figure 3: Timing of Systems Engineering for Successful and 

Problematic Programs 9

Figure 4: Requirements and Systems Engineering for Problematic 

and Successful Product Development Start 10

Figure 5: KC-46 Tanker Modernization Program 14

Figure 6: Small Diameter Bomb Increment I 15

Figure 7: Joint Light Tactical Vehicle 16

Contents
Page ii GAO-17-77 DOD Systems Engineering

Figure 8: Global Positioning System III 17

Figure 9: Paladin Integrated Management 19

Figure 10: P-8A Poseidon Multi-Mission Maritime Aircraft 

Increment I 20

Figure 11: F-35 Lightning II 22

Figure 12: CH-53K Heavy Lift Replacement Helicopter 23

Figure 13: Integrated Air and Missile Defense 24

Figure 14: Key Systems Engineering Stages Needed for Sound 

Business Case 28

Abbreviations 

CH-53K CH-53K Heavy Lift Replacement Helicopter

DOD Department of Defense 

F-35 F-35 Lightning II

GPS III Global Positioning System III

IAMD Integrated Air and Missile Defense 

JLTV Joint Light Tactical Vehicle

KC-46A KC-46A Tanker Modernization

MGUE Military GPS User Equipment

OCX Next Generation Operational Control System

P-8A Poseidon Multi-Mission Maritime Aircraft Increment l

PIM Paladin Integrated Management

SDB I Small Diameter Bomb Increment I

TRL Technology Readiness Level

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-17-77 DOD Systems Engineering

441 G St. N.W.

Washington, DC 20548

November 17, 2016

Congressional Committees

The Department of Defense (DOD) expects to have invested a total of 

more than $1.4 trillion in development and procurement of the major 

defense acquisition programs in its current portfolio. This portfolio has 

experienced cost growth of 48 percent since first full estimates and 

average delays in delivering initial capabilities of almost 30 months. Some 

acquisition reform proponents believe that unplanned changes in program 

requirements—”requirements creep”—after the Joint Requirements 

Oversight Council1 has approved the requirements and handed them off 

to the acquisition community are a primary cause of these ongoing cost 

and schedule problems in many programs. In June 2015, we found that 

cost and schedule growth in major acquisition programs were not 

necessarily a direct result of requirements changes, but were instead 

more directly related to a lack of discipline and rigor in the process of 

defining and understanding a program’s initial requirements.2

House Armed Services Committee Report 114-102 contained a provision 

for us to review the DOD requirements process.3 This report (1) identifies 

a framework for assessing the challenges and risks posed by 

requirements and the extent of systems engineering done before product 

development begins; (2) illustrates the relationship between systems 

engineering and program outcomes; and (3) assesses implications for 

program oversight.

To conduct our work, we selected a non-generalizable sample of nine 

major defense acquisition programs, made up of various types of systems 

from each military service, to conduct case study assessments. The 

specific case study programs we selected are identified in table 1.

 1 The Joint Requirements Oversight Council, which is chaired by the Vice Chairman of the 

Joint Chiefs of Staff and is comprised of the Vice Chiefs of Staff of each military service 

and the Combatant Commanders, has the mission and responsibility to identify, assess, 

and approve joint military requirements including existing systems and equipment to meet 

the National Military Strategy.

2 GAO, Defense Acquisition Process: Military Service Chiefs’ Concerns Reflect Need to 

Better Define Requirements before Programs Start, GAO-15-469 (Washington, D.C.: June 

11, 2015).

3 H.R. Rep. No. 114-102, at 184-185 (May 5, 2015)

Letter
Page 2 GAO-17-77 DOD Systems Engineering

Table 1: Case Study Programs

Program Description

KC-46A Tanker Modernization Program Aerial refueling tanker aircraft

Global Positioning System III Satellites

Small Diameter Bomb Increment I Air-to-ground precision munitions

Integrated Air and Missile Defense Network of sensors and weapons with a 

common battle command system 

Paladin Integrated Management Self-propelled howitzer and tracked 

ammunition carrier 

F-35 Lightning II Program Stealthy, strike fighter aircraft

Joint Light Tactical Vehicle Tactical wheeled vehicles

CH-53K Heavy Lift Replacement Helicopter Personnel and equipment transport aircraft

P-8A Poseidon Multi-mission Maritime 

Aircraft Increment I

Anti-surface, anti-submarine and 

intelligence, surveillance and 

reconnaissance aircraft

Source: GAO analysis of DOD data. | GAO-17-77

To assess the challenges and risks associated with program 

requirements and the extent of the systems engineering done by each of 

the nine programs before development began, we analyzed top-level 

program requirements such as key performance parameters, key system 

attributes, and other system attributes, as well as lower-level, derived 

requirements like system design specifications and design drawings. In 

addition, we assessed documents and data from program system 

engineering reviews and reported technology readiness levels, and 

reviewed program acquisition strategies, acquisition baselines, and 

selected acquisition reports. We reviewed cost, schedule, and 

requirements data and interviewed agency officials knowledgeable about 

the data. We determined that the data were sufficiently reliable for the 

purposes of our reporting objectives. To assess the relationship between 

systems engineering and program outcomes, we analyzed requirements, 

cost, and schedule documentation for each of our case study programs, 

and then met with relevant program officials and prime contractors to 

obtain relevant program documents, data, and historical information. To 

identify program oversight implications, we also reviewed relevant 

acquisition statutes, DOD acquisition policy, systems engineering 

guidance, and previous GAO reports examining weapons systems 

acquisitions and best practices for product development. To obtain 

additional insights into these implications, we spoke with knowledgeable 

DOD officials and the prime contractors for our case study programs. See 

appendix I for additional information on our objectives, scope, and 

methodology.
Page 3 GAO-17-77 DOD Systems Engineering

We conducted this performance audit from June 2015 to November 2016 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.

In our past work examining weapon system acquisitions and best 

practices for product development, we have found that leading 

commercial firms and successful DOD programs pursue an acquisition 

approach that is anchored in knowledge, whereby high levels of product 

knowledge are demonstrated at critical points in the acquisition process. 

The first key point in this knowledge-based process occurs when a match 

is achieved between the customer’s needs or requirements and available 

resources—including technology, design, time, and funding. That match 

establishes a sound basis for a program business case and supports the 

decision to invest in product development. Achieving a match between 

requirements and resources prior to committing to a product development 

program reduces risk and uncertainty and sets the program up for 

success. We have previously found that key enablers of a good business 

case include:

• Firm, feasible requirements: Requirements should be clearly defined, 

affordable, and clearly informed—thus tempered—by systems 

engineering. Once programs begin, requirements should not change 

without assessing their potential disruption to the program.

• Mature technology: Science and technology organizations should 

shoulder the technology development burden, proving technologies 

can work as intended before they are included in a weapon system 

program. The principle is not to avoid technical risk but rather take risk 

early and resolve it ahead of the start of product development.

• Incremental, knowledge-based acquisition strategy: Rigorous systems 

engineering coupled with more achievable requirements are essential 

to achieve faster delivery of needed capability to the warfighter. 

Building on mature technologies, such a strategy provides time, 

money, and other resources for a stable design, building and testing 

of prototypes, and demonstration of mature production processes.

• Realistic cost estimate: Sound cost estimates depend on a 

knowledge-based acquisition strategy, independent assessments, and 

sound methodologies.

Background
Page 4 GAO-17-77 DOD Systems Engineering

Major defense acquisition programs go through a series of phases as 

they progress from the identification of the top-level requirements for new 

capability, through initial planning of a solution, to product development, 

and finally production and deployment of a fielded system. Within DOD, 

the process of identifying and understanding requirements typically 

begins when a sponsor, usually a military service, submits an Initial 

Capabilities Document that identifies the existence of a capability gap; the 

operational risks associated with the gap; and a recommended solution or 

preferred set of solutions for filling the gap. Potential solutions are then 

assessed in an Analysis of Alternatives, system capabilities are chosen, 

and top-level capability or operational requirements are then defined in a 

draft Capability Development Document that goes through several stages 

of service- and DOD-level review and validation. After the top-level 

capability requirements are defined, they are then decomposed into more 

specific capability requirements, known as the performance specification, 

and then into a series of more detailed design requirements. 

Progressively more detailed descriptions of the system design are 

established in what systems engineering literature refers to as 

configuration baselines. Within DOD, the configuration baselines are 

called the “functional baseline” that details the system-level performance 

requirements; the “allocated baseline” that establishes a preliminary 

design and defines the subsystems and how they are to work together; 

and the “product baseline” that describes the final design and provides 

component-level details (see figure 1).
Page 5 GAO-17-77 DOD Systems Engineering

Figure 1: Requirements Decomposition

Systems engineering is a disciplined process to transform top-level 

capability requirements into detailed, lower-level design requirements that 

can be achieved with available resources. Programs are required to 

prepare a systems engineering plan as a management tool to guide 

systems engineering activities over the life of a program. According to 

DOD, the discipline of this process provides the control and traceability to 

develop solutions that meet customer needs. The systems engineering 

process can be depicted in a V-shape that starts when a capability gap is 

identified and a proposed solution for filling that gap is selected. Top-level 

capability requirements are then defined and decomposed into lower-level 

design requirements until a final design is established. As this process 

takes place, requirements become more specific and the number of 

requirements at each successive level grows. This growth can be 

exponential, with tens of thousands of detailed design requirements 

derived from a relatively small number of capability requirements. Figure 

2 illustrates the relationship between requirements decomposition and 

systems engineering.
Page 6 GAO-17-77 DOD Systems Engineering

Figure 2: Requirements and Systems Engineering

Note: Because this review focused on the requirements process that leads to a final design, the 

product, validated solution, and delivered capability portions of the systems engineering process that 

focus on developing, testing, and delivering a final product are not discussed.

As more detailed design requirements are identified, decision makers can 

make informed trades between the requirements and available resources, 

potentially achieving a match and establishing a sound basis for a 

program business case. As the requirements decomposition process 

takes place, engineering and design knowledge grows and risk 

decreases, leading to more realistic cost and schedule estimates and 

more predictable program outcomes.
Page 7 GAO-17-77 DOD Systems Engineering

Our analysis of nine selected DOD programs, supported by our previous 

work examining best practices for product development, identified four 

key factors that provide insight into the challenge posed by a system’s 

top-level capability requirements and the related risk. While they are not 

the only factors that could be considered, we found that they were 

prominent and observable early in our case study programs. Those four 

factors are:

• Acquisition approach - indicates whether a program intends to take an 

incremental, evolutionary approach or a single-step approach to meet 

all capability requirements. According to GAO best practices, an 

incremental approach reduces risk by developing and delivering a 

product in a series of enhanced interim capabilities until final 

capability is reached. In contrast, a single-step approach increases 

risk by attempting to deliver the final capability all at once without any 

interim capabilities.

• Technology status - indicates the extent to which the critical 

technologies for a proposed system are mature at the start of product 

development.4 According to GAO best practices, technologies are 

ready for inclusion in a product development program when they have 

been demonstrated in a realistic, operational environment (i.e., 

Technology Readiness Level (TRL) 7), proving that they can work as 

intended. Demonstrating technologies in an operational environment 

provides a higher level of technology understanding and reduces risk 

prior to starting product development. Beginning system development 

with less mature technologies (i.e., TRL lower than 7) increases 

program risk because unexpected technology challenges could 

significantly affect product design.

• Design maturity - indicates the extent to which a proposed system’s 

design is either derived from an existing commercial or DOD 

system—whether operational or prototype—or is new and 

unprecedented. According to GAO best practices, designs derived 

from existing systems, whether commercial or DOD, by nature have 

more knowledge available when product development begins, thus 

 4 DOD policy requires critical technologies to be assessed on a scale from 1 to 9 with 9 

being the most mature. Demonstration in a relevant environment is TRL 6 and 

demonstration in an operational environment is TRL 7. In addition, a major defense 

acquisition program generally may not receive approval for development start until the 

milestone decision authority certifies that the technology in the program has been 

demonstrated in a relevant environment. 10 U.S.C. § 2366b.

Four Factors Frame 

the Challenge Posed 

by Requirements, and 

Detailed Systems 

Engineering Is Key to 

Knowing Whether 

and How the 

Challenge Can Be 

Met
Page 8 GAO-17-77 DOD Systems Engineering

reducing risk. In contrast, unprecedented designs are by nature more 

complex and inherently higher risk.

• System interdependency - indicates the extent to which a system 

relies on another system or group of systems being developed and 

managed separately to provide its required capabilities. According to 

DOD acquisition guidance, although DOD programs should not be 

acquired in isolation, a program office developing a more independent 

system generally has more control over requirements, schedule, 

funding, and interfaces, among other factors. In contrast, a program 

office developing a system that is more dependent on other 

systems—like a system of systems—generally has less control, thus 

making its requirements more challenging to achieve and increasing 

risk.5 For example, systems of systems tend to be more complex and 

must manage (1) interfaces with other systems that have potentially 

unaligned performance requirements, (2) the need for collaborative 

governance with separately managed programs, and (3) differences in 

program phases of development.

As we discuss in the following pages, the case study programs with toplevel capability requirements that posed a more modest challenge took an 

incremental acquisition approach, incorporated more mature 

technologies, used a design that was derived from an existing product or 

prototype, and limited dependence on systems being developed and 

managed separately, reducing the amount of risk being carried forward 

into product development. In contrast, the programs with top-level 

capability requirements that were more challenging generally assumed 

full capability would be delivered in a single-step, incorporated immature 

critical technologies, used a new and unprecedented system design, and 

had full capability that in some cases was critically dependent on 

separately developed and managed systems without sufficiently 

mitigating risk before starting product development.

The fact that some requirements can be more challenging than others 

does not necessarily mean that problems are unavoidable. Rather, the 

ensuing systems engineering effort, if timely and sufficiently vigorous, can 

provide the needed confidence that the requirements are achievable. Our 

previous work in best practices has found that conducting detailed 

systems engineering analysis before starting product development 

contributes to understanding the requirements challenge and identifying 

 5 Department of Defense Acquisition Guidebook, Ch. 4, para. 4.1.3.
Page 9 GAO-17-77 DOD Systems Engineering

and mitigating associated risks.6 Systems engineering is the primary 

means for determining whether and how the challenge posed by a 

program’s requirements can be met with available resources. It is a 

disciplined learning process that translates capability requirements into 

specific design features and thus identifies key risks to be resolved. Our 

prior best practices work has indicated that if detailed systems 

engineering is done before the start of product development, the program 

can resolve these risks through trade-offs and additional investments, 

ensuring that risks have been sufficiently retired or that they are clearly 

understood and adequately resourced if they are being carried forward. 

Figure 3 depicts this relationship between the timing of systems 

engineering and the start of product development for successful and 

problematic programs.

Figure 3: Timing of Systems Engineering for Successful and Problematic Programs

In our case-study analysis, we found that systems engineering is a 

natural learning process that takes place at some point in all programs. 

 6 GAO, Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and 

Schedule Problems under DOD’s Revised Policy, GAO-06-368 (Washington, D.C.: Apr. 

13, 2006); Best Practices: Capturing Design and Manufacturing Knowledge Early 

Improves Acquisition Outcomes. GAO-02-701 (Washington, D.C.: July 15, 2002); Best 

Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System 

Outcomes, GAO-01-288 (Washington, D.C.: Mar. 8, 2001).
Page 10 GAO-17-77 DOD Systems Engineering

Our work in commercial best practices has found that, at a minimum, 

programs need to conduct systems engineering to decompose capability 

requirements into an allocated baseline—that is, establish a preliminary 

design—before beginning a product development program.7 When the 

bulk of the detailed systems engineering is done after starting a product 

development program it often results in cost and schedule growth and 

creates inefficiency. Figure 4 compares the points in the requirements 

and systems engineering processes at which problematic and successful 

programs would begin product development.

Figure 4: Requirements and Systems Engineering for Problematic and Successful Product Development Start

Within the current defense acquisition framework, DOD decision makers 

should assess a program’s risk factors and systems engineering progress 

well in advance of the start of product development. For example, DOD’s 

acquisition policy states that program requirements must be firm and 

 7 GAO-06-368, GAO-02-701, GAO-01-288.
Page 11 GAO-17-77 DOD Systems Engineering

clearly stated and that the risk of committing to development is or will be 

adequately reduced prior to awarding a development contract. The policy 

also emphasizes the value in establishing an allocated baseline—i.e. 

preliminary design—prior to the start of product development, which is a 

best practice that contributed to good outcomes in our case study 

programs. This practice was also emphasized by Congress in the 

Weapon System Acquisition Reform Act of 2009, which required 

programs to hold a preliminary design review or obtain a waiver prior to 

starting a development program.8

We have historically found that many of DOD’s major weapon system 

development programs tended to conduct the bulk of systems 

engineering after a development contract had been awarded and product 

development had begun.9 In many cases, the top-level capability 

requirements established by the government at that time still needed to 

be analyzed by the contractor and decomposed into a functional baseline 

(system-level design), an allocated baseline (preliminary design), and a 

product baseline (final design). As a result, those programs had started 

product development with a limited understanding of the challenge posed 

by requirements.

 8 Pub. L. No. 111-23, § 205(a) (May 22, 2009).

9 GAO, Defense Acquisition Process: Military Service Chiefs’ Concerns Reflect Need to 

Better Define Requirements before Programs Start, GAO-15-469 (Washington, D.C.: June 

11, 2015); Defense Acquisitions: Assessments of Selected Weapon Programs, 

GAO-12-400SP (Washington, D.C.: Mar. 29, 2012); Defense Acquisitions: Assessments 

of Selected Weapon Programs, GAO-09-326SP (Washington, D.C.: Mar. 30, 2009); and 

Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon 

System Outcomes, GAO-01-288 (Washington, D.C.: Mar. 8, 2001).
Page 12 GAO-17-77 DOD Systems Engineering

Our case studies illustrate the relationship among program outcomes, the 

four key factors that frame the challenge posed by requirements, and 

timely systems engineering. Our analysis of development cost and 

schedule data for the nine case studies shows that, as of December 

2015, about half of the programs had reported relatively good 

development cost and schedule outcomes, while the others had 

experienced significant cost and schedule growth (see table 2). Of the 

programs we studied, the three that began development with more 

modest requirements and had conducted detailed systems engineering 

generally had good outcomes. The three programs with some 

requirements challenges that conducted systems engineering analysis to 

mitigate associated risks experienced moderate cost and schedule 

outcomes. Finally, the three programs in our sample that began 

development with challenging requirements and had done little systems 

engineering generally reported poor outcomes.

Table 2: Development Cost and Schedule Outcomes of Nine Case Study Programs 

(then-year dollars in millions)

Program

Initial 

estimate 

Current 

estimate

Percent 

change 

Acquisition cycle time 

growth since initial 

estimates (in months)

KC-46A Tanker 

Modernization Program 

$7,149.6 $6,259.6 -12% 14

Joint Light Tactical 

Vehicle 

$1,009.8 $948.9 -6% 19

Small Diameter Bomb 

Increment I 

$381.3 $367.7 -4% -1

Paladin Integrated 

Management/M109A7 

Family of Vehicles

$1,041.7 $1,098.6 5% 2

P-8A Poseidon Multimission Maritime Aircraft 

Increment I

$6,975.5 $7,940.4 14% 4

Global Positioning 

System III

$2,512.0 $3,018.6 20% n/a

CH-53K Heavy Lift 

Replacement Helicopter

$4,366.4 $6,598.3 51% 51

F-35 Lightning II Program $34,400.0 $55,133.0 60% 62

Integrated Air and Missile 

Defense

$1,672.5 $2,632.9 62% 22

Source: GAO analysis of DOD data. | GAO-17-77

Note: Acquisition cycle time is calculated from the start of product development to initial operational 

capability. We could not calculate acquisition cycle times for the first increment of the Global 

Positioning System III program because initial operational capability will not occur until satellites from 

Case Studies 

Illustrate the 

Relationship between 

Requirements 

Challenge, the Timing 

of Systems 

Engineering, and 

Program Outcomes
Page 13 GAO-17-77 DOD Systems Engineering

a future increment are fielded. For the P-8A Increment I current estimate, we used the P-8A budget 

estimate from February 2016 to separate increment l cost from increment ll.

These results are consistent with our recent analysis of DOD’s overall 

major defense acquisition portfolio. In March 2016, we reported that while 

program outcomes across the portfolio remained mixed, DOD’s newer 

programs were reporting better cost outcomes.10 We found that some of 

those programs had taken steps to retire risks through systems 

engineering before starting product development and thus began with a 

clearer understanding of the challenge posed by their requirements.

The KC-46A Tanker Modernization (KC-46), Small Diameter Bomb 

Increment I (SDB I), and Joint Light Tactical Vehicle (JLTV) programs 

provide examples in which top-level requirements generally posed 

moderate challenges, steps were taken to identify and retire risks through 

detailed systems engineering, which enabled development of sound 

business cases before product development began. In these cases, the 

government and the prime contractor conducted systems engineering to 

decompose the requirements and identify an allocated baseline by the 

time they started product development. As a result, the programs 

established cost and schedule baselines that were well informed, 

contributing to relatively good outcomes.

 10 GAO, Defense Acquisitions: Assessments of Selected Weapons Programs. 

GAO-16-329SP (Washington, D.C.: Mar. 31, 2016).

Three More Successful 

Programs Had Modest 

Requirements and 

Conducted Detailed 

Systems Engineering 

Analysis before 

Development Start
Page 14 GAO-17-77 DOD Systems Engineering

Our analysis of the KC-46 requirements challenge and the ensuing 

systems engineering effort is summarized in figure 5.

Figure 5: KC-46 Tanker Modernization Program

Officials from the Air Force and the KC-46 prime contractor told us that at 

the start of product development in 2011, they had conducted detailed 

systems engineering and the system’s critical technologies were nearing 

maturity. The KC-46 was designed to improve on the KC-135’s refueling 

capacity, efficiency, and capabilities for cargo and aeromedical 

evacuation, and to integrate defensive systems. The Air Force planned an 

incremental acquisition approach focused on integrating mature military 

technologies onto a commercial aircraft derivative design. Although the 

government did not hold a preliminary design review until after 

development began, prime contractor officials noted that they had 

identified the allocated baseline as part of their internal risk reduction 

efforts before the development contract was awarded. As a result the KC46 program took steps that enabled a sound business case. Because KC46 development was considered by the government to be a relatively lowrisk effort, the prime contractor was awarded a fixed price incentive 

contract at the start of product development, which mitigated risk to the 

government.11 The prime contractor subsequently discovered problems 

with the aircraft wiring and aerial refueling systems and encountered a 

fuel contamination incident, all of which contributed to a 14-month delay 

 11 The KC-46 fixed-price incentive contract for product development limits the 

government’s financial liability and provides the contractor incentives to reduce costs in 

order to earn more profit. Barring any changes to KC-46 requirements by the Air Force, 

the contract specifies a target price of $4.4 billion and a ceiling price of $4.9 billion, at 

which point Boeing must assume responsibility for all additional costs.

KC-46 Tanker Modernization 

Program
Page 15 GAO-17-77 DOD Systems Engineering

in the delivery of initial operating capability. However, the government’s 

development costs have decreased by 12 percent largely because the 

contract is fixed price and the government’s initial cost estimate assumed 

a greater number of engineering changes than have actually occurred, 

indicating that the Air Force understood that development challenges 

remained and reflected that understanding in its initial cost estimates.

Our analysis of the SDB I requirements challenge and the ensuing 

systems engineering effort is summarized in figure 6.

Figure 6: Small Diameter Bomb Increment I

At the start of product development in 2003, the Air Force and the SDB I 

prime contractor had performed detailed systems engineering to 

understand the challenge posed by requirements and remaining risks. 

The SDB was designed to meet a pressing Air Force need for a lowcollateral damage weapon small enough to maximize the number that can 

be carried aboard a single aircraft. The Air Force planned an incremental 

acquisition approach to the overall SDB program, with SDB I—the first 

increment—using mature technology and based on a design derived from 

competitive prototypes developed and tested by the prime contractor. 

According to program officials, all of the system’s critical technologies 

were mature and they had production representative hardware when 

product development started. The Air Force and competing contractors 

conducted systems engineering over the course of 2 years prior to 

development to refine requirements in light of available resources. During 

that time live fire testing was conducted and the Air Force held 

preliminary and critical design reviews—key systems engineering 

events—with each contractor. According to prime contractor officials, their 

interactions with the Air Force prior to the start of development allowed 

them to get a clear understanding of design requirements. As a result, 

Small Diameter Bomb 

Increment I
Page 16 GAO-17-77 DOD Systems Engineering

they were able to achieve a stable design and provide a production 

representative system at the start of product development. With all of the 

detailed systems engineering done before development started, risks 

were understood. The Air Force had a sound business case that reflected 

a realistic understanding of the challenge facing the SDB I development 

program. The program’s total development cost decreased by almost 4 

percent and the system was available for use 1 month earlier than initially 

estimated.

Our analysis of the JLTV requirements challenge and the ensuing 

systems engineering effort is summarized in figure 7.

Figure 7: Joint Light Tactical Vehicle

At the start of product development in 2012, the Army and Marine Corps 

and the JLTV prime contractor had performed detailed systems 

engineering to understand the challenge posed by requirements and 

remaining risks. The JLTV was expected to be a family of vehicles built 

on a common automotive vehicle platform to replace the High Mobility 

Multipurpose Wheeled Vehicle for some missions. In response to 

concerns raised by DOD’s acquisition executive regarding technical 

maturity, shifting requirements, and affordability, and in light of available 

resources, the Army and Marine Corps worked with three contractors 

prior to the start of product development to refine requirements. The Army 

and Marine Corps planned an incremental acquisition approach using 

mature technology and based on a design derived from prototypes 

developed and tested by the winning contractor prior to the start of 

product development. Basic capability would be provided initially, with 

enhanced force protection, increased fuel efficiency, greater payload, and 

other improvements to be added in later increments. As a result, the 

prime contractor was able to identify an allocated baseline and provide a 

Joint Light Tactical Vehicle
Page 17 GAO-17-77 DOD Systems Engineering

mature product based on a demonstrated design at the start of 

development. With the bulk of the systems engineering done before 

development started, the JLTV program had taken steps to reduce risk 

and enable a sound business case that reflected a realistic understanding 

of the challenge it faced. The program has decreased development costs 

by 6 percent. Although acquisition cycle time has increased 19 months 

from initial estimates, program officials noted that the schedule growth 

was due to issues unrelated to systems engineering or program 

requirements, specifically production contract bid protests that delayed 

operational testing and the need to add time for Army users to complete 

critical training.

The Global Positioning System III (GPS III), Paladin Integrated 

Management (PIM), and P-8A Poseidon Multi-Mission Maritime Aircraft 

(P-8A) programs provide examples in which top-level requirements 

generally posed moderate challenges with some risks that were identified 

and generally mitigated through systems engineering. In these cases, the 

government and the prime contractor conducted some detailed systems 

engineering to inform their program business cases before starting 

product development but not all requirements risks were mitigated, which 

contributed to problems and some cost and schedule growth.

Our analysis of the GPS III requirements challenge and the ensuing 

systems engineering effort is summarized in figure 8.

Figure 8: Global Positioning System III

Although the Air Force and GPS III prime contractor conducted some 

detailed systems engineering to inform the program’s business case 

before the start of development in May 2008, additional risks to fielding 

Three Programs Had 

Some Requirements 

Challenges but Conducted 

Systems Engineering to 

Support Risk Mitigation

Global Positioning System III
Page 18 GAO-17-77 DOD Systems Engineering

the capability existed due to interdependencies associated with a critical 

ground control system and military user equipment that were being 

developed and managed as separate programs. The GPS III program is 

expected to develop and field a new generation of satellites to 

supplement and eventually replace currently operational GPS satellites. 

As the space segment of DOD’s effort to sustain and modernize the GPS 

system, the GPS III program is one part of a system of systems, which 

also includes the Next Generation Operational Control System (OCX), 

intended to replace the existing GPS ground system and to operate both 

current and future satellites, and the Military GPS User Equipment 

(MGUE), intended to provide the military services with new GPS 

receivers. In order to avoid problems associated with a previous GPS 

satellite program, the Air Force planned an incremental acquisition 

approach to GPS III, using technologies that were assessed as mature 

and performed some systems engineering to gain an early understanding 

of the challenge posed by requirements. According to contractor officials, 

this systems engineering, conducted during the competitive requirements 

development phase, helped them better understand design requirements. 

Regardless of these efforts, the program was negatively impacted by 

dependency on other systems.

Design and manufacturing problems with a key GPS III navigation 

subsystem delayed the program by almost 2 years; however, the 

satellites’ dependence on the much delayed OCX proved to be a greater 

challenge to sustaining and modernizing the GPS system. The delivery of 

the OCX Block 1, which is required to operate the GPS III satellites with 

both legacy and new capabilities, has been delayed by 6 years to July 

2021 and may be delayed further because of ongoing developmental 

issues. As a result of the OCX program’s delays, the Air Force is pursuing 

a smaller scale program to modify the existing GPS operational control 

system to enable the operational use of GPS III satellites for all legacy 

GPS functions until delivery of the OCX Block 1, which will then permit 

the operational testing and use of the GPS III satellites’ new capabilities. 

Due to OCX delays and the likely timing of the contingency operations 

system the Air Force may need to delay the launch of multiple GPS III 

satellites or launch several without fully testing them. Moreover, although 

testing the satellites’ functionality is not dependent on new user 

equipment, timing of MGUE capability delivery will further postpone—by 

about a decade—the warfighter’s ability to take advantage of the 

upgraded system’s new military code, which offers greater resistance to 

jamming.
Page 19 GAO-17-77 DOD Systems Engineering

Our analysis of the PIM requirements challenge and the ensuing systems 

engineering effort is summarized in figure 9.

Figure 9: Paladin Integrated Management

The Army’s Paladin Integrated Management (PIM) program—officially the 

M109A7 Family of Vehicles—began formal product development as a 

major defense acquisition program in June 2011. 12 The Army began 

early systems engineering and development work in 2007. At that time, 

PIM was a service life extension program for the existing Paladin M109A6 

and according to program officials, was expected to be a relatively minor 

upgrade to update the fire control system and replace the power pack of 

the existing system. The Army expected these requirements to neither 

result in a significant or complicated design change, nor be costly. As part 

of the service life extension program, the Army had identified both 

functional and allocated baselines. By 2011, the Army had made several 

changes to the program, with significant implications for the design and 

cost of PIM. Specifically, the Army increased force protection and 

survivability requirements, as directed by DOD, and added a cannon that 

had been in development as part of the Army’s Future Combat Systems 

program before it was cancelled in 2009. These changes essentially 

resulted in a new design versus an upgrade, and the associated costs 

increased enough to make PIM a major defense acquisition program.

 12 Major defense acquisition programs are those identified by DOD with a dollar value for 

all increments estimated to require eventual total expenditure for research, development, 

test, and evaluation of more than $480 million, or for procurement of more than $2.79 

billion, in fiscal year 2014 constant dollars. See DOD Instruction 5000.02, Operation of 

Defense Acquisition System, enclosure 1, table 1 (Jan. 7, 2015).

Paladin Integrated 

Management
Page 20 GAO-17-77 DOD Systems Engineering

The program’s functional and allocated baselines were re-established in 

2012, reflecting both the design changes and the systems engineering 

done since 2007, and ensured cost and schedule estimates were 

sufficiently informed for a sound business case. Since 2011, development 

cost has increased 5 percent and the program has experienced a 

schedule delay of 2 months. While not a textbook example of how to keep 

requirements stable during a program, PIM’s experience shows that 

changes can be accommodated when accompanied by a suitable 

systems engineering effort.

Our analysis of the P-8A Increment I requirements challenge and the 

ensuing systems engineering effort is summarized in figure 10.

Figure 10: P-8A Poseidon Multi-Mission Maritime Aircraft Increment I

At the start of P-8A development in 2004, the Navy and the prime 

contractor had completed some detailed systems engineering but an 

allocated baseline was not yet established. The P-8A is expected to 

replace an existing system, the P-3C Orion, and meet aspects of the 

Navy’s anti-submarine warfare, anti-surface warfare, and intelligence, 

surveillance, and reconnaissance capability requirements. The Navy 

planned an incremental, evolutionary acquisition approach focused on 

providing a first increment of capability to users in the quickest and most 

cost efficient manner. The Navy also expected P-8A Increment I to be 

built on open systems architecture to allow subsequent increments to 

more easily deliver increased capabilities.13 In addition, the airframe was 

 13 An open system allows system components to be added, removed, modified, replaced, 

or sustained by consumers or different manufacturers in addition to the manufacturer that 

developed the system. It also allows independent suppliers to build components that can 

plug into the existing system through the open connections.

P-8A Poseidon Multi-Mission 

Maritime Aircraft Increment I
Page 21 GAO-17-77 DOD Systems Engineering

to be derived from an existing commercial aircraft. According to prime 

contractor officials, the P-8A airframe design was about 50 percent 

common with the commercial airframe from which it was derived. At the 

start of development, none of P-8A Increment I’s four critical technologies 

were mature, increasing risk associated with the program’s business 

case. However, as part of its systems engineering analysis, the Navy and 

the prime contractor identified mature backup technologies that could be 

used if the critical technologies did not mature as expected thus mitigating 

some of the technology risk. P-8A Increment I experienced some cost 

and schedule growth. That growth is largely due to delays in the 

completion of the final system design. As of February 2016, the program’s 

development cost estimate had increased 14 percent, and its acquisition 

cycle time had increased by 4 months.

The F-35 Lightning II (F-35) and CH-53K Heavy Lift Replacement 

Helicopter (CH-53K) did not conduct adequate systems engineering prior 

to starting product development, and as result began development with 

significant risks and a limited understanding of the challenge posed by 

their requirements. Neither program had established a functional or 

allocated baseline before development began. In fact, their allocated 

baselines were not established until years into product development. The 

government and prime contractor for the Integrated Air and Missile 

Defense (IAMD) program did some systems engineering before the start 

of product development, but the Army made high-level changes during 

the first year of development to integrate additional systems, 

necessitating additional detailed systems engineering work after the start 

of development.

Three Programs with 

Problematic Outcomes 

Had Challenging 

Requirements and Limited 

Systems Engineering 

before Development Start
Page 22 GAO-17-77 DOD Systems Engineering

Our analysis of the F-35 requirements challenge and the ensuing systems 

engineering effort is summarized in figure 11.

Figure 11: F-35 Lightning II

At the start of development in 2001, neither DOD nor the F-35 prime 

contractor had conducted detailed systems engineering to adequately 

retire risk, establish an allocated baseline, and truly understand the 

challenge posed by their requirements. As a result, the F-35 program did 

not have a sound, executable business case. The F-35 was a next 

generation, highly complex stealth fighter with integrated avionics and 

software intensive mission systems being developed in three variants to 

meet the needs of three U.S. military services and various international 

partners. When development began, DOD planned to achieve full 

capability using a single-step acquisition approach and the F-35’s critical 

technologies were immature. DOD’s initial program plans showed that 

key systems engineering analyses to support an allocated baseline would 

not be complete until the program was years into development.

The bulk of F-35 systems engineering was done after development began 

and the program experienced significant cost and schedule growth—

development costs increased 60 percent over initial estimates and initial 

operational capability was delayed over 5 years—and was restructured 

three times. The first restructuring was driven by the discovery of 

significant weight issues with the F-35B variant as the program developed 

its preliminary design, resulting in costly design changes. DOD and prime 

contractor data indicate that the system’s functional baseline—detailing 

the system-level design—was not finalized until 4 years into development. 

As systems engineering continued and testing began, additional 

discoveries were made, requiring further design changes and contributing 

to the two additional restructurings. Ultimately, the program realized that 

F-35 Lightning II
Page 23 GAO-17-77 DOD Systems Engineering

some capabilities originally envisioned when development began would 

not work as expected and would have to be completed as part of followon development program.

Our analysis of the CH-53K requirements challenge and the ensuing 

systems engineering effort is summarized in figure 12.

Figure 12: CH-53K Heavy Lift Replacement Helicopter

At the start of development in 2005, neither the Marine Corps nor the CH53K prime contractor had conducted detailed systems engineering to 

adequately retire risk, establish an allocated baseline, and fully 

understand the challenge posed by their requirements. They had 

established neither a performance specification nor a functional baseline. 

As a result, the CH-53K program did not have a sound, executable 

business case. The Marine Corps and the prime contractor disagreed on 

what specific systems engineering tasks were needed, and initial plans 

showed that key analyses would not be complete until the system was 

well into development. The Marine Corps expected the CH-53K to provide 

improvements relative to the CH-53E in range and payload, survivability 

and force protection, reliability and maintainability, coordination with other 

assets, and overall cost of ownership. As a result and in order to meet 

requirements, an entirely new aircraft design was needed. According to 

the prime contractor, none of the parts in the new design were common 

parts with the CH-53E. When development began, the Marine Corps 

planned to achieve full capability using a single-step acquisition 

approach. At that same time, the system’s critical technologies were not 

mature.

Nearly all CH-53K systems engineering was done after development 

began, and the program experienced significant cost and schedule 

CH-53K Heavy Lift 

Replacement Helicopter
Page 24 GAO-17-77 DOD Systems Engineering

growth. Because the program office and prime contractor disagreed on 

what systems engineering tasks needed to be accomplished, problems 

began immediately following the start of development. Once systems 

engineering began and problems were identified, the Marine Corps chose 

to defer some key performance capabilities to reduce development costs, 

and, in 2010, the program was restructured with a new baseline approved 

in 2013. Discoveries during ground testing and qualification drove 

additional unanticipated design changes and delays. As a result, the 

program’s initial production decision was delayed 8 months, and the 

program now plans to establish a new cost baseline. In total, the 

program’s development costs have increased 51 percent over initial 

estimates, and its initial operational capability has been delayed by over 4 

years.

Our analysis of the IAMD requirements challenge and the ensuing 

systems engineering effort is summarized in figure 13.

Figure 13: Integrated Air and Missile Defense

Although the Army and IAMD prime contractor had conducted some 

detailed systems engineering and technologies were assessed as nearly 

mature at the start of product development, the IAMD program’s business 

case was critically dependent on sensors and weapons that had been 

developed and managed as separate programs. IAMD is expected to 

network sensors, weapons, and a common battle command system 

across an integrated fire control network to support the engagement of air 

and missile threats. The Army planned an incremental acquisition 

approach and held a competition for preliminary design to mitigate risk in 

May 2009. Prime contractor officials told us they conducted systems 

engineering prior to the start of development, which may have helped 

them better understand design requirements. While a functional baseline 

Integrated Air and Missile 

Defense
Page 25 GAO-17-77 DOD Systems Engineering

was established prior to the start of product development and the prime 

contractor built a basic design prototype to demonstrate for the 

government that the design architecture would work, the system’s 

allocated baseline was not fully established until after the start of 

development.

System interdependency has posed challenges to the IAMD program in 

several ways. The Army made high-level changes in IAMD during the first 

year, which resulted in the program having to integrate additional systems 

and causing significant changes in how the systems were to interact. 

According to prime contractor officials, three of the four IAMD critical 

technologies rely on interfacing with other systems and many of these 

interfaces were not defined until after the development contract was 

awarded. IAMD has encountered challenges integrating new software 

from the contractor with other acquisition programs critical to its 

functionality. IAMD development cost has increased over 60 percent 

since the start of development, with about half of the increase occurring in 

the first year. The program has been rebaselined and initial operational 

capability has been deferred by 2 years. This experience is somewhat 

similar to that of the GPS III with an important distinction. While 

challenges with GPS III’s interdependence with other systems did not 

significantly affect the design of the satellites themselves, the 

interdependence of IAMD did have significant implications for its software 

design, which were not recognized until after the start of product 

development.

Systems engineering, if done well, can be a key enabler for establishing a 

sound business case for a program before the start of product 

development. By the same token, it can provide indications to Congress 

and other organizations responsible for oversight as to the soundness of 

DOD’s approach to undertaking a new program. The decision to begin 

product development is the last opportunity in the acquisition process—

prior to committing to a major financial investment—to level a product’s 

requirements to available resources, thereby establishing a good 

business case. A sound business case provides credible evidence that 

(1) the warfighter’s needs are valid and that they can best be met with the 

chosen concept; and (2) the chosen concept can be developed and 

produced within existing resources—that is, proven technologies, 

sufficient design knowledge, adequate funding, and adequate time exist 

to deliver the product when it is needed. A program should not go forward 

into product development, nor should this step be funded, unless a sound 

business case can be made. Therefore, Congress, which is responsible 

Knowledge of 

Systems Engineering 

Can Enhance 

Oversight before a 

Program Starts 

Product Development
Page 26 GAO-17-77 DOD Systems Engineering

for making that funding decision, should also have insight into the 

soundness of the business case. However, given current budget 

processes and mechanics in DOD today, these congressional decisions 

must be made well in advance of the start of product development and 

establishment of the final business case.

Systems engineering is essential to establishing a sound business case. 

We have previously found that key enablers of a good business case 

include firm, feasible requirements, mature technology, an incremental, 

knowledge-based acquisition strategy, and a realistic cost estimate. Our 

work examining commercial companies’ best practices has found that 

requirements should be clearly defined, affordable, and clearly 

informed—thus tempered—by systems engineering prior to the start of 

product development. Once programs begin, requirements should not 

change without assessing their potential disruption to the program. In 

addition, science and technology organizations should shoulder the 

technology development burden, proving technologies can work as 

intended before they are included in a weapon system program. The 

principle is not to avoid technical risk, but rather take risk early and 

resolve it ahead of the start of product development. Programs should 

use an incremental, knowledge-based acquisition strategy. We have 

found that rigorous systems engineering coupled with more achievable 

requirements are essential to achieve faster delivery of needed capability 

to the warfighter. Building on mature technologies, such a strategy 

provides time, money, and other resources for a stable design, building 

and testing of prototypes, and demonstration of mature production 

processes. Finally, a good business case will have a realistic cost 

estimate based on a knowledge-based acquisition strategy, independent 

assessments, and sound methodologies.

Over the years, and as previously noted in this report, we have found that 

for a program to deliver a successful product with available resources, 

high levels of knowledge—informed by systems engineering—must be 

demonstrated before significant commitments are made.14 This requires 

the user and developer to negotiate whatever trade-offs are needed to 

achieve a match between the user’s requirements and the developer’s 

resources before product development begins. While it would seem that 

taking such an approach would be axiomatic, we have found that it is 

 14GAO-15-469, GAO-12-400SP, GAO-09-326SP, GAO-02-701, and GAO-01-288.
Page 27 GAO-17-77 DOD Systems Engineering

not.15 On the contrary, we have previously found that there are strong 

incentives within the culture of weapon system acquisition to overpromise 

a prospective weapon’s performance while understating its likely cost and 

schedule demands.16 Competition with other programs vying for defense 

dollars puts pressure on program sponsors to project unprecedented 

levels of performance (often by counting on unproven technologies) while 

promising low cost and short schedules. These incentives, coupled with a 

marketplace that is characterized by a single buyer (DOD), low volume, 

and limited number of major sources, create a culture in weapon system 

acquisition that encourages undue optimism about program risks and 

costs.

A particular challenge for Congress is the fact that committees must often 

consider requests to authorize and fund a new program well ahead of the 

start of product development—the point at which key business case 

information would be presented. For example, if DOD has scheduled a 

decision to start a new development program in August 2017, the funding 

needed for the first year of development would have to be included in 

DOD’s fiscal year 2017 budget request. This budget request would be 

submitted to Congress in February 2016, or 18 months before the actual 

program decision. Given this time lag, Congress could be making critical 

funding decisions—which in effect authorize the start of development—

with limited information about program risk factors, systems engineering 

progress, and the soundness of the program’s business case.

The nine case studies we examined for this report suggest that 

understanding the dynamic between program requirements, risks, and the 

requisite systems engineering analysis can facilitate early oversight.

Specifically, when the top-level requirements for what could be a new 

major weapon system are initially identified in a draft Capability 

Development Document, the challenge those requirements pose and how 

it relates to the four factors we identified—acquisition approach, 

technology status, design maturity, and system interdependence—can be 

observed. Once the challenge is framed, the systems engineering plan 

 15 GAO, Defense Acquisitions: Assessments of Selected Weapons Programs, 

GAO-16-329SP (Washington, D.C.: Mar. 31, 2016); Defense Acquisitions: Assessments 

of Selected Weapons Programs, GAO-15-342SP (Washington, D.C.: Mar. 12, 2015); and 

Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule 

Problems under DOD’s Revised Policy, GAO-06-368 (Washington, D.C.: Apr. 13, 2006).

16 GAO, Defense Acquisitions: Joint Action Needed by DOD and Congress to Improve 

Outcomes, GAO-16-187T (Washington, D.C.: Oct. 27, 2015).
Page 28 GAO-17-77 DOD Systems Engineering

can be refined to obtain the requisite knowledge to (1) identify expectation 

gaps and other risks, and (2) resolve them by the start of product 

development. Ultimately, the final Capability Development Document 

could then be informed by an allocated baseline (the preliminary design) 

and provide a sound basis for an executable business case. This 

sequence of events is notionally depicted in figure 14 along with when a 

budget to start a new program is submitted to Congress.

Figure 14: Key Systems Engineering Stages Needed for Sound Business Case

While congressional insight at the time of the funding decision is limited, 

DOD policy directs program managers to provide certain documents to 

DOD decision makers that when taken together could provide a picture of 

a proposed program’s risk factors and systems engineering status. 

Programs are generally required to provide versions of these documents 

to DOD decision makers prior to each major milestone review, including 

the start of development. However, it is not clear whether Congress is 

given the same information DOD decision makers are given when DOD 

submits its budget request for funding to begin a new program. Those 

documents include an acquisition strategy, a test and evaluation master 

plan, a technology readiness assessment, and a systems engineering 

plan.

The systems engineering plan is of particular interest because it brings 

the risk factors that we have identified together with a proposed 

program’s systems engineering knowledge. According to DOD acquisition 

policy, the plan will be submitted for approval for each milestone review, 

beginning with the milestone review to approve the start of the technology 

maturation and risk reduction phase—the acquisition phase that precedes 

the start of product development. DOD views the plan as a “living 

document” that it expects to be updated as needed throughout the 

acquisition process. The plan is expected to support the acquisition 

strategy, including the program interdependencies, and communicate the 

overall technical approach to balance system performance, life-cycle cost, 
Page 29 GAO-17-77 DOD Systems Engineering

and risk in addressing warfighter needs. It should describe the program’s 

overall technical approach, including key technical risks, processes, 

resources, organization, metrics, and design considerations. It should 

also detail the timing and criteria for conducting systems engineering 

technical reviews, including those that will establish the functional, 

allocated, and product level baselines. In addition, the plan should 

address system integration risks, including risks related to external 

dependencies which are outside the program manager’s span of control 

in order to ensure timely design, development, deployment, and 

sustainment of the system. Key aspects of the plan are intended to 

support more detailed technical planning in order to provide effective 

management and control of the program’s technical progress and the 

execution of risk mitigation activities.

If the key factors that frame the requirements challenge are understood, a 

functional baseline is established, and a realistic plan exists to complete 

the allocated baseline, Congress will have more assurance that a 

program is on a path to a sound business case—informed by an allocated 

baseline—as it considers funding requests for development start, and a 

systems engineering plan could help to pull these elements together.

Weapon system development programs involve unknowns that translate 

into cost and schedule risk. This is the nature of any product development 

program. The case studies in this report illustrate a strong relationship 

between requirements, systems engineering, and program outcomes. 

They show that one key to avoid poor outcomes in major weapon system 

development programs is not necessarily to eliminate all risks, but instead 

to invest in detailed systems engineering analysis to understand the 

design implications of requirements and reconcile them with the 

resources that can be reasonably expected. At a minimum, this 

reconciliation, or match, should be evidenced by an allocated baseline—

that is, a preliminary design—before committing to a product development 

program. Our cases indicate that regardless of the challenge posed by 

initial requirements, systems engineering, if done well, can be a key 

enabler of a sound business case for starting a new development 

program. This helps to ensure that if a program takes risks, those risks 

are clearly identified, and any resource consequences are acknowledged 

and accounted for in cost and schedule estimates. In some cases—as 

seen with JLTV and PIM—this process could take time and result in 

requirements changes, but, if done before committing to a product 

development, it is more likely to result in a sound and executable 

business case.

Conclusions
Page 30 GAO-17-77 DOD Systems Engineering

Understanding the dynamic between program requirements, risks, and 

systems engineering can facilitate early oversight—thus offering a 

potential curb to overpromising. When the top-level requirements for what 

could be a new major weapon system are initially identified in a draft 

Capability Development Document, DOD should clearly understand the 

challenge presented by those requirements as framed by the four factors 

we identified. Having a clear understanding of the challenge presented by 

the initial requirements, coupled with a systems engineering plan laid out 

against a schedule to achieve a sound, executable business case before

the start of product development would provide Congress and other 

oversight organizations a means to assess the soundness of a proposed 

acquisition program. Importantly, it would be possible to assess progress 

made by the program against the plan, using the identification of key 

systems engineering baselines as waypoints. As progress is made, 

artifacts of knowledge—in terms of the systems engineering baselines as

well as decisions made to trade off requirements, choose alternate design 

or technology solutions, and to add time and money to close gaps—

should also be available. While Congress would still likely be in the 

position of considering a budget request to start a program well before 

DOD’s actual decision to start product development, the systems 

engineering plan and progress made against it would provide more robust 

input to Congress’s deliberations. This could also provide Congress with 

better insight into the risks facing a proposed development program—

including the risks a program may be taking after the start of product 

development in the event that the desired level of systems engineering is 

not complete.

To enhance program oversight and provide more robust input to budget 

deliberations, Congress should consider requiring DOD to report on each 

major acquisition program’s systems engineering status in the 

department’s annual budget request, beginning with the budget 

requesting funds to start development. The information could be 

presented on a simple timeline—as done for the case studies in this 

report—and at a minimum should reflect the status of a program’s

functional and allocated baselines as contained in the most current 

version of the program’s systems engineering plan. 

We provided a draft of this report to DOD for review and comment. DOD’s

written comments are reprinted in appendix II of this report. That draft 

report included a recommendation that DOD submit the systems 

engineering plans of each new proposed development program to 

Matter for 

Congressional 

Consideration

Agency Comments 

and Our Evaluation
Page 31 GAO-17-77 DOD Systems Engineering

Congress at the same time the budget requesting funds to begin 

development is sent to Congress. DOD did not agree with our 

recommendation. In its comments, DOD agreed that early systems 

engineering reduces risk and establishes a solid foundation for program 

success. However, DOD noted that the systems engineering plan is a 

management tool to guide a program’s system engineering activities and 

the timing of the plan and any updates are not aligned to inform a budget 

decision that could occur as much as 18 months prior to program 

initiation. DOD also noted that it believes that existing statutory 

certifications and reports submitted to Congress contain adequate 

information regarding program risk and technical maturity. We appreciate 

DOD’s thoughtful response and agree that existing statutory certifications 

and reports provide some level of insight into program risks and general 

assurance that plans are in place to address those risks. However, it is 

not clear whether they provide adequate detail about progress against 

those plans, such as whether a functional baseline has been established 

or when an allocated baseline will be established. In addition, those 

certifications and reports are linked to program milestone reviews and are 

not specifically aligned to inform program budget decisions. Documents 

such as the systems engineering plan are “living documents” that are 

updated as needed throughout the acquisition process and could be 

made available to inform the budget process. Therefore, we continue to 

believe that linking robust insight into systems engineering progress, like 

the information contained in the systems engineering plan, with the timing 

of congressional budget deliberations would be beneficial and as such 

are now including a matter for congressional consideration. Given that the 

systems engineering plan is updated during the acquisition process, 

linking systems engineering progress to budget requests need not be 

onerous. As noted above, this could be accomplished using a top-level 

timeline of less than a page, as done in the case studies in this report.

We are sending copies of this report to interested congressional 

committees and offices; the Secretary of Defense; the Secretaries of the 

Army, Navy, and Air Force. In addition, the report will be made available 

at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff has any questions concerning this report, please 

contact me at (202) 512-4841. Contact points for our offices of 

Congressional Relations and Public Affairs may be found on the last page
Page 32 GAO-17-77 DOD Systems Engineering

of this report. Staff members making key contributions to this report are 

listed in appendix III.

Michael J. Sullivan

Director, Acquisition and Sourcing Management
Page 33 GAO-17-77 DOD Systems Engineering

List of Committees

The Honorable John McCain

Chairman

The Honorable Jack Reed

Ranking Member

Committee on Armed Services 

United States Senate

The Honorable Thad Cochran

Chairman

The Honorable Richard Durbin

Ranking Member

Subcommittee on Defense

Committee on Appropriations

United States Senate

The Honorable Mac Thornberry

Chairman

The Honorable Adam Smith

Ranking Member

Committee on Armed Services

House of Representatives

The Honorable Rodney Frelinghuysen

Chairman

The Honorable Pete Visclosky

Ranking Member

Subcommittee on Defense

Committee on Appropriations

House of Representatives
Appendix I: Objectives, Scope and 

Methodology

Page 34 GAO-17-77 DOD Systems Engineering

This report (1) identifies a framework for assessing the challenges and 

risks associated with program requirements and the extent of systems 

engineering done before product development begins; (2) illustrates the 

relationship between systems engineering and program outcomes; and 

(3) assesses implications for program management and oversight.

To conduct our work, we selected a non-generalizable sample of 9 major 

defense acquisition programs, out of a total portfolio of 79 programs, to 

include at least 2 from each service and Department of Defense (DOD) 

wide programs. The specific case study programs we selected are 

identified in table 1.

Table 3: Case Study Programs

Lead Service Program Description

Air Force KC-46A Tanker Modernization Program Aerial refueling tanker aircraft

Global Positioning System III Satellites

Small Diameter Bomb Increment I Air-to-ground precision munitions

Army Integrated Air and Missile Defense Network of sensors and weapons with a 

common battle command system 

Paladin Integrated Management Self-propelled howitzer and tracked 

ammunition carrier 

DOD F-35 Lightning II Program Stealthy, strike fighter aircraft

Joint Light Tactical Vehicle Tactical wheeled vehicles

Navy/Marine Corps CH-53K Heavy Lift Replacement Helicopter Personnel and equipment transport aircraft

P-8A Poseidon Multi-mission Maritime Aircraft 

Increment I

Anti-surface, anti-submarine and 

intelligence, surveillance and 

reconnaissance aircraft

Source: GAO analysis of DOD data | GAO-17-77

To assess the challenges and risks associated with program 

requirements and the extent of systems engineering done by each of the 

9 programs before development began, we analyzed top-level program 

requirements such as key performance parameters, key system 

attributes, and other system attributes, as well as lower-level derived 

requirements like system design specifications and design drawings. For 

each case study program, we requested and analyzed the number of 

requirements identified for each baseline (functional, allocated, and 

product level) at the time of system engineering reviews and program 

milestones and compared the results to DOD systems engineering 

guidance, DOD acquisition policy, and our previous GAO reports 

Appendix I: Objectives, Scope and 

Methodology
Appendix I: Objectives, Scope and 

Methodology

Page 35 GAO-17-77 DOD Systems Engineering

examining weapons systems acquisitions and best practices for product 

development.1 To ensure our understanding of commercial best practices 

remained current, we interviewed officials from leading companies 

including Caterpillar, Inc., Cummins, Inc., Proctor and Gamble, and 

Motorola Solutions. In addition, we assessed documents and data from 

system engineering reviews and program reported technology readiness 

levels; and we reviewed program acquisition strategies, capability 

development documents or operational requirements documents, 

acquisition baselines, and selected acquisition reports.

To assess the relationship between systems engineering and program 

outcomes, we analyzed requirements, cost, and schedule documentation 

for each of our case study programs, and then met with relevant program 

officials and prime contractors to obtain relevant program documents, 

data, and historical information. We reviewed selected acquisition and 

Defense Acquisition Executive Summary reports, budget data, DOD 

Systems Engineering assessments of preliminary and critical design 

reviews, and previous GAO reports such as our annual reviews of major 

defense acquisition programs and program specific reports.2

To identify program oversight implications, we also reviewed relevant 

acquisition statutes, DOD acquisition policy, systems engineering 

guidance, and previous GAO reports examining weapons systems 

 1 GAO, Defense Acquisition Process: Military Service Chiefs’ Concerns Reflect Need to 

Better Define Requirements before Programs Start, GAO-15-469 (Washington, D.C.: June 

11, 2015); Defense Acquisitions: Major Weapon Systems Continue to Experience Cost 

and Schedule Problems under DOD’s Revised Policy, GAO-06-368 (Washington, D.C.: 

Apr. 13, 2006); Best Practices: Capturing Design and Manufacturing Knowledge Early 

Improves Acquisition Outcomes. GAO-02-701 (Washington, D.C.: July 15, 2002); and 

Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon 

System Outcomes, GAO-01-288 (Washington, D.C.: Mar. 8, 2001).

2 GAO, Defense Acquisitions: Assessments of Selected Weapon Programs 

GAO-16-329SP (Washington, D.C.: Mar. 31, 2016); F-35 Joint Strike Fighter: Continued 

Oversight Needed as Program Plans to Begin Development of New Capabilities

GAO-16-390 (Washington, D.C.: Apr. 14, 2016); KC-46 Tanker Aircraft: Challenging 

Testing and Delivery Schedules Lie Ahead GAO-16-346 (Washington, D.C.: Apr. 8, 2016); 

Defense Acquisitions: CH-53K Helicopter Program Has Addressed Early Difficulties and 

Adopted Strategies to Address Future Risks GAO-11-332 (Washington, D.C.: Apr. 4, 

2011); Defense Acquisitions: Issues to Be Considered as DOD Modernizes Its Fleet of 

Tactical Wheeled Vehicles GAO-11-83 (Washington, D.C.: Nov. 5, 2010); Global 

Positioning System: Challenges in Sustaining and Upgrading Capabilities Persist

GAO-10-636 (Washington, D.C.: Sept. 15, 2010); and Defense Acquisitions: Strong 

Leadership Is Key to Planning and Executing Stable Weapon Program GAO-10-522

(Washington, D.C.: May 6, 2010).
Appendix I: Objectives, Scope and 

Methodology

Page 36 GAO-17-77 DOD Systems Engineering

acquisitions and best practices for product development. To obtain 

additional insights into these implications, we spoke with knowledgeable 

DOD officials including program managers and prime contractors for our 

case study programs. We assessed the reliability of DOD and contractor 

data by reviewing existing information about the data and interviewing 

agency officials knowledgeable about the data. We spoke to both 

program and prime contractor officials for each of the case study 

programs and determined that the data were sufficiently reliable for the 

purposes of our reporting objectives.

We conducted this performance audit from June 2015 to November 2016 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.
Appendix II: Comments from the Department 

of Defense

Page 37 GAO-17-77 DOD Systems Engineering

Appendix II: Comments from the Department 

of Defense
Appendix II: Comments from the Department 

of Defense

Page 38 GAO-17-77 DOD Systems Engineering
Appendix III: GAO Contact and Staff 

Acknowledgments

Page 39 GAO-17-77 DOD Systems Engineering

Michael J. Sullivan, 202-512-4841 or sullivanm@gao.gov

In addition to the contact named above, Travis Masters (Assistant 

Director); Peter Anderson; Marvin Bonner; Robert Bullock; Kurt Gurka; 

Scott Hiromoto; Jean McSween; Robert Miller; LeAnna Parkey; and 

Roxanna T. Sun made key contributions to this report.

Appendix III: GAO Contact and Staff 

Acknowledgments

GAO Contact

Acknowledgments

(100167)
The Government Accountability Office, the audit, evaluation, and investigative 

arm of Congress, exists to support Congress in meeting its constitutional 

responsibilities and to help improve the performance and accountability of the 

federal government for the American people. GAO examines the use of public 

funds; evaluates federal programs and policies; and provides analyses, 

recommendations, and other assistance to help Congress make informed 

oversight, policy, and funding decisions. GAO’s commitment to good government 

is reflected in its core values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through GAO’s website (http://www.gao.gov). Each weekday afternoon, GAO 

posts on its website newly released reports, testimony, and correspondence. To 

have GAO e-mail you a list of newly posted products, go to http://www.gao.gov

and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of production and 

distribution and depends on the number of pages in the publication and whether 

the publication is printed in color or black and white. Pricing and ordering 

information is posted on GAO’s website, http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, MasterCard, 

Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.

Visit GAO on the web at www.gao.gov.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400, 

U.S. Government Accountability Office, 441 G Street NW, Room 7125, 

Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707

U.S. Government Accountability Office, 441 G Street NW, Room 7814, 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Strategic Planning and 

External Liaison

Please Print on Recycled Paper.