Weapon System Requirements: Detailed Systems Engineering Prior to Product Development Positions Programs for Success
Released: 2016-11-17T00:00:00

Letter

Background

In our past work examining weapon system acquisitions and best practices for product development, we have found that leading commercial firms and successful DOD programs pursue an acquisition approach that is anchored in knowledge, whereby high levels of product knowledge are demonstrated at critical points in the acquisition process. The first key point in this knowledge-based process occurs when a match is achieved between the customer’s needs or requirements and available resources—including technology, design, time, and funding. That match establishes a sound basis for a program business case and supports the decision to invest in product development. Achieving a match between requirements and resources prior to committing to a product development program reduces risk and uncertainty and sets the program up for success. We have previously found that key enablers of a good business case include:

Firm, feasible requirements: Requirements should be clearly defined, affordable, and clearly informed—thus tempered—by systems engineering. Once programs begin, requirements should not change without assessing their potential disruption to the program.

Mature technology: Science and technology organizations should shoulder the technology development burden, proving technologies can work as intended before they are included in a weapon system program. The principle is not to avoid technical risk but rather take risk early and resolve it ahead of the start of product development.

Incremental, knowledge-based acquisition strategy: Rigorous systems engineering coupled with more achievable requirements are essential to achieve faster delivery of needed capability to the warfighter. Building on mature technologies, such a strategy provides time, money, and other resources for a stable design, building and testing of prototypes, and demonstration of mature production processes.

Realistic cost estimate: Sound cost estimates depend on a knowledge-based acquisition strategy, independent assessments, and sound methodologies.

Major defense acquisition programs go through a series of phases as they progress from the identification of the top-level requirements for new capability, through initial planning of a solution, to product development, and finally production and deployment of a fielded system. Within DOD, the process of identifying and understanding requirements typically begins when a sponsor, usually a military service, submits an Initial Capabilities Document that identifies the existence of a capability gap; the operational risks associated with the gap; and a recommended solution or preferred set of solutions for filling the gap. Potential solutions are then assessed in an Analysis of Alternatives, system capabilities are chosen, and top-level capability or operational requirements are then defined in a draft Capability Development Document that goes through several stages of service- and DOD-level review and validation. After the top-level capability requirements are defined, they are then decomposed into more specific capability requirements, known as the performance specification, and then into a series of more detailed design requirements. Progressively more detailed descriptions of the system design are established in what systems engineering literature refers to as configuration baselines. Within DOD, the configuration baselines are called the “functional baseline” that details the system-level performance requirements; the “allocated baseline” that establishes a preliminary design and defines the subsystems and how they are to work together; and the “product baseline” that describes the final design and provides component-level details (see figure 1).

Systems engineering is a disciplined process to transform top-level capability requirements into detailed, lower-level design requirements that can be achieved with available resources. Programs are required to prepare a systems engineering plan as a management tool to guide systems engineering activities over the life of a program. According to DOD, the discipline of this process provides the control and traceability to develop solutions that meet customer needs. The systems engineering process can be depicted in a V-shape that starts when a capability gap is identified and a proposed solution for filling that gap is selected. Top-level capability requirements are then defined and decomposed into lower-level design requirements until a final design is established. As this process takes place, requirements become more specific and the number of requirements at each successive level grows. This growth can be exponential, with tens of thousands of detailed design requirements derived from a relatively small number of capability requirements. Figure 2 illustrates the relationship between requirements decomposition and systems engineering.

As more detailed design requirements are identified, decision makers can make informed trades between the requirements and available resources, potentially achieving a match and establishing a sound basis for a program business case. As the requirements decomposition process takes place, engineering and design knowledge grows and risk decreases, leading to more realistic cost and schedule estimates and more predictable program outcomes.

Four Factors Frame the Challenge Posed by Requirements, and Detailed Systems Engineering Is Key to Knowing Whether and How the Challenge Can Be Met

Our analysis of nine selected DOD programs, supported by our previous work examining best practices for product development, identified four key factors that provide insight into the challenge posed by a system’s top-level capability requirements and the related risk. While they are not the only factors that could be considered, we found that they were prominent and observable early in our case study programs. Those four factors are:

Acquisition approach - indicates whether a program intends to take an incremental, evolutionary approach or a single-step approach to meet all capability requirements. According to GAO best practices, an incremental approach reduces risk by developing and delivering a product in a series of enhanced interim capabilities until final capability is reached. In contrast, a single-step approach increases risk by attempting to deliver the final capability all at once without any interim capabilities.

Technology status - indicates the extent to which the critical technologies for a proposed system are mature at the start of product development. According to GAO best practices, technologies are ready for inclusion in a product development program when they have been demonstrated in a realistic, operational environment (i.e., Technology Readiness Level (TRL) 7), proving that they can work as intended. Demonstrating technologies in an operational environment provides a higher level of technology understanding and reduces risk prior to starting product development. Beginning system development with less mature technologies (i.e., TRL lower than 7) increases program risk because unexpected technology challenges could significantly affect product design.

Design maturity - indicates the extent to which a proposed system’s design is either derived from an existing commercial or DOD system—whether operational or prototype—or is new and unprecedented. According to GAO best practices, designs derived from existing systems, whether commercial or DOD, by nature have more knowledge available when product development begins, thus reducing risk. In contrast, unprecedented designs are by nature more complex and inherently higher risk.

System interdependency - indicates the extent to which a system relies on another system or group of systems being developed and managed separately to provide its required capabilities. According to DOD acquisition guidance, although DOD programs should not be acquired in isolation, a program office developing a more independent system generally has more control over requirements, schedule, funding, and interfaces, among other factors. In contrast, a program office developing a system that is more dependent on other systems—like a system of systems—generally has less control, thus making its requirements more challenging to achieve and increasing risk. For example, systems of systems tend to be more complex and must manage (1) interfaces with other systems that have potentially unaligned performance requirements, (2) the need for collaborative governance with separately managed programs, and (3) differences in program phases of development.

As we discuss in the following pages, the case study programs with top- level capability requirements that posed a more modest challenge took an incremental acquisition approach, incorporated more mature technologies, used a design that was derived from an existing product or prototype, and limited dependence on systems being developed and managed separately, reducing the amount of risk being carried forward into product development. In contrast, the programs with top-level capability requirements that were more challenging generally assumed full capability would be delivered in a single-step, incorporated immature critical technologies, used a new and unprecedented system design, and had full capability that in some cases was critically dependent on separately developed and managed systems without sufficiently mitigating risk before starting product development.

The fact that some requirements can be more challenging than others does not necessarily mean that problems are unavoidable. Rather, the ensuing systems engineering effort, if timely and sufficiently vigorous, can provide the needed confidence that the requirements are achievable. Our previous work in best practices has found that conducting detailed systems engineering analysis before starting product development contributes to understanding the requirements challenge and identifying and mitigating associated risks. Systems engineering is the primary means for determining whether and how the challenge posed by a program’s requirements can be met with available resources. It is a disciplined learning process that translates capability requirements into specific design features and thus identifies key risks to be resolved. Our prior best practices work has indicated that if detailed systems engineering is done before the start of product development, the program can resolve these risks through trade-offs and additional investments, ensuring that risks have been sufficiently retired or that they are clearly understood and adequately resourced if they are being carried forward. Figure 3 depicts this relationship between the timing of systems engineering and the start of product development for successful and problematic programs.

In our case-study analysis, we found that systems engineering is a natural learning process that takes place at some point in all programs.

Our work in commercial best practices has found that, at a minimum, programs need to conduct systems engineering to decompose capability requirements into an allocated baseline—that is, establish a preliminary design—before beginning a product development program. When the bulk of the detailed systems engineering is done after starting a product development program it often results in cost and schedule growth and creates inefficiency. Figure 4 compares the points in the requirements and systems engineering processes at which problematic and successful programs would begin product development.

Within the current defense acquisition framework, DOD decision makers should assess a program’s risk factors and systems engineering progress well in advance of the start of product development. For example, DOD’s acquisition policy states that program requirements must be firm and clearly stated and that the risk of committing to development is or will be adequately reduced prior to awarding a development contract. The policy also emphasizes the value in establishing an allocated baseline—i.e. preliminary design—prior to the start of product development, which is a best practice that contributed to good outcomes in our case study programs. This practice was also emphasized by Congress in the Weapon System Acquisition Reform Act of 2009, which required programs to hold a preliminary design review or obtain a waiver prior to starting a development program.

We have historically found that many of DOD’s major weapon system development programs tended to conduct the bulk of systems engineering after a development contract had been awarded and product development had begun. In many cases, the top-level capability requirements established by the government at that time still needed to be analyzed by the contractor and decomposed into a functional baseline (system-level design), an allocated baseline (preliminary design), and a product baseline (final design). As a result, those programs had started product development with a limited understanding of the challenge posed by requirements.

Case Studies Illustrate the Relationship between Requirements Challenge, the Timing of Systems Engineering, and Program Outcomes

Our case studies illustrate the relationship among program outcomes, the four key factors that frame the challenge posed by requirements, and timely systems engineering. Our analysis of development cost and schedule data for the nine case studies shows that, as of December 2015, about half of the programs had reported relatively good development cost and schedule outcomes, while the others had experienced significant cost and schedule growth (see table 2). Of the programs we studied, the three that began development with more modest requirements and had conducted detailed systems engineering generally had good outcomes. The three programs with some requirements challenges that conducted systems engineering analysis to mitigate associated risks experienced moderate cost and schedule outcomes. Finally, the three programs in our sample that began development with challenging requirements and had done little systems engineering generally reported poor outcomes.

These results are consistent with our recent analysis of DOD’s overall major defense acquisition portfolio. In March 2016, we reported that while program outcomes across the portfolio remained mixed, DOD’s newer programs were reporting better cost outcomes. We found that some of those programs had taken steps to retire risks through systems engineering before starting product development and thus began with a clearer understanding of the challenge posed by their requirements.

Three More Successful Programs Had Modest Requirements and Conducted Detailed Systems Engineering Analysis before Development Start

The KC-46A Tanker Modernization (KC-46), Small Diameter Bomb Increment I (SDB I), and Joint Light Tactical Vehicle (JLTV) programs provide examples in which top-level requirements generally posed moderate challenges, steps were taken to identify and retire risks through detailed systems engineering, which enabled development of sound business cases before product development began. In these cases, the government and the prime contractor conducted systems engineering to decompose the requirements and identify an allocated baseline by the time they started product development. As a result, the programs established cost and schedule baselines that were well informed, contributing to relatively good outcomes.

KC-46 Tanker Modernization Program

Our analysis of the KC-46 requirements challenge and the ensuing systems engineering effort is summarized in figure 5.

Officials from the Air Force and the KC-46 prime contractor told us that at the start of product development in 2011, they had conducted detailed systems engineering and the system’s critical technologies were nearing maturity. The KC-46 was designed to improve on the KC-135’s refueling capacity, efficiency, and capabilities for cargo and aeromedical evacuation, and to integrate defensive systems. The Air Force planned an incremental acquisition approach focused on integrating mature military technologies onto a commercial aircraft derivative design. Although the government did not hold a preliminary design review until after development began, prime contractor officials noted that they had identified the allocated baseline as part of their internal risk reduction efforts before the development contract was awarded. As a result the KC- 46 program took steps that enabled a sound business case. Because KC- 46 development was considered by the government to be a relatively low- risk effort, the prime contractor was awarded a fixed price incentive contract at the start of product development, which mitigated risk to the government. The prime contractor subsequently discovered problems with the aircraft wiring and aerial refueling systems and encountered a fuel contamination incident, all of which contributed to a 14-month delay in the delivery of initial operating capability. However, the government’s development costs have decreased by 12 percent largely because the contract is fixed price and the government’s initial cost estimate assumed a greater number of engineering changes than have actually occurred, indicating that the Air Force understood that development challenges remained and reflected that understanding in its initial cost estimates.

Small Diameter Bomb Increment I

Our analysis of the SDB I requirements challenge and the ensuing systems engineering effort is summarized in figure 6.

At the start of product development in 2003, the Air Force and the SDB I prime contractor had performed detailed systems engineering to understand the challenge posed by requirements and remaining risks. The SDB was designed to meet a pressing Air Force need for a low- collateral damage weapon small enough to maximize the number that can be carried aboard a single aircraft. The Air Force planned an incremental acquisition approach to the overall SDB program, with SDB I—the first increment—using mature technology and based on a design derived from competitive prototypes developed and tested by the prime contractor. According to program officials, all of the system’s critical technologies were mature and they had production representative hardware when product development started. The Air Force and competing contractors conducted systems engineering over the course of 2 years prior to development to refine requirements in light of available resources. During that time live fire testing was conducted and the Air Force held preliminary and critical design reviews—key systems engineering events—with each contractor. According to prime contractor officials, their interactions with the Air Force prior to the start of development allowed them to get a clear understanding of design requirements. As a result, they were able to achieve a stable design and provide a production representative system at the start of product development. With all of the detailed systems engineering done before development started, risks were understood. The Air Force had a sound business case that reflected a realistic understanding of the challenge facing the SDB I development program. The program’s total development cost decreased by almost 4 percent and the system was available for use 1 month earlier than initially estimated.

Joint Light Tactical Vehicle

Our analysis of the JLTV requirements challenge and the ensuing systems engineering effort is summarized in figure 7.

At the start of product development in 2012, the Army and Marine Corps and the JLTV prime contractor had performed detailed systems engineering to understand the challenge posed by requirements and remaining risks. The JLTV was expected to be a family of vehicles built on a common automotive vehicle platform to replace the High Mobility Multipurpose Wheeled Vehicle for some missions. In response to concerns raised by DOD’s acquisition executive regarding technical maturity, shifting requirements, and affordability, and in light of available resources, the Army and Marine Corps worked with three contractors prior to the start of product development to refine requirements. The Army and Marine Corps planned an incremental acquisition approach using mature technology and based on a design derived from prototypes developed and tested by the winning contractor prior to the start of product development. Basic capability would be provided initially, with enhanced force protection, increased fuel efficiency, greater payload, and other improvements to be added in later increments. As a result, the prime contractor was able to identify an allocated baseline and provide a mature product based on a demonstrated design at the start of development. With the bulk of the systems engineering done before development started, the JLTV program had taken steps to reduce risk and enable a sound business case that reflected a realistic understanding of the challenge it faced. The program has decreased development costs by 6 percent. Although acquisition cycle time has increased 19 months from initial estimates, program officials noted that the schedule growth was due to issues unrelated to systems engineering or program requirements, specifically production contract bid protests that delayed operational testing and the need to add time for Army users to complete critical training.

Three Programs Had Some Requirements Challenges but Conducted Systems Engineering to Support Risk Mitigation

The Global Positioning System III (GPS III), Paladin Integrated Management (PIM), and P-8A Poseidon Multi-Mission Maritime Aircraft (P-8A) programs provide examples in which top-level requirements generally posed moderate challenges with some risks that were identified and generally mitigated through systems engineering. In these cases, the government and the prime contractor conducted some detailed systems engineering to inform their program business cases before starting product development but not all requirements risks were mitigated, which contributed to problems and some cost and schedule growth.

Global Positioning System III

Our analysis of the GPS III requirements challenge and the ensuing systems engineering effort is summarized in figure 8.

Although the Air Force and GPS III prime contractor conducted some detailed systems engineering to inform the program’s business case before the start of development in May 2008, additional risks to fielding the capability existed due to interdependencies associated with a critical ground control system and military user equipment that were being developed and managed as separate programs. The GPS III program is expected to develop and field a new generation of satellites to supplement and eventually replace currently operational GPS satellites. As the space segment of DOD’s effort to sustain and modernize the GPS system, the GPS III program is one part of a system of systems, which also includes the Next Generation Operational Control System (OCX), intended to replace the existing GPS ground system and to operate both current and future satellites, and the Military GPS User Equipment (MGUE), intended to provide the military services with new GPS receivers. In order to avoid problems associated with a previous GPS satellite program, the Air Force planned an incremental acquisition approach to GPS III, using technologies that were assessed as mature and performed some systems engineering to gain an early understanding of the challenge posed by requirements. According to contractor officials, this systems engineering, conducted during the competitive requirements development phase, helped them better understand design requirements. Regardless of these efforts, the program was negatively impacted by dependency on other systems.

Design and manufacturing problems with a key GPS III navigation subsystem delayed the program by almost 2 years; however, the satellites’ dependence on the much delayed OCX proved to be a greater challenge to sustaining and modernizing the GPS system. The delivery of the OCX Block 1, which is required to operate the GPS III satellites with both legacy and new capabilities, has been delayed by 6 years to July 2021 and may be delayed further because of ongoing developmental issues. As a result of the OCX program’s delays, the Air Force is pursuing a smaller scale program to modify the existing GPS operational control system to enable the operational use of GPS III satellites for all legacy GPS functions until delivery of the OCX Block 1, which will then permit the operational testing and use of the GPS III satellites’ new capabilities. Due to OCX delays and the likely timing of the contingency operations system the Air Force may need to delay the launch of multiple GPS III satellites or launch several without fully testing them. Moreover, although testing the satellites’ functionality is not dependent on new user equipment, timing of MGUE capability delivery will further postpone—by about a decade—the warfighter’s ability to take advantage of the upgraded system’s new military code, which offers greater resistance to jamming.

Paladin Integrated Management

Our analysis of the PIM requirements challenge and the ensuing systems engineering effort is summarized in figure 9.

The Army’s Paladin Integrated Management (PIM) program—officially the M109A7 Family of Vehicles—began formal product development as a major defense acquisition program in June 2011. The Army began early systems engineering and development work in 2007. At that time, PIM was a service life extension program for the existing Paladin M109A6 and according to program officials, was expected to be a relatively minor upgrade to update the fire control system and replace the power pack of the existing system. The Army expected these requirements to neither result in a significant or complicated design change, nor be costly. As part of the service life extension program, the Army had identified both functional and allocated baselines. By 2011, the Army had made several changes to the program, with significant implications for the design and cost of PIM. Specifically, the Army increased force protection and survivability requirements, as directed by DOD, and added a cannon that had been in development as part of the Army’s Future Combat Systems program before it was cancelled in 2009. These changes essentially resulted in a new design versus an upgrade, and the associated costs increased enough to make PIM a major defense acquisition program.

The program’s functional and allocated baselines were re-established in 2012, reflecting both the design changes and the systems engineering done since 2007, and ensured cost and schedule estimates were sufficiently informed for a sound business case. Since 2011, development cost has increased 5 percent and the program has experienced a schedule delay of 2 months. While not a textbook example of how to keep requirements stable during a program, PIM’s experience shows that changes can be accommodated when accompanied by a suitable systems engineering effort.

P-8A Poseidon Multi-Mission Maritime Aircraft Increment I

Our analysis of the P-8A Increment I requirements challenge and the ensuing systems engineering effort is summarized in figure 10.

At the start of P-8A development in 2004, the Navy and the prime contractor had completed some detailed systems engineering but an allocated baseline was not yet established. The P-8A is expected to replace an existing system, the P-3C Orion, and meet aspects of the Navy’s anti-submarine warfare, anti-surface warfare, and intelligence, surveillance, and reconnaissance capability requirements. The Navy planned an incremental, evolutionary acquisition approach focused on providing a first increment of capability to users in the quickest and most cost efficient manner. The Navy also expected P-8A Increment I to be built on open systems architecture to allow subsequent increments to more easily deliver increased capabilities. In addition, the airframe was to be derived from an existing commercial aircraft. According to prime contractor officials, the P-8A airframe design was about 50 percent common with the commercial airframe from which it was derived. At the start of development, none of P-8A Increment I’s four critical technologies were mature, increasing risk associated with the program’s business case. However, as part of its systems engineering analysis, the Navy and the prime contractor identified mature backup technologies that could be used if the critical technologies did not mature as expected thus mitigating some of the technology risk. P-8A Increment I experienced some cost and schedule growth. That growth is largely due to delays in the completion of the final system design. As of February 2016, the program’s development cost estimate had increased 14 percent, and its acquisition cycle time had increased by 4 months.

Three Programs with Problematic Outcomes Had Challenging Requirements and Limited Systems Engineering before Development Start

The F-35 Lightning II (F-35) and CH-53K Heavy Lift Replacement Helicopter (CH-53K) did not conduct adequate systems engineering prior to starting product development, and as result began development with significant risks and a limited understanding of the challenge posed by their requirements. Neither program had established a functional or allocated baseline before development began. In fact, their allocated baselines were not established until years into product development. The government and prime contractor for the Integrated Air and Missile Defense (IAMD) program did some systems engineering before the start of product development, but the Army made high-level changes during the first year of development to integrate additional systems, necessitating additional detailed systems engineering work after the start of development.

F-35 Lightning II

Our analysis of the F-35 requirements challenge and the ensuing systems engineering effort is summarized in figure 11.

At the start of development in 2001, neither DOD nor the F-35 prime contractor had conducted detailed systems engineering to adequately retire risk, establish an allocated baseline, and truly understand the challenge posed by their requirements. As a result, the F-35 program did not have a sound, executable business case. The F-35 was a next generation, highly complex stealth fighter with integrated avionics and software intensive mission systems being developed in three variants to meet the needs of three U.S. military services and various international partners. When development began, DOD planned to achieve full capability using a single-step acquisition approach and the F-35’s critical technologies were immature. DOD’s initial program plans showed that key systems engineering analyses to support an allocated baseline would not be complete until the program was years into development.

The bulk of F-35 systems engineering was done after development began and the program experienced significant cost and schedule growth— development costs increased 60 percent over initial estimates and initial operational capability was delayed over 5 years—and was restructured three times. The first restructuring was driven by the discovery of significant weight issues with the F-35B variant as the program developed its preliminary design, resulting in costly design changes. DOD and prime contractor data indicate that the system’s functional baseline—detailing the system-level design—was not finalized until 4 years into development. As systems engineering continued and testing began, additional discoveries were made, requiring further design changes and contributing to the two additional restructurings. Ultimately, the program realized that some capabilities originally envisioned when development began would not work as expected and would have to be completed as part of follow- on development program.

CH-53K Heavy Lift Replacement Helicopter

Our analysis of the CH-53K requirements challenge and the ensuing systems engineering effort is summarized in figure 12.

At the start of development in 2005, neither the Marine Corps nor the CH- 53K prime contractor had conducted detailed systems engineering to adequately retire risk, establish an allocated baseline, and fully understand the challenge posed by their requirements. They had established neither a performance specification nor a functional baseline. As a result, the CH-53K program did not have a sound, executable business case. The Marine Corps and the prime contractor disagreed on what specific systems engineering tasks were needed, and initial plans showed that key analyses would not be complete until the system was well into development. The Marine Corps expected the CH-53K to provide improvements relative to the CH-53E in range and payload, survivability and force protection, reliability and maintainability, coordination with other assets, and overall cost of ownership. As a result and in order to meet requirements, an entirely new aircraft design was needed. According to the prime contractor, none of the parts in the new design were common parts with the CH-53E. When development began, the Marine Corps planned to achieve full capability using a single-step acquisition approach. At that same time, the system’s critical technologies were not mature.

Nearly all CH-53K systems engineering was done after development began, and the program experienced significant cost and schedule growth. Because the program office and prime contractor disagreed on what systems engineering tasks needed to be accomplished, problems began immediately following the start of development. Once systems engineering began and problems were identified, the Marine Corps chose to defer some key performance capabilities to reduce development costs, and, in 2010, the program was restructured with a new baseline approved in 2013. Discoveries during ground testing and qualification drove additional unanticipated design changes and delays. As a result, the program’s initial production decision was delayed 8 months, and the program now plans to establish a new cost baseline. In total, the program’s development costs have increased 51 percent over initial estimates, and its initial operational capability has been delayed by over 4 years.

Integrated Air and Missile Defense

Our analysis of the IAMD requirements challenge and the ensuing systems engineering effort is summarized in figure 13.

Although the Army and IAMD prime contractor had conducted some detailed systems engineering and technologies were assessed as nearly mature at the start of product development, the IAMD program’s business case was critically dependent on sensors and weapons that had been developed and managed as separate programs. IAMD is expected to network sensors, weapons, and a common battle command system across an integrated fire control network to support the engagement of air and missile threats. The Army planned an incremental acquisition approach and held a competition for preliminary design to mitigate risk in May 2009. Prime contractor officials told us they conducted systems engineering prior to the start of development, which may have helped them better understand design requirements. While a functional baseline was established prior to the start of product development and the prime contractor built a basic design prototype to demonstrate for the government that the design architecture would work, the system’s allocated baseline was not fully established until after the start of development.

System interdependency has posed challenges to the IAMD program in several ways. The Army made high-level changes in IAMD during the first year, which resulted in the program having to integrate additional systems and causing significant changes in how the systems were to interact. According to prime contractor officials, three of the four IAMD critical technologies rely on interfacing with other systems and many of these interfaces were not defined until after the development contract was awarded. IAMD has encountered challenges integrating new software from the contractor with other acquisition programs critical to its functionality. IAMD development cost has increased over 60 percent since the start of development, with about half of the increase occurring in the first year. The program has been rebaselined and initial operational capability has been deferred by 2 years. This experience is somewhat similar to that of the GPS III with an important distinction. While challenges with GPS III’s interdependence with other systems did not significantly affect the design of the satellites themselves, the interdependence of IAMD did have significant implications for its software design, which were not recognized until after the start of product development.

Knowledge of Systems Engineering Can Enhance Oversight before a Program Starts Product Development

Systems engineering, if done well, can be a key enabler for establishing a sound business case for a program before the start of product development. By the same token, it can provide indications to Congress and other organizations responsible for oversight as to the soundness of DOD’s approach to undertaking a new program. The decision to begin product development is the last opportunity in the acquisition process— prior to committing to a major financial investment—to level a product’s requirements to available resources, thereby establishing a good business case. A sound business case provides credible evidence that (1) the warfighter’s needs are valid and that they can best be met with the chosen concept; and (2) the chosen concept can be developed and produced within existing resources—that is, proven technologies, sufficient design knowledge, adequate funding, and adequate time exist to deliver the product when it is needed. A program should not go forward into product development, nor should this step be funded, unless a sound business case can be made. Therefore, Congress, which is responsible for making that funding decision, should also have insight into the soundness of the business case. However, given current budget processes and mechanics in DOD today, these congressional decisions must be made well in advance of the start of product development and establishment of the final business case.

Systems engineering is essential to establishing a sound business case. We have previously found that key enablers of a good business case include firm, feasible requirements, mature technology, an incremental, knowledge-based acquisition strategy, and a realistic cost estimate. Our work examining commercial companies’ best practices has found that requirements should be clearly defined, affordable, and clearly informed—thus tempered—by systems engineering prior to the start of product development. Once programs begin, requirements should not change without assessing their potential disruption to the program. In addition, science and technology organizations should shoulder the technology development burden, proving technologies can work as intended before they are included in a weapon system program. The principle is not to avoid technical risk, but rather take risk early and resolve it ahead of the start of product development. Programs should use an incremental, knowledge-based acquisition strategy. We have found that rigorous systems engineering coupled with more achievable requirements are essential to achieve faster delivery of needed capability to the warfighter. Building on mature technologies, such a strategy provides time, money, and other resources for a stable design, building and testing of prototypes, and demonstration of mature production processes. Finally, a good business case will have a realistic cost estimate based on a knowledge-based acquisition strategy, independent assessments, and sound methodologies.

Over the years, and as previously noted in this report, we have found that for a program to deliver a successful product with available resources, high levels of knowledge—informed by systems engineering—must be demonstrated before significant commitments are made. This requires the user and developer to negotiate whatever trade-offs are needed to achieve a match between the user’s requirements and the developer’s resources before product development begins. While it would seem that taking such an approach would be axiomatic, we have found that it is not. On the contrary, we have previously found that there are strong incentives within the culture of weapon system acquisition to overpromise a prospective weapon’s performance while understating its likely cost and schedule demands. Competition with other programs vying for defense dollars puts pressure on program sponsors to project unprecedented levels of performance (often by counting on unproven technologies) while promising low cost and short schedules. These incentives, coupled with a marketplace that is characterized by a single buyer (DOD), low volume, and limited number of major sources, create a culture in weapon system acquisition that encourages undue optimism about program risks and costs.

A particular challenge for Congress is the fact that committees must often consider requests to authorize and fund a new program well ahead of the start of product development—the point at which key business case information would be presented. For example, if DOD has scheduled a decision to start a new development program in August 2017, the funding needed for the first year of development would have to be included in DOD’s fiscal year 2017 budget request. This budget request would be submitted to Congress in February 2016, or 18 months before the actual program decision. Given this time lag, Congress could be making critical funding decisions—which in effect authorize the start of development— with limited information about program risk factors, systems engineering progress, and the soundness of the program’s business case.

The nine case studies we examined for this report suggest that understanding the dynamic between program requirements, risks, and the requisite systems engineering analysis can facilitate early oversight. Specifically, when the top-level requirements for what could be a new major weapon system are initially identified in a draft Capability Development Document, the challenge those requirements pose and how it relates to the four factors we identified—acquisition approach, technology status, design maturity, and system interdependence—can be observed. Once the challenge is framed, the systems engineering plan can be refined to obtain the requisite knowledge to (1) identify expectation gaps and other risks, and (2) resolve them by the start of product development. Ultimately, the final Capability Development Document could then be informed by an allocated baseline (the preliminary design) and provide a sound basis for an executable business case. This sequence of events is notionally depicted in figure 14 along with when a budget to start a new program is submitted to Congress.

While congressional insight at the time of the funding decision is limited, DOD policy directs program managers to provide certain documents to DOD decision makers that when taken together could provide a picture of a proposed program’s risk factors and systems engineering status. Programs are generally required to provide versions of these documents to DOD decision makers prior to each major milestone review, including the start of development. However, it is not clear whether Congress is given the same information DOD decision makers are given when DOD submits its budget request for funding to begin a new program. Those documents include an acquisition strategy, a test and evaluation master plan, a technology readiness assessment, and a systems engineering plan.

The systems engineering plan is of particular interest because it brings the risk factors that we have identified together with a proposed program’s systems engineering knowledge. According to DOD acquisition policy, the plan will be submitted for approval for each milestone review, beginning with the milestone review to approve the start of the technology maturation and risk reduction phase—the acquisition phase that precedes the start of product development. DOD views the plan as a “living document” that it expects to be updated as needed throughout the acquisition process. The plan is expected to support the acquisition strategy, including the program interdependencies, and communicate the overall technical approach to balance system performance, life-cycle cost, and risk in addressing warfighter needs. It should describe the program’s overall technical approach, including key technical risks, processes, resources, organization, metrics, and design considerations. It should also detail the timing and criteria for conducting systems engineering technical reviews, including those that will establish the functional, allocated, and product level baselines. In addition, the plan should address system integration risks, including risks related to external dependencies which are outside the program manager’s span of control in order to ensure timely design, development, deployment, and sustainment of the system. Key aspects of the plan are intended to support more detailed technical planning in order to provide effective management and control of the program’s technical progress and the execution of risk mitigation activities.

If the key factors that frame the requirements challenge are understood, a functional baseline is established, and a realistic plan exists to complete the allocated baseline, Congress will have more assurance that a program is on a path to a sound business case—informed by an allocated baseline—as it considers funding requests for development start, and a systems engineering plan could help to pull these elements together.

Conclusions

Weapon system development programs involve unknowns that translate into cost and schedule risk. This is the nature of any product development program. The case studies in this report illustrate a strong relationship between requirements, systems engineering, and program outcomes. They show that one key to avoid poor outcomes in major weapon system development programs is not necessarily to eliminate all risks, but instead to invest in detailed systems engineering analysis to understand the design implications of requirements and reconcile them with the resources that can be reasonably expected. At a minimum, this reconciliation, or match, should be evidenced by an allocated baseline— that is, a preliminary design—before committing to a product development program. Our cases indicate that regardless of the challenge posed by initial requirements, systems engineering, if done well, can be a key enabler of a sound business case for starting a new development program. This helps to ensure that if a program takes risks, those risks are clearly identified, and any resource consequences are acknowledged and accounted for in cost and schedule estimates. In some cases—as seen with JLTV and PIM—this process could take time and result in requirements changes, but, if done before committing to a product development, it is more likely to result in a sound and executable business case.

Understanding the dynamic between program requirements, risks, and systems engineering can facilitate early oversight—thus offering a potential curb to overpromising. When the top-level requirements for what could be a new major weapon system are initially identified in a draft Capability Development Document, DOD should clearly understand the challenge presented by those requirements as framed by the four factors we identified. Having a clear understanding of the challenge presented by the initial requirements, coupled with a systems engineering plan laid out against a schedule to achieve a sound, executable business case before the start of product development would provide Congress and other oversight organizations a means to assess the soundness of a proposed acquisition program. Importantly, it would be possible to assess progress made by the program against the plan, using the identification of key systems engineering baselines as waypoints. As progress is made, artifacts of knowledge—in terms of the systems engineering baselines as well as decisions made to trade off requirements, choose alternate design or technology solutions, and to add time and money to close gaps— should also be available. While Congress would still likely be in the position of considering a budget request to start a program well before DOD’s actual decision to start product development, the systems engineering plan and progress made against it would provide more robust input to Congress’s deliberations. This could also provide Congress with better insight into the risks facing a proposed development program— including the risks a program may be taking after the start of product development in the event that the desired level of systems engineering is not complete.

Matter for Congressional Consideration

To enhance program oversight and provide more robust input to budget deliberations, Congress should consider requiring DOD to report on each major acquisition program’s systems engineering status in the department’s annual budget request, beginning with the budget requesting funds to start development. The information could be presented on a simple timeline—as done for the case studies in this report—and at a minimum should reflect the status of a program’s functional and allocated baselines as contained in the most current version of the program’s systems engineering plan.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD for review and comment. DOD’s written comments are reprinted in appendix II of this report. That draft report included a recommendation that DOD submit the systems engineering plans of each new proposed development program to Congress at the same time the budget requesting funds to begin development is sent to Congress. DOD did not agree with our recommendation. In its comments, DOD agreed that early systems engineering reduces risk and establishes a solid foundation for program success. However, DOD noted that the systems engineering plan is a management tool to guide a program’s system engineering activities and the timing of the plan and any updates are not aligned to inform a budget decision that could occur as much as 18 months prior to program initiation. DOD also noted that it believes that existing statutory certifications and reports submitted to Congress contain adequate information regarding program risk and technical maturity. We appreciate DOD’s thoughtful response and agree that existing statutory certifications and reports provide some level of insight into program risks and general assurance that plans are in place to address those risks. However, it is not clear whether they provide adequate detail about progress against those plans, such as whether a functional baseline has been established or when an allocated baseline will be established. In addition, those certifications and reports are linked to program milestone reviews and are not specifically aligned to inform program budget decisions. Documents such as the systems engineering plan are “living documents” that are updated as needed throughout the acquisition process and could be made available to inform the budget process. Therefore, we continue to believe that linking robust insight into systems engineering progress, like the information contained in the systems engineering plan, with the timing of congressional budget deliberations would be beneficial and as such are now including a matter for congressional consideration. Given that the systems engineering plan is updated during the acquisition process, linking systems engineering progress to budget requests need not be onerous. As noted above, this could be accomplished using a top-level timeline of less than a page, as done in the case studies in this report.

We are sending copies of this report to interested congressional committees and offices; the Secretary of Defense; the Secretaries of the Army, Navy, and Air Force. In addition, the report will be made available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff has any questions concerning this report, please contact me at (202) 512-4841. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope and Methodology

This report (1) identifies a framework for assessing the challenges and risks associated with program requirements and the extent of systems engineering done before product development begins; (2) illustrates the relationship between systems engineering and program outcomes; and (3) assesses implications for program management and oversight.

To conduct our work, we selected a non-generalizable sample of 9 major defense acquisition programs, out of a total portfolio of 79 programs, to include at least 2 from each service and Department of Defense (DOD) wide programs. The specific case study programs we selected are identified in table 1.

To assess the challenges and risks associated with program requirements and the extent of systems engineering done by each of the 9 programs before development began, we analyzed top-level program requirements such as key performance parameters, key system attributes, and other system attributes, as well as lower-level derived requirements like system design specifications and design drawings. For each case study program, we requested and analyzed the number of requirements identified for each baseline (functional, allocated, and product level) at the time of system engineering reviews and program milestones and compared the results to DOD systems engineering guidance, DOD acquisition policy, and our previous GAO reports examining weapons systems acquisitions and best practices for product development. To ensure our understanding of commercial best practices remained current, we interviewed officials from leading companies including Caterpillar, Inc., Cummins, Inc., Proctor and Gamble, and Motorola Solutions. In addition, we assessed documents and data from system engineering reviews and program reported technology readiness levels; and we reviewed program acquisition strategies, capability development documents or operational requirements documents, acquisition baselines, and selected acquisition reports.

To assess the relationship between systems engineering and program outcomes, we analyzed requirements, cost, and schedule documentation for each of our case study programs, and then met with relevant program officials and prime contractors to obtain relevant program documents, data, and historical information. We reviewed selected acquisition and Defense Acquisition Executive Summary reports, budget data, DOD Systems Engineering assessments of preliminary and critical design reviews, and previous GAO reports such as our annual reviews of major defense acquisition programs and program specific reports.

To identify program oversight implications, we also reviewed relevant acquisition statutes, DOD acquisition policy, systems engineering guidance, and previous GAO reports examining weapons systems acquisitions and best practices for product development. To obtain additional insights into these implications, we spoke with knowledgeable DOD officials including program managers and prime contractors for our case study programs. We assessed the reliability of DOD and contractor data by reviewing existing information about the data and interviewing agency officials knowledgeable about the data. We spoke to both program and prime contractor officials for each of the case study programs and determined that the data were sufficiently reliable for the purposes of our reporting objectives.

We conducted this performance audit from June 2015 to November 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Travis Masters (Assistant Director); Peter Anderson; Marvin Bonner; Robert Bullock; Kurt Gurka; Scott Hiromoto; Jean McSween; Robert Miller; LeAnna Parkey; and Roxanna T. Sun made key contributions to this report.