Weapon System Requirements: Detailed Systems Engineering Prior to Product Development Positions Programs for Success
Released: 2016-11-17T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: Cost and schedule growth in DOD major defense acquisition programs persist, and some acquisition reform proponents believe such growth is due to unplanned changes in program requirements (commonly referred to as "requirements creep"). GAO found in June 2015 that cost and schedule growth are often more directly related to a lack of systems engineering, which, if done, would reduce risk by introducing discipline and rigor into the process of defining and understanding a program's initial requirements.

House Armed Services Committee Report 114-102 contained a provision for GAO to review the DOD requirements process. This report (1) identifies a framework for assessing the challenge posed by weapon system requirements and the extent of systems engineering done before product development begins; (2) illustrates the relationship between systems engineering and program outcomes; and (3) assesses implications for program oversight. GAO analyzed a non-generalizable sample of nine case studies. GAO assessed the extent to which systems engineering was conducted before development by reviewing program requirements and analyzing cost and schedule documentation for each case study. GAO also reviewed prior GAO work and interviewed DOD officials.

section_title: What GAO Found
paragraphs: GAO's analysis of nine case studies identified four factors that frame the challenge posed by a given weapon system's requirements: acquisition approach, technology status, design maturity, and system interdependency. Systems engineering is the primary means for determining whether and how that challenge can be met. It is a disciplined learning process that translates requirements into specific design features and thus identifies key risks to be resolved. GAO's prior best practices work has found that if detailed systems engineering is done early, a program can resolve such risks through trade-offs and additional investments before a program starts. A key point in systems engineering where this match can be assessed is the preliminary design. As shown below, establishing a preliminary design through early detailed systems engineering portends better program outcomes than doing so after program start.

GAO's analysis of selected Department of Defense (DOD) programs illustrates the relationship among the four factors, systems engineering, and program outcomes. Programs with modest requirements and early detailed systems engineering had better outcomes. For example, the Small Diameter Bomb Increment I program, which delivered within cost and schedule estimates, had an incremental approach, mature technologies, a derivative design, and detailed systems engineering before development began. Programs that began with more challenging requirements and insufficient systems engineering reported worse outcomes. For example, the F-35 Lightning II, which has encountered significant cost and schedule problems, began development with a single-step approach, a highly complex design, immature technologies, and little systems engineering.

Understanding the dynamic between a program's requirements, risks, and the requisite systems engineering effort has important implications for oversight. A particular challenge is that Congress often must consider requests to authorize and fund a new program in advance of the start of product development, when the business case would be better established. DOD policy requires that DOD decision makers have information about a proposed program's risk factors and systems engineering status, in a systems engineering plan, at the start of a new program. However, it is not clear whether Congress also has this information at that time. A systems engineering plan could provide more robust information to Congress when considering a budget request to start a new program. In commenting on a draft of this report DOD disagreed.

section_title: What GAO Recommends
paragraphs: To support oversight and inform budget decisions, Congress should consider requiring DOD to report on systems engineering status of each major acquisition program in the department's annual budget request.

section_title: Letter
paragraphs: The Department of Defense (DOD) expects to have invested a total of  more than $1.4 trillion in development and procurement of the major  defense acquisition programs in its current portfolio. This portfolio has  experienced cost growth of 48 percent since first full estimates and  average delays in delivering initial capabilities of almost 30 months. Some  acquisition reform proponents believe that unplanned changes in program  requirements—”requirements creep”—after the Joint Requirements  Oversight Council has approved the requirements and handed them off  to the acquisition community are a primary cause of these ongoing cost  and schedule problems in many programs. In June 2015, we found that  cost and schedule growth in major acquisition programs were not  necessarily a direct result of requirements changes, but were instead  more directly related to a lack of discipline and rigor in the process of  defining and understanding a program’s initial requirements.

House Armed Services Committee Report 114-102 contained a provision  for us to review the DOD requirements process. This report (1) identifies  a framework for assessing the challenges and risks posed by  requirements and the extent of systems engineering done before product  development begins; (2) illustrates the relationship between systems  engineering and program outcomes; and (3) assesses implications for  program oversight.

To conduct our work, we selected a non-generalizable sample of nine  major defense acquisition programs, made up of various types of systems  from each military service, to conduct case study assessments. The  specific case study programs we selected are identified in table 1.

To assess the challenges and risks associated with program  requirements and the extent of the systems engineering done by each of  the nine programs before development began, we analyzed top-level  program requirements such as key performance parameters, key system  attributes, and other system attributes, as well as lower-level, derived  requirements like system design specifications and design drawings. In  addition, we assessed documents and data from program system  engineering reviews and reported technology readiness levels, and  reviewed program acquisition strategies, acquisition baselines, and  selected acquisition reports. We reviewed cost, schedule, and  requirements data and interviewed agency officials knowledgeable about  the data. We determined that the data were sufficiently reliable for the  purposes of our reporting objectives. To assess the relationship between  systems engineering and program outcomes, we analyzed requirements,  cost, and schedule documentation for each of our case study programs,  and then met with relevant program officials and prime contractors to  obtain relevant program documents, data, and historical information. To  identify program oversight implications, we also reviewed relevant  acquisition statutes, DOD acquisition policy, systems engineering  guidance, and previous GAO reports examining weapons systems  acquisitions and best practices for product development. To obtain  additional insights into these implications, we spoke with knowledgeable  DOD officials and the prime contractors for our case study programs. See  appendix I for additional information on our objectives, scope, and  methodology.

We conducted this performance audit from June 2015 to November 2016  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: In our past work examining weapon system acquisitions and best  practices for product development, we have found that leading  commercial firms and successful DOD programs pursue an acquisition  approach that is anchored in knowledge, whereby high levels of product  knowledge are demonstrated at critical points in the acquisition process.  The first key point in this knowledge-based process occurs when a match  is achieved between the customer’s needs or requirements and available  resources—including technology, design, time, and funding. That match  establishes a sound basis for a program business case and supports the  decision to invest in product development. Achieving a match between  requirements and resources prior to committing to a product development  program reduces risk and uncertainty and sets the program up for  success. We have previously found that key enablers of a good business  case include:

Firm, feasible requirements: Requirements should be clearly defined,  affordable, and clearly informed—thus tempered—by systems  engineering. Once programs begin, requirements should not change  without assessing their potential disruption to the program.

Mature technology: Science and technology organizations should  shoulder the technology development burden, proving technologies  can work as intended before they are included in a weapon system  program. The principle is not to avoid technical risk but rather take risk  early and resolve it ahead of the start of product development.

Incremental, knowledge-based acquisition strategy: Rigorous systems  engineering coupled with more achievable requirements are essential  to achieve faster delivery of needed capability to the warfighter.  Building on mature technologies, such a strategy provides time,  money, and other resources for a stable design, building and testing  of prototypes, and demonstration of mature production processes.

Realistic cost estimate: Sound cost estimates depend on a  knowledge-based acquisition strategy, independent assessments, and  sound methodologies.

Major defense acquisition programs go through a series of phases as  they progress from the identification of the top-level requirements for new  capability, through initial planning of a solution, to product development,  and finally production and deployment of a fielded system. Within DOD,  the process of identifying and understanding requirements typically  begins when a sponsor, usually a military service, submits an Initial  Capabilities Document that identifies the existence of a capability gap; the  operational risks associated with the gap; and a recommended solution or  preferred set of solutions for filling the gap. Potential solutions are then  assessed in an Analysis of Alternatives, system capabilities are chosen,  and top-level capability or operational requirements are then defined in a  draft Capability Development Document that goes through several stages  of service- and DOD-level review and validation. After the top-level  capability requirements are defined, they are then decomposed into more  specific capability requirements, known as the performance specification,  and then into a series of more detailed design requirements.  Progressively more detailed descriptions of the system design are  established in what systems engineering literature refers to as  configuration baselines. Within DOD, the configuration baselines are  called the “functional baseline” that details the system-level performance  requirements; the “allocated baseline” that establishes a preliminary  design and defines the subsystems and how they are to work together;  and the “product baseline” that describes the final design and provides  component-level details (see figure 1).

Systems engineering is a disciplined process to transform top-level  capability requirements into detailed, lower-level design requirements that  can be achieved with available resources. Programs are required to  prepare a systems engineering plan as a management tool to guide  systems engineering activities over the life of a program. According to  DOD, the discipline of this process provides the control and traceability to  develop solutions that meet customer needs. The systems engineering  process can be depicted in a V-shape that starts when a capability gap is  identified and a proposed solution for filling that gap is selected. Top-level  capability requirements are then defined and decomposed into lower-level  design requirements until a final design is established. As this process  takes place, requirements become more specific and the number of  requirements at each successive level grows. This growth can be  exponential, with tens of thousands of detailed design requirements  derived from a relatively small number of capability requirements. Figure  2 illustrates the relationship between requirements decomposition and  systems engineering.

As more detailed design requirements are identified, decision makers can  make informed trades between the requirements and available resources,  potentially achieving a match and establishing a sound basis for a  program business case. As the requirements decomposition process  takes place, engineering and design knowledge grows and risk  decreases, leading to more realistic cost and schedule estimates and  more predictable program outcomes.

section_title: Four Factors Frame the Challenge Posed by Requirements, and Detailed Systems Engineering Is Key to Knowing Whether and How the Challenge Can Be Met
paragraphs: Our analysis of nine selected DOD programs, supported by our previous  work examining best practices for product development, identified four  key factors that provide insight into the challenge posed by a system’s  top-level capability requirements and the related risk. While they are not  the only factors that could be considered, we found that they were  prominent and observable early in our case study programs. Those four  factors are:

Acquisition approach - indicates whether a program intends to take an  incremental, evolutionary approach or a single-step approach to meet  all capability requirements. According to GAO best practices, an  incremental approach reduces risk by developing and delivering a  product in a series of enhanced interim capabilities until final  capability is reached. In contrast, a single-step approach increases  risk by attempting to deliver the final capability all at once without any  interim capabilities.

Technology status - indicates the extent to which the critical  technologies for a proposed system are mature at the start of product  development. According to GAO best practices, technologies are  ready for inclusion in a product development program when they have  been demonstrated in a realistic, operational environment (i.e.,  Technology Readiness Level (TRL) 7), proving that they can work as  intended. Demonstrating technologies in an operational environment  provides a higher level of technology understanding and reduces risk  prior to starting product development. Beginning system development  with less mature technologies (i.e., TRL lower than 7) increases  program risk because unexpected technology challenges could  significantly affect product design.

Design maturity - indicates the extent to which a proposed system’s  design is either derived from an existing commercial or DOD  system—whether operational or prototype—or is new and  unprecedented. According to GAO best practices, designs derived  from existing systems, whether commercial or DOD, by nature have  more knowledge available when product development begins, thus  reducing risk. In contrast, unprecedented designs are by nature more  complex and inherently higher risk.

System interdependency - indicates the extent to which a system  relies on another system or group of systems being developed and  managed separately to provide its required capabilities. According to  DOD acquisition guidance, although DOD programs should not be  acquired in isolation, a program office developing a more independent  system generally has more control over requirements, schedule,  funding, and interfaces, among other factors. In contrast, a program  office developing a system that is more dependent on other  systems—like a system of systems—generally has less control, thus  making its requirements more challenging to achieve and increasing  risk. For example, systems of systems tend to be more complex and  must manage (1) interfaces with other systems that have potentially  unaligned performance requirements, (2) the need for collaborative  governance with separately managed programs, and (3) differences in  program phases of development.

As we discuss in the following pages, the case study programs with top- level capability requirements that posed a more modest challenge took an  incremental acquisition approach, incorporated more mature  technologies, used a design that was derived from an existing product or  prototype, and limited dependence on systems being developed and  managed separately, reducing the amount of risk being carried forward  into product development. In contrast, the programs with top-level  capability requirements that were more challenging generally assumed  full capability would be delivered in a single-step, incorporated immature  critical technologies, used a new and unprecedented system design, and  had full capability that in some cases was critically dependent on  separately developed and managed systems without sufficiently  mitigating risk before starting product development.

The fact that some requirements can be more challenging than others  does not necessarily mean that problems are unavoidable. Rather, the  ensuing systems engineering effort, if timely and sufficiently vigorous, can  provide the needed confidence that the requirements are achievable. Our  previous work in best practices has found that conducting detailed  systems engineering analysis before starting product development  contributes to understanding the requirements challenge and identifying  and mitigating associated risks. Systems engineering is the primary  means for determining whether and how the challenge posed by a  program’s requirements can be met with available resources. It is a  disciplined learning process that translates capability requirements into  specific design features and thus identifies key risks to be resolved. Our  prior best practices work has indicated that if detailed systems  engineering is done before the start of product development, the program  can resolve these risks through trade-offs and additional investments,  ensuring that risks have been sufficiently retired or that they are clearly  understood and adequately resourced if they are being carried forward.  Figure 3 depicts this relationship between the timing of systems  engineering and the start of product development for successful and  problematic programs.

In our case-study analysis, we found that systems engineering is a  natural learning process that takes place at some point in all programs.

Our work in commercial best practices has found that, at a minimum,  programs need to conduct systems engineering to decompose capability  requirements into an allocated baseline—that is, establish a preliminary  design—before beginning a product development program. When the  bulk of the detailed systems engineering is done after starting a product  development program it often results in cost and schedule growth and  creates inefficiency. Figure 4 compares the points in the requirements  and systems engineering processes at which problematic and successful  programs would begin product development.

Within the current defense acquisition framework, DOD decision makers  should assess a program’s risk factors and systems engineering progress  well in advance of the start of product development. For example, DOD’s  acquisition policy states that program requirements must be firm and  clearly stated and that the risk of committing to development is or will be  adequately reduced prior to awarding a development contract. The policy  also emphasizes the value in establishing an allocated baseline—i.e.  preliminary design—prior to the start of product development, which is a  best practice that contributed to good outcomes in our case study  programs. This practice was also emphasized by Congress in the  Weapon System Acquisition Reform Act of 2009, which required  programs to hold a preliminary design review or obtain a waiver prior to  starting a development program.

We have historically found that many of DOD’s major weapon system  development programs tended to conduct the bulk of systems  engineering after a development contract had been awarded and product  development had begun. In many cases, the top-level capability  requirements established by the government at that time still needed to  be analyzed by the contractor and decomposed into a functional baseline  (system-level design), an allocated baseline (preliminary design), and a  product baseline (final design). As a result, those programs had started  product development with a limited understanding of the challenge posed  by requirements.

section_title: Case Studies Illustrate the Relationship between Requirements Challenge, the Timing of Systems Engineering, and Program Outcomes
paragraphs: Our case studies illustrate the relationship among program outcomes, the  four key factors that frame the challenge posed by requirements, and  timely systems engineering. Our analysis of development cost and  schedule data for the nine case studies shows that, as of December  2015, about half of the programs had reported relatively good  development cost and schedule outcomes, while the others had  experienced significant cost and schedule growth (see table 2). Of the  programs we studied, the three that began development with more  modest requirements and had conducted detailed systems engineering  generally had good outcomes. The three programs with some  requirements challenges that conducted systems engineering analysis to  mitigate associated risks experienced moderate cost and schedule  outcomes. Finally, the three programs in our sample that began  development with challenging requirements and had done little systems  engineering generally reported poor outcomes.

These results are consistent with our recent analysis of DOD’s overall  major defense acquisition portfolio. In March 2016, we reported that while  program outcomes across the portfolio remained mixed, DOD’s newer  programs were reporting better cost outcomes. We found that some of  those programs had taken steps to retire risks through systems  engineering before starting product development and thus began with a  clearer understanding of the challenge posed by their requirements.
subsections: section_title: Three More Successful Programs Had Modest Requirements and Conducted Detailed Systems Engineering Analysis before Development Start
paragraphs: The KC-46A Tanker Modernization (KC-46), Small Diameter Bomb  Increment I (SDB I), and Joint Light Tactical Vehicle (JLTV) programs  provide examples in which top-level requirements generally posed  moderate challenges, steps were taken to identify and retire risks through  detailed systems engineering, which enabled development of sound  business cases before product development began. In these cases, the  government and the prime contractor conducted systems engineering to  decompose the requirements and identify an allocated baseline by the  time they started product development. As a result, the programs  established cost and schedule baselines that were well informed,  contributing to relatively good outcomes.
subsections: section_title: KC-46 Tanker Modernization Program
paragraphs: Our analysis of the KC-46 requirements challenge and the ensuing  systems engineering effort is summarized in figure 5.

Officials from the Air Force and the KC-46 prime contractor told us that at  the start of product development in 2011, they had conducted detailed  systems engineering and the system’s critical technologies were nearing  maturity. The KC-46 was designed to improve on the KC-135’s refueling  capacity, efficiency, and capabilities for cargo and aeromedical  evacuation, and to integrate defensive systems. The Air Force planned an  incremental acquisition approach focused on integrating mature military  technologies onto a commercial aircraft derivative design. Although the  government did not hold a preliminary design review until after  development began, prime contractor officials noted that they had  identified the allocated baseline as part of their internal risk reduction  efforts before the development contract was awarded. As a result the KC- 46 program took steps that enabled a sound business case. Because KC- 46 development was considered by the government to be a relatively low- risk effort, the prime contractor was awarded a fixed price incentive  contract at the start of product development, which mitigated risk to the  government. The prime contractor subsequently discovered problems  with the aircraft wiring and aerial refueling systems and encountered a  fuel contamination incident, all of which contributed to a 14-month delay  in the delivery of initial operating capability. However, the government’s  development costs have decreased by 12 percent largely because the  contract is fixed price and the government’s initial cost estimate assumed  a greater number of engineering changes than have actually occurred,  indicating that the Air Force understood that development challenges  remained and reflected that understanding in its initial cost estimates.

section_title: Small Diameter Bomb Increment I
paragraphs: Our analysis of the SDB I requirements challenge and the ensuing  systems engineering effort is summarized in figure 6.

At the start of product development in 2003, the Air Force and the SDB I  prime contractor had performed detailed systems engineering to  understand the challenge posed by requirements and remaining risks.  The SDB was designed to meet a pressing Air Force need for a low- collateral damage weapon small enough to maximize the number that can  be carried aboard a single aircraft. The Air Force planned an incremental  acquisition approach to the overall SDB program, with SDB I—the first  increment—using mature technology and based on a design derived from  competitive prototypes developed and tested by the prime contractor.  According to program officials, all of the system’s critical technologies  were mature and they had production representative hardware when  product development started. The Air Force and competing contractors  conducted systems engineering over the course of 2 years prior to  development to refine requirements in light of available resources. During  that time live fire testing was conducted and the Air Force held  preliminary and critical design reviews—key systems engineering  events—with each contractor. According to prime contractor officials, their  interactions with the Air Force prior to the start of development allowed  them to get a clear understanding of design requirements. As a result,  they were able to achieve a stable design and provide a production  representative system at the start of product development. With all of the  detailed systems engineering done before development started, risks  were understood. The Air Force had a sound business case that reflected  a realistic understanding of the challenge facing the SDB I development  program. The program’s total development cost decreased by almost 4  percent and the system was available for use 1 month earlier than initially  estimated.

section_title: Joint Light Tactical Vehicle
paragraphs: Our analysis of the JLTV requirements challenge and the ensuing  systems engineering effort is summarized in figure 7.

At the start of product development in 2012, the Army and Marine Corps  and the JLTV prime contractor had performed detailed systems  engineering to understand the challenge posed by requirements and  remaining risks. The JLTV was expected to be a family of vehicles built  on a common automotive vehicle platform to replace the High Mobility  Multipurpose Wheeled Vehicle for some missions. In response to  concerns raised by DOD’s acquisition executive regarding technical  maturity, shifting requirements, and affordability, and in light of available  resources, the Army and Marine Corps worked with three contractors  prior to the start of product development to refine requirements. The Army  and Marine Corps planned an incremental acquisition approach using  mature technology and based on a design derived from prototypes  developed and tested by the winning contractor prior to the start of  product development. Basic capability would be provided initially, with  enhanced force protection, increased fuel efficiency, greater payload, and  other improvements to be added in later increments. As a result, the  prime contractor was able to identify an allocated baseline and provide a  mature product based on a demonstrated design at the start of  development. With the bulk of the systems engineering done before  development started, the JLTV program had taken steps to reduce risk  and enable a sound business case that reflected a realistic understanding  of the challenge it faced. The program has decreased development costs  by 6 percent. Although acquisition cycle time has increased 19 months  from initial estimates, program officials noted that the schedule growth  was due to issues unrelated to systems engineering or program  requirements, specifically production contract bid protests that delayed  operational testing and the need to add time for Army users to complete  critical training.

section_title: Three Programs Had Some Requirements Challenges but Conducted Systems Engineering to Support Risk Mitigation
paragraphs: The Global Positioning System III (GPS III), Paladin Integrated  Management (PIM), and P-8A Poseidon Multi-Mission Maritime Aircraft  (P-8A) programs provide examples in which top-level requirements  generally posed moderate challenges with some risks that were identified  and generally mitigated through systems engineering. In these cases, the  government and the prime contractor conducted some detailed systems  engineering to inform their program business cases before starting  product development but not all requirements risks were mitigated, which  contributed to problems and some cost and schedule growth.
subsections: section_title: Global Positioning System III
paragraphs: Our analysis of the GPS III requirements challenge and the ensuing  systems engineering effort is summarized in figure 8.

Although the Air Force and GPS III prime contractor conducted some  detailed systems engineering to inform the program’s business case  before the start of development in May 2008, additional risks to fielding  the capability existed due to interdependencies associated with a critical  ground control system and military user equipment that were being  developed and managed as separate programs. The GPS III program is  expected to develop and field a new generation of satellites to  supplement and eventually replace currently operational GPS satellites.  As the space segment of DOD’s effort to sustain and modernize the GPS  system, the GPS III program is one part of a system of systems, which  also includes the Next Generation Operational Control System (OCX),  intended to replace the existing GPS ground system and to operate both  current and future satellites, and the Military GPS User Equipment  (MGUE), intended to provide the military services with new GPS  receivers. In order to avoid problems associated with a previous GPS  satellite program, the Air Force planned an incremental acquisition  approach to GPS III, using technologies that were assessed as mature  and performed some systems engineering to gain an early understanding  of the challenge posed by requirements. According to contractor officials,  this systems engineering, conducted during the competitive requirements  development phase, helped them better understand design requirements.  Regardless of these efforts, the program was negatively impacted by  dependency on other systems.

Design and manufacturing problems with a key GPS III navigation  subsystem delayed the program by almost 2 years; however, the  satellites’ dependence on the much delayed OCX proved to be a greater  challenge to sustaining and modernizing the GPS system. The delivery of  the OCX Block 1, which is required to operate the GPS III satellites with  both legacy and new capabilities, has been delayed by 6 years to July  2021 and may be delayed further because of ongoing developmental  issues. As a result of the OCX program’s delays, the Air Force is pursuing  a smaller scale program to modify the existing GPS operational control  system to enable the operational use of GPS III satellites for all legacy  GPS functions until delivery of the OCX Block 1, which will then permit  the operational testing and use of the GPS III satellites’ new capabilities.  Due to OCX delays and the likely timing of the contingency operations  system the Air Force may need to delay the launch of multiple GPS III  satellites or launch several without fully testing them. Moreover, although  testing the satellites’ functionality is not dependent on new user  equipment, timing of MGUE capability delivery will further postpone—by  about a decade—the warfighter’s ability to take advantage of the  upgraded system’s new military code, which offers greater resistance to  jamming.

section_title: Paladin Integrated Management
paragraphs: Our analysis of the PIM requirements challenge and the ensuing systems  engineering effort is summarized in figure 9.

The Army’s Paladin Integrated Management (PIM) program—officially the  M109A7 Family of Vehicles—began formal product development as a  major defense acquisition program in June 2011.  The Army began  early systems engineering and development work in 2007. At that time,  PIM was a service life extension program for the existing Paladin M109A6  and according to program officials, was expected to be a relatively minor  upgrade to update the fire control system and replace the power pack of  the existing system. The Army expected these requirements to neither  result in a significant or complicated design change, nor be costly. As part  of the service life extension program, the Army had identified both  functional and allocated baselines. By 2011, the Army had made several  changes to the program, with significant implications for the design and  cost of PIM. Specifically, the Army increased force protection and  survivability requirements, as directed by DOD, and added a cannon that  had been in development as part of the Army’s Future Combat Systems  program before it was cancelled in 2009. These changes essentially  resulted in a new design versus an upgrade, and the associated costs  increased enough to make PIM a major defense acquisition program.

The program’s functional and allocated baselines were re-established in  2012, reflecting both the design changes and the systems engineering  done since 2007, and ensured cost and schedule estimates were  sufficiently informed for a sound business case. Since 2011, development  cost has increased 5 percent and the program has experienced a  schedule delay of 2 months. While not a textbook example of how to keep  requirements stable during a program, PIM’s experience shows that  changes can be accommodated when accompanied by a suitable  systems engineering effort.

section_title: P-8A Poseidon Multi-Mission Maritime Aircraft Increment I
paragraphs: Our analysis of the P-8A Increment I requirements challenge and the  ensuing systems engineering effort is summarized in figure 10.

At the start of P-8A development in 2004, the Navy and the prime  contractor had completed some detailed systems engineering but an  allocated baseline was not yet established. The P-8A is expected to  replace an existing system, the P-3C Orion, and meet aspects of the  Navy’s anti-submarine warfare, anti-surface warfare, and intelligence,  surveillance, and reconnaissance capability requirements. The Navy  planned an incremental, evolutionary acquisition approach focused on  providing a first increment of capability to users in the quickest and most  cost efficient manner. The Navy also expected P-8A Increment I to be  built on open systems architecture to allow subsequent increments to  more easily deliver increased capabilities. In addition, the airframe was to be derived from an existing commercial aircraft. According to prime  contractor officials, the P-8A airframe design was about 50 percent  common with the commercial airframe from which it was derived. At the  start of development, none of P-8A Increment I’s four critical technologies  were mature, increasing risk associated with the program’s business  case. However, as part of its systems engineering analysis, the Navy and  the prime contractor identified mature backup technologies that could be  used if the critical technologies did not mature as expected thus mitigating  some of the technology risk. P-8A Increment I experienced some cost  and schedule growth. That growth is largely due to delays in the  completion of the final system design. As of February 2016, the program’s  development cost estimate had increased 14 percent, and its acquisition  cycle time had increased by 4 months.

section_title: Three Programs with Problematic Outcomes Had Challenging Requirements and Limited Systems Engineering before Development Start
paragraphs: The F-35 Lightning II (F-35) and CH-53K Heavy Lift Replacement  Helicopter (CH-53K) did not conduct adequate systems engineering prior  to starting product development, and as result began development with  significant risks and a limited understanding of the challenge posed by  their requirements. Neither program had established a functional or  allocated baseline before development began. In fact, their allocated  baselines were not established until years into product development. The  government and prime contractor for the Integrated Air and Missile  Defense (IAMD) program did some systems engineering before the start  of product development, but the Army made high-level changes during  the first year of development to integrate additional systems,  necessitating additional detailed systems engineering work after the start  of development.
subsections: section_title: F-35 Lightning II
paragraphs: Our analysis of the F-35 requirements challenge and the ensuing systems  engineering effort is summarized in figure 11.

At the start of development in 2001, neither DOD nor the F-35 prime  contractor had conducted detailed systems engineering to adequately  retire risk, establish an allocated baseline, and truly understand the  challenge posed by their requirements. As a result, the F-35 program did  not have a sound, executable business case. The F-35 was a next  generation, highly complex stealth fighter with integrated avionics and  software intensive mission systems being developed in three variants to  meet the needs of three U.S. military services and various international  partners. When development began, DOD planned to achieve full  capability using a single-step acquisition approach and the F-35’s critical  technologies were immature. DOD’s initial program plans showed that  key systems engineering analyses to support an allocated baseline would  not be complete until the program was years into development.

The bulk of F-35 systems engineering was done after development began  and the program experienced significant cost and schedule growth— development costs increased 60 percent over initial estimates and initial  operational capability was delayed over 5 years—and was restructured  three times. The first restructuring was driven by the discovery of  significant weight issues with the F-35B variant as the program developed  its preliminary design, resulting in costly design changes. DOD and prime  contractor data indicate that the system’s functional baseline—detailing  the system-level design—was not finalized until 4 years into development.  As systems engineering continued and testing began, additional  discoveries were made, requiring further design changes and contributing  to the two additional restructurings. Ultimately, the program realized that  some capabilities originally envisioned when development began would  not work as expected and would have to be completed as part of follow- on development program.

section_title: CH-53K Heavy Lift Replacement Helicopter
paragraphs: Our analysis of the CH-53K requirements challenge and the ensuing  systems engineering effort is summarized in figure 12.

At the start of development in 2005, neither the Marine Corps nor the CH- 53K prime contractor had conducted detailed systems engineering to  adequately retire risk, establish an allocated baseline, and fully  understand the challenge posed by their requirements. They had  established neither a performance specification nor a functional baseline.  As a result, the CH-53K program did not have a sound, executable  business case. The Marine Corps and the prime contractor disagreed on  what specific systems engineering tasks were needed, and initial plans  showed that key analyses would not be complete until the system was  well into development. The Marine Corps expected the CH-53K to provide  improvements relative to the CH-53E in range and payload, survivability  and force protection, reliability and maintainability, coordination with other  assets, and overall cost of ownership. As a result and in order to meet  requirements, an entirely new aircraft design was needed. According to  the prime contractor, none of the parts in the new design were common  parts with the CH-53E. When development began, the Marine Corps  planned to achieve full capability using a single-step acquisition  approach. At that same time, the system’s critical technologies were not  mature.

Nearly all CH-53K systems engineering was done after development  began, and the program experienced significant cost and schedule  growth. Because the program office and prime contractor disagreed on  what systems engineering tasks needed to be accomplished, problems  began immediately following the start of development. Once systems  engineering began and problems were identified, the Marine Corps chose  to defer some key performance capabilities to reduce development costs,  and, in 2010, the program was restructured with a new baseline approved  in 2013. Discoveries during ground testing and qualification drove  additional unanticipated design changes and delays. As a result, the  program’s initial production decision was delayed 8 months, and the  program now plans to establish a new cost baseline. In total, the  program’s development costs have increased 51 percent over initial  estimates, and its initial operational capability has been delayed by over 4  years.

section_title: Integrated Air and Missile Defense
paragraphs: Our analysis of the IAMD requirements challenge and the ensuing  systems engineering effort is summarized in figure 13.

Although the Army and IAMD prime contractor had conducted some  detailed systems engineering and technologies were assessed as nearly  mature at the start of product development, the IAMD program’s business  case was critically dependent on sensors and weapons that had been  developed and managed as separate programs. IAMD is expected to  network sensors, weapons, and a common battle command system  across an integrated fire control network to support the engagement of air  and missile threats. The Army planned an incremental acquisition  approach and held a competition for preliminary design to mitigate risk in  May 2009. Prime contractor officials told us they conducted systems  engineering prior to the start of development, which may have helped  them better understand design requirements. While a functional baseline  was established prior to the start of product development and the prime  contractor built a basic design prototype to demonstrate for the  government that the design architecture would work, the system’s  allocated baseline was not fully established until after the start of  development.

System interdependency has posed challenges to the IAMD program in  several ways. The Army made high-level changes in IAMD during the first  year, which resulted in the program having to integrate additional systems  and causing significant changes in how the systems were to interact.  According to prime contractor officials, three of the four IAMD critical  technologies rely on interfacing with other systems and many of these  interfaces were not defined until after the development contract was  awarded. IAMD has encountered challenges integrating new software  from the contractor with other acquisition programs critical to its  functionality. IAMD development cost has increased over 60 percent  since the start of development, with about half of the increase occurring in  the first year. The program has been rebaselined and initial operational  capability has been deferred by 2 years. This experience is somewhat  similar to that of the GPS III with an important distinction. While  challenges with GPS III’s interdependence with other systems did not  significantly affect the design of the satellites themselves, the  interdependence of IAMD did have significant implications for its software  design, which were not recognized until after the start of product  development.

section_title: Knowledge of Systems Engineering Can Enhance Oversight before a Program Starts Product Development
paragraphs: Systems engineering, if done well, can be a key enabler for establishing a  sound business case for a program before the start of product  development. By the same token, it can provide indications to Congress  and other organizations responsible for oversight as to the soundness of  DOD’s approach to undertaking a new program. The decision to begin  product development is the last opportunity in the acquisition process— prior to committing to a major financial investment—to level a product’s  requirements to available resources, thereby establishing a good  business case. A sound business case provides credible evidence that  (1) the warfighter’s needs are valid and that they can best be met with the  chosen concept; and (2) the chosen concept can be developed and  produced within existing resources—that is, proven technologies,  sufficient design knowledge, adequate funding, and adequate time exist  to deliver the product when it is needed. A program should not go forward  into product development, nor should this step be funded, unless a sound  business case can be made. Therefore, Congress, which is responsible  for making that funding decision, should also have insight into the  soundness of the business case. However, given current budget  processes and mechanics in DOD today, these congressional decisions  must be made well in advance of the start of product development and  establishment of the final business case.

Systems engineering is essential to establishing a sound business case.  We have previously found that key enablers of a good business case  include firm, feasible requirements, mature technology, an incremental,  knowledge-based acquisition strategy, and a realistic cost estimate. Our  work examining commercial companies’ best practices has found that  requirements should be clearly defined, affordable, and clearly  informed—thus tempered—by systems engineering prior to the start of  product development. Once programs begin, requirements should not  change without assessing their potential disruption to the program. In  addition, science and technology organizations should shoulder the  technology development burden, proving technologies can work as  intended before they are included in a weapon system program. The  principle is not to avoid technical risk, but rather take risk early and  resolve it ahead of the start of product development. Programs should  use an incremental, knowledge-based acquisition strategy. We have  found that rigorous systems engineering coupled with more achievable  requirements are essential to achieve faster delivery of needed capability  to the warfighter. Building on mature technologies, such a strategy  provides time, money, and other resources for a stable design, building  and testing of prototypes, and demonstration of mature production  processes. Finally, a good business case will have a realistic cost  estimate based on a knowledge-based acquisition strategy, independent  assessments, and sound methodologies.

Over the years, and as previously noted in this report, we have found that  for a program to deliver a successful product with available resources,  high levels of knowledge—informed by systems engineering—must be  demonstrated before significant commitments are made. This requires  the user and developer to negotiate whatever trade-offs are needed to  achieve a match between the user’s requirements and the developer’s  resources before product development begins. While it would seem that  taking such an approach would be axiomatic, we have found that it is  not. On the contrary, we have previously found that there are strong  incentives within the culture of weapon system acquisition to overpromise  a prospective weapon’s performance while understating its likely cost and  schedule demands. Competition with other programs vying for defense  dollars puts pressure on program sponsors to project unprecedented  levels of performance (often by counting on unproven technologies) while  promising low cost and short schedules. These incentives, coupled with a  marketplace that is characterized by a single buyer (DOD), low volume,  and limited number of major sources, create a culture in weapon system  acquisition that encourages undue optimism about program risks and  costs.

A particular challenge for Congress is the fact that committees must often  consider requests to authorize and fund a new program well ahead of the  start of product development—the point at which key business case  information would be presented. For example, if DOD has scheduled a  decision to start a new development program in August 2017, the funding  needed for the first year of development would have to be included in  DOD’s fiscal year 2017 budget request. This budget request would be  submitted to Congress in February 2016, or 18 months before the actual  program decision. Given this time lag, Congress could be making critical  funding decisions—which in effect authorize the start of development— with limited information about program risk factors, systems engineering  progress, and the soundness of the program’s business case.

The nine case studies we examined for this report suggest that  understanding the dynamic between program requirements, risks, and the  requisite systems engineering analysis can facilitate early oversight.  Specifically, when the top-level requirements for what could be a new  major weapon system are initially identified in a draft Capability  Development Document, the challenge those requirements pose and how  it relates to the four factors we identified—acquisition approach,  technology status, design maturity, and system interdependence—can be  observed. Once the challenge is framed, the systems engineering plan  can be refined to obtain the requisite knowledge to (1) identify expectation  gaps and other risks, and (2) resolve them by the start of product  development. Ultimately, the final Capability Development Document  could then be informed by an allocated baseline (the preliminary design)  and provide a sound basis for an executable business case. This  sequence of events is notionally depicted in figure 14 along with when a  budget to start a new program is submitted to Congress.

While congressional insight at the time of the funding decision is limited,  DOD policy directs program managers to provide certain documents to  DOD decision makers that when taken together could provide a picture of  a proposed program’s risk factors and systems engineering status.  Programs are generally required to provide versions of these documents  to DOD decision makers prior to each major milestone review, including  the start of development. However, it is not clear whether Congress is  given the same information DOD decision makers are given when DOD  submits its budget request for funding to begin a new program. Those  documents include an acquisition strategy, a test and evaluation master  plan, a technology readiness assessment, and a systems engineering  plan.

The systems engineering plan is of particular interest because it brings  the risk factors that we have identified together with a proposed  program’s systems engineering knowledge. According to DOD acquisition  policy, the plan will be submitted for approval for each milestone review,  beginning with the milestone review to approve the start of the technology  maturation and risk reduction phase—the acquisition phase that precedes  the start of product development. DOD views the plan as a “living  document” that it expects to be updated as needed throughout the  acquisition process. The plan is expected to support the acquisition  strategy, including the program interdependencies, and communicate the  overall technical approach to balance system performance, life-cycle cost,  and risk in addressing warfighter needs. It should describe the program’s  overall technical approach, including key technical risks, processes,  resources, organization, metrics, and design considerations. It should  also detail the timing and criteria for conducting systems engineering  technical reviews, including those that will establish the functional,  allocated, and product level baselines. In addition, the plan should  address system integration risks, including risks related to external  dependencies which are outside the program manager’s span of control  in order to ensure timely design, development, deployment, and  sustainment of the system. Key aspects of the plan are intended to  support more detailed technical planning in order to provide effective  management and control of the program’s technical progress and the  execution of risk mitigation activities.

If the key factors that frame the requirements challenge are understood, a  functional baseline is established, and a realistic plan exists to complete  the allocated baseline, Congress will have more assurance that a  program is on a path to a sound business case—informed by an allocated  baseline—as it considers funding requests for development start, and a  systems engineering plan could help to pull these elements together.

section_title: Conclusions
paragraphs: Weapon system development programs involve unknowns that translate  into cost and schedule risk. This is the nature of any product development  program. The case studies in this report illustrate a strong relationship  between requirements, systems engineering, and program outcomes.  They show that one key to avoid poor outcomes in major weapon system  development programs is not necessarily to eliminate all risks, but instead  to invest in detailed systems engineering analysis to understand the  design implications of requirements and reconcile them with the  resources that can be reasonably expected. At a minimum, this  reconciliation, or match, should be evidenced by an allocated baseline— that is, a preliminary design—before committing to a product development  program. Our cases indicate that regardless of the challenge posed by  initial requirements, systems engineering, if done well, can be a key  enabler of a sound business case for starting a new development  program. This helps to ensure that if a program takes risks, those risks  are clearly identified, and any resource consequences are acknowledged  and accounted for in cost and schedule estimates. In some cases—as  seen with JLTV and PIM—this process could take time and result in  requirements changes, but, if done before committing to a product  development, it is more likely to result in a sound and executable  business case.

Understanding the dynamic between program requirements, risks, and  systems engineering can facilitate early oversight—thus offering a  potential curb to overpromising. When the top-level requirements for what  could be a new major weapon system are initially identified in a draft  Capability Development Document, DOD should clearly understand the  challenge presented by those requirements as framed by the four factors  we identified. Having a clear understanding of the challenge presented by  the initial requirements, coupled with a systems engineering plan laid out  against a schedule to achieve a sound, executable business case before  the start of product development would provide Congress and other  oversight organizations a means to assess the soundness of a proposed  acquisition program. Importantly, it would be possible to assess progress  made by the program against the plan, using the identification of key  systems engineering baselines as waypoints. As progress is made,  artifacts of knowledge—in terms of the systems engineering baselines as  well as decisions made to trade off requirements, choose alternate design  or technology solutions, and to add time and money to close gaps— should also be available. While Congress would still likely be in the  position of considering a budget request to start a program well before  DOD’s actual decision to start product development, the systems  engineering plan and progress made against it would provide more robust  input to Congress’s deliberations. This could also provide Congress with  better insight into the risks facing a proposed development program— including the risks a program may be taking after the start of product  development in the event that the desired level of systems engineering is  not complete.

section_title: Matter for Congressional Consideration
paragraphs: To enhance program oversight and provide more robust input to budget  deliberations, Congress should consider requiring DOD to report on each  major acquisition program’s systems engineering status in the  department’s annual budget request, beginning with the budget  requesting funds to start development. The information could be  presented on a simple timeline—as done for the case studies in this  report—and at a minimum should reflect the status of a program’s  functional and allocated baselines as contained in the most current  version of the program’s systems engineering plan.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOD for review and comment. DOD’s  written comments are reprinted in appendix II of this report. That draft  report included a recommendation that DOD submit the systems  engineering plans of each new proposed development program to  Congress at the same time the budget requesting funds to begin  development is sent to Congress. DOD did not agree with our  recommendation. In its comments, DOD agreed that early systems  engineering reduces risk and establishes a solid foundation for program  success. However, DOD noted that the systems engineering plan is a  management tool to guide a program’s system engineering activities and  the timing of the plan and any updates are not aligned to inform a budget  decision that could occur as much as 18 months prior to program  initiation. DOD also noted that it believes that existing statutory  certifications and reports submitted to Congress contain adequate  information regarding program risk and technical maturity. We appreciate  DOD’s thoughtful response and agree that existing statutory certifications  and reports provide some level of insight into program risks and general  assurance that plans are in place to address those risks. However, it is  not clear whether they provide adequate detail about progress against  those plans, such as whether a functional baseline has been established  or when an allocated baseline will be established. In addition, those  certifications and reports are linked to program milestone reviews and are  not specifically aligned to inform program budget decisions. Documents  such as the systems engineering plan are “living documents” that are  updated as needed throughout the acquisition process and could be  made available to inform the budget process. Therefore, we continue to  believe that linking robust insight into systems engineering progress, like  the information contained in the systems engineering plan, with the timing  of congressional budget deliberations would be beneficial and as such  are now including a matter for congressional consideration. Given that the  systems engineering plan is updated during the acquisition process,  linking systems engineering progress to budget requests need not be  onerous.  As noted above, this could be accomplished using a top-level  timeline of less than a page, as done in the case studies in this report.

We are sending copies of this report to interested congressional  committees and offices; the Secretary of Defense; the Secretaries of the  Army, Navy, and Air Force. In addition, the report will be made available  at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff has any questions concerning this report, please  contact me at (202) 512-4841. Contact points for our offices of  Congressional Relations and Public Affairs may be found on the last page   of this report. Staff members making key contributions to this report are  listed in appendix III.

section_title: Appendix I: Objectives, Scope and Methodology
paragraphs: This report (1) identifies a framework for assessing the challenges and  risks associated with program requirements and the extent of systems  engineering done before product development begins; (2) illustrates the  relationship between systems engineering and program outcomes; and  (3) assesses implications for program management and oversight.

To conduct our work, we selected a non-generalizable sample of 9 major  defense acquisition programs, out of a total portfolio of 79 programs, to  include at least 2 from each service and Department of Defense (DOD)  wide programs. The specific case study programs we selected are  identified in table 1.

To assess the challenges and risks associated with program  requirements and the extent of systems engineering done by each of the  9 programs before development began, we analyzed top-level program  requirements such as key performance parameters, key system  attributes, and other system attributes, as well as lower-level derived  requirements like system design specifications and design drawings. For  each case study program, we requested and analyzed the number of  requirements identified for each baseline (functional, allocated, and  product level) at the time of system engineering reviews and program  milestones and compared the results to DOD systems engineering  guidance, DOD acquisition policy, and our previous GAO reports  examining weapons systems acquisitions and best practices for product  development. To ensure our understanding of commercial best practices  remained current, we interviewed officials from leading companies  including Caterpillar, Inc., Cummins, Inc., Proctor and Gamble, and  Motorola Solutions. In addition, we assessed documents and data from  system engineering reviews and program reported technology readiness  levels; and we reviewed program acquisition strategies, capability  development documents or operational requirements documents,  acquisition baselines, and selected acquisition reports.

To assess the relationship between systems engineering and program  outcomes, we analyzed requirements, cost, and schedule documentation  for each of our case study programs, and then met with relevant program  officials and prime contractors to obtain relevant program documents,  data, and historical information. We reviewed selected acquisition and  Defense Acquisition Executive Summary reports, budget data, DOD  Systems Engineering assessments of preliminary and critical design  reviews, and previous GAO reports such as our annual reviews of major  defense acquisition programs and program specific reports.

To identify program oversight implications, we also reviewed relevant  acquisition statutes, DOD acquisition policy, systems engineering  guidance, and previous GAO reports examining weapons systems  acquisitions and best practices for product development. To obtain  additional insights into these implications, we spoke with knowledgeable  DOD officials including program managers and prime contractors for our  case study programs. We assessed the reliability of DOD and contractor  data by reviewing existing information about the data and interviewing  agency officials knowledgeable about the data. We spoke to both  program and prime contractor officials for each of the case study  programs and determined that the data were sufficiently reliable for the  purposes of our reporting objectives.

We conducted this performance audit from June 2015 to November 2016  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

section_title: Appendix II: Comments from the Department of Defense

section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Acknowledgments
paragraphs: In addition to the contact named above, Travis Masters (Assistant  Director); Peter Anderson; Marvin Bonner; Robert Bullock; Kurt Gurka;  Scott Hiromoto; Jean McSween; Robert Miller; LeAnna Parkey; and  Roxanna T. Sun made key contributions to this report.

Fast facts: The Department of Defense expects to spend more than $1.4 trillion to develop and procure its current portfolio of major weapon system programs, such as the F-35 Joint Strike Fighter and the KC-46A refueling tanker. DOD's major programs have historically come in over cost and behind schedule.

Our latest look at 9 programs found that delays and overruns might be averted through early, detailed systems engineering—a disciplined process for ensuring that program requirements can be met with available resources.

We recommended DOD submit systems engineering plans to Congress when requesting new program funding.

Question-summary annotations: questions: question: What challenges do a given weapon system's requirements pose?
answer: GAO's analysis of nine case studies identified four factors that frame the challenge posed by a given weapon system's requirements: acquisition approach, technology status, design maturity, and system interdependency.

question: Why is systems engineering crucial to the success of a program?
answer: Systems engineering is the primary means for determining whether and how that challenge can be met. It is a disciplined learning process that translates requirements into specific design features and thus identifies key risks to be resolved.
child_questions: question: How does the timing of systems engineering negatively or positively impact a program?
answer: GAO's prior best practices work has found that if detailed systems engineering is done early, a program can resolve such risks through trade-offs and additional investments before a program starts.

question: What is the advantage of utilizing preliminary designs?
answer: A key point in systems engineering where this match can be assessed is the preliminary design. As shown below, establishing a preliminary design through early detailed systems engineering portends better program outcomes than doing so after program start.

questions: question: What kinds of programs had better outcomes?
answer: Programs with modest requirements and early detailed systems engineering had better outcomes.
child_questions: question: What specifically about such programs led to desired outcomes?
answer: For example, the Small Diameter Bomb Increment I program, which delivered within cost and schedule estimates, had an incremental approach, mature technologies, a derivative design, and detailed systems engineering before development began.

question: Why did some programs fail to produce the desired outcome?
answer: Programs that began with more challenging requirements and insufficient systems engineering reported worse outcomes. For example, the F-35 Lightning II, which has encountered significant cost and schedule problems, began development with a single-step approach, a highly complex design, immature technologies, and little systems engineering.

questions: question: What challenges must Congress overcome in overseeing product development?
answer: Understanding the dynamic between a program's requirements, risks, and the requisite systems engineering effort has important implications for oversight. A particular challenge is that Congress often must consider requests to authorize and fund a new program in advance of the start of product development, when the business case would be better established.

question: How does DOD policy address the need to take into account risk factors of a program?
answer: DOD policy requires that DOD decision makers have information about a proposed program's risk factors and systems engineering status, in a systems engineering plan, at the start of a new program.

question: How could DOD better communicate its process to Congress?
answer: However, it is not clear whether Congress also has this information at that time. A systems engineering plan could provide more robust information to Congress when considering a budget request to start a new program.

question: What was DOD's response to this report?
answer: In commenting on a draft of this report DOD disagreed.

questions: question: What does Report 114-102 address?
answer: This report (1) identifies a framework for assessing the challenge posed by weapon system requirements and the extent of systems engineering done before product development begins; (2) illustrates the relationship between systems engineering and program outcomes; and (3) assesses implications for program oversight.
child_questions: question: How did GAO assess the role of system engineering in product development?
answer: GAO analyzed a non-generalizable sample of nine case studies. GAO assessed the extent to which systems engineering was conducted before development by reviewing program requirements and analyzing cost and schedule documentation for each case study.