Defense Acquisition Process: Military Service Chiefs' Concerns Reflect Need to Better Define Requirements before Programs Start
Released: 2015-06-11T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: GAO has reported extensively on problems in cost, schedule, and performance for major defense acquisition programs. According to some acquisition reform advocates, expanding the role of the military service chiefs in the process to acquire weapon systems may improve acquisition outcomes.

Following a 2014 GAO report on the service chiefs' role in the acquisition chain of command, GAO was asked to review further related issues and concerns the service chiefs have with the acquisition process and its outcomes. This report examines: (1) the views of current and former military service chiefs on the acquisition process, and (2) key problems or factors the service chiefs identified with the acquisition process and GAO's assessment of these issues.

GAO conducted interviews with 12 current and former military service chiefs and vice chiefs, and with other current and former DOD leadership to discuss the acquisition process. GAO also drew upon its extensive body of work on defense acquisitions and best practices. To assess key problems with the current process, GAO reviewed program execution information on all 78 current major defense programs.

section_title: What GAO Found
paragraphs: Most current and former military service chiefs and vice chiefs GAO interviewed from the Army, Air Force, Navy, and Marine Corps collectively expressed dissatisfaction with acquisition program outcomes and believed that the Department of Defense's (DOD) requirements development and acquisition processes need to be better integrated. The service chiefs are largely responsible for developing the services' requirements for weapon systems, while the service acquisition executives are responsible for overseeing programs to plan and develop systems. Most service chiefs told GAO they were concerned that after weapon system requirements are handed to the acquisition process, requirements are changed or added by the acquisition community (sometimes referred to as “creep”), increasing the capabilities and cost of the system. Some service chiefs stated that they are not always involved in the acquisition process and are frequently caught by surprise when cost, schedule, and performance problems emerge in programs. Current and former chiefs agreed that the chiefs should be more involved in programs, but their views varied on how best to achieve this.

GAO analyzed requirements for all 78 major defense acquisition programs and found that creep—or growth—in the high-level requirements is rare. Instead, it is after a program has formally started development that the myriad lower-level, technical requirements needed to complete a weapon system's design are defined (see figure). It is the definition of these requirements—most of which occurs after the service chiefs' primary involvement—that leads to the realization that much more time and resources are needed to build the weapon system.

The process of systems engineering translates high-level requirements, such as range, into specifics, like fuel tank size. GAO has previously reported on the importance of conducting systems engineering early so that the consequences of high-level requirements can be confronted before a program starts. When GAO presented its analysis of the problem to the service chiefs, they generally agreed with it. Several noted that trade-offs informed by systems engineering must take place before programs start so that requirements are better defined and more realistic cost, schedule, and performance commitments can be made.

section_title: What GAO Recommends
paragraphs: GAO recommends that DOD ensure sufficient systems engineering is conducted to better define requirements and assess resource trade-offs before a program starts. DOD concurred with the recommendations, citing recent policy changes. GAO believes more focus is needed on implementing actions.

section_title: Letter
paragraphs: For years, GAO has reported on problems in cost, schedule, and  performance for major defense acquisition programs within the  Department of Defense (DOD). Poor program outcomes can be traced to  a culture in which the military services begin programs with unrealistic  requirements, immature technologies, and overly optimistic cost and  schedule estimates. In recent years, acquisition reform advocates have  suggested that acquisition outcomes could be improved by expanding the  role of military service chiefs with respect to managing and overseeing  weapon system programs. In 2014, GAO issued a report on DOD’s  acquisition chain of command and found that the military service chiefs  currently have several opportunities to contribute to the management and   However, the extent to which  oversight of defense acquisition programs.the chiefs are actively involved and influencing acquisition programs  remained unclear.

You asked GAO to conduct additional work on this topic and review what  concerns, if any, the military service chiefs have with the department’s  acquisition process and outcomes it produces. the views of current and former military service chiefs on the current  acquisition process, and (2) key problems or factors the service chiefs  identified with the acquisition process and our assessment of these  issues.

This report examines: (1)

Senator Inhofe’s August 2014 request was in his role as Ranking Member of the Senate  Armed Services Committee in the 113th Congress. former military service chiefs and vice chiefs. We met with all current  military service chiefs and vice chiefs as of September 2014, including the  Chief and Vice Chief of Staff of the Air Force, the Chief and Vice Chief of  Staff of the Army, the Chief and Vice Chief of Naval Operations, and the  Commandant and Assistant Commandant of the Marine Corps. We also  sought the perspectives of other DOD leadership, including the service  acquisition executives. We analyzed evidence and examples collected  from our interviews with current and former DOD leadership, and  assessed them in relation to findings from acquisition reform studies and  prior GAO work on DOD weapon systems and commercial best practices.  To assess key problems or factors the service chiefs identified with the  acquisition process and our assessment of these issues, we drew upon  our extensive body of work in defense acquisitions and best practices,  and reviewed program execution information from all 78 ongoing major  defense acquisition programs identified in annual Selected Acquisition  Reports. We assessed the reliability of the data by reviewing existing  information about the system and determined that the data were  sufficiently reliable for the purposes of this report. In December 2014,  after completing our interviews and analysis, we sent a summary of the  problems with the acquisition process identified during our interviews to  the seven current and former military service chiefs whom we interviewed  for their review and comment. We received responses from five of the  seven service chiefs. See appendix I for additional information on our  objectives, scope, and methodology.

We conducted this audit from October 2014 to June 2015 in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.
subsections: section_title: Background
paragraphs: Within DOD’s overall acquisition framework, there are three key decision- support processes—the acquisition management system, requirements  determination, and resource allocation—that must work closely together  for acquisition programs to successfully deliver the right weapon systems  at the right time and right price. Each process is managed and overseen  by different organizations and leaders within DOD and the military  departments. At the DOD level, the Under Secretary of Defense for  Acquisition, Technology, and Logistics (USD (AT&L)) is responsible for  the acquisition function and is the Milestone Decision Authority (MDA) for  major defense acquisition programs, whereas the Joint Chiefs of Staff are  responsible for implementing the requirements process, and the Under  Secretary of Defense (Comptroller) is responsible for the resource  process. At the military department level, where programs are largely  planned and executed, the civilian service acquisition executive is  responsible for the acquisition process, while the service chiefs have  responsibility for assisting the military departments in the development of  requirements and the resourcing processes. We have previously found  that these processes are fragmented, making it difficult for the department  to achieve a balanced mix of weapon systems that are affordable and  feasible and provide the best military value to the warfighter. In recent  years, Congress and DOD have taken steps to better integrate the  requirements and acquisition processes. For example, the department  added new decision points and reviews for weapon programs as they  progress through the acquisition process. Additionally, USD (AT&L) now  serves as an advisor to the council that reviews requirements for major  weapon programs. Furthermore, the Fiscal Year 2011 National Defense  Authorization Act delineated that the service chiefs have a responsibility  to assist the secretaries of the military departments concerned in carrying  out the acquisition function.

Generally, major defense acquisition programs go through a series of  phases as they progress from the identification of the need for a new  capability, through initial planning of a solution, to system development,  and finally production and deployment of a fielded system. High-level,  operational requirements of major weapon systems are first generated,  vetted, and put forward for DOD-level review and approval, generally by  the military services. These requirements are prioritized based on how  critical the associated system characteristics are to delivering the military  capability. Key performance parameters are considered most critical by  the sponsor military organization, while key system attributes and other  performance attributes are considered essential for an effective military  capability. Through systems engineering efforts, these high-level  requirements must then be translated into lower-level technical  requirements and specifications to design and build the weapon system.  Figure 1 illustrates the notional types and levels of requirements for  weapon system development.

Following military service-level reviews and approvals, the high-level  operational requirements, which are specified in a capability development  document, go through several key stages where DOD-level reviews and  validations are required, a process accomplished for joint military  requirements within the department’s Joint Capabilities Integration and  Development System (JCIDS) process. Capability requirements  documents for these programs are assessed and validated within JCIDS  by the Chairman of the Joint Chiefs of Staff with the advice of the Joint  Requirements Oversight Council (JROC), which is chaired by the Vice  Chairman of the Joint Chiefs of Staff and is comprised of the Vice Chiefs  of Staff of each military service and the Combatant Commanders, when  directed by the chairman.

These high-level requirements along with several other acquisition-related  analyses and documents (e.g., acquisition strategy, cost estimates, and  test and evaluation plan) are required for approval at Milestone B, when  an acquisition program formally starts system development. As major  defense acquisition programs go through the iterative phases of the  acquisition process, they are reviewed by the Defense Acquisition Board,  which is chaired by USD (AT&L) and includes the secretaries of the  military departments and other senior leaders. However, prior to these  DOD-level reviews, programs have reviews and approvals at the military  service level where the service acquisition executives and service chiefs  are involved. In our prior report on the acquisition chain of command, we  found that service chiefs and their supporting offices have multiple  opportunities to be involved in major defense acquisition programs  throughout the acquisition process, including participation in integrated  product teams, service-level reviews during system development, and  requirements review and approval prior to a program’s production  decision. Figure 2 illustrates DOD’s current acquisition process and  where the military service chiefs and service acquisition executives have  primary responsibilities. Generally, after Milestone B, when system  development begins in earnest, the chief’s role diminishes whereas the  service acquisition executive’s role becomes more prominent.

For more than a decade, we have recommended numerous actions to  improve the way DOD acquires its defense systems. Our work in  commercial best practices and defense acquisitions has consistently  found that, at the program level, a key cause of poor program outcomes  is the approval of programs with business cases that contain inadequate  knowledge about requirements and the resources—funding, time,  technologies, and people—needed to execute them.programs run into problems during system development because  requirements are unrealistic, technologies are immature, cost and  schedule are underestimated, and design and production risks are high.  Some key recommendations that we have made in the past to improve  DOD’s acquisition process include the following:

Require that systems engineering that is needed to evaluate the  sufficiency of available resources be conducted before weapon  system requirements are formalized;

Require, as a condition for starting a new weapon system program,  that sufficient evidence exists to show there is a match between a  weapon’s system requirements and the resources the program  manager has to develop that weapon;

Require program officials to demonstrate that they have captured  appropriate knowledge at program start (Milestone B), which includes  ensuring that requirements for the product are informed by the  systems engineering process, and establishing cost and schedule  estimates on the basis of knowledge from preliminary design using  system engineering tools;

Have contractors perform more detailed systems engineering analysis  to develop sound requirements before DOD selects a prime contractor  for the systems development contract; and

Define a shipbuilding approach that calls for (1) demonstrating  balance among program requirements, technology demands, and cost  considerations by preliminary design review, and (2) retiring technical  risk and closing any remaining gaps in design requirements before a  contract for detail design is awarded.

GAO, Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and  Schedule Problems under DOD’s Revised Policy, GAO-06-368 (Washington, D.C.: Apr.  13, 2006).

section_title: Military Service Chiefs Are Dissatisfied with Acquisition Outcomes, Citing Requirements Growth as a Key Problem
paragraphs: Most current and former military service chiefs that we interviewed  collectively expressed dissatisfaction with the current acquisition process  and the outcomes it produces. They were concerned that after validated  requirements are handed over to the acquisition process, requirements  are frequently added or changed to increase the scope and capabilities of  a weapon system. Some current and former service chiefs said that  because they lack visibility into programs, they are unable to influence  trade-offs between requirements and resources. However, their views  differed on how best to be more involved in the management of  acquisitions and improve the integration between DOD’s requirements  and acquisition functions.
subsections: section_title: Service Chiefs Cite Requirements Growth as a Key Problem in Acquisitions
paragraphs: Most of the current and former service chiefs that we interviewed were  dissatisfied with the current acquisition process and stated that programs  often fail to deliver needed operational capabilities to the warfighter with  expected resources—such as technologies and funding—and in expected  time frames. They were concerned that requirements are developed  within a military service, validated by the JROC, handed over to the  acquisition process and then, later on—years later—program cost,  schedule, and performance problems materialize. According to a number  of both current and former service chiefs, they are not always involved in  the acquisition process and are frequently caught by surprise when these  problems emerge. Several service chiefs saw a key factor contributing to  this condition as unplanned requirements growth—sometimes referred to  as “creep”—that occurs during program execution.

Several current and former service chiefs expressed the view that, after a  program is approved and system development is underway, requirements  are frequently added or changed to increase the scope and capabilities of  a weapon system beyond the requirements originally agreed upon when  the program started. One current service chief cited an example where  program officials unnecessarily created a lower-level requirement for an  aircraft system that did not meet any validated operational need. The  service chief attributed the problem, in part, to a lack of military officers  with acquisition expertise and a corresponding absence of acquisition  officials with operational expertise. A former DOD official pointed to the  lengthy timeframe usually involved in developing major weapon systems  and how requirements increases occur because programs want to  introduce the latest technology advances into a system, such as  information technology and electronics equipment. Some current and  former service chiefs stated that because they lack visibility into  programs, they are unable to influence trade-offs between requirements  and resources. One current service chief provided an example when  program officials, in an effort to meet a validated operational requirement  for speed, were developing an engine that led to cost increases, while he  believed there was an existing engine available that would have required  a minor reduction in capability in favor of reducing cost.

The service chiefs also had concerns that requirements growth is a  function of too many stakeholders within DOD having the ability to  influence acquisition programs, making it difficult to hold anyone  accountable for program outcomes. Many of these service chiefs believed  that cultural factors and incentives within the department make it difficult  for program managers to manage requirements growth and execute  programs effectively. These chiefs said that program managers and other  acquisition officials often lack experience and expertise to manage  requirements and acquisitions, are incentivized to meet internal  milestones and not raise issues, and rely too much on contractors to  figure out what is needed to develop a weapon system. Further, they  noted that high turnover in program manager tenure—approximately  every 2-3 years according to several service chiefs—make it difficult to  hold managers accountable when problems emerge.

section_title: Service Chiefs Propose a Range of Solutions to Be More Involved in Acquisitions
paragraphs: To improve weapon system program outcomes, both current and former  service chiefs agree that they should be more involved in the acquisition  process. However, their views differed on the measures needed to  achieve more involvement and improve integration between DOD’s  requirements and acquisition functions. Most current service chiefs said  that better collaboration does not require restructuring the chain of  command. These service chiefs cited examples of ongoing collaboration  between requirements and acquisition offices, and programs where they  worked closely with acquisition leadership to address problems. In one  case, a service chief pointed to an initiative that he and the service  acquisition executive instituted to provide technical training and  assistance to uniformed requirements officers as an example of  formalized collaboration before and after the start of system development.  Another service chief indicated that, faced with rising program costs and  the possibility of cancellation, he actively monitored program progress  through regular meetings with the program manager and contractor.

Current service chiefs and other acquisition leadership generally indicated  the service chiefs have the ability to be more involved in the current  process, such as by attending service and DOD level program reviews.  However, some chiefs indicated that involvement in acquisition programs,  in general, varies by service chief based on their priorities and the other  personalities involved. Several current and former service chiefs agreed  that they have been involved in the oversight of some programs, but their  level of involvement is dependent on the importance of the program and  established working relationships with the service acquisition executive.  One service chief stated that, at times, service chiefs have not been  involved due to unfamiliarity with the acquisition process, their own  perceived role in the process, or a lack of interest in an acquisition.

Several former service chiefs thought that establishing co-chairmanship  for key decision reviews and co-signature of key acquisition documents,  particularly at the military department level, may improve collaboration,  encourage requirements trade-offs during development, and force the  service chiefs to share the burden of responsibility for acquisition  programs. One suggestion from an outside expert for implementing this  solution was to have the service chief and the service acquisition  executive co-chair the service-level acquisition review board. Some  military and acquisition leaders noted, however, that requiring co- chairmanship of acquisition meetings and co-signature of decision  documents could slow an already complex process and further  discourage program managers from raising issues and concerns.

In general, the former service chiefs we interviewed emphasized the need  for a stronger role in the acquisition chain of command with more formal  authority and mechanisms in place to ensure that the service chiefs are  consistently involved and sufficiently able to influence program decisions.  However, as we found in our prior review, studies that have advocated for  a stronger role for the service chiefs in the acquisition process provide   Several of  little evidence that this would improve program outcomes.these former service chiefs advocated for changes to DOD policy and  statute, including the Goldwater-Nichols Act.service chief believed that DOD acquisition policy should require service  chief approval on all major defense acquisition programs prior to program  start. Some acquisition experts have observed that, in giving sole  responsibility for acquisitions to the military secretaries through the  service acquisition executives, DOD created an unintended wall when  implementing the Goldwater-Nichols Act reforms between the military- controlled requirements process and civilian-driven acquisition process.  These acquisition experts note, however, that while service chiefs had  significant influence on certain acquisition programs in the past, their  close involvement did not always result in successful cost, schedule, or  performance outcomes. For example, service chiefs had significant  involvement in the Navy’s Littoral Combat Ship and the Army’s Future  Combat System and, in both cases, viewed the programs as providing  vital operational capabilities and needing to be fielded quickly.  Consequently, the programs pursued aggressive acquisition strategies  that pushed the programs through development with ill-defined  requirements and unstable designs, which contributed to significant cost  and schedule increases, and in the case of the Future Combat System,  program cancellation.

section_title: Defining Lower-Level Requirements Is Often the More Significant Contributing Factor to Program Cost Increases
paragraphs: Acting on the chiefs’ concerns, we analyzed all 78 major defense  acquisition programs and found that growth in high-level requirements— and consequent cost growth—was rare. Rather, we found that cost  growth and other problems are more directly related to deriving lower- level requirements after a program has started. The distinction between  high-level and lower-level requirements is key. Growth in high-level  requirements could be attributable to a lack of discipline, but growth in  lower-level requirements is not the result of additions, but rather the  definition and realization of the details necessary to meet the high-level  requirements. The process of defining lower-level requirements is an  essential function of systems engineering, much of which is done late— after a development contract has been signed and a program has started.

In other words, requirements are insufficiently defined at program start;  when their full consequences are realized, trade-offs are harder to  make—cost increases and schedule delays become the preferred  solutions. We presented our assessment of the requirements problem to  current and former service chiefs and they generally agreed with it.  Several service chiefs noted that more integration, collaboration, and  communication during the requirements and acquisition processes needs  to take place to ensure that trade-offs between desired capabilities and  expected costs are made and that requirements are essential, technically  feasible, and affordable before programs get underway. Some service  chiefs believed that applying systems engineering to arrive at well-defined  requirements before the start of system development at Milestone B can  go a long way towards solving some of their dissatisfaction with the  acquisition process and improving outcomes.
subsections: section_title: Key Requirements Have Not Changed Significantly in Major Defense Acquisition Programs Experiencing Cost Growth since 2009
paragraphs: We found few instances of requirements changes between 2009 and  2013 that involved increasing capabilities on major defense programs  during system development. Seventeen programs in the current portfolio  of 78 major defense acquisition programs experienced system  development cost growth of more than 20 percent between 2009 and  2013, but 13 of them did not report associated key requirements  increases (see table 1). A number of factors other than requirements  increases contributed to the cost growth in these programs.

We found that, within the current portfolio of major defense acquisition  programs, 5 of 78 programs reported increases to key performance  parameters between 2009 and 2013. In these 5 programs, the changes  involved adding a new component, technology, or other subsystem to  increase the capabilities of the weapon system. Table 2 describes the  requirement changes reported by these 5 programs. In 4 programs,  development cost increases were more than 20 percent during the same  time period.

section_title: Insufficiently Defined Requirements Led to Problems during Weapons System Development
paragraphs: A key factor consistently identified by GAO in prior reports is the  mismatch between the requirements for a new weapon system and the  resources—technologies, time, and funding—that are planned to develop   Requirements, especially at the lower levels, are often  the new system.not fully developed or well-defined when passed over to the acquisition  process at Milestone B, at which time a system development contract is  awarded and a program begins. During system development, the high- level operational requirements, such as key performance parameters and  key system attributes, usually need to be further analyzed by the  contractor using systems engineering techniques to fully understand,  break down, and translate them into technical weapon system-level  requirements and contract specifications. Systems engineering analysis  translates operational requirements into detailed system requirements for  which requisite technological, software, engineering, and production  capabilities have been identified. It also provides knowledge to enable the  developer to identify and resolve gaps before system development  begins. It is often at this point—when the technical specifications are  finally understood and design challenges are recognized—that cost and  schedule increases materialize in a program. What may appear to be  requirements growth is the recognition that the weapon system will  require considerably more time and money than expected to build to  these derived technical specifications to meet the validated operational  requirements.

The process of translating high-level operational requirements into low- level requirements and technical specifications in many programs does  not usually occur until well after Milestone B approval (see figure 3 for a  notional depiction).

The number of requirements can expand greatly over time, as the  designs of the subsystems and components become defined. In the case  of the Army’s Future Combat System, a large program that was intended  to equip combat brigades with an advanced set of integrated systems,  requirements were still being defined when the program was canceled  beginning in 2009—after 6 years and $18 billion had been spent on initial  system development. The program was approved to start system  development with 7 key performance parameters. In order to meet these  key performance parameters—which did not change—the program  ultimately translated them into over 50,000 lower-level requirements  before it was canceled. Requirements definition remains a challenge  facing current major defense acquisition programs. For example, the F-35  program, which was conceptualized around three aircraft design variants  to achieve cost efficiencies, has had difficulty reconciling different  requirements imposed by the military services. According to program  officials, in order to meet the nine validated key performance parameters,  the program developed approximately 3,600 specifications. While the  operational requirements for the F-35 have not increased, factors such as  poorly defined requirements, significant concurrency between  development and production, and immature technologies have  contributed to significant cost growth and delays in the program.

We found that several of the major defense acquisition programs that  experienced cost growth, but did not report changing key performance  parameters, had a significant number of engineering change orders and  other configuration changes. As operational requirements become better  understood during system development, contract specifications change to  reflect what is needed to build the weapon system. Changes show up in  engineering change orders and other design configuration changes,  which contribute to cost growth. For example, between 2009 and 2013,  the Littoral Combat Ship program reported 487 changes to its system  configuration or design. Similarly, the Joint Tactical Radio System  Handheld, Manpack, and Small Form Fit Radios program reported  making 29 engineering changes and 11,573 software changes between  2009 and 2013. In neither case were the high-level requirements  increased. While some configuration changes are necessary to manage  obsolescence and other issues, the pursuit of poorly defined  requirements results in overly optimistic cost and schedule estimates that  are sometimes unachievable—leading to cost and schedule growth as  programs encounter increased technical challenges necessary to achieve  operational requirements.

GAO’s prior work as well as DOD’s own policy emphasizes that the  translation of operational requirements into technical weapon system  specifications, which are informed by systems engineering, should take  place prior to approving a program at Milestone B and awarding a  contract that locks in the requirements. This allows trade-offs between  requirements and resources to take place, and the establishment of more  realistic cost, schedule, and performance commitments before programs  get underway. However, DOD often does not perform sufficient up-front  requirements analysis via systems engineering on programs to determine  whether the requirements are feasible and there is a sound business  case to move forward. Programs are proposed with unachievable  requirements and overly optimistic cost and schedule estimates and,  usually, participants on both the requirements side and the acquisition  side are loathe to trade away performance. For example, a preliminary  design review is a key systems engineering event that should be held  before the start of system development to ensure requirements are  defined and feasible, and the proposed design can meet the requirements  within cost, schedule, and other system constraints. In 2013, GAO  reviewed the 38 major defense acquisition programs that held preliminary  design reviews that year. Only 11 of these programs held design  reviews prior to the start of system development. The remaining 27  programs completed or planned to complete their design reviews  approximately 24 months, on average, after the start of development.  Thus, the resource consequences of deriving lower-level requirements  are similarly deferred.

section_title: Service Chiefs Generally Agree That Poorly Defined Requirements Contribute to Program Cost Growth
paragraphs: We shared a summary of our assessment of the requirements problem,  namely that high-level requirements are poorly defined when passed over  to the acquisition process at the start of development, with the current  and former service chiefs and they generally agreed with our findings.  Several current and former service chiefs indicated that requirement and  resource trade-offs, informed by systems engineering, do not consistently  take place before programs get underway. Some chiefs also noted that  reassessments of requirements, acquisition, and funding are not  conducted often enough during program execution. According to one  service chief, under the current acquisition process, there are too few  points of collaboration among requirements officers, acquisition  professionals, systems engineers, and cost estimators to work out  requirements early in the process or to address problems and limitations  associated with meeting operational requirements after programs are  underway. Another service chief noted that the acquisition workforce  lacks experience in operational and tactical settings and that his  requirements community lacks technical acquisition skills, so it is  important that collaboration regularly occurs between the two  communities. Further, one chief emphasized that requirements officers  are too dependent upon the acquisition community and its contractors to  work out requirements.

Several current and former service chiefs voiced concern that cost and  schedule problems that acquisition programs experience are due to the  failure to make appropriate trade-offs during system development. They  indicated that too often programs encounter cost and schedule problems  because in striving to meet challenging requirements the programs end  up making technical and design changes to the weapon system. For  example, one former service chief highlighted a combat vehicle program  in development which had fallen short of meeting its vehicle speed  requirement by a small percentage. Instead of making trade-offs, and  perhaps seeking requirements relief, the program manager requested  additional funding so the contractor could make design changes to the  engine. Another service chief stated that requirement changes made  during weapon system development are often viewed as sacrificing  capability rather than reconciling requirements with operational  conditions. The chief was concerned that program managers too often  take the view that requirements cannot be changed and avoid elevating  problems to leadership before they become critical, forgoing the  opportunity to make needed trade-offs. In addition, one service chief  described this problem as “cost creep” to meet requirements, not  “requirements creep”. We have previously found that incentives within the  current acquisition process create pressure on defense system  requirements and are geared toward delaying knowledge so as not to  jeopardize program funding.

Several current and former service chiefs agreed that there needs to be  more integration, collaboration, and communication during the  requirements and acquisition processes to ensure trade-offs are made  and the requirements that get approved are essential, technically feasible,  and affordable prior to the start of system development. Some service  chiefs said that conducting systems engineering analyses during  requirements setting and, again, early on during an acquisition program’s  planning phase to inform trade-offs between cost and capability could go  a long way toward establishing better defined requirements and improving  program outcomes. Almost all of the service chiefs stated that there is a  need to further enhance expertise within the government, and several  specified expertise in systems engineering. Several service chiefs  indicated that systems engineering capabilities are generally lacking in  the requirements development process, and do not become available  until after requirements are validated and an expensive and risky system  development program is underway. Some service chiefs advocated that  having systems engineering capabilities available to the military services  during requirements development could help to ensure earlier  assessment of requirements feasibility. The service chiefs’ views on the  importance of systems engineering is consistent with our prior acquisition  work, which calls for DOD to implement a knowledge-based approach to  guide the match of defense program needs with available technology and  resources.

The service chiefs expressed a willingness to be more involved in the  management and oversight of acquisition programs. Enhancing  collaboration between the requirements and acquisition processes could  be one of several steps needed to address the underlying culture and  incentives that exist in DOD that lead to programs that are not feasible  and affordable. We have found in prior work that characteristics of DOD’s  processes and incentives create pressure to push for unrealistic defense  system requirements and lead to poor decisions and mismatches   This culture has become  between requirements and resources. ingrained over several decades and a number of studies and reforms  have been directed at changing the incentives underlying the culture,  without much success.

section_title: Conclusions
paragraphs: GAO-01-288. be overly optimistic and to minimize the difficulty and resources needed to  deliver the capability. Many of our prior recommendations have been  aimed at this problem and, while one could argue whether more formal  authority should be granted to the service chiefs, the current acquisition  process allows for the service chiefs to be more involved in the  management and oversight of acquisition programs. Regardless, the  solution must involve investing in systems engineering expertise sooner— while developing requirements—to enable technological knowledge to  better shape and define operational requirements.

Recommendations such as holding preliminary design reviews before the  start of system development have been made as a means to improve  program outcomes. After initial support, the enthusiasm for these  practices wanes and the old pressures to continue with insufficient  knowledge prevail, because the old practices allow programs to proceed  and funding to flow. Importantly, the negative consequences of  proceeding with limited knowledge are not sufficient to counteract these  pressures, as accountability for the initial poor decisions is lost by the  time problems emerge. Information and expertise will not result in good  outcomes unless the need for a solid business case is reinforced. In order  to improve program outcomes, DOD must focus its efforts on better  integrating the requirements and acquisition processes, which can be  achieved through better collaboration between these communities from  the generation of requirements through system development, coupled  with a greater emphasis on systems engineering and knowledge  attainment early in a program’s life cycle. Without sufficient systems  engineering input to better define requirements and examine trade-offs  early on, there is no assurance that acquisition programs going forward  have a sound basis to start system development.

section_title: Recommendations for Executive Action
paragraphs: To help ensure that requirements are well defined and well understood  before a program is approved to start system development, we  recommend that the Secretary of Defense direct the military service  chiefs and service acquisition executives to work together to take the  following two actions:

Assess whether sufficient systems engineering expertise is available  during the requirements development process; and

Develop a better way to make sure sufficient systems engineering is  conducted and opportunities exist to better define requirements and  assess resource trade-offs before a program starts.

section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us with written comments on a draft of this report, which  are reprinted in appendix II. The department concurred with both of our  recommendations, stating that the early application of systems  engineering expertise and ensuring the availability of appropriately skilled  personnel are critical to successful program outcomes. DOD noted that  recent changes to department-wide policies, such as DOD Instruction  5000.02, strengthen the department’s focus on conducting systems  engineering and making trade-offs during requirements development and  pre-program planning. DOD further agreed that continuing to improve  engagement between the requirements and acquisition communities will  result in better informed program initiation and resourcing decisions.

We are encouraged that DOD agrees with our recommendations and has  recently taken steps to strengthen its policies and identify the need for  early systems engineering. However, for many years DOD policies have  emphasized the importance of a knowledge-based approach to acquiring  weapon systems, but practice does not always follow policy. Instead,  incentives exist that encourage deviation from sound policies and  practices. We believe that DOD must focus on achieving better  collaboration between the requirements and acquisition communities  such as by ensuring that more systems engineering and other expertise  are applied when requirements are being defined. It is through informed  collaboration that knowledge will be attained, trade-offs between  requirements and resources can be made earlier, and acquisition  programs will begin development with realistic cost and schedule  estimates, ultimately leading to improved outcomes.

We are sending copies of this report to appropriate congressional  committees; the Secretary of Defense; the Secretaries of the Air Force,  Army, and Navy; the Chief of Staff of the Air Force; the Chief of Staff of  the Army; the Chief of Naval Operations; the Commandant of the Marine  Corps; and the Under Secretary of Defense for Acquisition, Technology,  and Logistics. In addition, this report also is available at no charge on the  GAO website at http://www.gao.gov.

If you have any questions about this report or need additional information,  please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. Key contributors to this report are  listed in appendix III.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: GAO issued a report in 2014 on the military service chiefs’ role in the  acquisition chain of command. This report reviews further related issues  and concerns the military service chiefs have with the Department of  Defense’s (DOD) acquisition process and outcomes it produces.  Specifically, we examined (1) the views of current and former military  service chiefs on the current acquisition process, and (2) key problems or  factors the service chiefs identified with the acquisition process and our  assessment of these issues.

To obtain the views of current and former military service chiefs on the  current acquisition process, we conducted interviews with 12 current and  former military service chiefs and vice chiefs between August and  December 2014. We met with all current military service chiefs and vice  chiefs as of September 2014, including the Chief and Vice Chief of Staff  of the Air Force, the Chief and Vice Chief of Staff of the Army, the Chief  and Vice Chief of Naval Operations, and the Commandant and Assistant  Commandant of the Marine Corps. These individuals possessed joint-  and service-level experience, including positions as the Chairman of the  Joint Chiefs of Staff and Combatant Commander. In December 2014,  after completing our interviews, we analyzed our findings and sent a  summary to the four current and three former military service chiefs, but  not the vice chiefs, for their review and comment. We received responses  from five of the seven service chiefs, all of whom concurred with our  findings. We reviewed any comments and made changes to the summary  document, as appropriate. We also interviewed or sought the  perspectives of additional current and former DOD leadership, including  the Service Acquisition Executive of the Air Force, Army, and Navy;  officials from the Office of the Secretary of Defense, Joint Staff; and  another former member of the Joint Chiefs of Staff. We analyzed  evidence and examples collected from our interviews with current and  former military service chiefs and DOD leadership. We also reviewed  findings from existing reports and compendiums focused on the  acquisition chain of command and interviewed acquisition subject matter  experts to discuss the current acquisition process, the role of the military  service chiefs in the acquisition chain of command, and potential  solutions to improve program outcomes. We reviewed prior GAO work on  weapon system acquisition and commercial best practices and analyzed  the extent to which evidence exists that would demonstrate that these  potential solutions may improve program outcomes.

To assess key problems or factors the service chiefs identified with the  acquisition process and our assessment of these issues, we drew upon  our extensive body of work in defense acquisitions and best practices,  and reviewed program execution information from ongoing major defense  acquisition programs. We reviewed the annual Selected Acquisition  Reports (SAR) from 2009 to 2013 for the 78 programs in DOD’s current  portfolio of major defense acquisition programs. SAR data was collected  from the Defense Acquisition Management Information Retrieval (DAMIR)  Purview system, referred to as DAMIR. We assessed the reliability of the  data by reviewing existing information about DAMIR and determined that  the data were sufficiently reliable for the purposes of this report. We  analyzed the performance metrics to determine the extent to which  programs were reporting changes to key performance parameters. Our  analysis was limited to unclassified requirements that are included as part  of the SAR. In October 2014, we developed and submitted a  questionnaire to 28 major defense acquisition programs that had reported  key requirement changes in their respective SAR from 2009 to 2013,  sought requirement relief from the Joint Requirements Oversight Council,  or experienced a development cost increase or decrease of 10 percent or  more between 2011 and 2013. We conducted two pretests of the  questionnaire prior to distribution to ensure that our questions were clear,  unbiased, and consistently interpreted. We obtained responses from all  28 programs, and in cases where questionnaire results differed from  previously collected SAR data, we submitted follow-up questions to the  program office to adjudicate any discrepancies. To determine the extent  to which programs that experienced development cost growth also  changed key requirements, we compared the research, development, test  and evaluation cost estimates from 2009 and 2013 for DOD’s current  portfolio of major defense acquisition programs, as reported in their  annual SAR. In instances where a program began development after  2009, we compared the program’s initial research, development, test and  evaluation cost estimate with its 2013 current estimate. We then reviewed  any program that had a cost increase of more than 20 percent to  determine if this program also reported key requirement changes in its  annual reports for the same time period. We also leveraged prior and  ongoing GAO work on weapon system acquisition.

We conducted this audit from October 2014 to June 2015 in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.

section_title: Appendix II: Comments from the Department of Defense

section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact:

section_title: Staff Acknowledgments:
paragraphs: In addition to the contact named above, John E. Oppenheim, Assistant  Director; Jacob Leon Beier; Brandon H. Greene; Laura M. Jezewski;  Megan Porter; Abby C. Volk; Marie P. Ahearn; Peter W. Anderson; Jean  L. McSween; and Kristy E. Williams made key contributions to this report.


Question-summary annotations: questions: question: What were the reactions of the chiefs and vice chiefs GAO interviewed?
answer: Most current and former military service chiefs and vice chiefs GAO interviewed from the Army, Air Force, Navy, and Marine Corps collectively expressed dissatisfaction with acquisition program outcomes and believed that the Department of Defense's (DOD) requirements development and acquisition processes need to be better integrated.
child_questions: question: What concerns did service chiefs have when interviewed?
answer: Most service chiefs told GAO they were concerned that after weapon system requirements are handed to the acquisition process, requirements are changed or added by the acquisition community (sometimes referred to as “creep”), increasing the capabilities and cost of the system. Some service chiefs stated that they are not always involved in the acquisition process and are frequently caught by surprise when cost, schedule, and performance problems emerge in programs.
child_questions: question: What suggestions did the service chiefs make to solve these problems?
answer: Current and former chiefs agreed that the chiefs should be more involved in programs, but their views varied on how best to achieve this.

question: What are the responsibilities of service chiefs and acquisition executives?
answer: The service chiefs are largely responsible for developing the services' requirements for weapon systems, while the service acquisition executives are responsible for overseeing programs to plan and develop systems.

questions: question: How rare is creep in high-level requirements?
answer: GAO analyzed requirements for all 78 major defense acquisition programs and found that creep—or growth—in the high-level requirements is rare.

question: What happens after a program has formally started development?
answer: Instead, it is after a program has formally started development that the myriad lower-level, technical requirements needed to complete a weapon system's design are defined (see figure).
child_questions: question: What realizations are drawn from these requirements?
answer: It is the definition of these requirements—most of which occurs after the service chiefs' primary involvement—that leads to the realization that much more time and resources are needed to build the weapon system.

questions: question: What is systems engineering?
answer: The process of systems engineering translates high-level requirements, such as range, into specifics, like fuel tank size.

question: Why is it important to conduct systems engineering early?
answer: GAO has previously reported on the importance of conducting systems engineering early so that the consequences of high-level requirements can be confronted before a program starts.

question: What was the response to GAO's analysis of the problem?
answer: When GAO presented its analysis of the problem to the service chiefs, they generally agreed with it.
child_questions: question: What did the service chiefs note about systems engineering?
answer: Several noted that trade-offs informed by systems engineering must take place before programs start so that requirements are better defined and more realistic cost, schedule, and performance commitments can be made.