Defense Acquisition Process: Military Service Chiefs' Concerns Reflect Need to Better Define Requirements before Programs Start
Released: 2015-06-11T00:00:00

Letter

Background

Within DOD’s overall acquisition framework, there are three key decision- support processes—the acquisition management system, requirements determination, and resource allocation—that must work closely together for acquisition programs to successfully deliver the right weapon systems at the right time and right price. Each process is managed and overseen by different organizations and leaders within DOD and the military departments. At the DOD level, the Under Secretary of Defense for Acquisition, Technology, and Logistics (USD (AT&L)) is responsible for the acquisition function and is the Milestone Decision Authority (MDA) for major defense acquisition programs, whereas the Joint Chiefs of Staff are responsible for implementing the requirements process, and the Under Secretary of Defense (Comptroller) is responsible for the resource process. At the military department level, where programs are largely planned and executed, the civilian service acquisition executive is responsible for the acquisition process, while the service chiefs have responsibility for assisting the military departments in the development of requirements and the resourcing processes. We have previously found that these processes are fragmented, making it difficult for the department to achieve a balanced mix of weapon systems that are affordable and feasible and provide the best military value to the warfighter. In recent years, Congress and DOD have taken steps to better integrate the requirements and acquisition processes. For example, the department added new decision points and reviews for weapon programs as they progress through the acquisition process. Additionally, USD (AT&L) now serves as an advisor to the council that reviews requirements for major weapon programs. Furthermore, the Fiscal Year 2011 National Defense Authorization Act delineated that the service chiefs have a responsibility to assist the secretaries of the military departments concerned in carrying out the acquisition function.

Generally, major defense acquisition programs go through a series of phases as they progress from the identification of the need for a new capability, through initial planning of a solution, to system development, and finally production and deployment of a fielded system. High-level, operational requirements of major weapon systems are first generated, vetted, and put forward for DOD-level review and approval, generally by the military services. These requirements are prioritized based on how critical the associated system characteristics are to delivering the military capability. Key performance parameters are considered most critical by the sponsor military organization, while key system attributes and other performance attributes are considered essential for an effective military capability. Through systems engineering efforts, these high-level requirements must then be translated into lower-level technical requirements and specifications to design and build the weapon system. Figure 1 illustrates the notional types and levels of requirements for weapon system development.

Following military service-level reviews and approvals, the high-level operational requirements, which are specified in a capability development document, go through several key stages where DOD-level reviews and validations are required, a process accomplished for joint military requirements within the department’s Joint Capabilities Integration and Development System (JCIDS) process. Capability requirements documents for these programs are assessed and validated within JCIDS by the Chairman of the Joint Chiefs of Staff with the advice of the Joint Requirements Oversight Council (JROC), which is chaired by the Vice Chairman of the Joint Chiefs of Staff and is comprised of the Vice Chiefs of Staff of each military service and the Combatant Commanders, when directed by the chairman.

These high-level requirements along with several other acquisition-related analyses and documents (e.g., acquisition strategy, cost estimates, and test and evaluation plan) are required for approval at Milestone B, when an acquisition program formally starts system development. As major defense acquisition programs go through the iterative phases of the acquisition process, they are reviewed by the Defense Acquisition Board, which is chaired by USD (AT&L) and includes the secretaries of the military departments and other senior leaders. However, prior to these DOD-level reviews, programs have reviews and approvals at the military service level where the service acquisition executives and service chiefs are involved. In our prior report on the acquisition chain of command, we found that service chiefs and their supporting offices have multiple opportunities to be involved in major defense acquisition programs throughout the acquisition process, including participation in integrated product teams, service-level reviews during system development, and requirements review and approval prior to a program’s production decision. Figure 2 illustrates DOD’s current acquisition process and where the military service chiefs and service acquisition executives have primary responsibilities. Generally, after Milestone B, when system development begins in earnest, the chief’s role diminishes whereas the service acquisition executive’s role becomes more prominent.

For more than a decade, we have recommended numerous actions to improve the way DOD acquires its defense systems. Our work in commercial best practices and defense acquisitions has consistently found that, at the program level, a key cause of poor program outcomes is the approval of programs with business cases that contain inadequate knowledge about requirements and the resources—funding, time, technologies, and people—needed to execute them.programs run into problems during system development because requirements are unrealistic, technologies are immature, cost and schedule are underestimated, and design and production risks are high. Some key recommendations that we have made in the past to improve DOD’s acquisition process include the following:

Require that systems engineering that is needed to evaluate the sufficiency of available resources be conducted before weapon system requirements are formalized;

Require, as a condition for starting a new weapon system program, that sufficient evidence exists to show there is a match between a weapon’s system requirements and the resources the program manager has to develop that weapon;

Require program officials to demonstrate that they have captured appropriate knowledge at program start (Milestone B), which includes ensuring that requirements for the product are informed by the systems engineering process, and establishing cost and schedule estimates on the basis of knowledge from preliminary design using system engineering tools;

Have contractors perform more detailed systems engineering analysis to develop sound requirements before DOD selects a prime contractor for the systems development contract; and

Define a shipbuilding approach that calls for (1) demonstrating balance among program requirements, technology demands, and cost considerations by preliminary design review, and (2) retiring technical risk and closing any remaining gaps in design requirements before a contract for detail design is awarded.

GAO, Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD’s Revised Policy, GAO-06-368 (Washington, D.C.: Apr. 13, 2006).

Military Service Chiefs Are Dissatisfied with Acquisition Outcomes, Citing Requirements Growth as a Key Problem

Most current and former military service chiefs that we interviewed collectively expressed dissatisfaction with the current acquisition process and the outcomes it produces. They were concerned that after validated requirements are handed over to the acquisition process, requirements are frequently added or changed to increase the scope and capabilities of a weapon system. Some current and former service chiefs said that because they lack visibility into programs, they are unable to influence trade-offs between requirements and resources. However, their views differed on how best to be more involved in the management of acquisitions and improve the integration between DOD’s requirements and acquisition functions.

Service Chiefs Cite Requirements Growth as a Key Problem in Acquisitions

Most of the current and former service chiefs that we interviewed were dissatisfied with the current acquisition process and stated that programs often fail to deliver needed operational capabilities to the warfighter with expected resources—such as technologies and funding—and in expected time frames. They were concerned that requirements are developed within a military service, validated by the JROC, handed over to the acquisition process and then, later on—years later—program cost, schedule, and performance problems materialize. According to a number of both current and former service chiefs, they are not always involved in the acquisition process and are frequently caught by surprise when these problems emerge. Several service chiefs saw a key factor contributing to this condition as unplanned requirements growth—sometimes referred to as “creep”—that occurs during program execution.

Several current and former service chiefs expressed the view that, after a program is approved and system development is underway, requirements are frequently added or changed to increase the scope and capabilities of a weapon system beyond the requirements originally agreed upon when the program started. One current service chief cited an example where program officials unnecessarily created a lower-level requirement for an aircraft system that did not meet any validated operational need. The service chief attributed the problem, in part, to a lack of military officers with acquisition expertise and a corresponding absence of acquisition officials with operational expertise. A former DOD official pointed to the lengthy timeframe usually involved in developing major weapon systems and how requirements increases occur because programs want to introduce the latest technology advances into a system, such as information technology and electronics equipment. Some current and former service chiefs stated that because they lack visibility into programs, they are unable to influence trade-offs between requirements and resources. One current service chief provided an example when program officials, in an effort to meet a validated operational requirement for speed, were developing an engine that led to cost increases, while he believed there was an existing engine available that would have required a minor reduction in capability in favor of reducing cost.

The service chiefs also had concerns that requirements growth is a function of too many stakeholders within DOD having the ability to influence acquisition programs, making it difficult to hold anyone accountable for program outcomes. Many of these service chiefs believed that cultural factors and incentives within the department make it difficult for program managers to manage requirements growth and execute programs effectively. These chiefs said that program managers and other acquisition officials often lack experience and expertise to manage requirements and acquisitions, are incentivized to meet internal milestones and not raise issues, and rely too much on contractors to figure out what is needed to develop a weapon system. Further, they noted that high turnover in program manager tenure—approximately every 2-3 years according to several service chiefs—make it difficult to hold managers accountable when problems emerge.

Service Chiefs Propose a Range of Solutions to Be More Involved in Acquisitions

To improve weapon system program outcomes, both current and former service chiefs agree that they should be more involved in the acquisition process. However, their views differed on the measures needed to achieve more involvement and improve integration between DOD’s requirements and acquisition functions. Most current service chiefs said that better collaboration does not require restructuring the chain of command. These service chiefs cited examples of ongoing collaboration between requirements and acquisition offices, and programs where they worked closely with acquisition leadership to address problems. In one case, a service chief pointed to an initiative that he and the service acquisition executive instituted to provide technical training and assistance to uniformed requirements officers as an example of formalized collaboration before and after the start of system development. Another service chief indicated that, faced with rising program costs and the possibility of cancellation, he actively monitored program progress through regular meetings with the program manager and contractor.

Current service chiefs and other acquisition leadership generally indicated the service chiefs have the ability to be more involved in the current process, such as by attending service and DOD level program reviews. However, some chiefs indicated that involvement in acquisition programs, in general, varies by service chief based on their priorities and the other personalities involved. Several current and former service chiefs agreed that they have been involved in the oversight of some programs, but their level of involvement is dependent on the importance of the program and established working relationships with the service acquisition executive. One service chief stated that, at times, service chiefs have not been involved due to unfamiliarity with the acquisition process, their own perceived role in the process, or a lack of interest in an acquisition.

Several former service chiefs thought that establishing co-chairmanship for key decision reviews and co-signature of key acquisition documents, particularly at the military department level, may improve collaboration, encourage requirements trade-offs during development, and force the service chiefs to share the burden of responsibility for acquisition programs. One suggestion from an outside expert for implementing this solution was to have the service chief and the service acquisition executive co-chair the service-level acquisition review board. Some military and acquisition leaders noted, however, that requiring co- chairmanship of acquisition meetings and co-signature of decision documents could slow an already complex process and further discourage program managers from raising issues and concerns.

In general, the former service chiefs we interviewed emphasized the need for a stronger role in the acquisition chain of command with more formal authority and mechanisms in place to ensure that the service chiefs are consistently involved and sufficiently able to influence program decisions. However, as we found in our prior review, studies that have advocated for a stronger role for the service chiefs in the acquisition process provide Several of little evidence that this would improve program outcomes.these former service chiefs advocated for changes to DOD policy and statute, including the Goldwater-Nichols Act.service chief believed that DOD acquisition policy should require service chief approval on all major defense acquisition programs prior to program start. Some acquisition experts have observed that, in giving sole responsibility for acquisitions to the military secretaries through the service acquisition executives, DOD created an unintended wall when implementing the Goldwater-Nichols Act reforms between the military- controlled requirements process and civilian-driven acquisition process. These acquisition experts note, however, that while service chiefs had significant influence on certain acquisition programs in the past, their close involvement did not always result in successful cost, schedule, or performance outcomes. For example, service chiefs had significant involvement in the Navy’s Littoral Combat Ship and the Army’s Future Combat System and, in both cases, viewed the programs as providing vital operational capabilities and needing to be fielded quickly. Consequently, the programs pursued aggressive acquisition strategies that pushed the programs through development with ill-defined requirements and unstable designs, which contributed to significant cost and schedule increases, and in the case of the Future Combat System, program cancellation.

Defining Lower-Level Requirements Is Often the More Significant Contributing Factor to Program Cost Increases

Acting on the chiefs’ concerns, we analyzed all 78 major defense acquisition programs and found that growth in high-level requirements— and consequent cost growth—was rare. Rather, we found that cost growth and other problems are more directly related to deriving lower- level requirements after a program has started. The distinction between high-level and lower-level requirements is key. Growth in high-level requirements could be attributable to a lack of discipline, but growth in lower-level requirements is not the result of additions, but rather the definition and realization of the details necessary to meet the high-level requirements. The process of defining lower-level requirements is an essential function of systems engineering, much of which is done late— after a development contract has been signed and a program has started.

In other words, requirements are insufficiently defined at program start; when their full consequences are realized, trade-offs are harder to make—cost increases and schedule delays become the preferred solutions. We presented our assessment of the requirements problem to current and former service chiefs and they generally agreed with it. Several service chiefs noted that more integration, collaboration, and communication during the requirements and acquisition processes needs to take place to ensure that trade-offs between desired capabilities and expected costs are made and that requirements are essential, technically feasible, and affordable before programs get underway. Some service chiefs believed that applying systems engineering to arrive at well-defined requirements before the start of system development at Milestone B can go a long way towards solving some of their dissatisfaction with the acquisition process and improving outcomes.

Key Requirements Have Not Changed Significantly in Major Defense Acquisition Programs Experiencing Cost Growth since 2009

We found few instances of requirements changes between 2009 and 2013 that involved increasing capabilities on major defense programs during system development. Seventeen programs in the current portfolio of 78 major defense acquisition programs experienced system development cost growth of more than 20 percent between 2009 and 2013, but 13 of them did not report associated key requirements increases (see table 1). A number of factors other than requirements increases contributed to the cost growth in these programs.

We found that, within the current portfolio of major defense acquisition programs, 5 of 78 programs reported increases to key performance parameters between 2009 and 2013. In these 5 programs, the changes involved adding a new component, technology, or other subsystem to increase the capabilities of the weapon system. Table 2 describes the requirement changes reported by these 5 programs. In 4 programs, development cost increases were more than 20 percent during the same time period.

Insufficiently Defined Requirements Led to Problems during Weapons System Development

A key factor consistently identified by GAO in prior reports is the mismatch between the requirements for a new weapon system and the resources—technologies, time, and funding—that are planned to develop Requirements, especially at the lower levels, are often the new system.not fully developed or well-defined when passed over to the acquisition process at Milestone B, at which time a system development contract is awarded and a program begins. During system development, the high- level operational requirements, such as key performance parameters and key system attributes, usually need to be further analyzed by the contractor using systems engineering techniques to fully understand, break down, and translate them into technical weapon system-level requirements and contract specifications. Systems engineering analysis translates operational requirements into detailed system requirements for which requisite technological, software, engineering, and production capabilities have been identified. It also provides knowledge to enable the developer to identify and resolve gaps before system development begins. It is often at this point—when the technical specifications are finally understood and design challenges are recognized—that cost and schedule increases materialize in a program. What may appear to be requirements growth is the recognition that the weapon system will require considerably more time and money than expected to build to these derived technical specifications to meet the validated operational requirements.

The process of translating high-level operational requirements into low- level requirements and technical specifications in many programs does not usually occur until well after Milestone B approval (see figure 3 for a notional depiction).

The number of requirements can expand greatly over time, as the designs of the subsystems and components become defined. In the case of the Army’s Future Combat System, a large program that was intended to equip combat brigades with an advanced set of integrated systems, requirements were still being defined when the program was canceled beginning in 2009—after 6 years and $18 billion had been spent on initial system development. The program was approved to start system development with 7 key performance parameters. In order to meet these key performance parameters—which did not change—the program ultimately translated them into over 50,000 lower-level requirements before it was canceled. Requirements definition remains a challenge facing current major defense acquisition programs. For example, the F-35 program, which was conceptualized around three aircraft design variants to achieve cost efficiencies, has had difficulty reconciling different requirements imposed by the military services. According to program officials, in order to meet the nine validated key performance parameters, the program developed approximately 3,600 specifications. While the operational requirements for the F-35 have not increased, factors such as poorly defined requirements, significant concurrency between development and production, and immature technologies have contributed to significant cost growth and delays in the program.

We found that several of the major defense acquisition programs that experienced cost growth, but did not report changing key performance parameters, had a significant number of engineering change orders and other configuration changes. As operational requirements become better understood during system development, contract specifications change to reflect what is needed to build the weapon system. Changes show up in engineering change orders and other design configuration changes, which contribute to cost growth. For example, between 2009 and 2013, the Littoral Combat Ship program reported 487 changes to its system configuration or design. Similarly, the Joint Tactical Radio System Handheld, Manpack, and Small Form Fit Radios program reported making 29 engineering changes and 11,573 software changes between 2009 and 2013. In neither case were the high-level requirements increased. While some configuration changes are necessary to manage obsolescence and other issues, the pursuit of poorly defined requirements results in overly optimistic cost and schedule estimates that are sometimes unachievable—leading to cost and schedule growth as programs encounter increased technical challenges necessary to achieve operational requirements.

GAO’s prior work as well as DOD’s own policy emphasizes that the translation of operational requirements into technical weapon system specifications, which are informed by systems engineering, should take place prior to approving a program at Milestone B and awarding a contract that locks in the requirements. This allows trade-offs between requirements and resources to take place, and the establishment of more realistic cost, schedule, and performance commitments before programs get underway. However, DOD often does not perform sufficient up-front requirements analysis via systems engineering on programs to determine whether the requirements are feasible and there is a sound business case to move forward. Programs are proposed with unachievable requirements and overly optimistic cost and schedule estimates and, usually, participants on both the requirements side and the acquisition side are loathe to trade away performance. For example, a preliminary design review is a key systems engineering event that should be held before the start of system development to ensure requirements are defined and feasible, and the proposed design can meet the requirements within cost, schedule, and other system constraints. In 2013, GAO reviewed the 38 major defense acquisition programs that held preliminary design reviews that year. Only 11 of these programs held design reviews prior to the start of system development. The remaining 27 programs completed or planned to complete their design reviews approximately 24 months, on average, after the start of development. Thus, the resource consequences of deriving lower-level requirements are similarly deferred.

Service Chiefs Generally Agree That Poorly Defined Requirements Contribute to Program Cost Growth

We shared a summary of our assessment of the requirements problem, namely that high-level requirements are poorly defined when passed over to the acquisition process at the start of development, with the current and former service chiefs and they generally agreed with our findings. Several current and former service chiefs indicated that requirement and resource trade-offs, informed by systems engineering, do not consistently take place before programs get underway. Some chiefs also noted that reassessments of requirements, acquisition, and funding are not conducted often enough during program execution. According to one service chief, under the current acquisition process, there are too few points of collaboration among requirements officers, acquisition professionals, systems engineers, and cost estimators to work out requirements early in the process or to address problems and limitations associated with meeting operational requirements after programs are underway. Another service chief noted that the acquisition workforce lacks experience in operational and tactical settings and that his requirements community lacks technical acquisition skills, so it is important that collaboration regularly occurs between the two communities. Further, one chief emphasized that requirements officers are too dependent upon the acquisition community and its contractors to work out requirements.

Several current and former service chiefs voiced concern that cost and schedule problems that acquisition programs experience are due to the failure to make appropriate trade-offs during system development. They indicated that too often programs encounter cost and schedule problems because in striving to meet challenging requirements the programs end up making technical and design changes to the weapon system. For example, one former service chief highlighted a combat vehicle program in development which had fallen short of meeting its vehicle speed requirement by a small percentage. Instead of making trade-offs, and perhaps seeking requirements relief, the program manager requested additional funding so the contractor could make design changes to the engine. Another service chief stated that requirement changes made during weapon system development are often viewed as sacrificing capability rather than reconciling requirements with operational conditions. The chief was concerned that program managers too often take the view that requirements cannot be changed and avoid elevating problems to leadership before they become critical, forgoing the opportunity to make needed trade-offs. In addition, one service chief described this problem as “cost creep” to meet requirements, not “requirements creep”. We have previously found that incentives within the current acquisition process create pressure on defense system requirements and are geared toward delaying knowledge so as not to jeopardize program funding.

Several current and former service chiefs agreed that there needs to be more integration, collaboration, and communication during the requirements and acquisition processes to ensure trade-offs are made and the requirements that get approved are essential, technically feasible, and affordable prior to the start of system development. Some service chiefs said that conducting systems engineering analyses during requirements setting and, again, early on during an acquisition program’s planning phase to inform trade-offs between cost and capability could go a long way toward establishing better defined requirements and improving program outcomes. Almost all of the service chiefs stated that there is a need to further enhance expertise within the government, and several specified expertise in systems engineering. Several service chiefs indicated that systems engineering capabilities are generally lacking in the requirements development process, and do not become available until after requirements are validated and an expensive and risky system development program is underway. Some service chiefs advocated that having systems engineering capabilities available to the military services during requirements development could help to ensure earlier assessment of requirements feasibility. The service chiefs’ views on the importance of systems engineering is consistent with our prior acquisition work, which calls for DOD to implement a knowledge-based approach to guide the match of defense program needs with available technology and resources.

The service chiefs expressed a willingness to be more involved in the management and oversight of acquisition programs. Enhancing collaboration between the requirements and acquisition processes could be one of several steps needed to address the underlying culture and incentives that exist in DOD that lead to programs that are not feasible and affordable. We have found in prior work that characteristics of DOD’s processes and incentives create pressure to push for unrealistic defense system requirements and lead to poor decisions and mismatches This culture has become between requirements and resources. ingrained over several decades and a number of studies and reforms have been directed at changing the incentives underlying the culture, without much success.

Conclusions

GAO-01-288. be overly optimistic and to minimize the difficulty and resources needed to deliver the capability. Many of our prior recommendations have been aimed at this problem and, while one could argue whether more formal authority should be granted to the service chiefs, the current acquisition process allows for the service chiefs to be more involved in the management and oversight of acquisition programs. Regardless, the solution must involve investing in systems engineering expertise sooner— while developing requirements—to enable technological knowledge to better shape and define operational requirements.

Recommendations such as holding preliminary design reviews before the start of system development have been made as a means to improve program outcomes. After initial support, the enthusiasm for these practices wanes and the old pressures to continue with insufficient knowledge prevail, because the old practices allow programs to proceed and funding to flow. Importantly, the negative consequences of proceeding with limited knowledge are not sufficient to counteract these pressures, as accountability for the initial poor decisions is lost by the time problems emerge. Information and expertise will not result in good outcomes unless the need for a solid business case is reinforced. In order to improve program outcomes, DOD must focus its efforts on better integrating the requirements and acquisition processes, which can be achieved through better collaboration between these communities from the generation of requirements through system development, coupled with a greater emphasis on systems engineering and knowledge attainment early in a program’s life cycle. Without sufficient systems engineering input to better define requirements and examine trade-offs early on, there is no assurance that acquisition programs going forward have a sound basis to start system development.

Recommendations for Executive Action

To help ensure that requirements are well defined and well understood before a program is approved to start system development, we recommend that the Secretary of Defense direct the military service chiefs and service acquisition executives to work together to take the following two actions:

Assess whether sufficient systems engineering expertise is available during the requirements development process; and

Develop a better way to make sure sufficient systems engineering is conducted and opportunities exist to better define requirements and assess resource trade-offs before a program starts.

Agency Comments and Our Evaluation

DOD provided us with written comments on a draft of this report, which are reprinted in appendix II. The department concurred with both of our recommendations, stating that the early application of systems engineering expertise and ensuring the availability of appropriately skilled personnel are critical to successful program outcomes. DOD noted that recent changes to department-wide policies, such as DOD Instruction 5000.02, strengthen the department’s focus on conducting systems engineering and making trade-offs during requirements development and pre-program planning. DOD further agreed that continuing to improve engagement between the requirements and acquisition communities will result in better informed program initiation and resourcing decisions.

We are encouraged that DOD agrees with our recommendations and has recently taken steps to strengthen its policies and identify the need for early systems engineering. However, for many years DOD policies have emphasized the importance of a knowledge-based approach to acquiring weapon systems, but practice does not always follow policy. Instead, incentives exist that encourage deviation from sound policies and practices. We believe that DOD must focus on achieving better collaboration between the requirements and acquisition communities such as by ensuring that more systems engineering and other expertise are applied when requirements are being defined. It is through informed collaboration that knowledge will be attained, trade-offs between requirements and resources can be made earlier, and acquisition programs will begin development with realistic cost and schedule estimates, ultimately leading to improved outcomes.

We are sending copies of this report to appropriate congressional committees; the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; the Chief of Staff of the Air Force; the Chief of Staff of the Army; the Chief of Naval Operations; the Commandant of the Marine Corps; and the Under Secretary of Defense for Acquisition, Technology, and Logistics. In addition, this report also is available at no charge on the GAO website at http://www.gao.gov.

If you have any questions about this report or need additional information, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

GAO issued a report in 2014 on the military service chiefs’ role in the acquisition chain of command. This report reviews further related issues and concerns the military service chiefs have with the Department of Defense’s (DOD) acquisition process and outcomes it produces. Specifically, we examined (1) the views of current and former military service chiefs on the current acquisition process, and (2) key problems or factors the service chiefs identified with the acquisition process and our assessment of these issues.

To obtain the views of current and former military service chiefs on the current acquisition process, we conducted interviews with 12 current and former military service chiefs and vice chiefs between August and December 2014. We met with all current military service chiefs and vice chiefs as of September 2014, including the Chief and Vice Chief of Staff of the Air Force, the Chief and Vice Chief of Staff of the Army, the Chief and Vice Chief of Naval Operations, and the Commandant and Assistant Commandant of the Marine Corps. These individuals possessed joint- and service-level experience, including positions as the Chairman of the Joint Chiefs of Staff and Combatant Commander. In December 2014, after completing our interviews, we analyzed our findings and sent a summary to the four current and three former military service chiefs, but not the vice chiefs, for their review and comment. We received responses from five of the seven service chiefs, all of whom concurred with our findings. We reviewed any comments and made changes to the summary document, as appropriate. We also interviewed or sought the perspectives of additional current and former DOD leadership, including the Service Acquisition Executive of the Air Force, Army, and Navy; officials from the Office of the Secretary of Defense, Joint Staff; and another former member of the Joint Chiefs of Staff. We analyzed evidence and examples collected from our interviews with current and former military service chiefs and DOD leadership. We also reviewed findings from existing reports and compendiums focused on the acquisition chain of command and interviewed acquisition subject matter experts to discuss the current acquisition process, the role of the military service chiefs in the acquisition chain of command, and potential solutions to improve program outcomes. We reviewed prior GAO work on weapon system acquisition and commercial best practices and analyzed the extent to which evidence exists that would demonstrate that these potential solutions may improve program outcomes.

To assess key problems or factors the service chiefs identified with the acquisition process and our assessment of these issues, we drew upon our extensive body of work in defense acquisitions and best practices, and reviewed program execution information from ongoing major defense acquisition programs. We reviewed the annual Selected Acquisition Reports (SAR) from 2009 to 2013 for the 78 programs in DOD’s current portfolio of major defense acquisition programs. SAR data was collected from the Defense Acquisition Management Information Retrieval (DAMIR) Purview system, referred to as DAMIR. We assessed the reliability of the data by reviewing existing information about DAMIR and determined that the data were sufficiently reliable for the purposes of this report. We analyzed the performance metrics to determine the extent to which programs were reporting changes to key performance parameters. Our analysis was limited to unclassified requirements that are included as part of the SAR. In October 2014, we developed and submitted a questionnaire to 28 major defense acquisition programs that had reported key requirement changes in their respective SAR from 2009 to 2013, sought requirement relief from the Joint Requirements Oversight Council, or experienced a development cost increase or decrease of 10 percent or more between 2011 and 2013. We conducted two pretests of the questionnaire prior to distribution to ensure that our questions were clear, unbiased, and consistently interpreted. We obtained responses from all 28 programs, and in cases where questionnaire results differed from previously collected SAR data, we submitted follow-up questions to the program office to adjudicate any discrepancies. To determine the extent to which programs that experienced development cost growth also changed key requirements, we compared the research, development, test and evaluation cost estimates from 2009 and 2013 for DOD’s current portfolio of major defense acquisition programs, as reported in their annual SAR. In instances where a program began development after 2009, we compared the program’s initial research, development, test and evaluation cost estimate with its 2013 current estimate. We then reviewed any program that had a cost increase of more than 20 percent to determine if this program also reported key requirement changes in its annual reports for the same time period. We also leveraged prior and ongoing GAO work on weapon system acquisition.

We conducted this audit from October 2014 to June 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact:

Staff Acknowledgments:

In addition to the contact named above, John E. Oppenheim, Assistant Director; Jacob Leon Beier; Brandon H. Greene; Laura M. Jezewski; Megan Porter; Abby C. Volk; Marie P. Ahearn; Peter W. Anderson; Jean L. McSween; and Kristy E. Williams made key contributions to this report.