Weapon Systems: Prototyping Has Benefited Acquisition Programs, but More Can Be Done to Support Innovation Initiatives
Released: 2017-06-27T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: DOD invests roughly $70 billion annually in weapon system research, development, test, and evaluation, including prototyping activities. Prototyping can help reduce risk in weapon system acquisition programs by improving understanding of technologies, requirements, and proposed solutions. It can also contribute to innovation by demonstrating the value of new technologies or systems.

House Conference Report 114-102 accompanying a bill for the fiscal year 2016 National Defense Authorization Act included a provision for GAO to review how DOD's research and development funds are used and whether this approach effectively supports activities such as prototyping. This report assesses (1) how DOD has used prototyping prior to system development on major defense acquisition programs, and (2) what steps DOD has taken to increase innovation through prototyping activities outside of major defense acquisition programs. GAO examined prototyping activities for 22 MDAPs that planned to enter system development between December 2009 and February 2016 and 7 prototyping-focused initiatives with the stated purpose of promoting innovation.

section_title: What GAO Found
paragraphs: The Department of Defense (DOD) has used prototyping on its major defense acquisition programs (MDAP) primarily to reduce technical risk, investigate integration challenges, validate designs, mature technologies, and refine performance requirements. Of the 22 programs GAO reviewed, 17 used prototyping before starting system development. For many of those programs, prototyping provided information that helped introduce realism into their business cases by providing information on technology maturity, the feasibility of the design concepts, potential costs, and the achievability of planned performance requirements.

DOD has developed new initiatives that are outside of major defense acquisition programs to increase prototyping and further innovation. However, these initiatives face barriers, such as limited funding, a risk averse culture, and competing priorities. Literature on private sector innovation identifies key enablers for these types of efforts, such as developing an innovation strategy, aligning investments with innovation goals, and protecting funding for riskier projects. DOD has taken steps that are consistent with a few, but not all, of these enablers. For example, DOD does not have a department-wide strategy that communicates strategic goals and priorities and delineates roles and responsibilities to guide the prototyping initiatives. This could lead to unproductive or poorly coordinated investments later. DOD's initiatives also face competition for funding, particularly with acquisition programs. One strategy to address funding issues called “strategic buckets” involves allocating resources to different types of projects based on an organization's strategy (see figure). DOD has not set strategic funding targets for its initiatives. Failing to do so could prevent them from gaining traction and puts their long-term success at risk.

section_title: What GAO Recommends
paragraphs: GAO is making four recommendations, including that DOD develop a department-wide innovation strategy that includes prototyping and adopt a more strategic approach for funding prototyping efforts across DOD. DOD concurred with the recommendations and is currently working on this strategy.

section_title: Letter
paragraphs: The Department of Defense (DOD) is faced with a variety of challenges  as it seeks to maximize the return on its roughly $70 billion annual  investment in weapon system research and development. While it is  generally acknowledged that U.S. weapon systems are the best in the  world, there is concern within DOD and elsewhere that American  dominance in key warfighting domains is eroding as potential adversaries  modernize their weapon system capabilities at a rapid rate. At risk of  having its dominance disrupted, DOD has been looking for innovative  ways to sustain and advance its military superiority.

Prototyping—the development and testing of a model or system design  that uses available or emerging technologies—can be employed in  weapon system acquisition programs to reduce risk by maturing  technologies and to improve understanding of a potential system’s likely  design, feasibility, cost, and schedule before making major funding  commitments. It can also help ensure that new, innovative, and disruptive  technologies are available for inclusion into potential future weapons and  demonstrate the value of new technologies or systems. Over the past few  years, DOD has acknowledged the importance of prototyping to maintain  the U.S. military’s capability to rapidly address emerging threats and  provide technological surprise, as it has in the past through capabilities  including stealth and precision weapons. Congress has enacted  legislation with numerous prototyping-related provisions.

House Report 114-102 accompanying a bill for the National Defense  Authorization Act (NDAA) for Fiscal Year 2016 included a provision that  we review how DOD’s research and development funds are used and  whether this approach effectively supports activities such as prototyping.  This report assesses (1) how DOD has used prototyping prior to system  development on major defense acquisition programs, and (2) what steps  DOD has taken to increase innovation through prototyping activities  conducted outside of major defense acquisition programs. A major  defense acquisition program (MDAP) is a program that is designated as  such or is estimated by DOD to require an eventual total expenditure for  research, development, test, and evaluation (RDT&E) of more than $480  million, or for procurement of more than $2.79 billion, in fiscal year 2014  constant dollars, for all increments.

To determine how DOD has used prototyping in MDAPs, we examined 22  MDAPs that entered system development between December 2009— which is when DOD implemented new prototyping requirements enacted  by Congress in the Weapon Systems Acquisition Reform Act of 2009  (WSARA)—and February 2016. For those 22 programs, we reviewed  program documents and conducted semi-structured interviews with  program officials to determine what prototyping approaches they used, if  any, and to identify costs, benefits, challenges, and lessons learned.  Table 1 includes a list of the programs we reviewed.

To assess steps DOD has taken to increase innovation through  prototyping outside of MDAPs, we reviewed fiscal year 2017 budget  documentation and interviewed military service officials to identify  initiatives that DOD has begun in the past 5 years to increase innovation  through prototyping and experimentation. We focused on seven broad- based initiatives, rather than ones focused on a specific technology area.  We reviewed documentation and interviewed officials or obtained written  responses from each of those seven initiatives to identify their goals,  focus areas, scope, approaches, funding characteristics, strategies,  coordination mechanisms, and barriers, if any, they face. To determine  DOD’s overall strategy for these initiatives, we analyzed DOD  memorandums and other relevant documentation, including those related  to the management of its science and technology investments. Finally, we  conducted a review of literature on private sector innovation, including the  use of prototyping, to identify key enablers and barriers. See appendix I  for additional information on our objectives, scope, and methodology.

We conducted this performance audit from September 2015 to June 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DOD’s science and technology community—including research  laboratories, test facilities, industry, and academia—conducts initial  research, development, and testing of new technologies to improve  military operations and ensure technological superiority over potential  adversaries. Afterwards, the acquisition community typically manages  product development, in which technologies are further advanced and  system development begins. These activities are generally supported by  DOD’s RDT&E budget, which DOD groups into seven budget activity  categories for its budget estimates and the President’s Budget. The  categories follow a mostly sequential path for developing technologies  from basic research to operational system development, as is shown in  figure 1. The first three budget activity categories represent DOD’s  science and technology activities to advance research and technology  development, while the remaining budget activity categories are typically  associated with product development for acquisition programs. See  appendix II for a description of each budget activity.

Funding for prototyping is mostly found in advanced technology  development and advanced component development, budget activity 6.3  and 6.4 respectively. Appendix III provides a breakdown of budget activity  6.3 and 6.4 funding by organization for fiscal year 2016. Funding in  budget activity 6.3 is not directly tied to acquisition programs whereas  budget activity 6.4 is typically used for that purpose. Funding for  acquisition programs, including budget activity 6.5, was $28 billion in  fiscal year 2016 and has varied over time, whereas science and  technology funding was $13 billion and has remained relatively flat, as is  shown in figure 2.

There are numerous types and definitions of prototyping. One construct  used by parts of DOD refers to conceptual, developmental, and  operational prototypes, each of which has a different purpose and time  horizon for when they can be expected to be incorporated into or become  their own acquisition program. Figure 3 includes more information about  each of these types of prototypes. Although each type is more mature or  closer to a capability that can be fielded than its predecessor, prototyping  does not have to proceed sequentially. For example, an operational  prototype might not be preceded by a conceptual or developmental  prototype, if it is based on existing mature technologies or concepts.

Prototyping can involve a variety of different approaches, in terms of what  is being developed and demonstrated, who is building the prototype, and  how it is being acquired or managed. System prototyping is when a  prototype that includes components for an entire system is developed,  such as a prototype of a ground vehicle or missile. Subsystem prototyping  is when a prototype is developed that includes a group of components  that combine to perform a major function for a system, such as a power  supply system for a radar. In a DOD context, prototypes can be  developed by contractors or groups of contractors, government labs, or  both, and efforts can be managed by the science and technology  community, acquisition programs, or other types of research and  development organizations. When two or more contractors or other  entities prototype the same component, subsystem, or system, the effort  is referred to as competitive prototyping.

Over the years, DOD and Congress have taken steps to encourage  prototyping during the technology development phase of weapon system  acquisition programs. In 2007, the Office of the Under Secretary of  Defense for Acquisitions, Technology, and Logistics issued a  memorandum on prototyping and competition that expressed concern  that DOD decisions on acquisition programs were being based largely on  paper proposals that provided inadequate knowledge of technical risk and  a weak foundation for estimating development and procurement cost. To  help address these concerns, the memorandum required pending and  future acquisition programs to formulate acquisition strategies that call for  conducting competitive prototyping up through the start of system  development. Not long after, in 2009, Congress passed WSARA, which  included a provision on competitive prototyping for MDAPs as well as  many other reforms. WSARA called for competitive prototyping at the  system level or—if not feasible—for critical subsystems, and allowed  competitive prototyping to be waived only if the cost of producing  competitive prototypes would exceed the life-cycle benefits of producing  them or if without a waiver, DOD would be unable to meet critical national  security objectives. If competitive prototyping was waived, WSARA  required that programs produce a prototype before milestone B, if the  expected life-cycle benefits of producing such prototypes exceeded the  cost and its production was consistent with achieving critical national  security objectives. Originally implemented by DOD in December 2009,  Congress repealed WSARA’s competitive prototyping provision in 2015.  However, as of the time of this review, DOD still required program officials  to consider using prototyping and competitive prototyping at the system or  subsystem level as a risk mitigation technique. Congress also included  several new prototyping-related provisions in the fiscal year 2017 NDAA,  which are discussed later in this report.

DOD prototyping also occurs outside or independent of acquisition  programs. One of the purposes of this type of prototyping can be to  further disruptive innovation. Disruptive innovation attempts to shift the  balance of military power in our favor by providing new capabilities,  potentially unforeseen by the warfighter. The capabilities can be a result  of new technologies, new ways to integrate existing technologies, or  changes to how systems are employed. Disruptive innovation can also  include providing existing capabilities at substantially lower cost, thereby  increasing military advantage. Examples of potentially disruptive  technologies include directed energy, hypersonics, and low cost missile  defense capabilities. Prototyping can be a way to “test the waters” or  experiment with new and potentially disruptive concepts and technologies  without the level of commitment associated with starting acquisition  programs. Prototyping in this environment may involve more risk,  including less mature technologies. There may also be no residual value  at the end of a project other than increased knowledge and potentially a  prototype “on the shelf” for further maturation.

section_title: DOD Used Prototyping to Reduce Technical Risks, Validate Designs, and Investigate Integration Challenges on Major Weapon Acquisition Programs
paragraphs: Most major weapon system acquisition programs we examined used  prototyping to reduce technical risk, investigate integration challenges,  and validate designs, among other things. Program officials chose  prototyping approaches to align with their assessments of program risks,  with riskier programs prototyping more extensively. They generally found  that prototyping provided a good return on investment. It helped programs  better understand requirements, the feasibility of proposed solutions, and  cost—the key elements of a program’s business case. Identifying key  risks early and structuring prototyping efforts to inform key decisions  helped maximize the utility of programs’ prototyping efforts.
subsections: section_title: Major Acquisition Programs Selected Prototyping Approaches That Addressed Key Risks
paragraphs: The programs we reviewed used prototyping primarily to reduce technical  risks, investigate integration challenges, validate designs, and mature  technologies. Of the 22 MDAPs we reviewed, 17 used some form of  prototyping during technology development and 5 did not prototype. Of  the 17 programs that prototyped, officials from 15 programs told us they  chose to prototype because it made the most sense given the program’s  needs. Officials from the other 2 programs told us their programs  prototyped at the direction of the Under Secretary of Defense (USD) for  Acquisition, Technology and Logistics (AT&L)—the milestone decision  authority for their programs. All 17 programs prototyped for multiple  reasons and officials from 11 programs identified four or more reasons.  The reasons cited by each program are depicted in figure 4.

Program officials stated that they tailored their prototyping approaches to  their program’s risks, with riskier programs prototyping more extensively.  Ten of the programs we reviewed conducted system-level prototyping, 7  programs conducted subsystem prototyping, and 5 did not prototype (see  appendix IV for a brief overview of each program’s prototyping efforts or  the reasons it did not prototype). Prototyping approaches varied within  these categories. Some programs prototyped one or two subsystems  while others used multiple contractors and multiple-phased prototyping  efforts at the system and subsystem level. The five programs that did not  conduct any prototyping used known designs and existing technologies,  which DOD generally considers less risky. Four of the five entered the  acquisition process without a technology development phase and most  obtained waivers from the competitive prototyping requirements in DOD  policy. Figure 5 shows examples of programs’ prototyping efforts—or lack  thereof—and how they align with the program officials’ understanding of  their risks.

The 10 programs that developed system level prototypes ranged from the  Joint Light Tactical Vehicle, which involves less expensive, lower  complexity items that will eventually be purchased in the tens of  thousands, to the Space Fence Ground-Based Radar, which will only  produce one, large ground-based radar to detect, track, and provide  information about objects in Earth’s orbit. System level prototyping led to  improved understanding of: (1) integration challenges and the feasibility  of system designs; (2) significant unknowns, such as costs for ambitious  requirements; (3) uncertainties related to integrating mature technologies  in new ways; and (4) new technologies. Six of these programs first  conducted subsystem prototyping to mature new or existing technologies  used in new ways before using system-level prototyping to investigate  integration challenges.

The seven programs that developed subsystem prototypes ranged from  an F-22 modernization program that was upgrading the aircraft’s weapon  and communication systems to the Ship to Shore Connector Amphibious  Craft, an air-cushioned landing craft that transports personnel, weapon  systems, and cargo. Subsystem level prototyping efforts focused on  narrower areas of perceived risk, such as maturing critical technologies,  integrating a subsystem with other hardware or software, or testing  specific components that are being considered for use in a system  design. The programs that conducted subsystem prototyping were often  building on or well-positioned to leverage existing weapon systems. Five  of the seven programs in this category used existing platforms, either  hardware or software, for their subsystem prototyping efforts. For  example, the Ship to Shore Connector Amphibious Craft utilized the  Navy’s existing landing craft as a platform to prototype and test new  components.

Of the 17 programs that prototyped, 12 used competitive prototyping.  They did so for a variety of reasons, including to prove out potential  solutions when more than one solution could be feasible and to gain  knowledge from multiple sources about uncertainties, such as integration  challenges, design feasibility, or cost. We found that most of the  competitive prototyping efforts included system-level designs and almost  all programs that conducted system-level prototyping used competitive  approaches, as is shown in figure 6. Competitive prototyping may  generate more information about proposed solutions because contractors  sometimes propose different design approaches or system concepts to  meet DOD’s capability needs. DOD’s 2007 prototyping and competition  memorandum also noted that competition would generate more  knowledge about technical risk and build a stronger foundation for  estimating costs.

section_title: Major Acquisition Program Officials Stated That Prototyping Provided a Good Return on Investment
paragraphs: According to officials from 16 of the 17 programs that prototyped, the  benefits gained from their prototyping effort were worth the cost and  provided a positive return on investment. The benefits gained were  central to the development of a sound business case, which includes  evidence that (1) the customer’s needs are valid and can best be met with  the chosen concept and (2) the chosen concept can be developed and  produced with existing resources, such as time, money, and available  technology. We have previously found that establishing a sound business  case is essential to achieving better program outcomes. See appendix  V for an analysis of acquisition outcomes to date for the programs we  examined. Prototyping provided programs with information on technology  maturity, the feasibility of the design concepts, potential costs, and the  achievability of planned performance requirements, which helped inject  realism into their business cases. Appendix VI includes examples of  these benefits, several of which we highlight below:

Prototyping demonstrated key technologies or proposed design  solutions. For example, Space Fence officials stated that prototyping  helped them determine that a riskier, cutting edge design involving a  15 percent smaller radar was feasible. Without prototyping, the  program would not have had sufficient information to be confident in  the riskier option, nor would the contractor have proposed it without  the opportunity to demonstrate that it worked.

Prototyping informed programs’ understanding of prices and  helped validate business case cost estimates. During the  prototyping process, contractors select vendors, develop supplier  relationships, purchase materials, and build a version of the system or  parts of the system, all of which provide information on potential costs.  Air and Missile Defense Radar program officials stated that  prototyping increased the cost information available to the program  and led to cost reductions. They explained that competitive  prototyping incentivized the contractors to determine their cost drivers  in order to be more competitive in the next phase.

Prototyping helped programs better understand requirements  and—in most cases—helped them make performance tradeoffs  to meet cost targets. In the case of Joint Light Tactical Vehicle,  prototyping helped program officials determine that two versions of  the vehicle were too heavy to be transported as planned. The  program then lowered its transportability requirements by eliminating  the need to airlift the vehicles in extreme conditions. This change  allowed the vehicles to be heavier and resulted in $35,000 in savings  per vehicle, according to the Army.

Prototyping provided a variety of other benefits as well. For example,  prototyping helped programs improve system performance. Small  Diameter Bomb II officials said that data collected during its prototype  testing set the stage for improvements in its target classification software.  It also helped identify potential reliability issues early. The Next  Generation Jammer Increment 1 and Ship to Shore Connector  Amphibious Craft programs changed certain subsystem materials based  on information learned about wear during prototype testing. Further, for  11 programs, the prototypes served as test assets during system  development or were used to continue development efforts.

Competitive prototyping approaches generated additional benefits, such  as enabling more favorable business terms. According to the Office of the  Under Secretary of Defense for Acquisition, Technology, and Logistics,  the Air and Missile Defense Radar program’s use of competition resulted  in over $100 million in savings and will reduce operation and support  costs over the life of the program. Air and Missile Defense program  officials explained that having three competitors was helpful because it  reduced the likelihood that contractors would team up in the next phase,  leaving the government with only one proposal. In other cases,  competition improved the quality of the systems being offered to DOD.  For example, Space Fence program officials told us that competition  spurred contractors to introduce cutting edge designs and continue  refining those designs in order to remain competitive in the next phase of  the program. Finally, officials from several programs stated that  contractors supplemented prototyping efforts using their own funds and  believed contractors did this in order to make their subsequent offers  more competitive.

A common perception is that competitive prototyping might cost more up  front because multiple contracts are awarded, but our analysis showed  that programs using multiple contractors did not have higher relative costs  for technology development. Programs that used a competitive approach  planned to spend a similar percentage or less of their total expected  RDT&E funding prior to the start of system development as those  programs that did not competitively prototype. Using competitive  prototyping approaches did create additional administrative burdens in  the short term because program offices had to manage multiple  contractors and maintain firewalls to ensure fair competitions, but officials  from across the programs we examined stated it was worth the  investment.

section_title: Several Practices Helped Programs Get More Out of Their Prototyping Efforts
paragraphs: Officials from all 17 programs that prototyped told us their prototyping  efforts were useful; however, we found some programs got more out of  their prototyping efforts than others. Based on those programs’  experiences and lessons other programs shared, we identified the  following five practices that helped programs maximize the utility of their  prototyping efforts.  1.  Identify risks early and target prototyping efforts to address  them. Officials from seven programs described early activities that  helped identify risks and shape their prototyping efforts. The Air and  Missile Defense Radar program planned a big leap in technology over  the existing radar, which was fielded over 30 years ago. The program  conducted early technology maturity assessments that identified the  large aperture digital beamforming and calibration critical technology  as the program’s key risk area. The program focused its prototyping  efforts on maturing this and other critical technologies and  demonstrating them in a relevant environment. It is on track to  complete system development on time and within its estimated  research and development cost.  2.  Structure prototyping efforts to be completed in time to inform  key decisions—particularly source selection. We found that many  of the programs prototyped before selecting their system development  contractor and about three-quarters of the programs also held  preliminary design review before entering system development.  Prototyping helped inform these decisions and related assessments.  Common Infrared Countermeasure program officials told us they  required contractors’ proposals for the program’s system development  contract to include solutions to address reliability failures identified  during prototyping.  3.  Specify the level of fidelity needed to provide the necessary  information about which risks to address. Officials from 15  programs told us they prototyped designs similar to the actual design  of the system they will develop. This is known as high fidelity  prototyping. Officials from a few programs noted that doing so  enabled them to understand what the system entailed and, if needed,  make trade-offs accordingly. In contrast, Integrated Air and Missile  Defense program officials stated that the results from prototyping  were not as helpful to make programmatic decisions because they  were limited to demonstrating the feasibility of a certain system  concept using a generic government design.  4.  Ensure the appropriate level of insight into the design and cost  information. Officials from each program said they had sufficient  visibility into the prototyping efforts, but some officials described  having more insight into the efforts and their results than others. The  level of insight can be affected by factors, such as the type of  information a program requires a contractor to provide under a  prototyping contract. Space Fence officials noted that in addition to  conducting a live prototype demonstration, they had access to reports  and data from contractors’ efforts. The officials said that the data  obtained through prototyping were helpful for pricing. The program  used the data to mature its cost estimate and was able to use a firm  fixed price contract for system development. Space Fence finalized its  design just over a year after entering system development, has had  no cost growth to date, and anticipates that the contractor will deliver  the system earlier than initially planned.  5.  Keep plans flexible to adapt to information learned during the  effort. Officials from four programs told us they used multi-phased  prototyping approaches and a few described adding or removing  contractors. Officials from at least three programs told us they  changed strategies or modified their approach based on information  learned or in response to a tighter budget environment. For example,  after the Common Infrared Countermeasure program determined its  technologies were less mature than expected, it added two  prototyping phases to continue maturing the technologies, testing the  system with related systems, and demonstrating manufacturing  processes. The program entered system development with its  technologies nearing maturity and completed its system design in just  over a year. It has had about 6 percent cost growth as of October  2016 and is estimating completing system development on time.

section_title: DOD Has Established New Initiatives to Increase Prototyping and Innovation, but Does Not Have a Strategy to Guide Its Investments
paragraphs: To help ensure U.S. military capabilities outpace those of potential  adversaries, DOD has expanded prototyping efforts focused on  innovation, including disruptive innovation, and has started several new  initiatives outside of major acquisition programs to address gaps in its  innovation portfolio. However, these initiatives face barriers, such as  limited funding and competing priorities. Literature on private sector  innovation, including the use of prototyping, identifies key enablers for  these types of efforts, such as developing a strategy for innovation,  identifying relative levels of investments that align with innovation goals,  and protecting funding for technology investments that have higher risk,  but perhaps more reward across the enterprise. DOD has taken steps  that are consistent with a few of these enablers, but lacks others, such as  an innovation strategy that could also address the role of prototyping.
subsections: section_title: DOD Has Started New Initiatives to Expand Prototyping and Experimentation
paragraphs: Since 2012, DOD and the military departments have established seven  new offices to increase prototyping and experimentation and further  innovation. Prototyping can be a way to “test the waters” with new and  potentially disruptive concepts and technologies. Experimentation puts  prototypes into the warfighter’s hands, so that the capabilities they  provide can be assessed in an operational context. Most of the efforts we  examined aim to mature technologies for future capabilities, but without  the rigidity, commitment, and additional cost associated with starting new  acquisition programs. Other than the experimentation initiatives, all of  them involve demonstrations that seek to improve DOD’s or the military  services’ understanding of the viability, maturity, and potential utility of the  technologies, subsystems, or systems being prototyped. The  demonstrations also inform decisions regarding potential next steps, such  as transition to a military service in the case of mature capabilities that  are ready to be put into use or to an acquisition program for those that  need further development. Table 2 provides an overview of these  initiatives.

The new initiatives help address gaps in DOD’s science and technology  and weapon system investments and expand efforts to identify and  mature potentially innovative and disruptive technologies. For example,  the Army Technology Maturation Initiative uses budget activity 6.4  funding, which is typically associated with acquisition programs, to  conduct higher-fidelity prototyping and further mature technology outside  of those programs. Other initiatives focus on modifying already fielded  equipment and technologies to use them in new ways, combining  prototyping and rapid acquisition practices to field capabilities faster, and  encouraging experimentation to explore how capabilities being prototyped  could be employed in an operational setting.

The two most mature prototyping-related initiatives have made some  progress. For example, the Strategic Capabilities Office reported that it is  currently in the process of transitioning six of its technology  demonstration and prototyping projects to the military services. The  Army’s Technology Maturation Initiative has also demonstrated some  progress—it has six projects that have either transitioned to a program of  record or are in the process of transitioning. However, with the exception  of the Navy’s Technology Innovation Games, the other four initiatives are  still in the early planning phases. Some of them are still in the process of  developing charters, determining project selection processes, and  documenting priorities. Most of the new rapid capabilities offices were  developed so recently that they were also not in the fiscal year 2017  budget request, but the Army plans to temporarily support its office with  funding from existing Army accounts.

section_title: New Prototyping Initiatives Face a Variety of Barriers
paragraphs: DOD’s new prototyping initiatives face several barriers that can make it  challenging to obtain funding to start projects, manage the initiatives to  achieve innovation, and transition the prototypes to acquisition programs.  Literature on private sector innovation, including the use of prototyping,  suggests that private sector firms face some of these same barriers. Key  barriers we identified include:

Funding structure: Several studies have suggested that maturing  technologies outside and independent of acquisition programs to  higher technology readiness levels can promote innovation and  facilitate technology transition. However, DOD’s funding structure and  how it is commonly interpreted may limit the amount of higher fidelity  prototyping conducted outside of acquisition programs. DOD’s  science and technology community manages and invests research  and development funding in budget activities 6.1-6.3, but does not  typically use budget activity 6.4 funds. According to DOD regulation,  projects funded with budget activity 6.3 are to mature technologies to  technology readiness levels 4, 5, or 6, while those funded with budget  activity 6.4 are to result in the achievement of technology readiness  level 6 or 7 (see app. VIII). Due in part to this budget activity structure,  the science and technology community typically sees its role as  maturing technologies to no higher than technology readiness level 6.  As a result, until DOD and the military services’ recent prototyping  initiatives, there were not many offices focused on further maturing  technologies outside of acquisition programs.

Risk averse culture: Although it is appropriate to minimize risks in  acquisition programs, some officials stated that excessive risk  aversion outside of acquisition programs can stifle innovation.  According to the Defense Science Board, over time, DOD has  become increasingly risk averse and experimentation has moved  towards scripted demonstrations, testing, and training. Pressure to  justify budgets, demonstrate utility to the warfighter, and advance  careers all contribute to this risk aversion. Many prototyping and  innovation initiatives we reviewed emphasized high transition rates of  between 80 and 100 percent. Generally speaking, transition means  that a technology has been sufficiently matured and is ready to  transition to a user such as a weapon acquisition program or the  warfighter in the field. On one hand, a high transition rate can be an  indicator that an initiative is generating a good return on investment  and is developing capabilities that meet customers’ needs. But, for  prototyping initiatives with the stated purpose of encouraging  innovation, particularly disruptive innovation, making high transition  rates a goal could be counterproductive and lead to a lower tolerance  for risk or failure. For private sector projects focused on innovation,  companies can aim for transition rates as low as 20 to 50 percent.

Competing priorities: Officials identified competition with projects the  military services have previously funded and prioritized as a barrier to  innovation efforts—both when requesting funding to prototype and  later when trying to transition. Innovation literature suggests that  companies frequently face this same problem. Resources are often  not available for bolder projects because funds are consumed by pre- existing projects; furthermore, companies are more likely to devote  resources to sustaining innovation, which gradually improves on  existing products, rather than riskier disruptive innovation. The  Secretary of Defense testified to Congress in September 2016 that he  has seen the constant temptation over the years to starve new and  future-oriented defense investments in favor of more established and  therefore well-entrenched programs. He expressed concern that  funding was being taken away from initiatives such as the Strategic  Capabilities Office, to instead pay for existing acquisitions. In fiscal  year 2016, 6.4-funded initiatives that focus on prototyping and  innovation represented less than 4 percent of budget activity 6.4  funds.

Long budget timelines: Long budget timelines make it difficult to start  prototyping projects that address emerging threats in a timely  manner. For example, as is illustrated in figure 8, a project  conceived in February 2017 might not be authorized and appropriated  funding until October 2018. Projects that are expected to take 3 to 5  years to complete in effect require 5 to7 years from conception to  completion. If there is a continuing resolution, it could take longer.  These long timelines make it difficult to achieve the adaptability and  faster capability development and fielding times that DOD seeks to  keep pace with rapidly evolving threats. DOD can take special steps  to provide funding in other ways, such as through reprogramming; but,  in general, long budget timelines not only make it difficult to succeed  fast, they also make it difficult for initiatives to “fail fast” and for DOD  to move on to potentially more promising projects.

Synchronization with acquisition programs: Prototyping efforts may  not be complete at the most opportune time for acquisition programs,  as is reflected in figure 9. If the effort is completed too early,  technology can rapidly become obsolete before a relevant acquisition  program is begun. If a prototyping effort is completed after an  acquisition program has begun, the program may not be willing to  adopt it. Defense Advanced Research Projects Agency officials noted  that partners must budget 2 years in advance to further mature or  transition technologies, which exacerbates this problem. Congress  included a provision in the Fiscal Year 2017 NDAA that, depending on  how it is implemented by DOD, could help make it easier to transition  new technologies and components to programs that have already  begun system development. The NDAA requires that certain MDAPs  be designed and developed, to the maximum extent practicable, with  a modular open systems approach.This type of approach, which  includes a modular design and standard interfaces, enables system  components to be more readily replaced.

section_title: DOD Actions Partially Align with Practices for Fostering Innovation in the Private Sector
paragraphs: The literature we reviewed on private sector innovation highlights several  key practices related to how to organize and manage innovation units,  fund projects, and address potential culture barriers. These partially align  with recent DOD actions. Some of these practices apply directly to  prototyping, while others address innovation more broadly. The key  practices or enablers we identified are listed in table 3 below.

We compared DOD’s new prototyping initiatives with these enablers to  determine whether DOD is well-positioned to generate the type of  innovation, including disruptive innovation, that it is seeking.
subsections: section_title: Developing an Innovation Strategy
paragraphs: DOD has issued multiple memorandums related to prototyping and  innovation, as reflected in table 4, but these documents fall short of a  strategy. Specifically, with regard to prototyping and innovation, none of  the documents we reviewed communicate strategic goals and priorities or  delineate roles and responsibilities among DOD and the military services’  initiatives, which are elements of the innovation strategies described in  the literature as well as standards for internal control. Congress  included a provision in the Fiscal Year 2017 NDAA that provides some  strategic direction for certain prototyping projects. It calls for the military  services to establish or identify oversight boards that will develop triennial  strategic plans to prioritize capability and weapon system component  portfolio areas for prototyping projects, among other things. However, it is  not yet clear whether there will be a mechanism to tie these efforts into a  department-wide strategy.

DOD’s lack of an innovation strategy means it has to rely on other  mechanisms to coordinate and provide strategic direction for its  prototyping initiatives, although those mechanisms do not cover some of  DOD’s prototyping and innovation activities and do not establish  department-wide priorities. For example, Communities of Interest (COI),  which are organized by the Assistant Secretary of Defense for Research  and Engineering (ASD(R&E)), help plan, coordinate, and share  knowledge on science and technology activities for budget activities 6.2  and 6.3, but there are no analogous mechanisms for 6.4-funded activities,  including those related to prototyping and innovation. The 17 COI working  groups are generally organized by portfolio—for example, advanced  electronics—and include representatives from across DOD. They  periodically develop roadmaps for their portfolios. ASD(R&E) officials  explained that the road mapping process is not directly tied to budget  decisions and does not establish department-wide science and  technology priorities. However, it does help identify investment gaps,  opportunities for collaboration, and areas of potential overlap or  overinvestment. DOD’s Long Range Research and Development  Planning Program suggested using COIs to prioritize the technology  investments it identified, which would expand their focus beyond science  and technology. We have previously found that one way to better manage  potentially fragmented activities is to improve collaboration and  coordination. Without an approach that covers relevant 6.4-funded  activities, DOD may be missing out on opportunities to take a more  strategic approach to prototyping and innovation across the department,  including sharing information and identifying areas of potential under- or  overinvestment related to prototyping and experimentation.

DOD is undergoing organizational changes that could provide more  focused leadership, strategy development, and coordination for  prototyping and innovation-related activities:

The NDAA for Fiscal Year 2017 establishes the position of Under  Secretary of Defense for Research and Engineering. According to  the conferees, the creation of this position was part of organizational  changes to DOD that seek to, among other things, advance  technology and innovation. The duties of the Under Secretary of  Defense for Research and Engineering include advancing technology  and innovation and establishing policies on all defense research and  engineering.

DOD’s Defense Innovation Board has recommended that DOD  establish the position of Chief Innovation Officer, to coordinate,  oversee, and synchronize innovation activities across the  department.

DOD has established the position of Deputy Director for Prototyping  and Experimentation to oversee program execution, provide technical  and programmatic advice, and work with DOD entities to identify  shortfalls and potential technologies and projects to address them.  However, the position only has authority over prototyping and  experimentation efforts within the office of the Deputy Assistant  Secretary of Defense for Emerging Capability and Prototyping. To  influence prototyping activities outside of that office including military  service led initiatives, the Deputy Director stated that he has to  leverage his personal relationships and experience.

With DOD’s increased level of effort and investment in prototyping and  innovation comes the potential for inefficiencies if efforts are not  coordinated and aligned with an overarching strategy. Although these  offices are generally attempting to meet different needs and are using a  variety of approaches to achieve innovation, without an articulated  strategy, there is a potential for overlap if their goals and approaches  evolve over time.

section_title: Ensuring Adequate Funding to Address Innovation Goals
paragraphs: With the exception of the Strategic Capabilities Office, DOD and the  military services have not allocated large amounts of funding to their new  prototyping and innovation initiatives in their budget requests and they will  have to compete with other priorities to receive funding in the future.  DOD’s fiscal year 2017 budget request included less than $100 million for  each of the six other initiatives we examined. One approach identified in  the academic literature that helps ensure innovative projects receive  sufficient funding, in the face of competing priorities and a risk averse  culture, is called a “strategic buckets” approach. Under this approach,  management makes a strategic decision to allocate set “buckets” of  resources for different types of projects, including breakthroughs, and  then takes steps to ensure adequate funding for innovation efforts. The  distribution of resources among different buckets is dictated by the  organization’s strategy. This approach is consistent with portfolio  management best practices, which call for organizations to use an  integrated approach to prioritize needs and allocate resources in  accordance with strategic goals. Figure 10 includes a notional depiction  of how this approach could be adapted to the basic tenets of DOD’s  prototyping and innovation efforts.

To implement a strategic buckets approach for innovation, an  organization needs to develop innovation goals, reflect those goals in its  innovation strategy, inventory current projects and funding allocations,  and then adjust funding levels, if needed, to make sure they align with its  goals and strategy. Decisions to change relative levels of investment in  different buckets may be made over time in response to changing world  events. For example, DOD’s concern about losing its eroding warfighting  edge in certain areas could cause it to place a higher priority on  prototyping systems that could lead to disruptive innovations.

The Navy’s approach to managing its science and technology  investments, including its prototyping and innovation initiatives, has  elements of a “strategic buckets” approach. It maps out roughly the  percentage of funding that it plans to request for different parts of its  science and technology portfolio, including some of its prototyping and  innovation initiatives, as is reflected in figure 11 below. The Navy uses  this information to help develop its science and technology budgets. The  Navy has not extended the concept to other research and development  budget activities, such as 6.4, which are largely driven by decisions on  individual acquisition programs.

DOD has also employed aspects of this approach to set and enforce  minimum funding levels for its science and technology investments, but it  lacks certain prerequisites needed to apply it more broadly to prototyping.  ASD(R&E) officials explained that, in recent years, the Office of the  Secretary of Defense has communicated an investment floor for budget  activities 6.1 through 6.3 in the Defense Planning Guidance, which  provides strategic direction for DOD budget formulation. ASD(R&E)  enforces these levels by reviewing military service budget requests and  directing funding increases or other shifts to ensure the floor is met  across the department. Although ASD(R&E) has responsibility for  overseeing research and development activities under budget activities  6.1 through 6.4, ASD(R&E) officials stated that they do not review budget  requests for budget activity 6.4 because they are primarily allocated to  acquisition programs. By not exercising its authority over the full range of  budget activity 6.1 through 6.4 funding, ASD(R&E) is missing an  opportunity to assess prototyping activities collectively from an enterprise  level to determine if and how this funding might best be used to support  DOD’s prototyping and innovation initiatives and its strategic goals.

DOD also lacks the innovation strategy and baseline understanding of its  prototyping projects and their associated funding, to identify areas of  potential over- and under-investment as well as appropriate investment  targets. Better Buying Power 3.0 called for (1) the USD(AT&L) and Vice  Chairman of the Joint Chiefs of Staff, who oversees the weapon system  requirements process, to conduct annual reviews of each service’s  budget activity 6.3- and 6.4-funded prototyping and experimentation  activities, and (2) the ASD(R&E) to develop, maintain, and publish a  database of government and industry experimentation capabilities and  events, and make annual recommendations to the military services and  USD(AT&L) for additional prototyping. USD(AT&L) and military service  officials stated that the reviews have not been held due to difficulties with  scheduling. In addition, although ASD(R&E) officials took steps to  develop the database, it was not completed and efforts to make the  information available through a different database were unsuccessful.  Both the reviews and the database could have provided useful  information about prototyping and experimentation activities and  opportunities across the department.

The NDAA for Fiscal Year 2017 included a provision that could provide  more information about funding for select prototyping initiatives outside of  acquisition programs. This provision could better position DOD to set  and track prototyping investment targets. In budget requests after fiscal  year 2017, for budget activity 6.4, the provision requires DOD to state the  amounts requested for prototyping and experimentation of weapon  system components and technologies separate from acquisition programs  of record. Those requests are to reflect priority areas for prototyping.  Furthermore, the legislation calls for military services to establish or  identify prototyping oversight boards to, among other things, annually  recommend funding levels for prototype projects across capability or  weapon system component portfolios although no analogous  recommendations are required for efforts outside of the military  services.

section_title: Balancing Demand Pull and Supply Push
paragraphs: Most of DOD and the military services’ prototyping and innovation  initiatives use more of a “demand pull’ approach to selecting projects,  which could limit their likelihood of generating disruptive capabilities (see  app. IX for a list of these initiatives). This was the case for both older and  newer initiatives. Demand pull initiatives focus on prototyping  technologies or systems to address validated requirements, which means  they have built in constituencies ready to support them. On the other  hand, “supply push” initiatives take on projects without a stated customer  need and do not align with existing organizational structures. This can  make it difficult to gain support for supply push type projects, particularly  when it is time to transition them into programs. For example, the Navy  had to establish an Unmanned Maritime Systems program office when  unmanned underwater capabilities languished because there were no  “customers” given existing organizational structures. DARPA, with an  annual budget of over $1 billion, is DOD’s largest example of an  organization that primarily uses a supply push model.

An overreliance on demand pull can lead to incremental improvements in  capabilities without ever achieving a more disruptive breakthrough. The  Deputy Assistant Secretary of Defense for Research stated that DOD’s  requirements process is a model for slightly improving how DOD  conducts operations now rather than thinking outside the box of the art of  the possible. DOD does not have a similar process designed to foster  more innovative solutions. Without an innovation strategy that sets goals  and aligns funding for demand pull and supply push projects accordingly,  DOD’s prototyping and innovation initiatives might not produce the types  of disruptive capabilities and breakthroughs the department is seeking.

section_title: Learning Quickly Through Prototyping and Experimentation
paragraphs: Most DOD prototyping and innovation initiatives we reviewed took steps  so that they could learn quickly through their projects. Almost all of them  have expected project turnarounds of 3 to 5 years or less. Initiatives such  as the Army’s Technology Maturation Initiative and the Strategic  Capabilities Office also regularly reviewed projects to determine whether  they were still needed or feasible based on initial efforts and, if they were  not, terminated projects accordingly. Two longstanding initiatives  employed approaches to speed up the funding process. DOD’s Joint  Concept Technology Demonstration program notifies Congress about  new projects via letter prior to starting them rather than waiting to request  approval in each budget request. Officials from the Future Naval  Capabilities Program stated that they use funding left over from projects  completed in a given year for other projects, as long as the amount falls  below a certain threshold.

section_title: Fostering Greater Risk Tolerance
paragraphs: DOD and military department officials acknowledge that there is a risk  averse culture across the department, even with respect to prototyping  and innovation initiatives. However, neither the officials we spoke with nor  recent memorandums have described ways DOD is changing its metrics  or incentives to encourage more risk tolerance within these initiatives,  which is one of the enablers highlighted in the literature on private sector  innovation. Other enablers, such as developing an innovation strategy  and ensuring adequate funding to support it, could also help foster a more  risk tolerant environment. DOD’s Defense Innovation Board is also in the  process of identifying ways to develop a culture of innovation in DOD in  which new ideas can be tested and fail without fear of ending or derailing  the career of a science and technology manager, acquisition professional,  or military officer.

section_title: Conclusions
paragraphs: Prototyping is a tool that can help DOD address a variety of both long- standing and recent weapon system acquisition and modernization  challenges. When used effectively, it can help reduce risks and improve  the likelihood that a weapon system acquisition program will be  completed on time and on budget. Furthermore, it helps keep DOD’s  technology pipeline stocked with new and innovative technologies that  might provide the next great leap ahead in military capabilities and may  even deter adversaries by demonstrating advanced capabilities.

In the period since the Weapon System Acquisition Reform Act (WSARA)  of 2009 was implemented, DOD acquisition programs have used  prototyping to reduce risk and inject realism into their business cases,  which has helped place them on sound footing for future success. The  results were notable on the programs we reviewed—lower technical risk,  better understanding of requirements, and more information on potential  costs, among other benefits. With the recent repeal of WSARA’s  competitive prototyping requirements, there is a risk that programs will  choose not to prototype. In doing so, those programs would forfeit the  significant benefits that early prototyping can offer.

DOD’s efforts to expand prototyping and experimentation to help achieve  the innovation and disruption needed to maintain its technological and  military advantage are in a more nascent stage. However, challenges,  such as limited funding, a risk averse culture, and competing priorities,  are already apparent and may make it difficult for the efforts to gain  momentum. Pending organizational changes, including the creation of the  positions of Under Secretary of Defense for Research and Engineering  and Chief Innovation Officer, provide an opportunity for DOD to elevate  and take a more strategic approach to the mission of advancing  technology and innovation.

The literature on private sector practices provides a roadmap for how this  new DOD leadership can enable innovation, including through the use of  prototyping. But DOD will need to fully embrace certain key enablers that  are not currently present in the department, including a strategy that  addresses its disparate prototyping and innovation efforts and strategic  goals that can be used to guide resource decisions. It will also need to  work across funding structures for science and technology and more  advanced development work that usually separate certain types of  prototyping efforts. The recent increased level of effort and investment in  prototyping and innovation comes with the potential for inefficiencies if  efforts are not strategic and coordinated. Other high-risk investments in  categories such as disruptive technologies may need to be protected from  a risk averse culture, as well.

DOD has taken several steps to adopt aspects of private sector  innovation practices and has developed mechanisms to coordinate and  review its science and technology investments, but without a more  strategic, inclusive, and deliberate approach overall, its new prototyping  and experimentation initiatives might not generate the levels and types of  innovation the department is seeking.

section_title: Recommendations for Executive Action
paragraphs: To help ensure DOD takes a strategic approach for its prototyping and  innovation initiatives and overcomes funding and cultural barriers, we  recommend that the Secretary of Defense direct the Assistant Secretary  of Defense for Research and Engineering to take the following four  actions:

Develop a high-level DOD-wide strategy, in collaboration with the  military services and other appropriate DOD components, to  communicate strategic goals and priorities and delineate roles and  responsibilities among DOD’s prototyping and innovation initiatives.

Take steps, such as adopting a “strategic buckets” approach, to help  ensure adequate investments in innovation that align with DOD-wide  strategy.

Review budget activity 6.4 funding requests to help maintain a level of  investment for budget activity 6.4-funded prototyping and innovation  efforts that is consistent with DOD-wide strategy.

Expand the Community of Interest working groups to include budget  activity 6.4-funded prototyping and innovation initiatives in their  science and technology planning and coordination processes or  employ a similar coordination mechanism for budget activity 6.4- funded prototyping and innovation initiatives.

section_title: Agency Comments
paragraphs: We provided a draft of this report to DOD for comment. In its comments,  reproduced in appendix X, DOD concurred with our four  recommendations. DOD also provided technical comments, which we  incorporated as appropriate.

We are sending copies of the report to the appropriate congressional  committees; the Secretary of Defense; the Secretaries of the Army, Navy,  and Air Force; and the Assistant Secretary of Defense for Research and  Engineering. In addition, the report will be available at no charge on  GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at 202-512-4841 or sullivanm@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix XI.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to assess (1) how the Department of Defense (DOD)  has used prototyping prior to system development on major defense  acquisition programs, and (2) what steps DOD has taken to increase  innovation through prototyping activities conducted outside of major  defense acquisition programs.
subsections: section_title: DOD’s Use of Prototyping on Major Programs Prior to System Development
paragraphs: To address our first objective, we examined 22 major defense acquisition  programs that had a Milestone B decision, which approves entry into  system development, between December 2009 and February 2016, or  anticipated receiving Milestone B approval by February 2016 when we  began our selection process. We selected December 2009 as the starting  date because it was when DOD implemented the Weapon System  Acquisition Reform Act of 2009 and its associated prototyping provisions.  These programs and the dates they entered system development are  included in table 5.

To determine what prototyping approaches the 22 programs used, if any,  and to identify costs, benefits, challenges, and lessons learned from their  prototyping efforts, we reviewed program documents, such as technology  development strategies, acquisition strategies, prototyping waivers,  acquisition decision memorandums, independent cost estimates, and  budget requests. We also conducted semi-structured interviews with  officials from each of the 22 programs and reviewed prior GAO reports.  We also examined prior GAO work related to acquisition program  outcomes and the technology development phase.

To examine the proportion of research, development, test, and evaluation  (RDT&E) funds planned for development prior to each program’s entry  into system development, we reviewed program funding stream data  obtained from December 2015 Selected Acquisition Reports. We  calculated the RDT&E funds planned as of the month prior to the  program’s Milestone B approval date, which we obtained from program  documents. We then divided the prorated amount by the program’s  current RDT&E cost estimates to obtain the proportion of RDT&E funds  planned for use prior to system development. We excluded six programs  that did not have complete data available. See appendix VII.

To examine how these 22 programs fared in terms of cost and schedule  performance, technology maturity, and design stability, we compared  prototyping programs’ data with non-prototyping programs’ data.  Specifically, for programs’ cost outcomes, we examined the difference  between programs’ first full and current RDT&E cost estimates.  Programs’ first full estimates are typically developed upon program entry  into system development at Milestone B. For programs’ schedule  outcomes, we examined the growth between when the program entered  and completed system development using programs’ first full and current  estimates. Completion of system development usually occurs when  Milestone C is achieved. For first full estimate data, we leveraged data  collected as part of our annual assessment of DOD weapon systems.  This included cost, quantity, and schedule data from the Defense  Acquisition Management Information Retrieval Purview system, referred  to as DAMIR. The team entered this data into a database and verified  that the data were entered correctly. We converted all cost information to  fiscal year 2017 dollars using conversion factors from the DOD  Comptroller’s National Defense Budget Estimates for Fiscal Year 2017  (tables 5-9). To assess the reliability of the data the annual assessment  team talked with a DOD official responsible for DAMIR’s data quality  control procedures and reviewed relevant documentation. They also  confirmed selected data reliability with program offices. For current  estimate data, we obtained RDT&E, total acquisition cost, quantity, and  schedule estimates from the August 2016 Defense Acquisition Executive  Summary reports. We determined that data were sufficiently reliable for  the purposes of this report.

The selected programs in our review entered system development in  December 2009 or after and are generally newer. To address concerns  about examining outcomes given the relative newness of many of the  programs, we excluded the following six programs from our cost and  schedule analyses because they are too recent to have current estimates  separate from the program’s baseline or do not have approved first full  estimates: Amphibious Combat Vehicle, B-2 Defense Modernization  System Modification, Common Infrared Countermeasure, Next  Generation Jammer Increment 1, Offensive Anti-Surface Warfare  Increment 1, and Three-Dimensional Expeditionary Long-Range Radar.  We excluded two additional programs—Enhanced Polar System and  Space Fence Increment 1—from our schedule analysis because these  programs will not hold a Milestone C.

To examine the technology maturity and design stability of programs, we  leveraged survey response data provided in support of our annual  assessments of selected weapon programs. These assessments rely on  data collected from program offices related to the technology readiness  levels of their critical technologies and their percentage of completed  design drawings.

Our best-practices work has shown that a technology readiness level  (TRL) 7—demonstration of a technology in an operational  environment—is the level of technology maturity that constitutes a low  risk for starting a product development program. For shipbuilding  programs, we have recommended that this level of maturity be  achieved by the contract award for detailed design. In our  assessment, the technologies that have reached TRL 7, a prototype  demonstrated in an operational environment, are referred to as  mature or fully mature. Those technologies that have reached TRL 6,  a prototype demonstrated in a relevant environment, are referred to  as approaching or nearing maturity. Satellite technologies that have  achieved TRL 6 are assessed as fully mature due to the difficulty of  demonstrating maturity in an operational environment—space. No  programs needed to be excluded from the technology maturity  analysis. See appendix VIII for TRL definitions.

Our best practices work shows that completion of at least 90 percent  of engineering drawings at critical design review provides tangible  evidence that the product’s design is stable. Completed design  drawings were defined as the number of drawings released or  deemed releasable to manufacturing that can be considered the “build  to” drawings. For shipbuilding programs, they asked program officials  to provide the percentage of the three-dimensional product model that  had been completed by the start of lead ship fabrication, and as of our  annual assessment. Five programs were excluded from this analysis.  The Joint Light Tactical Vehicle program does not track the percent of  releasable drawings and the Combat Rescue Helicopter, Next  Generation Jammer Increment 1, Global Positioning System Next  Generation Operational Control System, and Three-Dimensional  Expeditionary Long-Range Radar programs have not yet held their  critical design reviews.

Although the technology maturity and design stability information provided  at key knowledge points provide excellent indicators of potential risks, by  themselves they do not cover all elements of risk that a program  encounters during development, such as funding instability. See appendix  V for a summary of program outcomes.

section_title: DOD’s Use of Prototyping to Increase Innovation Outside of Major Programs
paragraphs: To address our second objective, we reviewed fiscal year 2017 budget  documentation and interviewed DOD and military service officials  responsible for research and development to identify initiatives that DOD  started in the past five years with the stated purpose of promoting  innovation through prototyping and experimentation. We focused on  broad-based initiatives, rather than ones focused on a specific technology  area. We also examined key preexisting initiatives for contrast. We only  included initiatives funded with budget activities 6.3 and 6.4, advanced  technology development and advanced component development and  prototypes respectively, because those budget activities fund the  development and testing of new concepts and capabilities using higher  fidelity prototypes that have the potential for short- or medium-term  application. We did not meet with rapid prototyping offices established for  direct support to the conflicts in Afghanistan and Iraq because they were  designed for a temporary contingency.

To identify the initiatives’ goals, focus areas, scope, approaches, funding  characteristics, strategies, coordination mechanisms, and barriers, if any,  they face, we reviewed documentation from the initiatives, such as budget  requests, charters, and briefings. We also interviewed program officials  and obtained written responses to questions. To determine what direction  and strategy DOD has provided for the initiatives, we analyzed DOD  memorandums on the following subjects: Long Range Research and  Development Plan, Defense Innovation Initiative, Wargaming and  Innovation, and Better Buying Power 3.0 as well as Navy memorandums  on: Task Force Innovation, Wargaming, and Innovation Funding within  the Naval Research and Development Establishment. We also reviewed  DOD’s Long Range Research and Development Planning Program  briefing. We examined DOD’s process for coordinating science and  technology investments, called “Reliance 21,” to determine the extent to  which it addressed prototyping for innovation and whether it has the  potential to do so.

We also conducted a review of literature on innovation in the commercial  sector, including the use of prototyping, to identify enablers that could be  applicable in DOD and to identify barriers commercial sector  organizations face. The literature was primarily from academic sources,  but included some literature from the private sector. Specifically, we  began with recognized experts in the field of innovation. We then used a  snowball methodology to identify other key authors on innovation through  databases such as ProQuest and WorldCat. We also asked DOD officials  for recommendations regarding relevant authors and articles. Our  literature search covered articles published from 1996 onward, with a  majority written between 2005 and 2016. We identified 19 sources that  were specific to our work. They primarily relied on interviews, surveys,  and case studies. Through the literature search, we identified a number of  general themes about spurring innovation across articles and interviews.  We then developed a list of key enablers from these themes that could  potentially apply to DOD prototyping for innovation activities. We also  noted when these sources identified barriers to innovation that aligned  with the barriers we identified as existing in DOD. To determine whether  DOD’s practices are consistent with these enablers, we compared them  with memorandums related to prototyping for innovation, the Navy’s and  DOD’s approach to managing funding for innovative research and  development as reflected in the Navy Science and Technology Strategy  and in DOD briefings, demand pull and supply push emphases of  prototyping for innovation initiatives, and initiatives’ approaches to  learning quickly as reflected in their documentation. When applicable, we  also compared DOD’s approach to its prototyping and innovation  initiatives with additional sources including the Standards for Internal  Control in the Federal Government (for strategy and goals); GAO work on  fragmentation, duplication, and overlap (for coordination); and portfolio  management best practices (for funding and prioritization).

To inform all assessments for this objective, we interviewed officials from  the Office of the Under Secretary of Defense (Comptroller) and officials  from each military department’s comptroller’s office; Office of the  Assistant Secretary of Defense for Research and Engineering; Office of  the Deputy Assistant Secretary of Defense for Emerging Capabilities and  Prototyping; Office of the Assistant Secretary of the Air Force for  Acquisition; Office of the Assistant Secretary of the Army (Acquisitions,  Logistics, and Technology); and Office of the Deputy Assistant Secretary  of the Navy for Research, Development, Test, and Evaluation.

We conducted this performance audit from September 2015 to June 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

section_title: Appendix II: Department of Defense Research, Development, Test, and Evaluation Budget Activities

section_title: Appendix III: Budget Activity 6.3 and 6.4 in Fiscal Year 2016
paragraphs: The technology transition-related budget activities 6.3 and 6.4—which  received $5.8 billion and $14.6 billion respectively in fiscal year 2016— support numerous entities across the Department of Defense (DOD). In  fiscal year 2016, the Defense Advanced Research Projects Agency, the  Army, and the Office of the Secretary of Defense received the highest  levels of budget activity 6.3 funds—with each component receiving about  20 percent of total funding—and the remaining 38 percent was divided  among several organizations (see figure 12). That same year, the Missile  Defense Agency received the largest percentage of budget activity 6.4  funds, with 42 percent, while the Navy received 35 percent and the  remaining 23 percent was shared among several organizations (see  figure 13).

section_title: Appendix IV: Overview of Selected Programs’ Prototyping Efforts and Reasons for Not Prototyping

section_title: Appendix V: Programs’ Cost and Schedule Performance and Knowledge Attainment
paragraphs: Overall, prototyping appears to have reduced risks on more complex  programs, as evidenced by the fact that most programs that prototyped  had similar cost and schedule outcomes and attained key knowledge at  similar rates as non-prototyping programs that were generally considered  less complex and risky. See appendix I for the programs included in each  of the following analyses. Table 9 below includes individual program  outcomes to date.

Our analysis of cost and schedule performance to date showed that for  the programs we reviewed, most prototyping and non-prototyping  programs’ outcomes were similar (see figures 14 and 15). Nine of 11  prototyping programs had similar levels of research, development, test  and evaluation cost growth as the five non-prototyping programs. Officials  from the two prototyping programs with higher levels of cost growth— Global Positioning System Next Generation Operational Control System  (GPS OCX) and Integrated Air and Missile Defense (IAMD)—told us that  they could have learned more about their prototyping efforts if they  produced higher fidelity prototypes or better addressed key risk areas. Six  of nine prototyping programs with comparable data experienced similar  schedule growth between the start and completion of system  development as the five non-prototyping programs. The IAMD and GPS  OCX programs were also among the prototyping programs with the  largest schedule growth.

We also found similar outcomes between prototyping and non-prototyping  programs with respect to the technology and design knowledge they had  at key points in the acquisition process (see tables 10 and 11). Most  programs that prototyped during technology development matured or  nearly matured their critical technologies before entering system  development and released 90 percent of the system’s drawings at critical  design review (see appendix VIII for definitions of technology readiness  levels). Demonstrating high levels of technology and design knowledge  by critical points in the acquisition process is a GAO best practice that  helps to reduce program risk.

section_title: Appendix VI: Examples of Prototyping Benefits that Contributed to Sound Business Cases

section_title: Appendix VII: Relative Cost of Prototyping Efforts
paragraphs: To get a general sense of the relative cost of programs’ prototyping  efforts, we compared programs’ planned research, development, test and  evaluation (RDT&E) funding during technology development to their  overall RDT&E cost estimate. Table 12 presents the programs’  prototyping approaches and percent of RDT&E acquisition costs planned  prior to system development for 16 of the programs we examined. Non- competitive programs ranged from 50 to 80 percent, whereas competitive  programs ranged from 10 to 77 percent.

section_title: Appendix VIII: Technology Readiness Levels

section_title: Appendix IX: Department of Defense Initiatives Using Prototyping to Further Innovation
paragraphs: Since 2012, DOD has expanded its prototyping and experimentation  efforts, in order to increase innovation, rapidly field new technologies,  explore operational concepts, and hedge against threats from potential  adversaries, among other purposes. Table 13 below includes newer  prototyping and innovation initiatives and select older initiatives. We  focused on the newer initiatives in the body of the report. Although  broadly speaking, these initiatives are all seeking to achieve the same  goal of innovating to maintain military superiority, we found that they differ  in the types of innovation they are trying to achieve, the scope of their  efforts, and their approaches.

The initiatives we examined varied in the primary types of innovation they  sought to achieve (see table 14). Initiatives that primarily focus on  incremental or sustaining innovation gradually improve existing products  and capabilities. Initiatives focused on disruptive innovation attempt to  shift the balance of military power by providing new capabilities,  potentially unforeseen by customers or adversaries. The type of  innovation sought should not necessarily be equated with risk. For  example, the Strategic Capabilities Office (SCO) considers itself to be  focusing on low technical risk solutions that are intended to have  potentially disruptive results. We excluded experimentation initiatives from  this analysis.

Most of the prototyping and experimentation initiatives we examined,  including all but one of the newer initiatives, were focused on addressing  military service specific needs (see table 15). Three initiatives address  capabilities that could potentially benefit multiple services or that are seen  as “falling through the cracks” of military service efforts. Those initiatives  are among the most well-funded, with the SCO requesting $902 million  for budget activities 6.3 and 6.4 and Defense Advanced Research  Projects Agency (DARPA) requesting $1.2 billion for budget activity 6.3  for fiscal year 2017. We excluded experimentation initiatives from this  analysis.

Most of the initiatives we examined focused on prototyping technologies  or systems that address validated customer or warfighter requirements— meaning they have more of a “demand pull” approach to selecting  projects (see table 16). In contrast, “supply push” initiatives take on  projects without a stated customer need and may not align with existing  organizational structures or warfighting concepts. DARPA has historically  focused almost exclusively on “supply push” type projects, as does the  Navy’s Innovative Naval Prototypes initiative. We excluded  experimentation initiatives from this analysis as well as the Marine Corps  Rapid Capabilities Office and the Navy’s Rapid Prototyping  Experimentation and Demonstration initiative because it is not yet clear  which approach they will take.

DOD’s newer prototyping and innovation initiatives tended to differ from  older initiatives in several ways, such as the DOD budget activity used,  the transition paths available, and senior leadership involvement.

The new initiatives are almost exclusively funded with budget activity  6.4. Budget activity 6.4 funds are viewed as allowing for higher-fidelity  prototyping efforts, which are closer to the intended end item. Some  previous studies have suggested that maturing technologies outside  of acquisition programs to higher readiness levels using budget  activity 6.4 can promote innovation and facilitate technology transition.  Older initiatives primarily rely on budget activity 6.3 funds.

The Army Rapid Capabilities Office and Navy Rapid Prototyping  Experimentation and Demonstration plan to conduct rapid prototyping  and then oversee a modified acquisition process for the most  promising prototypes. These offices may take responsibility for the  rapid prototyping pathway called for in the National Defense  Authorization Act for Fiscal Year 2016. Older initiatives typically  transition technologies to a different organization or acquisition  program.

Many of the new prototyping initiatives report to senior department  leaders, which can help maintain support for investments and  streamline decision-making, according to the literature on innovation  in the private sector. The SCO director reports to the Deputy  Secretary of Defense, which provides him flexibility to work throughout  the department.

Furthermore, several new initiatives have unique characteristics or  functions that the department is trying to invigorate.

The SCO focuses on retooling fielded equipment and technologies to  employ them in new ways. This reflects a new tactic in DOD’s  innovation efforts because no other initiative has focused on this  approach. The office limits its efforts to specific platform-centric  solutions with the intention of delivering capabilities quickly. According  to SCO’s director, one advantage of this approach is that it can help  “buy time” for other initiatives to develop and field the next generation  of systems. SCO covers all development costs initially to prove out  the idea and develop a prototype, then turns projects over to the  services for consideration in their budget deliberation processes. DOD  has increased SCO funding since fiscal year 2014 from $140 million  to a planned $902 million in fiscal year 2017. This increase is driven  by new projects as well as testing and development efforts on existing  projects.

Two offices—the Air Force’s Experimentation Initiative and the Navy’s  Technology Innovation Games—reflect a renewed emphasis on  experimentation in DOD to explore how new capabilities being  prototyped could be employed in an operational setting, among other  purposes. These offices use virtual or conceptual environments to  explore early ideas for implementing new technologies. They can also  help identify promising concepts, or allow officials to discuss how to  adjust tactics, doctrine, and training prior to development of new  technologies. For example, the Air Force’s Experimentation Initiative  is using experimentation to explore how directed energy could be  used in an operational context and plans to use this information to  inform decisions about pursuing directed energy systems. The Air  Force is also considering conducting hardware prototyping under this  initiative to demonstrate the operational value of proposed concepts.  The Navy’s Technology Innovation Games employ a progression from  workshops and discussions up through wargames and  demonstrations.

section_title: Appendix X: Comments from the Department of Defense

section_title: Appendix XI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Ron Schwenn, Assistant Director;  Pete Anderson; Leslie Ashton; Lorraine Ettaro; Daniel Glickstein; Laura  Holliday; Richard Hung; Katherine Lenane; Loren Lipsey; Michael Sweet;  Alyssa Weir; and Robin Wilson made significant contributions to this  report.

Fast facts: DOD invests roughly $70 billion annually in weapon system research and development, including prototyping—the development and testing of a model or weapon system design.

When used effectively, prototyping reduces risk and improves the likelihood that weapon systems will be completed on time and within budget. It also keeps DOD’s technology pipeline stocked with innovative technologies that may provide a leap ahead in military capability. DOD has several new prototyping initiatives under way, but has not developed a comprehensive strategy for these efforts.

Among other things, we recommend that DOD develop such a department-wide strategy.

Question-summary annotations: questions: question: Why does DOD use prototyping on its MDAP?
answer: The Department of Defense (DOD) has used prototyping on its major defense acquisition programs (MDAP) primarily to reduce technical risk, investigate integration challenges, validate designs, mature technologies, and refine performance requirements.
child_questions: question: What did GAO review with regards to prototyping?
answer: Of the 22 programs GAO reviewed, 17 used prototyping before starting system development.

question: What information did prototyping provide?
answer: For many of those programs, prototyping provided information that helped introduce realism into their business cases by providing information on technology maturity, the feasibility of the design concepts, potential costs, and the achievability of planned performance requirements.

questions: question: What initiatives has DOD developed?
answer: DOD has developed new initiatives that are outside of major defense acquisition programs to increase prototyping and further innovation.
child_questions: question: What problems do these initiatives face?
answer: However, these initiatives face barriers, such as limited funding, a risk averse culture, and competing priorities.

question: What are enablers for DOD's efforts?
answer: Literature on private sector innovation identifies key enablers for these types of efforts, such as developing an innovation strategy, aligning investments with innovation goals, and protecting funding for riskier projects.
child_questions: question: How is DOD not utilizing all of these enablers?
answer: For example, DOD does not have a department-wide strategy that communicates strategic goals and priorities and delineates roles and responsibilities to guide the prototyping initiatives. This could lead to unproductive or poorly coordinated investments later.

question: How can DOD address funding issues with its innovations?
answer: One strategy to address funding issues called “strategic buckets” involves allocating resources to different types of projects based on an organization's strategy (see figure).

questions: question: How was GAO able to review DOD's research and development funds?
answer: House Conference Report 114-102 accompanying a bill for the fiscal year 2016 National Defense Authorization Act included a provision for GAO to review how DOD's research and development funds are used and whether this approach effectively supports activities such as prototyping.
child_questions: question: What does GAO's report address?
answer: This report assesses (1) how DOD has used prototyping prior to system development on major defense acquisition programs, and (2) what steps DOD has taken to increase innovation through prototyping activities outside of major defense acquisition programs.

question: What did GAO examine?
answer: GAO examined prototyping activities for 22 MDAPs that planned to enter system development between December 2009 and February 2016 and 7 prototyping-focused initiatives with the stated purpose of promoting innovation.