Weapon Systems: Prototyping Has Benefited Acquisition Programs, but More Can Be Done to Support Innovation Initiatives
Released: 2017-06-27T00:00:00

Letter

Background

DOD’s science and technology community—including research laboratories, test facilities, industry, and academia—conducts initial research, development, and testing of new technologies to improve military operations and ensure technological superiority over potential adversaries. Afterwards, the acquisition community typically manages product development, in which technologies are further advanced and system development begins. These activities are generally supported by DOD’s RDT&E budget, which DOD groups into seven budget activity categories for its budget estimates and the President’s Budget. The categories follow a mostly sequential path for developing technologies from basic research to operational system development, as is shown in figure 1. The first three budget activity categories represent DOD’s science and technology activities to advance research and technology development, while the remaining budget activity categories are typically associated with product development for acquisition programs. See appendix II for a description of each budget activity.

Funding for prototyping is mostly found in advanced technology development and advanced component development, budget activity 6.3 and 6.4 respectively. Appendix III provides a breakdown of budget activity 6.3 and 6.4 funding by organization for fiscal year 2016. Funding in budget activity 6.3 is not directly tied to acquisition programs whereas budget activity 6.4 is typically used for that purpose. Funding for acquisition programs, including budget activity 6.5, was $28 billion in fiscal year 2016 and has varied over time, whereas science and technology funding was $13 billion and has remained relatively flat, as is shown in figure 2.

There are numerous types and definitions of prototyping. One construct used by parts of DOD refers to conceptual, developmental, and operational prototypes, each of which has a different purpose and time horizon for when they can be expected to be incorporated into or become their own acquisition program. Figure 3 includes more information about each of these types of prototypes. Although each type is more mature or closer to a capability that can be fielded than its predecessor, prototyping does not have to proceed sequentially. For example, an operational prototype might not be preceded by a conceptual or developmental prototype, if it is based on existing mature technologies or concepts.

Prototyping can involve a variety of different approaches, in terms of what is being developed and demonstrated, who is building the prototype, and how it is being acquired or managed. System prototyping is when a prototype that includes components for an entire system is developed, such as a prototype of a ground vehicle or missile. Subsystem prototyping is when a prototype is developed that includes a group of components that combine to perform a major function for a system, such as a power supply system for a radar. In a DOD context, prototypes can be developed by contractors or groups of contractors, government labs, or both, and efforts can be managed by the science and technology community, acquisition programs, or other types of research and development organizations. When two or more contractors or other entities prototype the same component, subsystem, or system, the effort is referred to as competitive prototyping.

Over the years, DOD and Congress have taken steps to encourage prototyping during the technology development phase of weapon system acquisition programs. In 2007, the Office of the Under Secretary of Defense for Acquisitions, Technology, and Logistics issued a memorandum on prototyping and competition that expressed concern that DOD decisions on acquisition programs were being based largely on paper proposals that provided inadequate knowledge of technical risk and a weak foundation for estimating development and procurement cost. To help address these concerns, the memorandum required pending and future acquisition programs to formulate acquisition strategies that call for conducting competitive prototyping up through the start of system development. Not long after, in 2009, Congress passed WSARA, which included a provision on competitive prototyping for MDAPs as well as many other reforms. WSARA called for competitive prototyping at the system level or—if not feasible—for critical subsystems, and allowed competitive prototyping to be waived only if the cost of producing competitive prototypes would exceed the life-cycle benefits of producing them or if without a waiver, DOD would be unable to meet critical national security objectives. If competitive prototyping was waived, WSARA required that programs produce a prototype before milestone B, if the expected life-cycle benefits of producing such prototypes exceeded the cost and its production was consistent with achieving critical national security objectives. Originally implemented by DOD in December 2009, Congress repealed WSARA’s competitive prototyping provision in 2015. However, as of the time of this review, DOD still required program officials to consider using prototyping and competitive prototyping at the system or subsystem level as a risk mitigation technique. Congress also included several new prototyping-related provisions in the fiscal year 2017 NDAA, which are discussed later in this report.

DOD prototyping also occurs outside or independent of acquisition programs. One of the purposes of this type of prototyping can be to further disruptive innovation. Disruptive innovation attempts to shift the balance of military power in our favor by providing new capabilities, potentially unforeseen by the warfighter. The capabilities can be a result of new technologies, new ways to integrate existing technologies, or changes to how systems are employed. Disruptive innovation can also include providing existing capabilities at substantially lower cost, thereby increasing military advantage. Examples of potentially disruptive technologies include directed energy, hypersonics, and low cost missile defense capabilities. Prototyping can be a way to “test the waters” or experiment with new and potentially disruptive concepts and technologies without the level of commitment associated with starting acquisition programs. Prototyping in this environment may involve more risk, including less mature technologies. There may also be no residual value at the end of a project other than increased knowledge and potentially a prototype “on the shelf” for further maturation.

DOD Used Prototyping to Reduce Technical Risks, Validate Designs, and Investigate Integration Challenges on Major Weapon Acquisition Programs

Most major weapon system acquisition programs we examined used prototyping to reduce technical risk, investigate integration challenges, and validate designs, among other things. Program officials chose prototyping approaches to align with their assessments of program risks, with riskier programs prototyping more extensively. They generally found that prototyping provided a good return on investment. It helped programs better understand requirements, the feasibility of proposed solutions, and cost—the key elements of a program’s business case. Identifying key risks early and structuring prototyping efforts to inform key decisions helped maximize the utility of programs’ prototyping efforts.

Major Acquisition Programs Selected Prototyping Approaches That Addressed Key Risks

The programs we reviewed used prototyping primarily to reduce technical risks, investigate integration challenges, validate designs, and mature technologies. Of the 22 MDAPs we reviewed, 17 used some form of prototyping during technology development and 5 did not prototype. Of the 17 programs that prototyped, officials from 15 programs told us they chose to prototype because it made the most sense given the program’s needs. Officials from the other 2 programs told us their programs prototyped at the direction of the Under Secretary of Defense (USD) for Acquisition, Technology and Logistics (AT&L)—the milestone decision authority for their programs. All 17 programs prototyped for multiple reasons and officials from 11 programs identified four or more reasons. The reasons cited by each program are depicted in figure 4.

Program officials stated that they tailored their prototyping approaches to their program’s risks, with riskier programs prototyping more extensively. Ten of the programs we reviewed conducted system-level prototyping, 7 programs conducted subsystem prototyping, and 5 did not prototype (see appendix IV for a brief overview of each program’s prototyping efforts or the reasons it did not prototype). Prototyping approaches varied within these categories. Some programs prototyped one or two subsystems while others used multiple contractors and multiple-phased prototyping efforts at the system and subsystem level. The five programs that did not conduct any prototyping used known designs and existing technologies, which DOD generally considers less risky. Four of the five entered the acquisition process without a technology development phase and most obtained waivers from the competitive prototyping requirements in DOD policy. Figure 5 shows examples of programs’ prototyping efforts—or lack thereof—and how they align with the program officials’ understanding of their risks.

The 10 programs that developed system level prototypes ranged from the Joint Light Tactical Vehicle, which involves less expensive, lower complexity items that will eventually be purchased in the tens of thousands, to the Space Fence Ground-Based Radar, which will only produce one, large ground-based radar to detect, track, and provide information about objects in Earth’s orbit. System level prototyping led to improved understanding of: (1) integration challenges and the feasibility of system designs; (2) significant unknowns, such as costs for ambitious requirements; (3) uncertainties related to integrating mature technologies in new ways; and (4) new technologies. Six of these programs first conducted subsystem prototyping to mature new or existing technologies used in new ways before using system-level prototyping to investigate integration challenges.

The seven programs that developed subsystem prototypes ranged from an F-22 modernization program that was upgrading the aircraft’s weapon and communication systems to the Ship to Shore Connector Amphibious Craft, an air-cushioned landing craft that transports personnel, weapon systems, and cargo. Subsystem level prototyping efforts focused on narrower areas of perceived risk, such as maturing critical technologies, integrating a subsystem with other hardware or software, or testing specific components that are being considered for use in a system design. The programs that conducted subsystem prototyping were often building on or well-positioned to leverage existing weapon systems. Five of the seven programs in this category used existing platforms, either hardware or software, for their subsystem prototyping efforts. For example, the Ship to Shore Connector Amphibious Craft utilized the Navy’s existing landing craft as a platform to prototype and test new components.

Of the 17 programs that prototyped, 12 used competitive prototyping. They did so for a variety of reasons, including to prove out potential solutions when more than one solution could be feasible and to gain knowledge from multiple sources about uncertainties, such as integration challenges, design feasibility, or cost. We found that most of the competitive prototyping efforts included system-level designs and almost all programs that conducted system-level prototyping used competitive approaches, as is shown in figure 6. Competitive prototyping may generate more information about proposed solutions because contractors sometimes propose different design approaches or system concepts to meet DOD’s capability needs. DOD’s 2007 prototyping and competition memorandum also noted that competition would generate more knowledge about technical risk and build a stronger foundation for estimating costs.

Major Acquisition Program Officials Stated That Prototyping Provided a Good Return on Investment

According to officials from 16 of the 17 programs that prototyped, the benefits gained from their prototyping effort were worth the cost and provided a positive return on investment. The benefits gained were central to the development of a sound business case, which includes evidence that (1) the customer’s needs are valid and can best be met with the chosen concept and (2) the chosen concept can be developed and produced with existing resources, such as time, money, and available technology. We have previously found that establishing a sound business case is essential to achieving better program outcomes. See appendix V for an analysis of acquisition outcomes to date for the programs we examined. Prototyping provided programs with information on technology maturity, the feasibility of the design concepts, potential costs, and the achievability of planned performance requirements, which helped inject realism into their business cases. Appendix VI includes examples of these benefits, several of which we highlight below:

Prototyping demonstrated key technologies or proposed design solutions. For example, Space Fence officials stated that prototyping helped them determine that a riskier, cutting edge design involving a 15 percent smaller radar was feasible. Without prototyping, the program would not have had sufficient information to be confident in the riskier option, nor would the contractor have proposed it without the opportunity to demonstrate that it worked.

Prototyping informed programs’ understanding of prices and helped validate business case cost estimates. During the prototyping process, contractors select vendors, develop supplier relationships, purchase materials, and build a version of the system or parts of the system, all of which provide information on potential costs. Air and Missile Defense Radar program officials stated that prototyping increased the cost information available to the program and led to cost reductions. They explained that competitive prototyping incentivized the contractors to determine their cost drivers in order to be more competitive in the next phase.

Prototyping helped programs better understand requirements and—in most cases—helped them make performance tradeoffs to meet cost targets. In the case of Joint Light Tactical Vehicle, prototyping helped program officials determine that two versions of the vehicle were too heavy to be transported as planned. The program then lowered its transportability requirements by eliminating the need to airlift the vehicles in extreme conditions. This change allowed the vehicles to be heavier and resulted in $35,000 in savings per vehicle, according to the Army.

Prototyping provided a variety of other benefits as well. For example, prototyping helped programs improve system performance. Small Diameter Bomb II officials said that data collected during its prototype testing set the stage for improvements in its target classification software. It also helped identify potential reliability issues early. The Next Generation Jammer Increment 1 and Ship to Shore Connector Amphibious Craft programs changed certain subsystem materials based on information learned about wear during prototype testing. Further, for 11 programs, the prototypes served as test assets during system development or were used to continue development efforts.

Competitive prototyping approaches generated additional benefits, such as enabling more favorable business terms. According to the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, the Air and Missile Defense Radar program’s use of competition resulted in over $100 million in savings and will reduce operation and support costs over the life of the program. Air and Missile Defense program officials explained that having three competitors was helpful because it reduced the likelihood that contractors would team up in the next phase, leaving the government with only one proposal. In other cases, competition improved the quality of the systems being offered to DOD. For example, Space Fence program officials told us that competition spurred contractors to introduce cutting edge designs and continue refining those designs in order to remain competitive in the next phase of the program. Finally, officials from several programs stated that contractors supplemented prototyping efforts using their own funds and believed contractors did this in order to make their subsequent offers more competitive.

A common perception is that competitive prototyping might cost more up front because multiple contracts are awarded, but our analysis showed that programs using multiple contractors did not have higher relative costs for technology development. Programs that used a competitive approach planned to spend a similar percentage or less of their total expected RDT&E funding prior to the start of system development as those programs that did not competitively prototype. Using competitive prototyping approaches did create additional administrative burdens in the short term because program offices had to manage multiple contractors and maintain firewalls to ensure fair competitions, but officials from across the programs we examined stated it was worth the investment.

Several Practices Helped Programs Get More Out of Their Prototyping Efforts

Officials from all 17 programs that prototyped told us their prototyping efforts were useful; however, we found some programs got more out of their prototyping efforts than others. Based on those programs’ experiences and lessons other programs shared, we identified the following five practices that helped programs maximize the utility of their prototyping efforts. 1. Identify risks early and target prototyping efforts to address them. Officials from seven programs described early activities that helped identify risks and shape their prototyping efforts. The Air and Missile Defense Radar program planned a big leap in technology over the existing radar, which was fielded over 30 years ago. The program conducted early technology maturity assessments that identified the large aperture digital beamforming and calibration critical technology as the program’s key risk area. The program focused its prototyping efforts on maturing this and other critical technologies and demonstrating them in a relevant environment. It is on track to complete system development on time and within its estimated research and development cost. 2. Structure prototyping efforts to be completed in time to inform key decisions—particularly source selection. We found that many of the programs prototyped before selecting their system development contractor and about three-quarters of the programs also held preliminary design review before entering system development. Prototyping helped inform these decisions and related assessments. Common Infrared Countermeasure program officials told us they required contractors’ proposals for the program’s system development contract to include solutions to address reliability failures identified during prototyping. 3. Specify the level of fidelity needed to provide the necessary information about which risks to address. Officials from 15 programs told us they prototyped designs similar to the actual design of the system they will develop. This is known as high fidelity prototyping. Officials from a few programs noted that doing so enabled them to understand what the system entailed and, if needed, make trade-offs accordingly. In contrast, Integrated Air and Missile Defense program officials stated that the results from prototyping were not as helpful to make programmatic decisions because they were limited to demonstrating the feasibility of a certain system concept using a generic government design. 4. Ensure the appropriate level of insight into the design and cost information. Officials from each program said they had sufficient visibility into the prototyping efforts, but some officials described having more insight into the efforts and their results than others. The level of insight can be affected by factors, such as the type of information a program requires a contractor to provide under a prototyping contract. Space Fence officials noted that in addition to conducting a live prototype demonstration, they had access to reports and data from contractors’ efforts. The officials said that the data obtained through prototyping were helpful for pricing. The program used the data to mature its cost estimate and was able to use a firm fixed price contract for system development. Space Fence finalized its design just over a year after entering system development, has had no cost growth to date, and anticipates that the contractor will deliver the system earlier than initially planned. 5. Keep plans flexible to adapt to information learned during the effort. Officials from four programs told us they used multi-phased prototyping approaches and a few described adding or removing contractors. Officials from at least three programs told us they changed strategies or modified their approach based on information learned or in response to a tighter budget environment. For example, after the Common Infrared Countermeasure program determined its technologies were less mature than expected, it added two prototyping phases to continue maturing the technologies, testing the system with related systems, and demonstrating manufacturing processes. The program entered system development with its technologies nearing maturity and completed its system design in just over a year. It has had about 6 percent cost growth as of October 2016 and is estimating completing system development on time.

DOD Has Established New Initiatives to Increase Prototyping and Innovation, but Does Not Have a Strategy to Guide Its Investments

To help ensure U.S. military capabilities outpace those of potential adversaries, DOD has expanded prototyping efforts focused on innovation, including disruptive innovation, and has started several new initiatives outside of major acquisition programs to address gaps in its innovation portfolio. However, these initiatives face barriers, such as limited funding and competing priorities. Literature on private sector innovation, including the use of prototyping, identifies key enablers for these types of efforts, such as developing a strategy for innovation, identifying relative levels of investments that align with innovation goals, and protecting funding for technology investments that have higher risk, but perhaps more reward across the enterprise. DOD has taken steps that are consistent with a few of these enablers, but lacks others, such as an innovation strategy that could also address the role of prototyping.

DOD Has Started New Initiatives to Expand Prototyping and Experimentation

Since 2012, DOD and the military departments have established seven new offices to increase prototyping and experimentation and further innovation. Prototyping can be a way to “test the waters” with new and potentially disruptive concepts and technologies. Experimentation puts prototypes into the warfighter’s hands, so that the capabilities they provide can be assessed in an operational context. Most of the efforts we examined aim to mature technologies for future capabilities, but without the rigidity, commitment, and additional cost associated with starting new acquisition programs. Other than the experimentation initiatives, all of them involve demonstrations that seek to improve DOD’s or the military services’ understanding of the viability, maturity, and potential utility of the technologies, subsystems, or systems being prototyped. The demonstrations also inform decisions regarding potential next steps, such as transition to a military service in the case of mature capabilities that are ready to be put into use or to an acquisition program for those that need further development. Table 2 provides an overview of these initiatives.

The new initiatives help address gaps in DOD’s science and technology and weapon system investments and expand efforts to identify and mature potentially innovative and disruptive technologies. For example, the Army Technology Maturation Initiative uses budget activity 6.4 funding, which is typically associated with acquisition programs, to conduct higher-fidelity prototyping and further mature technology outside of those programs. Other initiatives focus on modifying already fielded equipment and technologies to use them in new ways, combining prototyping and rapid acquisition practices to field capabilities faster, and encouraging experimentation to explore how capabilities being prototyped could be employed in an operational setting.

The two most mature prototyping-related initiatives have made some progress. For example, the Strategic Capabilities Office reported that it is currently in the process of transitioning six of its technology demonstration and prototyping projects to the military services. The Army’s Technology Maturation Initiative has also demonstrated some progress—it has six projects that have either transitioned to a program of record or are in the process of transitioning. However, with the exception of the Navy’s Technology Innovation Games, the other four initiatives are still in the early planning phases. Some of them are still in the process of developing charters, determining project selection processes, and documenting priorities. Most of the new rapid capabilities offices were developed so recently that they were also not in the fiscal year 2017 budget request, but the Army plans to temporarily support its office with funding from existing Army accounts.

New Prototyping Initiatives Face a Variety of Barriers

DOD’s new prototyping initiatives face several barriers that can make it challenging to obtain funding to start projects, manage the initiatives to achieve innovation, and transition the prototypes to acquisition programs. Literature on private sector innovation, including the use of prototyping, suggests that private sector firms face some of these same barriers. Key barriers we identified include:

Funding structure: Several studies have suggested that maturing technologies outside and independent of acquisition programs to higher technology readiness levels can promote innovation and facilitate technology transition. However, DOD’s funding structure and how it is commonly interpreted may limit the amount of higher fidelity prototyping conducted outside of acquisition programs. DOD’s science and technology community manages and invests research and development funding in budget activities 6.1-6.3, but does not typically use budget activity 6.4 funds. According to DOD regulation, projects funded with budget activity 6.3 are to mature technologies to technology readiness levels 4, 5, or 6, while those funded with budget activity 6.4 are to result in the achievement of technology readiness level 6 or 7 (see app. VIII). Due in part to this budget activity structure, the science and technology community typically sees its role as maturing technologies to no higher than technology readiness level 6. As a result, until DOD and the military services’ recent prototyping initiatives, there were not many offices focused on further maturing technologies outside of acquisition programs.

Risk averse culture: Although it is appropriate to minimize risks in acquisition programs, some officials stated that excessive risk aversion outside of acquisition programs can stifle innovation. According to the Defense Science Board, over time, DOD has become increasingly risk averse and experimentation has moved towards scripted demonstrations, testing, and training. Pressure to justify budgets, demonstrate utility to the warfighter, and advance careers all contribute to this risk aversion. Many prototyping and innovation initiatives we reviewed emphasized high transition rates of between 80 and 100 percent. Generally speaking, transition means that a technology has been sufficiently matured and is ready to transition to a user such as a weapon acquisition program or the warfighter in the field. On one hand, a high transition rate can be an indicator that an initiative is generating a good return on investment and is developing capabilities that meet customers’ needs. But, for prototyping initiatives with the stated purpose of encouraging innovation, particularly disruptive innovation, making high transition rates a goal could be counterproductive and lead to a lower tolerance for risk or failure. For private sector projects focused on innovation, companies can aim for transition rates as low as 20 to 50 percent.

Competing priorities: Officials identified competition with projects the military services have previously funded and prioritized as a barrier to innovation efforts—both when requesting funding to prototype and later when trying to transition. Innovation literature suggests that companies frequently face this same problem. Resources are often not available for bolder projects because funds are consumed by pre- existing projects; furthermore, companies are more likely to devote resources to sustaining innovation, which gradually improves on existing products, rather than riskier disruptive innovation. The Secretary of Defense testified to Congress in September 2016 that he has seen the constant temptation over the years to starve new and future-oriented defense investments in favor of more established and therefore well-entrenched programs. He expressed concern that funding was being taken away from initiatives such as the Strategic Capabilities Office, to instead pay for existing acquisitions. In fiscal year 2016, 6.4-funded initiatives that focus on prototyping and innovation represented less than 4 percent of budget activity 6.4 funds.

Long budget timelines: Long budget timelines make it difficult to start prototyping projects that address emerging threats in a timely manner. For example, as is illustrated in figure 8, a project conceived in February 2017 might not be authorized and appropriated funding until October 2018. Projects that are expected to take 3 to 5 years to complete in effect require 5 to7 years from conception to completion. If there is a continuing resolution, it could take longer. These long timelines make it difficult to achieve the adaptability and faster capability development and fielding times that DOD seeks to keep pace with rapidly evolving threats. DOD can take special steps to provide funding in other ways, such as through reprogramming; but, in general, long budget timelines not only make it difficult to succeed fast, they also make it difficult for initiatives to “fail fast” and for DOD to move on to potentially more promising projects.

Synchronization with acquisition programs: Prototyping efforts may not be complete at the most opportune time for acquisition programs, as is reflected in figure 9. If the effort is completed too early, technology can rapidly become obsolete before a relevant acquisition program is begun. If a prototyping effort is completed after an acquisition program has begun, the program may not be willing to adopt it. Defense Advanced Research Projects Agency officials noted that partners must budget 2 years in advance to further mature or transition technologies, which exacerbates this problem. Congress included a provision in the Fiscal Year 2017 NDAA that, depending on how it is implemented by DOD, could help make it easier to transition new technologies and components to programs that have already begun system development. The NDAA requires that certain MDAPs be designed and developed, to the maximum extent practicable, with a modular open systems approach.This type of approach, which includes a modular design and standard interfaces, enables system components to be more readily replaced.

DOD Actions Partially Align with Practices for Fostering Innovation in the Private Sector

The literature we reviewed on private sector innovation highlights several key practices related to how to organize and manage innovation units, fund projects, and address potential culture barriers. These partially align with recent DOD actions. Some of these practices apply directly to prototyping, while others address innovation more broadly. The key practices or enablers we identified are listed in table 3 below.

We compared DOD’s new prototyping initiatives with these enablers to determine whether DOD is well-positioned to generate the type of innovation, including disruptive innovation, that it is seeking.

Developing an Innovation Strategy

DOD has issued multiple memorandums related to prototyping and innovation, as reflected in table 4, but these documents fall short of a strategy. Specifically, with regard to prototyping and innovation, none of the documents we reviewed communicate strategic goals and priorities or delineate roles and responsibilities among DOD and the military services’ initiatives, which are elements of the innovation strategies described in the literature as well as standards for internal control. Congress included a provision in the Fiscal Year 2017 NDAA that provides some strategic direction for certain prototyping projects. It calls for the military services to establish or identify oversight boards that will develop triennial strategic plans to prioritize capability and weapon system component portfolio areas for prototyping projects, among other things. However, it is not yet clear whether there will be a mechanism to tie these efforts into a department-wide strategy.

DOD’s lack of an innovation strategy means it has to rely on other mechanisms to coordinate and provide strategic direction for its prototyping initiatives, although those mechanisms do not cover some of DOD’s prototyping and innovation activities and do not establish department-wide priorities. For example, Communities of Interest (COI), which are organized by the Assistant Secretary of Defense for Research and Engineering (ASD(R&E)), help plan, coordinate, and share knowledge on science and technology activities for budget activities 6.2 and 6.3, but there are no analogous mechanisms for 6.4-funded activities, including those related to prototyping and innovation. The 17 COI working groups are generally organized by portfolio—for example, advanced electronics—and include representatives from across DOD. They periodically develop roadmaps for their portfolios. ASD(R&E) officials explained that the road mapping process is not directly tied to budget decisions and does not establish department-wide science and technology priorities. However, it does help identify investment gaps, opportunities for collaboration, and areas of potential overlap or overinvestment. DOD’s Long Range Research and Development Planning Program suggested using COIs to prioritize the technology investments it identified, which would expand their focus beyond science and technology. We have previously found that one way to better manage potentially fragmented activities is to improve collaboration and coordination. Without an approach that covers relevant 6.4-funded activities, DOD may be missing out on opportunities to take a more strategic approach to prototyping and innovation across the department, including sharing information and identifying areas of potential under- or overinvestment related to prototyping and experimentation.

DOD is undergoing organizational changes that could provide more focused leadership, strategy development, and coordination for prototyping and innovation-related activities:

The NDAA for Fiscal Year 2017 establishes the position of Under Secretary of Defense for Research and Engineering. According to the conferees, the creation of this position was part of organizational changes to DOD that seek to, among other things, advance technology and innovation. The duties of the Under Secretary of Defense for Research and Engineering include advancing technology and innovation and establishing policies on all defense research and engineering.

DOD’s Defense Innovation Board has recommended that DOD establish the position of Chief Innovation Officer, to coordinate, oversee, and synchronize innovation activities across the department.

DOD has established the position of Deputy Director for Prototyping and Experimentation to oversee program execution, provide technical and programmatic advice, and work with DOD entities to identify shortfalls and potential technologies and projects to address them. However, the position only has authority over prototyping and experimentation efforts within the office of the Deputy Assistant Secretary of Defense for Emerging Capability and Prototyping. To influence prototyping activities outside of that office including military service led initiatives, the Deputy Director stated that he has to leverage his personal relationships and experience.

With DOD’s increased level of effort and investment in prototyping and innovation comes the potential for inefficiencies if efforts are not coordinated and aligned with an overarching strategy. Although these offices are generally attempting to meet different needs and are using a variety of approaches to achieve innovation, without an articulated strategy, there is a potential for overlap if their goals and approaches evolve over time.

Ensuring Adequate Funding to Address Innovation Goals

With the exception of the Strategic Capabilities Office, DOD and the military services have not allocated large amounts of funding to their new prototyping and innovation initiatives in their budget requests and they will have to compete with other priorities to receive funding in the future. DOD’s fiscal year 2017 budget request included less than $100 million for each of the six other initiatives we examined. One approach identified in the academic literature that helps ensure innovative projects receive sufficient funding, in the face of competing priorities and a risk averse culture, is called a “strategic buckets” approach. Under this approach, management makes a strategic decision to allocate set “buckets” of resources for different types of projects, including breakthroughs, and then takes steps to ensure adequate funding for innovation efforts. The distribution of resources among different buckets is dictated by the organization’s strategy. This approach is consistent with portfolio management best practices, which call for organizations to use an integrated approach to prioritize needs and allocate resources in accordance with strategic goals. Figure 10 includes a notional depiction of how this approach could be adapted to the basic tenets of DOD’s prototyping and innovation efforts.

To implement a strategic buckets approach for innovation, an organization needs to develop innovation goals, reflect those goals in its innovation strategy, inventory current projects and funding allocations, and then adjust funding levels, if needed, to make sure they align with its goals and strategy. Decisions to change relative levels of investment in different buckets may be made over time in response to changing world events. For example, DOD’s concern about losing its eroding warfighting edge in certain areas could cause it to place a higher priority on prototyping systems that could lead to disruptive innovations.

The Navy’s approach to managing its science and technology investments, including its prototyping and innovation initiatives, has elements of a “strategic buckets” approach. It maps out roughly the percentage of funding that it plans to request for different parts of its science and technology portfolio, including some of its prototyping and innovation initiatives, as is reflected in figure 11 below. The Navy uses this information to help develop its science and technology budgets. The Navy has not extended the concept to other research and development budget activities, such as 6.4, which are largely driven by decisions on individual acquisition programs.

DOD has also employed aspects of this approach to set and enforce minimum funding levels for its science and technology investments, but it lacks certain prerequisites needed to apply it more broadly to prototyping. ASD(R&E) officials explained that, in recent years, the Office of the Secretary of Defense has communicated an investment floor for budget activities 6.1 through 6.3 in the Defense Planning Guidance, which provides strategic direction for DOD budget formulation. ASD(R&E) enforces these levels by reviewing military service budget requests and directing funding increases or other shifts to ensure the floor is met across the department. Although ASD(R&E) has responsibility for overseeing research and development activities under budget activities 6.1 through 6.4, ASD(R&E) officials stated that they do not review budget requests for budget activity 6.4 because they are primarily allocated to acquisition programs. By not exercising its authority over the full range of budget activity 6.1 through 6.4 funding, ASD(R&E) is missing an opportunity to assess prototyping activities collectively from an enterprise level to determine if and how this funding might best be used to support DOD’s prototyping and innovation initiatives and its strategic goals.

DOD also lacks the innovation strategy and baseline understanding of its prototyping projects and their associated funding, to identify areas of potential over- and under-investment as well as appropriate investment targets. Better Buying Power 3.0 called for (1) the USD(AT&L) and Vice Chairman of the Joint Chiefs of Staff, who oversees the weapon system requirements process, to conduct annual reviews of each service’s budget activity 6.3- and 6.4-funded prototyping and experimentation activities, and (2) the ASD(R&E) to develop, maintain, and publish a database of government and industry experimentation capabilities and events, and make annual recommendations to the military services and USD(AT&L) for additional prototyping. USD(AT&L) and military service officials stated that the reviews have not been held due to difficulties with scheduling. In addition, although ASD(R&E) officials took steps to develop the database, it was not completed and efforts to make the information available through a different database were unsuccessful. Both the reviews and the database could have provided useful information about prototyping and experimentation activities and opportunities across the department.

The NDAA for Fiscal Year 2017 included a provision that could provide more information about funding for select prototyping initiatives outside of acquisition programs. This provision could better position DOD to set and track prototyping investment targets. In budget requests after fiscal year 2017, for budget activity 6.4, the provision requires DOD to state the amounts requested for prototyping and experimentation of weapon system components and technologies separate from acquisition programs of record. Those requests are to reflect priority areas for prototyping. Furthermore, the legislation calls for military services to establish or identify prototyping oversight boards to, among other things, annually recommend funding levels for prototype projects across capability or weapon system component portfolios although no analogous recommendations are required for efforts outside of the military services.

Balancing Demand Pull and Supply Push

Most of DOD and the military services’ prototyping and innovation initiatives use more of a “demand pull’ approach to selecting projects, which could limit their likelihood of generating disruptive capabilities (see app. IX for a list of these initiatives). This was the case for both older and newer initiatives. Demand pull initiatives focus on prototyping technologies or systems to address validated requirements, which means they have built in constituencies ready to support them. On the other hand, “supply push” initiatives take on projects without a stated customer need and do not align with existing organizational structures. This can make it difficult to gain support for supply push type projects, particularly when it is time to transition them into programs. For example, the Navy had to establish an Unmanned Maritime Systems program office when unmanned underwater capabilities languished because there were no “customers” given existing organizational structures. DARPA, with an annual budget of over $1 billion, is DOD’s largest example of an organization that primarily uses a supply push model.

An overreliance on demand pull can lead to incremental improvements in capabilities without ever achieving a more disruptive breakthrough. The Deputy Assistant Secretary of Defense for Research stated that DOD’s requirements process is a model for slightly improving how DOD conducts operations now rather than thinking outside the box of the art of the possible. DOD does not have a similar process designed to foster more innovative solutions. Without an innovation strategy that sets goals and aligns funding for demand pull and supply push projects accordingly, DOD’s prototyping and innovation initiatives might not produce the types of disruptive capabilities and breakthroughs the department is seeking.

Learning Quickly Through Prototyping and Experimentation

Most DOD prototyping and innovation initiatives we reviewed took steps so that they could learn quickly through their projects. Almost all of them have expected project turnarounds of 3 to 5 years or less. Initiatives such as the Army’s Technology Maturation Initiative and the Strategic Capabilities Office also regularly reviewed projects to determine whether they were still needed or feasible based on initial efforts and, if they were not, terminated projects accordingly. Two longstanding initiatives employed approaches to speed up the funding process. DOD’s Joint Concept Technology Demonstration program notifies Congress about new projects via letter prior to starting them rather than waiting to request approval in each budget request. Officials from the Future Naval Capabilities Program stated that they use funding left over from projects completed in a given year for other projects, as long as the amount falls below a certain threshold.

Fostering Greater Risk Tolerance

DOD and military department officials acknowledge that there is a risk averse culture across the department, even with respect to prototyping and innovation initiatives. However, neither the officials we spoke with nor recent memorandums have described ways DOD is changing its metrics or incentives to encourage more risk tolerance within these initiatives, which is one of the enablers highlighted in the literature on private sector innovation. Other enablers, such as developing an innovation strategy and ensuring adequate funding to support it, could also help foster a more risk tolerant environment. DOD’s Defense Innovation Board is also in the process of identifying ways to develop a culture of innovation in DOD in which new ideas can be tested and fail without fear of ending or derailing the career of a science and technology manager, acquisition professional, or military officer.

Conclusions

Prototyping is a tool that can help DOD address a variety of both long- standing and recent weapon system acquisition and modernization challenges. When used effectively, it can help reduce risks and improve the likelihood that a weapon system acquisition program will be completed on time and on budget. Furthermore, it helps keep DOD’s technology pipeline stocked with new and innovative technologies that might provide the next great leap ahead in military capabilities and may even deter adversaries by demonstrating advanced capabilities.

In the period since the Weapon System Acquisition Reform Act (WSARA) of 2009 was implemented, DOD acquisition programs have used prototyping to reduce risk and inject realism into their business cases, which has helped place them on sound footing for future success. The results were notable on the programs we reviewed—lower technical risk, better understanding of requirements, and more information on potential costs, among other benefits. With the recent repeal of WSARA’s competitive prototyping requirements, there is a risk that programs will choose not to prototype. In doing so, those programs would forfeit the significant benefits that early prototyping can offer.

DOD’s efforts to expand prototyping and experimentation to help achieve the innovation and disruption needed to maintain its technological and military advantage are in a more nascent stage. However, challenges, such as limited funding, a risk averse culture, and competing priorities, are already apparent and may make it difficult for the efforts to gain momentum. Pending organizational changes, including the creation of the positions of Under Secretary of Defense for Research and Engineering and Chief Innovation Officer, provide an opportunity for DOD to elevate and take a more strategic approach to the mission of advancing technology and innovation.

The literature on private sector practices provides a roadmap for how this new DOD leadership can enable innovation, including through the use of prototyping. But DOD will need to fully embrace certain key enablers that are not currently present in the department, including a strategy that addresses its disparate prototyping and innovation efforts and strategic goals that can be used to guide resource decisions. It will also need to work across funding structures for science and technology and more advanced development work that usually separate certain types of prototyping efforts. The recent increased level of effort and investment in prototyping and innovation comes with the potential for inefficiencies if efforts are not strategic and coordinated. Other high-risk investments in categories such as disruptive technologies may need to be protected from a risk averse culture, as well.

DOD has taken several steps to adopt aspects of private sector innovation practices and has developed mechanisms to coordinate and review its science and technology investments, but without a more strategic, inclusive, and deliberate approach overall, its new prototyping and experimentation initiatives might not generate the levels and types of innovation the department is seeking.

Recommendations for Executive Action

To help ensure DOD takes a strategic approach for its prototyping and innovation initiatives and overcomes funding and cultural barriers, we recommend that the Secretary of Defense direct the Assistant Secretary of Defense for Research and Engineering to take the following four actions:

Develop a high-level DOD-wide strategy, in collaboration with the military services and other appropriate DOD components, to communicate strategic goals and priorities and delineate roles and responsibilities among DOD’s prototyping and innovation initiatives.

Take steps, such as adopting a “strategic buckets” approach, to help ensure adequate investments in innovation that align with DOD-wide strategy.

Review budget activity 6.4 funding requests to help maintain a level of investment for budget activity 6.4-funded prototyping and innovation efforts that is consistent with DOD-wide strategy.

Expand the Community of Interest working groups to include budget activity 6.4-funded prototyping and innovation initiatives in their science and technology planning and coordination processes or employ a similar coordination mechanism for budget activity 6.4- funded prototyping and innovation initiatives.

Agency Comments

We provided a draft of this report to DOD for comment. In its comments, reproduced in appendix X, DOD concurred with our four recommendations. DOD also provided technical comments, which we incorporated as appropriate.

We are sending copies of the report to the appropriate congressional committees; the Secretary of Defense; the Secretaries of the Army, Navy, and Air Force; and the Assistant Secretary of Defense for Research and Engineering. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at 202-512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix XI.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to assess (1) how the Department of Defense (DOD) has used prototyping prior to system development on major defense acquisition programs, and (2) what steps DOD has taken to increase innovation through prototyping activities conducted outside of major defense acquisition programs.

DOD’s Use of Prototyping on Major Programs Prior to System Development

To address our first objective, we examined 22 major defense acquisition programs that had a Milestone B decision, which approves entry into system development, between December 2009 and February 2016, or anticipated receiving Milestone B approval by February 2016 when we began our selection process. We selected December 2009 as the starting date because it was when DOD implemented the Weapon System Acquisition Reform Act of 2009 and its associated prototyping provisions. These programs and the dates they entered system development are included in table 5.

To determine what prototyping approaches the 22 programs used, if any, and to identify costs, benefits, challenges, and lessons learned from their prototyping efforts, we reviewed program documents, such as technology development strategies, acquisition strategies, prototyping waivers, acquisition decision memorandums, independent cost estimates, and budget requests. We also conducted semi-structured interviews with officials from each of the 22 programs and reviewed prior GAO reports. We also examined prior GAO work related to acquisition program outcomes and the technology development phase.

To examine the proportion of research, development, test, and evaluation (RDT&E) funds planned for development prior to each program’s entry into system development, we reviewed program funding stream data obtained from December 2015 Selected Acquisition Reports. We calculated the RDT&E funds planned as of the month prior to the program’s Milestone B approval date, which we obtained from program documents. We then divided the prorated amount by the program’s current RDT&E cost estimates to obtain the proportion of RDT&E funds planned for use prior to system development. We excluded six programs that did not have complete data available. See appendix VII.

To examine how these 22 programs fared in terms of cost and schedule performance, technology maturity, and design stability, we compared prototyping programs’ data with non-prototyping programs’ data. Specifically, for programs’ cost outcomes, we examined the difference between programs’ first full and current RDT&E cost estimates. Programs’ first full estimates are typically developed upon program entry into system development at Milestone B. For programs’ schedule outcomes, we examined the growth between when the program entered and completed system development using programs’ first full and current estimates. Completion of system development usually occurs when Milestone C is achieved. For first full estimate data, we leveraged data collected as part of our annual assessment of DOD weapon systems. This included cost, quantity, and schedule data from the Defense Acquisition Management Information Retrieval Purview system, referred to as DAMIR. The team entered this data into a database and verified that the data were entered correctly. We converted all cost information to fiscal year 2017 dollars using conversion factors from the DOD Comptroller’s National Defense Budget Estimates for Fiscal Year 2017 (tables 5-9). To assess the reliability of the data the annual assessment team talked with a DOD official responsible for DAMIR’s data quality control procedures and reviewed relevant documentation. They also confirmed selected data reliability with program offices. For current estimate data, we obtained RDT&E, total acquisition cost, quantity, and schedule estimates from the August 2016 Defense Acquisition Executive Summary reports. We determined that data were sufficiently reliable for the purposes of this report.

The selected programs in our review entered system development in December 2009 or after and are generally newer. To address concerns about examining outcomes given the relative newness of many of the programs, we excluded the following six programs from our cost and schedule analyses because they are too recent to have current estimates separate from the program’s baseline or do not have approved first full estimates: Amphibious Combat Vehicle, B-2 Defense Modernization System Modification, Common Infrared Countermeasure, Next Generation Jammer Increment 1, Offensive Anti-Surface Warfare Increment 1, and Three-Dimensional Expeditionary Long-Range Radar. We excluded two additional programs—Enhanced Polar System and Space Fence Increment 1—from our schedule analysis because these programs will not hold a Milestone C.

To examine the technology maturity and design stability of programs, we leveraged survey response data provided in support of our annual assessments of selected weapon programs. These assessments rely on data collected from program offices related to the technology readiness levels of their critical technologies and their percentage of completed design drawings.

Our best-practices work has shown that a technology readiness level (TRL) 7—demonstration of a technology in an operational environment—is the level of technology maturity that constitutes a low risk for starting a product development program. For shipbuilding programs, we have recommended that this level of maturity be achieved by the contract award for detailed design. In our assessment, the technologies that have reached TRL 7, a prototype demonstrated in an operational environment, are referred to as mature or fully mature. Those technologies that have reached TRL 6, a prototype demonstrated in a relevant environment, are referred to as approaching or nearing maturity. Satellite technologies that have achieved TRL 6 are assessed as fully mature due to the difficulty of demonstrating maturity in an operational environment—space. No programs needed to be excluded from the technology maturity analysis. See appendix VIII for TRL definitions.

Our best practices work shows that completion of at least 90 percent of engineering drawings at critical design review provides tangible evidence that the product’s design is stable. Completed design drawings were defined as the number of drawings released or deemed releasable to manufacturing that can be considered the “build to” drawings. For shipbuilding programs, they asked program officials to provide the percentage of the three-dimensional product model that had been completed by the start of lead ship fabrication, and as of our annual assessment. Five programs were excluded from this analysis. The Joint Light Tactical Vehicle program does not track the percent of releasable drawings and the Combat Rescue Helicopter, Next Generation Jammer Increment 1, Global Positioning System Next Generation Operational Control System, and Three-Dimensional Expeditionary Long-Range Radar programs have not yet held their critical design reviews.

Although the technology maturity and design stability information provided at key knowledge points provide excellent indicators of potential risks, by themselves they do not cover all elements of risk that a program encounters during development, such as funding instability. See appendix V for a summary of program outcomes.

DOD’s Use of Prototyping to Increase Innovation Outside of Major Programs

To address our second objective, we reviewed fiscal year 2017 budget documentation and interviewed DOD and military service officials responsible for research and development to identify initiatives that DOD started in the past five years with the stated purpose of promoting innovation through prototyping and experimentation. We focused on broad-based initiatives, rather than ones focused on a specific technology area. We also examined key preexisting initiatives for contrast. We only included initiatives funded with budget activities 6.3 and 6.4, advanced technology development and advanced component development and prototypes respectively, because those budget activities fund the development and testing of new concepts and capabilities using higher fidelity prototypes that have the potential for short- or medium-term application. We did not meet with rapid prototyping offices established for direct support to the conflicts in Afghanistan and Iraq because they were designed for a temporary contingency.

To identify the initiatives’ goals, focus areas, scope, approaches, funding characteristics, strategies, coordination mechanisms, and barriers, if any, they face, we reviewed documentation from the initiatives, such as budget requests, charters, and briefings. We also interviewed program officials and obtained written responses to questions. To determine what direction and strategy DOD has provided for the initiatives, we analyzed DOD memorandums on the following subjects: Long Range Research and Development Plan, Defense Innovation Initiative, Wargaming and Innovation, and Better Buying Power 3.0 as well as Navy memorandums on: Task Force Innovation, Wargaming, and Innovation Funding within the Naval Research and Development Establishment. We also reviewed DOD’s Long Range Research and Development Planning Program briefing. We examined DOD’s process for coordinating science and technology investments, called “Reliance 21,” to determine the extent to which it addressed prototyping for innovation and whether it has the potential to do so.

We also conducted a review of literature on innovation in the commercial sector, including the use of prototyping, to identify enablers that could be applicable in DOD and to identify barriers commercial sector organizations face. The literature was primarily from academic sources, but included some literature from the private sector. Specifically, we began with recognized experts in the field of innovation. We then used a snowball methodology to identify other key authors on innovation through databases such as ProQuest and WorldCat. We also asked DOD officials for recommendations regarding relevant authors and articles. Our literature search covered articles published from 1996 onward, with a majority written between 2005 and 2016. We identified 19 sources that were specific to our work. They primarily relied on interviews, surveys, and case studies. Through the literature search, we identified a number of general themes about spurring innovation across articles and interviews. We then developed a list of key enablers from these themes that could potentially apply to DOD prototyping for innovation activities. We also noted when these sources identified barriers to innovation that aligned with the barriers we identified as existing in DOD. To determine whether DOD’s practices are consistent with these enablers, we compared them with memorandums related to prototyping for innovation, the Navy’s and DOD’s approach to managing funding for innovative research and development as reflected in the Navy Science and Technology Strategy and in DOD briefings, demand pull and supply push emphases of prototyping for innovation initiatives, and initiatives’ approaches to learning quickly as reflected in their documentation. When applicable, we also compared DOD’s approach to its prototyping and innovation initiatives with additional sources including the Standards for Internal Control in the Federal Government (for strategy and goals); GAO work on fragmentation, duplication, and overlap (for coordination); and portfolio management best practices (for funding and prioritization).

To inform all assessments for this objective, we interviewed officials from the Office of the Under Secretary of Defense (Comptroller) and officials from each military department’s comptroller’s office; Office of the Assistant Secretary of Defense for Research and Engineering; Office of the Deputy Assistant Secretary of Defense for Emerging Capabilities and Prototyping; Office of the Assistant Secretary of the Air Force for Acquisition; Office of the Assistant Secretary of the Army (Acquisitions, Logistics, and Technology); and Office of the Deputy Assistant Secretary of the Navy for Research, Development, Test, and Evaluation.

We conducted this performance audit from September 2015 to June 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Department of Defense Research, Development, Test, and Evaluation Budget Activities

Appendix III: Budget Activity 6.3 and 6.4 in Fiscal Year 2016

The technology transition-related budget activities 6.3 and 6.4—which received $5.8 billion and $14.6 billion respectively in fiscal year 2016— support numerous entities across the Department of Defense (DOD). In fiscal year 2016, the Defense Advanced Research Projects Agency, the Army, and the Office of the Secretary of Defense received the highest levels of budget activity 6.3 funds—with each component receiving about 20 percent of total funding—and the remaining 38 percent was divided among several organizations (see figure 12). That same year, the Missile Defense Agency received the largest percentage of budget activity 6.4 funds, with 42 percent, while the Navy received 35 percent and the remaining 23 percent was shared among several organizations (see figure 13).

Appendix IV: Overview of Selected Programs’ Prototyping Efforts and Reasons for Not Prototyping

Appendix V: Programs’ Cost and Schedule Performance and Knowledge Attainment

Overall, prototyping appears to have reduced risks on more complex programs, as evidenced by the fact that most programs that prototyped had similar cost and schedule outcomes and attained key knowledge at similar rates as non-prototyping programs that were generally considered less complex and risky. See appendix I for the programs included in each of the following analyses. Table 9 below includes individual program outcomes to date.

Our analysis of cost and schedule performance to date showed that for the programs we reviewed, most prototyping and non-prototyping programs’ outcomes were similar (see figures 14 and 15). Nine of 11 prototyping programs had similar levels of research, development, test and evaluation cost growth as the five non-prototyping programs. Officials from the two prototyping programs with higher levels of cost growth— Global Positioning System Next Generation Operational Control System (GPS OCX) and Integrated Air and Missile Defense (IAMD)—told us that they could have learned more about their prototyping efforts if they produced higher fidelity prototypes or better addressed key risk areas. Six of nine prototyping programs with comparable data experienced similar schedule growth between the start and completion of system development as the five non-prototyping programs. The IAMD and GPS OCX programs were also among the prototyping programs with the largest schedule growth.

We also found similar outcomes between prototyping and non-prototyping programs with respect to the technology and design knowledge they had at key points in the acquisition process (see tables 10 and 11). Most programs that prototyped during technology development matured or nearly matured their critical technologies before entering system development and released 90 percent of the system’s drawings at critical design review (see appendix VIII for definitions of technology readiness levels). Demonstrating high levels of technology and design knowledge by critical points in the acquisition process is a GAO best practice that helps to reduce program risk.

Appendix VI: Examples of Prototyping Benefits that Contributed to Sound Business Cases

Appendix VII: Relative Cost of Prototyping Efforts

To get a general sense of the relative cost of programs’ prototyping efforts, we compared programs’ planned research, development, test and evaluation (RDT&E) funding during technology development to their overall RDT&E cost estimate. Table 12 presents the programs’ prototyping approaches and percent of RDT&E acquisition costs planned prior to system development for 16 of the programs we examined. Non- competitive programs ranged from 50 to 80 percent, whereas competitive programs ranged from 10 to 77 percent.

Appendix VIII: Technology Readiness Levels

Appendix IX: Department of Defense Initiatives Using Prototyping to Further Innovation

Since 2012, DOD has expanded its prototyping and experimentation efforts, in order to increase innovation, rapidly field new technologies, explore operational concepts, and hedge against threats from potential adversaries, among other purposes. Table 13 below includes newer prototyping and innovation initiatives and select older initiatives. We focused on the newer initiatives in the body of the report. Although broadly speaking, these initiatives are all seeking to achieve the same goal of innovating to maintain military superiority, we found that they differ in the types of innovation they are trying to achieve, the scope of their efforts, and their approaches.

The initiatives we examined varied in the primary types of innovation they sought to achieve (see table 14). Initiatives that primarily focus on incremental or sustaining innovation gradually improve existing products and capabilities. Initiatives focused on disruptive innovation attempt to shift the balance of military power by providing new capabilities, potentially unforeseen by customers or adversaries. The type of innovation sought should not necessarily be equated with risk. For example, the Strategic Capabilities Office (SCO) considers itself to be focusing on low technical risk solutions that are intended to have potentially disruptive results. We excluded experimentation initiatives from this analysis.

Most of the prototyping and experimentation initiatives we examined, including all but one of the newer initiatives, were focused on addressing military service specific needs (see table 15). Three initiatives address capabilities that could potentially benefit multiple services or that are seen as “falling through the cracks” of military service efforts. Those initiatives are among the most well-funded, with the SCO requesting $902 million for budget activities 6.3 and 6.4 and Defense Advanced Research Projects Agency (DARPA) requesting $1.2 billion for budget activity 6.3 for fiscal year 2017. We excluded experimentation initiatives from this analysis.

Most of the initiatives we examined focused on prototyping technologies or systems that address validated customer or warfighter requirements— meaning they have more of a “demand pull” approach to selecting projects (see table 16). In contrast, “supply push” initiatives take on projects without a stated customer need and may not align with existing organizational structures or warfighting concepts. DARPA has historically focused almost exclusively on “supply push” type projects, as does the Navy’s Innovative Naval Prototypes initiative. We excluded experimentation initiatives from this analysis as well as the Marine Corps Rapid Capabilities Office and the Navy’s Rapid Prototyping Experimentation and Demonstration initiative because it is not yet clear which approach they will take.

DOD’s newer prototyping and innovation initiatives tended to differ from older initiatives in several ways, such as the DOD budget activity used, the transition paths available, and senior leadership involvement.

The new initiatives are almost exclusively funded with budget activity 6.4. Budget activity 6.4 funds are viewed as allowing for higher-fidelity prototyping efforts, which are closer to the intended end item. Some previous studies have suggested that maturing technologies outside of acquisition programs to higher readiness levels using budget activity 6.4 can promote innovation and facilitate technology transition. Older initiatives primarily rely on budget activity 6.3 funds.

The Army Rapid Capabilities Office and Navy Rapid Prototyping Experimentation and Demonstration plan to conduct rapid prototyping and then oversee a modified acquisition process for the most promising prototypes. These offices may take responsibility for the rapid prototyping pathway called for in the National Defense Authorization Act for Fiscal Year 2016. Older initiatives typically transition technologies to a different organization or acquisition program.

Many of the new prototyping initiatives report to senior department leaders, which can help maintain support for investments and streamline decision-making, according to the literature on innovation in the private sector. The SCO director reports to the Deputy Secretary of Defense, which provides him flexibility to work throughout the department.

Furthermore, several new initiatives have unique characteristics or functions that the department is trying to invigorate.

The SCO focuses on retooling fielded equipment and technologies to employ them in new ways. This reflects a new tactic in DOD’s innovation efforts because no other initiative has focused on this approach. The office limits its efforts to specific platform-centric solutions with the intention of delivering capabilities quickly. According to SCO’s director, one advantage of this approach is that it can help “buy time” for other initiatives to develop and field the next generation of systems. SCO covers all development costs initially to prove out the idea and develop a prototype, then turns projects over to the services for consideration in their budget deliberation processes. DOD has increased SCO funding since fiscal year 2014 from $140 million to a planned $902 million in fiscal year 2017. This increase is driven by new projects as well as testing and development efforts on existing projects.

Two offices—the Air Force’s Experimentation Initiative and the Navy’s Technology Innovation Games—reflect a renewed emphasis on experimentation in DOD to explore how new capabilities being prototyped could be employed in an operational setting, among other purposes. These offices use virtual or conceptual environments to explore early ideas for implementing new technologies. They can also help identify promising concepts, or allow officials to discuss how to adjust tactics, doctrine, and training prior to development of new technologies. For example, the Air Force’s Experimentation Initiative is using experimentation to explore how directed energy could be used in an operational context and plans to use this information to inform decisions about pursuing directed energy systems. The Air Force is also considering conducting hardware prototyping under this initiative to demonstrate the operational value of proposed concepts. The Navy’s Technology Innovation Games employ a progression from workshops and discussions up through wargames and demonstrations.

Appendix X: Comments from the Department of Defense

Appendix XI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Ron Schwenn, Assistant Director; Pete Anderson; Leslie Ashton; Lorraine Ettaro; Daniel Glickstein; Laura Holliday; Richard Hung; Katherine Lenane; Loren Lipsey; Michael Sweet; Alyssa Weir; and Robin Wilson made significant contributions to this report.