DEFENSE SCIENCE 

AND TECHNOLOGY

Adopting Best 

Practices Can 

Improve Innovation 

Investments and 

Management 

Report to Congressional Committees

June 2017

GAO-17-499

United States Government Accountability Office
United States Government Accountability Office

 

Highlights of GAO-17-499, a report to 

congressional committees

June 2017 

DEFENSE SCIENCE AND TECHNOLOGY

Adopting Best Practices Can Improve

Innovation Investments and Management

What GAO Found

The eight leading companies whose practices GAO assessed take a disciplined 

approach to organizing and executing their technology development activities by 

grouping them into two portfolios: incremental and disruptive, as shown in the 

figure. Incremental development improves product lines whereas disruptive 

development is for riskier innovative and potentially market-shifting technologies. 

Commercial Model Ensures Investments in Incremental and Disruptive Innovation

By separating these two portfolios, companies reported that they could promote 

existing product lines in the short term while exploring opportunities to remain 

competitive in the long term, and mitigate the financial risk associated with 

disruptive technology development. Moreover, GAO found that leading

companies also ensure technologies will be relevant in the marketplace by 

engaging a wide range of internal stakeholders. These companies also reported 

that they gain leadership buy-in by prototyping technologies before committing to 

further development and product integration. 

While some Department of Defense (DOD) practices closely mirror those of the 

companies GAO reviewed, DOD's ability to adopt leading commercial practices 

in its approach to managing science and technology (S&T) investments is limited 

by its funding policies and culture. Unlike the companies GAO reviewed, DOD 

leadership does not provide guidance on or assess the mix of incremental and 

disruptive innovation. As a result, officials reported that DOD labs struggle to find 

the right balance between these investment areas. Under DOD’s budget policy, 

projects are planned up to 2 years in advance, which can slow innovation and 

limit lab directors’ autonomy as compared to companies. Congress has provided 

a means for lab directors to initiate work outside of this lengthy process, but it 

has not been fully utilized. Additionally, responsibilities for technology versus 

product development also contribute to a culture that discourages collaboration 

and limits labs’ ability to prototype. Yet these issues are not insurmountable, as 

pockets of each military department have demonstrated, such as through recent 

efforts to expand advanced prototyping in the labs. Further, Congress has 

required that by February 2018 DOD create a new Under Secretary of Defense 

for Research and Engineering (USD(R&E)), which will be charged with 

developing policies to improve innovation. This position creates an opportunity to 

develop policies that further promote adoption of leading commercial practices.

View GAO-17-499. For more information, 

contact Mike Sullivan at (202) 512-4841 or 

sullivanm@gao.gov. 

Why GAO Did This Study

DOD relies on innovative technologies 

to ensure the superiority of its weapon 

systems and planned to invest about

$12.5 billion in fiscal year 2017 to 

achieve this aim. Recently, DOD’s 

leadership role in fostering innovation 

has been supplanted by the 

commercial sector. This has changed 

DOD’s approach to technology 

development by relying more on 

commercial innovation. 

Conference Report 112-329 included a 

provision for GAO to review DOD’s 

S&T enterprise. This report assesses 

(1) the practices leading companies 

employ to manage technology 

development and (2) the extent to 

which DOD can incorporate these 

practices into its own. GAO interviewed 

eight large, profitable, leading 

technology companies (Amazon, Dow 

Chemical, Honeywell, General Motors, 

IBM, Qualcomm, Siemens AG, and 

Valvoline) to identify practices they 

used to manage, prioritize, and assess 

their technology portfolios. GAO also 

met with DOD organizations that 

manage and execute S&T funds to 

identify their practices. 

.What GAO Recommends

GAO recommends that DOD 

annually define and assess the mix 

of innovation investments and define, 

in policy or guidance, an S&T 

management framework that 

comprehensively employs leading 

commercial practices. DOD did not 

agree with the recommendations, 

citing its ongoing deliberations on the 

new USD R&E’s role, but did identify 

some planned actions. GAO believes 

its recommendations are valid as 

discussed in the report. 
Page i GAO-17-499 Innovation Best Practices 

Letter 1

Background 3

Leading Companies Foster Innovation by Investing in Both Nearand Long-Term R&D and Taking Steps to Prove New 

Technologies Work 6

DOD Implements Some Practices Used by Leading Companies, 

but Policy Limitations and Culture Hinder Adopting Others 26

Conclusions 42

Recommendations for Executive Action 43

Agency Comments and Our Evaluation 43

Appendix I Objectives, Scope, and Methodology 47

Appendix II Department of Defense Research, Development, Test, and Evaluation 

(RDT&E) Budget Activities 52

Appendix III Comments from the Department of Defense 53

Appendix IV GAO Contact and Staff Acknowledgments 56

Table

Table 1: Department of Defense (DOD) Research, Development, 

Test, and Evaluation (RDT&E) Budget Activities 52

Figures

Figure 1: Department of Defense Technology Management 

Process 4

Figure 2: Department of Defense (DOD) Research, Development, 

Test, and Evaluation Budget Activities 6

Figure 3: Commercial Model for Incremental and Disruptive 

Research and Development Portfolios Linked to 

Company Strategy 7

Figure 4: Siemens Informs Its Research and Development 

Strategy by Projecting Pictures of the Future 10

Contents
Page ii GAO-17-499 Innovation Best Practices 

Figure 5: Commercial Model Protects Investments for Incremental 

and Disruptive Research and Development (R&D) 

Portfolios Linked to Company Strategy 11

Figure 6: Honeywell Aerospace Integrates Technology and 

Product Roadmaps 13

Figure 7: IBM Creates Disruptive Technologies through Grand 

Challenges 16

Figure 8: Collaboration Was Key to the Dow Chemical Company’s 

Development of Olefin-Block Co-polymers 19

Figure 9: Honeywell Aerospace Customers Helped It Develop 

Synthetic Vision for Aircraft 20

Figure 10: As Technologies Mature and Costs Grow, Management 

Oversight Increases 21

Figure 11: Product Developers Become Involved Earlier for 

Incremental Technologies than Disruptive Technologies 22

Figure 12: Companies Develop Technologies Iteratively Rather 

Than in a Single Step 24

Figure 13: Amazon Staff Inform Technology Development by 

Testing Prototypes in Their Homes 25

Figure 14: The Navy Reports It Ensures Investments in Disruptive 

Innovation by Using a “Strategic Buckets” Funding 

Approach 29

Figure 15: Notional Timeframes for the Department of Defense’s 

(DOD) Planning, Programming, Budgeting, and

Execution Process 31

Figure 16: Fiscal Year 2015 Section 219 Funds by Each Military 

Department (dollars in millions) 34

Figure 17: Close Collaboration Led to Successful Development of 

the Air Force’s Adaptive Engine Technologies 37
Page iii GAO-17-499 Innovation Best Practices 

Abbreviations

ASD(R&E) Assistant Secretary of Defense for Research and

 Engineering

BA budget activity

BES budget estimate submission

CEO Chief Executive Officer

COI Community of Interest

CTO Chief Technology Officer

DARPA Defense Advanced Research Projects Agency

DOD Department of Defense

IBM International Business Machines

NRL Naval Research Laboratory 

NDAA National Defense Authorization Act 

OSD Office of the Secretary of Defense

POM program objective memorandum

PPBE planning, programming, budgeting, and execution

R&D research and development

RDT&E research, development, test, and evaluation

RPED Rapid Prototyping, Experimentation, and Demonstration

S&T science and technology

USD(R&E) Under Secretary of Defense for Research and Engineering

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-17-499 Innovation Best Practices 

441 G St. N.W.

Washington, DC 20548

June 29, 2017

Congressional Committees

The Department of Defense (DOD) relies on the technological superiority 

of its weapon systems and armed forces to protect U.S. interests at home 

and abroad. This technological superiority is being challenged by 

potential adversaries in ways not seen since the Cold War. To counter 

these challenges, DOD’s science and technology (S&T) community is 

charged with identifying and maturing new technologies and making them 

available to DOD’s acquisition community for integration into weapon 

systems. In the President’s fiscal year 2017 budget submission, DOD 

requested approximately $12.5 billion for S&T activities aimed at 

developing technologies that meet both the short-term and long-term 

needs of current and future warfighters. In recent years, DOD has 

reported that its leadership role in fostering innovation has been 

supplanted by the commercial sector, which has changed the paradigm 

for DOD S&T. DOD now relies increasingly on the innovations 

commercial industry produces to guide its own S&T investments.

The conference report 112-329 accompanying the National Defense 

Authorization Act for Fiscal Year 2012 included a provision for us to 

review DOD’s S&T enterprise, including its interactions with industry, 

investment strategies, technology development methods, and transition 

activities—the process of migrating new technologies from the research 

environment to military users.1 We previously issued a report in March 

2013 to address the technology transition component of this provision.2 In 

this report, we analyze DOD’s approach to managing technology 

development as compared to best practices that leading companies use. 

Specifically, we identified (1) practices that selected leading companies 

use to manage their S&T portfolios and programs, and (2) the extent to 

which DOD can employ these practices within its S&T enterprise, 

including any barriers to adopting such practices.

 

1

H.R. Rep. No. 112-329, at 719 (2011). 

2

GAO, Defense Technology Development: Technology Transition Programs Support 

Military Users, but Opportunities Exist to Improve Measurement of Outcomes, 

GAO-13-286 (Washington, D.C.: Mar. 7, 2013).

Letter
Page 2 GAO-17-499 Innovation Best Practices 

To identify the practices that leading companies use to manage their S&T 

portfolios and programs, we selected eight companies to include as case 

studies in our review. These companies were

• Amazon.com,

• Dow Chemical Company,

• Honeywell International, Inc.,

• General Motors Co.,

• International Business Machines (IBM) Corporation,

• Qualcomm,

• Siemens AG, and

• Valvoline.

We selected these companies primarily on the basis of their having 

received awards or other recognition for technology innovation since 

2014, as well as factors such as profitability and industry type. All of the 

companies we selected were profitable, large Fortune 500-listed 

companies, or were owned by those companies. For each of the 

companies, we interviewed senior management officials and other 

representatives knowledgeable about research and development (R&D), 

which are technology development activities equivalent to S&T 

development activities at DOD. We collected documentation, when 

available, and conducted semi-structured interviews to gather consistent 

information about processes and practices these selected companies 

used to manage technology development. In particular, we obtained 

information on their (1) organizational structures and management 

cultures, (2) R&D portfolio management and investment strategies, (3) 

R&D project management practices, and (4) their technology transition 

processes, including when transitions occur, what organizations are 

involved, and how technologies are further funded. We synthesized each 

company’s processes and created summary documents, which the 

companies then reviewed for accuracy and completeness to validate our 

assessment of their specific practices. Using this validated information, 

we identified the practices that were consistent among the selected 

companies and which company representatives considered key to 

promoting innovation. We also presented our analysis of these leading 

practices to senior DOD S&T executives within the Office of the Secretary 

of Defense, the military services, and other defense research 

organizations to obtain their views on the practices.
Page 3 GAO-17-499 Innovation Best Practices 

To identify the extent to which DOD can employ practices that leading 

companies use to manage S&T investments, we met with officials from 

across the DOD S&T enterprise, including the Office of the Assistant 

Secretary of Defense for Research and Engineering (ASD(R&E)), the 

three military departments’ laboratories, and other defense research 

organizations that execute S&T funds. We conducted semi-structured 

interviews to gather consistent information about processes and practices 

these organization used to manage S&T development. In particular, we 

discussed their (1) organizational structure and management culture, (2) 

S&T portfolio management and investment strategy, (3) S&T project 

management practices, and (4) their technology transition process. From 

these, we identified the existing practices in S&T management being 

used throughout DOD. We compared and contrasted those practices with 

those identified through our meetings with selected leading companies to 

determine the extent to which DOD is employing commercial practices. 

We reviewed relevant policies and regulations used by DOD S&T 

organizations to manage the S&T enterprise. To further our 

understanding of the S&T management practices being used at the 

military department labs, we reviewed at least two recent S&T projects by 

each of these labs. These projects were identified by lab officials and 

included projects deemed successful as well as ones identified as 

unsuccessful. Finally, we hosted a forum in December 2016 that included 

12 DOD S&T executives, including those associated with the 

organizations we met with during our review, to identify what opportunities 

exist within DOD to adopt selected leading companies’ practices, as well 

as any obstacles that may preclude DOD from fully adopting these 

practices. Appendix I provides additional detail on our objectives, scope, 

and methodology.

We conducted this performance audit from February 2016 to June 2017 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.

DOD’s S&T community—including research laboratories, test facilities, 

industry, and academia—conducts initial research, development, and 

testing of new technologies to improve military operations and ensure 

technological superiority over potential adversaries. Key expectations 

DOD places on its S&T community include the following:

Background
Page 4 GAO-17-499 Innovation Best Practices 

• expand scientific knowledge and investigate technologies that may 

provide new warfighting capabilities,

• anticipate technological needs for an uncertain future, and

• produce relevant and feasible technologies that can transition into 

weapon system programs or go directly to the warfighter in the field.

As a result, some investments focus on conducting research to generate 

scientific knowledge, exploring new technologies, demonstrating the 

feasibility of a technology concept, and pursuing other science and 

technology endeavors. We have previously reported that the challenge is 

finding the right balance between developing breakthrough or “disruptive” 

technologies—those considered to be innovative—and investing in 

moderate, “incremental” technology enhancements.3

Figure 1 below provides a notional picture of how DOD’s S&T community 

manages technology investment, development, and transition to a user.

Figure 1: Department of Defense Technology Management Process

Following technology development, DOD’s acquisition community 

manages the next phase, product development, in which technologies are 

further advanced and system development begins. DOD has long 

reported the existence of a chasm between its S&T community and the 

acquisition community, which often precludes effective transitioning of 

 

3

GAO-13-286. 
Page 5 GAO-17-499 Innovation Best Practices 

technologies out of the S&T environment into weapon systems. In a 

series of reports, we found that technologies may not leave the lab 

because their potential has not been adequately demonstrated or 

recognized, acquisition programs may be unwilling to fund final stages of 

development, or private industry chooses to develop the technologies 

itself.4 Further, we found that the acquisition community frequently 

integrates technologies too early and takes on the task of maturing 

technologies—an activity that is the primary responsibility of the S&T 

community—at the start of an acquisition program. These challenges, in 

part, contribute to cost growth, schedule delays, and performance 

shortfalls that we have frequently found and reported on in DOD weapon 

programs.5

DOD funds technology and product development activities under its 

research, development, test, and evaluation (RDT&E) budget, which DOD 

groups into seven budget activity categories for its annual budget 

estimates. The categories follow a mostly sequential path for developing 

technologies from basic research to operational system development, as 

is shown in figure 2. The first three budget activity categories generally 

represent activities undertaken by DOD’s S&T enterprise to advance 

research and develop technology, while the remaining budget activity 

categories are typically associated with product development for 

acquisition programs. See Appendix II for a description of each budget 

activity.

 

4

See, e.g., GAO-13-286, GAO, Best Practices: Stronger Practices Needed to Improve 

DOD Technology Transition Processes, GAO-06-883 (Washington, D.C.: Sept. 14, 2006); 

Small Business Innovation Research: DOD's Program Supports Weapon Systems, but 

Lacks Comprehensive Data on Technology Transition Outcomes, GAO-14-96 

(Washington, D.C.: Dec. 20, 2013); and Defense Advanced Research Projects Agency: 

Key Factors Drive Transition of Technologies, but Better Training and Data Dissemination 

Can Increase Success, GAO-16-5 (Washington, D.C.: Nov. 18, 2015).

5

See, e.g., GAO, Defense Acquisitions: Assessments of Selected Weapons Programs, 

GAO-16-329SP (Washington, D.C.: Mar. 31, 2016); and Defense Acquisitions: 

Assessments of Selected Weapons Programs, GAO-17-333SP (Washington, D.C.: Mar. 

30, 2017). 
Page 6 GAO-17-499 Innovation Best Practices 

Figure 2: Department of Defense (DOD) Research, Development, Test, and Evaluation Budget Activities

Selected leading companies that we reviewed follow six key practices that 

together reflect a disciplined approach to managing their R&D activities—

those akin to DOD’s S&T activities. First, they define their corporate 

strategy by identifying desired markets. Next, they invest in technology 

programs to penetrate those desired markets. Effective management of 

these portfolios requires balancing investments between two types of 

R&D efforts: incremental R&D, which is tied to near-term products; and 

disruptive R&D, which is intended to deliver innovative technologies that 

can provide longer-term growth. According to company representatives, 

this balance is driven by the business imperative of sustaining current 

markets while also developing future ones. Leading companies align their 

goals for incremental technology development with product development, 

while also providing independent paths for developing disruptive 

technologies not tied to product development. In addition, these 

companies identify stakeholders outside the scientific realm and 

collaborate extensively with them to ensure that technologies are relevant 

and can be efficiently integrated into marketable products. Among the key 

R&D stakeholders are representatives from the business units who are 

responsible for identifying customer needs and getting products to 

market. They also scale the rigor in project oversight based on the 

amounts of time and money invested. Nonetheless, leading companies 

expect all R&D projects to include prototyping or other demonstrations to 

prove out the technology before it is integrated into a product for the 

company to sell. Figure 3 below summarizes the general management 

process these leading companies use to plan and execute their R&D 

investments.

Leading Companies 

Foster Innovation by 

Investing in Both 

Near- and Long-Term 

R&D and Taking 

Steps to Prove New 

Technologies Work
Page 7 GAO-17-499 Innovation Best Practices 

Figure 3: Commercial Model for Incremental and Disruptive Research and Development Portfolios Linked to Company 

Strategy 
Page 8 GAO-17-499 Innovation Best Practices 

Among the eight leading companies we reviewed, each manages R&D 

investments that are underpinned by defined strategies, markets, and 

financial goals. In addition, each company sets aside a percentage of 

company revenues to fund R&D. The company’s strategic direction is set 

by the Chief Executive Officer (CEO), in coordination with the company’s 

top executives, including the Chief Technology Officer (CTO) or other 

senior R&D executives.

These corporate strategies balance near-term profitability with long-term 

growth potential and market expansion. Companies stay competitive by 

dividing their collection of R&D projects, also known as their R&D 

portfolios, into two categories:

1. Incremental R&D: lower-risk projects to be integrated quickly into 

near-term products.

2. Disruptive R&D: projects that carry a higher risk of failure, but offer 

significant rewards for the company in the long-term. These 

investments may lead to non-incremental innovations that become an 

important piece of their portfolio. In some cases, these technologies 

render competing products obsolete by creating new markets or 

displacing existing product lines.

According to representatives of leading companies, around 80 percent of 

R&D funding is spent on incremental development, while the balance is 

spent on disruptive projects. Corporate leadership determines this 

percentage based on tolerance for risk and the company’s financial 

standing.

In addition to these two portfolio types, leading companies provide 

scientists with the flexibility to work on lower-cost exploratory projects. 

Such work is conducted by a few scientists or researchers and is not part 

of the annual process for approving projects within each portfolio. 

However, the work derived from these efforts could eventually become 

part of incremental or disruptive R&D portfolios.

In determining an appropriate balance between incremental and 

disruptive R&D investments, company leaders consider long-term 

scenarios based on current trends and technologies in the market. For 

example, R&D leaders at Siemens reported that they conduct an annual 

“Innovation Review” of the company’s entire technology development 

portfolio for the purpose of informing top leadership’s strategic decisions. 

These reviews evaluate Siemens’ technological competitiveness, 

Leading Companies Use 

Corporate Strategies to 

Guide Investments in 

Portfolios of Incremental 

and Disruptive 

Technologies

Setting Company Strategy
Page 9 GAO-17-499 Innovation Best Practices 

strategic resource allocation, and long-term corporate strategy. During 

this review, Siemens asks a number of questions, including the following:

• Are the overall resource allocations for R&D investments appropriate?

• Is the business unit’s technology position competitive, and will 

planned investments safeguard the business unit’s technological 

competitiveness?

• Is there a convincing long-term strategy for how to translate these 

investments into sustainable business success?

• Is there an adequate strategy for translating new technologies into 

winning offerings?

Siemens executives reported that they understand that these factors 

directly impact their ability to grow and profit as a company, which is why 

the desire to remain technologically competitive drives corporate strategy 

decisions.

Siemens also creates forecasting tools called “Pictures of the Future” that 

provide graphical representations of how future technologies could be 

used by customers 7 to 15 years in the future. Figure 4 provides an 

example of the elements that Siemens includes in a Picture of the Future.
Page 10 GAO-17-499 Innovation Best Practices 

Figure 4: Siemens Informs Its Research and Development Strategy by Projecting Pictures of the Future

According to Siemens, it uses Pictures of the Future to

• assess societal, technological, and other trends to guide visionary 

concepts for potential new markets and customer needs;

• consider existing product lines, technologies, and customer needs;
Page 11 GAO-17-499 Innovation Best Practices 

• analyze the opportunities and risks for the company’s core business; 

and

• identify what is required to allow the company to act upon potential 

future scenarios.

Most importantly, company representatives stated that these pictures help 

develop consensus within Siemens regarding the technologies the 

company needs to develop to drive innovation and remain a market 

leader.

After leading companies settle upon their corporate strategies for R&D 

investment, different units are charged with sponsoring—approving and 

funding—incremental and disruptive R&D projects. Incremental R&D 

projects are typically sponsored by business units, who are also 

responsible for product development. Disruptive R&D is often sponsored 

by a corporate research organization, which makes project investment 

decisions independently from the business units. Figure 5 shows this 

division of R&D that we observed in the private sector.

Figure 5: Commercial Model Protects Investments for Incremental and Disruptive 

Research and Development (R&D) Portfolios Linked to Company Strategy
Page 12 GAO-17-499 Innovation Best Practices 

Selected leading companies we reviewed align plans for developing new 

incremental technologies with plans for developing future products, which 

companies sometimes refer to as roadmapping. Individual business units 

are responsible for product sales in a company and have their own 

executive management teams that are charged with generating profits 

from their product lines. Business units sponsor incremental development 

projects intended to yield technologies to meet identified customer needs. 

Incremental R&D generally adds new capabilities to current products or 

next generation versions of existing products; therefore companies expect 

these to have lower risk of failure. Depending on the industry, business 

units generally do not look beyond a 5-year timeframe when making 

decisions about these new technologies due to the unpredictable nature 

of the markets in which they operate.

These leading companies document planned future products in product 

roadmaps, while the technologies that are to be integrated into those 

products are documented in technology roadmaps. By aligning these 

plans, business units can better identify and prioritize technology 

development investments. To develop these plans, companies solicit 

ideas and information from people across the organization to determine 

the composition of incremental R&D portfolios. Ultimately, however, 

technology development decisions come down to the management’s 

qualitative judgements regarding the merits of individual R&D projects, as 

well as on quantitative metrics like potential return on investment. Once a 

project is approved, it may be immediately funded and executed.

The process and the number of people involved in these R&D investment 

decisions vary depending on the company. In general, these leading 

companies solicit input from

• top leadership responsible for setting the company’s overall strategy 

and funding, including the CEO, CTO, and other corporate leaders;

• representatives from business units responsible for getting relevant 

products to market; and

• scientists and technologists who plan for future technology 

development and identify when technologies are ready for integration 

into products.

Technology and product development teams at Honeywell Aerospace, for 

example, complete an annual roadmapping process to align incremental 

technology development activities with the company’s product plans. As 

part of this process,

Leading Companies Align 

Incremental Technology 

Investments with Product 

Development Goals
Page 13 GAO-17-499 Innovation Best Practices 

• business units identify customer needs;

• marketing and product management staff review market trend 

information, and determine future products and when they must be 

completed; and

• Honeywell’s corporate research organization reviews external 

technology trends and creates technology roadmaps.

Honeywell Aerospace’s roadmapping process is illustrated in figure 6.

Figure 6: Honeywell Aerospace Integrates Technology and Product Roadmaps

Honeywell’s roadmapping process focuses on needs and trends 

projected for the next 10 years, with less emphasis on the more distant 

future, due to the difficulty in making reliable predictions that far in 

advance. In general, the leading companies we met with consider this 

timeframe to be realistic and manageable for planning incremental R&D 

investments. To assist with management decisions regarding what 
Page 14 GAO-17-499 Innovation Best Practices 

incremental R&D projects to start or change, Honeywell considers 

potential revenues in the next 5 years. However, certain technologies, 

such as those associated with jet propulsion engines, require longer-term 

plans because technology development takes many years to complete. 

Although the company’s technology development plans do not extend 

beyond the next 10 years, company officials reported they do review 

industry trends potentially leading to new developments further in the 

future and develop concept ideas for new technologies based on these 

trends.

Honeywell obtains input from a variety of sources to ensure technologies 

will be feasible for integration into future products and relevant to 

customer needs. Technology development plans, or roadmaps, are 

periodically revised based on changes in customer needs, prototyping 

results not meeting expectations, and other changes in circumstances 

requiring additional consideration. According to company representatives, 

Honeywell stays in close communication with customers to ensure the 

company’s understanding of market needs remains accurate, which helps 

them avoid wasting time and money on projects that have lost relevancy.

Selected leading companies we reviewed also ensure that a portion of 

their R&D is independently focused on futuristic concepts, which are 

intended to keep the companies competitive in the long term. Disruptive 

R&D includes significant technology development efforts addressing the 

anticipated customer needs of the future, potentially leading to products 

that render the competition’s products irrelevant in the marketplace.

The disruptive R&D portfolio is initiated separately from incremental 

portfolios and often managed by a corporate research organization. 

Corporate research looks for solutions that provide customers with 

capabilities they may not realize they need or want. By separately 

organizing disruptive R&D from incremental, companies are able to 

protect funds from near-term-focused business unit managers. Generally 

speaking, companies in our review ensure that disruptive R&D is planned 

and executed by management not averse to taking risks when significant 

long-term rewards are possible. Due to their near-term focus, business 

unit managers usually begin having significant influence over disruptive 

technologies only after they have been demonstrated and are ready to 

begin transitioning into products. Allowing exploratory and disruptive 

technology development to occur without requiring product development 

approval helps prevent the company’s products from becoming obsolete 

and gives potential to capture new markets.

Leading Companies Make 

Decisions to Fund 

Disruptive R&D 

Independently from 

Product Development 

Plans
Page 15 GAO-17-499 Innovation Best Practices 

These companies use various approaches to leverage the ideas of their 

own staff and external partners to innovate for futuristic technologies that 

look beyond product roadmaps. These approaches include

• challenging R&D staff to come up with feasible ideas to create 

disruptive technologies leading to entirely new product lines, and

• investing in external startup companies or leveraging externally 

developed technologies.

Companies seek to ensure longer-term competiveness by challenging 

their R&D staff to make scientific advances that will make existing 

products irrelevant in the future. IBM, for example, issues grand 

challenges to R&D staff to develop these kinds of technologies. Company 

representatives stated that about 15 to 20 percent of IBM’s R&D funding 

goes toward development of disruptive new technologies not aligned with 

known customer needs. While all of IBM’s disruptive technology 

development is executed by its corporate research organization, IBM 

business units provide funding to sponsor these efforts. To generate 

ideas for disruptive R&D projects, IBM management issues “Grand 

Challenges” asking for project proposals. Figure 7 below provides 

information on how IBM’s corporate R&D organization independently 

developed the IBM Watson supercomputer as a result of one such Grand 

Challenge.

Companies Challenge Staff to 

Develop Ideas for Disruptive 

Innovation
Page 16 GAO-17-499 Innovation Best Practices 

Figure 7: IBM Creates Disruptive Technologies through Grand Challenges

Amazon also provides a number of avenues for individual staff members 

to submit innovative R&D project proposals that are not necessarily tied 

to defined customer needs or market trends. For example, company 

representatives explained that Amazon holds week-long innovation 

forums, where R&D staff collaborates to develop new project ideas. R&D 

staff then vote on the best ideas, which are submitted to top company 

leadership for approval. Individual scientists or technicians can also 

propose new projects or ideas directly to their supervisors, who may help 

them develop formal proposals.

GM initiates disruptive projects through their corporate research 

organization by funding “Internal Startup” projects. These are disruptive 

technology development projects initiated solely at the discretion of GM’s 

CTO and the director of GM’s R&D labs. Aside from the project teams, 

these are the only GM employees that know the details of these projects. 

This level of confidentiality allows GM to take risks. GM expects these 

projects to

• have features that develop phenomenal value to customers,

• make existing technology obsolete, and
Page 17 GAO-17-499 Innovation Best Practices 

• make the competition irrelevant.

Any R&D staff member can propose this kind of project, although its 

potential value to the company must be defined in the proposal. GM also 

provides these projects with more funding than typical projects so that 

R&D work can progress about three times faster than usual. As these 

projects are generally high in technical risk, GM officials stated that they 

have about a 50 percent success rate, which they considered acceptable 

given the value associated with successful projects.

Leading companies also encourage their scientists to explore and initiate 

low-cost research projects, either as unfunded side-projects or using 

limited resources following approval from a supervisor. Providing this 

flexibility allows scientists to be creative, while also affording them access 

to the company’s laboratory or other resources. These less intensive and 

inexpensive projects are not required to be approved by senior 

management to avoid unnecessary administrative burden with scientists 

during exploratory development. If scientists and management deem a 

technology to be feasible, it will be referred to the company’s R&D project 

approval process for additional funding.

Sometimes leading companies develop technologies that may be useful 

in products beyond their preferred markets. In such instances, these 

companies seek to maximize the value of their R&D investments through 

external partnerships. This may occur through investments in startup 

companies or licensing arrangements. For example, Siemens provides 

alternate paths for innovative technologies to move into products outside 

of their own product lines by co-founding start-up companies that could 

turn them into products, thereby allowing Siemens to benefit financially 

when technologies they developed are used in other companies’ 

products. Siemens also licenses some of its technologies to other

companies so they can be used in their products.

Conversely, sometimes leading companies seek technologies from 

outside firms that show promise for use in the company’s own products. 

The Dow Chemical Company uses a Corporate Venture Fund to make 

investments in companies that have formed to commercialize new 

technologies. In many cases, Dow assumes a minority position, although 

in some cases, Dow may elect to acquire the company or partner without 

financial investment to mutually develop technologies. Dow scientists may 

work alongside R&D staff from these companies or Dow may just choose 

to be an investor. Dow employs technology scouts around the world 

responsible for finding opportunities to bring innovative technologies into 

Companies Maximize R&D 

Value through External 

Partnerships
Page 18 GAO-17-499 Innovation Best Practices 

Dow. These scouts are responsible for learning everything they can about 

a company before Dow proceeds with the investment or partnership.

At the selected leading companies we reviewed, once an R&D project is 

initiated, the research team in charge of the project actively collaborates 

with stakeholders outside the R&D office to help assist and inform project 

execution efforts. These stakeholders typically include

• product development staff and engineers who understand technical 

requirements for technologies to eventually transition beyond R&D,

• marketing staff familiar with how products might fit into the outside 

market,

• business unit staff who interface directly with customers, and

• potential users of the technology.

The level and timing of stakeholder involvement can vary based on the 

type of project. Stakeholders, such as product development staff, typically 

become involved early in incremental R&D projects. On the other hand, 

those same stakeholders might not get involved in disruptive projects until 

later phases of development. Collaboration between stakeholders 

continues even after the technology development effort concludes and a 

business unit begins product development. R&D staff may continue to 

assist product development efforts after technology development is 

completed as products are customized for different types of customers. 

For example, figure 8 describes how a cross-functional team at Dow 

Chemical collaborated to develop a new polymer.

R&D Project Teams at 

Leading Companies 

Collaborate Closely with 

Stakeholders to Produce 

Relevant and Feasible 

Technologies
Page 19 GAO-17-499 Innovation Best Practices 

Figure 8: Collaboration Was Key to the Dow Chemical Company’s Development of Olefin-Block Co-polymers

Leading companies also look outside the company when undertaking an 

R&D project to gain insights from potential customers. These customers 

provide input and perspectives that help inform refinements to 

technologies. Figure 9 details the process Honeywell Aerospace used to 

seek customer input when developing its Synthetic Vision System.
Page 20 GAO-17-499 Innovation Best Practices 

Figure 9: Honeywell Aerospace Customers Helped It Develop Synthetic Vision for Aircraft

Company representatives explained that input from both internal 

stakeholders and potential customers helps R&D staff to transition 

emerging technologies into product roadmaps. This collaboration also 

helps the R&D project team obtain the requisite information and 

resources it needs to develop a technology that is feasible for use as part 

of future products, while also being relevant to future customer needs so 

it is accepted in the marketplace.

At the selected leading companies we reviewed, R&D projects are more 

rigorously reviewed by higher levels of management as their needs for 

staff and funding grow. This leads to a subset of these projects continuing 

into later stages of development while others are ended. The Dow 

Chemical Company, for example, uses a stage-gate process to oversee 

R&D projects requiring significant investments. At each stage of 

development the project’s funding increases and the hurdles for moving 

forward become greater. Dow requires only minimal oversight for low-cost 

exploratory research. Once a scientist proves a technology’s feasibility, 

the project enters Dow’s normal processes for integrated project oversight 

and portfolio management involving all of the relevant Dow stakeholders. 

Later stage development projects generally have higher budgets and are 

more closely monitored to ensure they meet specific technical criteria and 

Leading Companies Tailor 

R&D Project Review 

Processes to the Scope of 

Investment
Page 21 GAO-17-499 Innovation Best Practices 

time-based milestones, according to company representatives. While 

specific review processes vary among leading companies, figure 10 

outlines the key principles that all these different processes embody.

Figure 10: As Technologies Mature and Costs Grow, Management Oversight Increases

IBM also emphasizes relevant stakeholder participation in reviews for 

R&D projects progressing beyond exploratory or disruptive corporate 

research. After projects are initiated, IBM leadership reviews R&D 

projects at least quarterly during business reviews, although company 

representatives noted they may be reviewed more frequently in some 

cases when more attention is warranted. IBM leadership believes—

according to representatives we interviewed—that it must have agile 

review processes that facilitate timely adjustments or, in certain cases, 

terminations to projects.

The selected leading companies we reviewed consider technology 

demonstrations, or prototyping, an inherent part of R&D and use 

demonstrations for a variety of purposes, including to

Leading Companies Prove 

Technologies Work before 

Integrating Them into 

Products
Page 22 GAO-17-499 Innovation Best Practices 

• create demand by convincing stakeholders or customers of the 

potential value of a future technology; and

• obtain feedback from potential end-users to add knowledge and 

improve technologies.

Both incremental and disruptive R&D projects receive funds to 

demonstrate technologies. Figure 11 depicts the technology 

demonstration process of the leading companies.

Figure 11: Product Developers Become Involved Earlier for Incremental Technologies than Disruptive Technologies

Leading companies demonstrate concepts for new disruptive 

technologies to stakeholders to generate demand so business units will 

contribute to technology development. Specifically, once corporate 

research organizations believe technology components are sufficiently 

mature for product development, they offer demonstrations to show their 

potential value to stakeholders in the business units. For example, 

Siemens use prototypes once a technology reaches the point that its 

components can be validated in a laboratory or relevant environment. 

Early Prototyping Creates 

Demand for New Disruptive 

R&D Projects
Page 23 GAO-17-499 Innovation Best Practices 

Siemens representatives explained that should the demonstration prove 

successful, a business unit assumes development responsibilities.

Leading companies also demonstrate these concepts for disruptive 

technologies to potential customers if business units are hesitant to invest 

in further development. For example, Qualcomm corporate research 

representatives told us they must sometimes overcome internal 

resistance to accepting new technologies by demonstrating their value to 

both internal and external customers. Barriers to transition of disruptive 

technologies may be even more prevalent when developing technologies 

that could lead to dropping an existing product or feature. In their role as 

a R&D component, not a business unit focused on product lines, 

corporate research works with both internal and external customers 

without committing to future products but with a clear pathway to adoption 

if successful. Qualcomm representatives explained that if these early 

demonstrations and advocacy for disruptive technologies proves 

successful, then customers may ask product developers to use them in 

future products.

Once a concept is proven, leading companies use technology 

demonstrations to inform developers how a technology needs to be 

improved. Company representatives explained that rapidly developing 

and demonstrating a series of iterations of a new technology provides 

early opportunities for improving the technology, rather than taking longer 

amounts of time to develop technologies without testing them. Figure 12 

depicts the iterative technology development and demonstration process 

that leading companies use.

Feedback from 

Demonstrations Add 

Knowledge and Improve 

Technologies
Page 24 GAO-17-499 Innovation Best Practices 

Figure 12: Companies Develop Technologies Iteratively Rather Than in a Single Step

When Amazon pursues a product or technology, the company already 

has an idea of its customers’ needs, but does not consider this 

information to be complete in the absence of user feedback from 

prototype demonstrations. To ensure technologies address these needs, 

Amazon representatives stated that the company builds iterations of 

prototypes during technology development in a facility specifically 

designed for doing so quickly and at minimal cost. Figure 13 illustrates 

how these early prototypes are generally used by Amazon employees in 

real world settings to help inform technology development.
Page 25 GAO-17-499 Innovation Best Practices 

Figure 13: Amazon Staff Inform Technology Development by Testing Prototypes in Their Homes

Similar to Amazon, IBM’s development method uses prototyping during 

technology and product development. IBM developers show early 

iterations of new technology to customers, improve the design based on 

feedback, and then produce another prototype after that. This process 

repeats until a releasable product is completed. IBM representatives 

found that development models with longer sequential steps and a single 

deliverable are less useful than faster paced and smaller deliverables 

under shorter timeframes.

Companies may also use external demonstrations of maturing 

technologies to prove they work in realistic environments. For example, 

Valvoline produces and tests small volumes of new motor oil formulas in 

its own laboratory or at a few select customers’ facilities. To obtain 

customer insights from prototyping, Valvoline

• develops data and uses cameras on test engines to demonstrate the 

formulas performance,

• uses customer test engines to conduct tests and provide specific data 

to customers on how new formulas perform in their engines; and
Page 26 GAO-17-499 Innovation Best Practices 

• provides test formulas to outside customers and obtains feedback 

through sales staff and focus groups.

Offering external customers an opportunity to test new technologies 

provides important feedback for companies such as Valvoline. By taking 

steps like these to involve outside customers in prototype testing, 

representatives of leading companies stated they are able to identify a 

new technology’s tangible benefits, while also encouraging eventual 

customer acceptance of new products.

While some DOD S&T practices closely mirror those of the selected 

leading companies we reviewed, DOD’s funding policies and culture limit 

its ability to adopt other practices for managing its S&T investments. 

Unlike the companies we reviewed, DOD does not organize and fund 

incremental and disruptive innovation separately. Nor does its leadership 

provide guidance on or assess how these innovation investments should 

be or are mixed. Instead, S&T officials explained that DOD labs face 

pressures to prioritize near-term requirements at the expense of 

potentially disruptive technologies. As a result of DOD funding policies, 

projects are planned 2 years in advance, which can slow innovation and 

limit lab directors’ autonomy to initiate work. While Congress has provided 

authority that, as implemented, has enabled the military department lab 

directors to initiate work outside of the normal lengthy process, DOD has 

not fully utilized these flexibilities. Additionally, we found that divided 

responsibilities for technology versus product development contribute to a 

culture that does not encourage collaboration between DOD’s S&T and 

acquisition communities and limits the S&T community’s ability to conduct 

advanced prototyping. These issues are not insurmountable, however, as 

demonstrated in pockets of each military department. In recognition of 

these and other issues, Congress has required that DOD create a new 

Under Secretary of Defense for Research and Engineering charged with 

advancing defense technology and innovation and establishing policies 

on technology development, prototyping, and experimentation, among 

other responsibilities, by February 2018.

Some of DOD’s practices for managing and executing S&T investments 

closely resemble those employed by the selected leading companies we 

reviewed. DOD has a corporate research organization for disruptive 

innovation and its leadership defines S&T strategies to guide investments 

which are consistent with elements that the companies we met with used 

to manage investments in incremental and disruptive portfolios. DOD 

DOD Implements 

Some Practices Used 

by Leading 

Companies, but 

Policy Limitations and 

Culture Hinder 

Adopting Others

Some DOD Practices for 

Managing and Executing 

S&T Investments Closely 

Resemble Practices Used 

by Leading Companies
Page 27 GAO-17-499 Innovation Best Practices 

project oversight is scaled based on the scope of investment, which 

aligns closely with leading company practices.

While differences exist between how these practices are implemented in 

DOD and at the companies we reviewed, we found that the outcomes are 

the same.

• The Defense Advanced Research Projects Agency (DARPA), for 

example, closely resembles the corporate research organization that 

many leading companies employ to foster disruptive innovation. In the 

President’s budget submission for fiscal year 2017, DARPA requested 

$2.9 billion, about 23 percent of DOD’s S&T budget request. Similar to 

a company’s corporate research organization, DARPA’s projects are 

generally not tied to existing DOD weapon systems or a specific 

military department requirement. Instead, their mission is to produce 

disruptive innovation that could support any military department. The 

DARPA Director makes all funding decisions and project 

prioritizations.

• DOD’s market research, which informs its S&T strategy, is based on 

near and far-term adversarial threats, capability needs, and warfighter 

requirements. While these inputs may differ from the companies we 

reviewed, they are likewise used to prioritize S&T projects.

• DOD does not necessarily use the same metrics as companies to 

evaluate projects, but its labs similarly scale the scope of their project 

reviews based on the maturity of the technology and scope of 

investment. For example, a lab typically reviews basic research 

projects once a year, while officials said more mature, larger 

investments are reviewed multiple times per year by the lab, its 

customers, and military department leadership. Similarly, both DOD 

and the companies we reviewed assess projects based on their cost, 

schedule, and technical performance requirements. One inherent 

difference is that leading companies are concerned with potential 

financial returns on investment, whereas DOD prioritizes for other 

reasons, such as whether the technology carries potential to reduce 

risk to the warfighter.
Page 28 GAO-17-499 Innovation Best Practices 

Although our review of selected leading companies found that they define 

in strategy their annual mix of investments in incremental and disruptive 

innovation, the military departments do not do this, nor do they assess 

such a mix. The office of the ASD(R&E)—the Office of the Secretary of 

Defense (OSD) organization responsible for establishing DOD S&T policy 

and guidance—does not provide guidance to the military departments on 

the mix of incremental and disruptive S&T investments. The military 

departments are responsible for defining their own S&T strategies, 

formulating and managing budgets, and developing technologies. Their 

S&T strategies, however, do not define the mix of incremental and 

disruptive investments each department should make annually. Instead, 

DOD S&T investments are organized and funded based on budget 

activities (BA) that reflect stages of technology maturity.6 DOD uses this 

approach, in part, because the Financial Management Regulation dictates 

how R&D activities are identified for the purposes of budgeting. We found 

that one limitation of funding under BAs is a lack of visibility into whether 

individual projects that labs and research centers invest in are geared 

toward disruptive or incremental innovation. The Financial Management 

Regulation, however, does not preclude DOD from developing investment 

targets for both incremental and disruptive R&D.

Military department lab and center officials we interviewed, however, 

identified certain projects they were working on that could lead to 

disruptive technologies. Officials from these labs and centers 

acknowledged that they struggle to determine the right balance between 

disruptive and incremental innovation projects. They expressed concern 

that military department leadership responsible for setting requirements 

for and approving S&T spending, at times, are more focused on nearterm, less risky, more incremental types of innovation investments at the 

expense of long-term, disruptive innovation.

The Navy is one military department that has taken steps to ensure 

funding for some investments in disruptive innovation. The Navy 

organizes S&T investments around “strategic buckets” to ensure it 

maintains investments in both near- and long-term projects and protects 

funding for potentially disruptive projects. The distribution of resources is 

determined by senior leadership based on the Navy’s S&T strategy. The 

strategy maps out roughly the minimum percentage of funding that the 

 

6

See Department of Defense Financial Management Regulation (DOD FMR), DOD 

7000.14-R (Dec. 2016).

Military Departments Do 

Not Define in Strategy or 

Assess the Mix of 

Investments in 

Incremental and Disruptive 

Innovation
Page 29 GAO-17-499 Innovation Best Practices 

Navy plans to request for high-priority, disruptive projects within its S&T 

portfolio, as reflected in figure 14 below. In fiscal year 2017, the Navy 

plans to invest more than $313 million for Leap Ahead Innovations which 

are intended to be disruptive technologies and deliver transformational 

warfighting capabilities. These are in addition to investments in other 

disruptive technologies that are categorized, but not quantified, under its 

other strategic buckets.

Figure 14: The Navy Reports It Ensures Investments in Disruptive Innovation by Using a “Strategic Buckets” Funding 

Approach

Note: Discovery and Innovation includes basic and applied research (6.1 and 6.2). Leap-Ahead 

Innovations refer to the technology portfolio that defines the future of naval warfighting. Acquisitions 

Enablers deliver critical component technologies to naval acquisition programs. Quick Reaction 

responds to urgent technology needs and solves problems for warfighters. Existing systems refers to 

projects responsive to immediate needs or compelling innovation identified by Navy leadership. 

In a June 2017 report, we recommended that DOD take steps—such as 

the Navy’s—to help ensure adequate investments in innovation that align 
Page 30 GAO-17-499 Innovation Best Practices 

with DOD-wide strategy to overcome the department’s risk-averse culture 

and pressures to focus on near-term projects.

7

In comparison to the practice at selected leading companies we reviewed, 

which annually align their investments to product goals, DOD’s process 

for prioritizing and funding projects takes longer—almost 2 years to 

complete—which we found can slow innovation. Like every other good 

and service DOD acquires, all S&T investments must follow DOD’s 

planning and budgeting policy.8 This policy is underpinned by DOD’s 

Planning, Programming, Budgeting, and Execution (PPBE) process. The 

PPBE process for S&T investments includes the following stages:

• Planning: DOD leadership, in guidance and planning documents, 

identifies strategic priorities, weapon system requirements, and 

adversarial threats. Collectively, these serve as DOD’s broad 

requirements for technology development.

• Programming: S&T organizations give consideration to those 

requirements and propose technology development projects to 

address them. Proposed projects and associated costs are 

documented in Program Objective Memorandums (POM). Each 

organization is tasked with determining which projects to propose in 

the POM, while maintaining balance across their portfolios of 

investment, as well as maintaining an appropriate mix of funding 

based on BA. POM documents are reviewed by senior officials across 

DOD—including those responsible for setting requirements and the 

budget—who also have a role in prioritizing S&T investments.

• Budgeting: Each S&T organization’s POM is used to formulate their 

respective military department’s Budget Estimate Submission (BES), 

which outlines the total funding needed, including how much it will 

need by budget activity. After the President’s budget is submitted, 

Congress enacts an appropriation. Once funds are appropriated, each 

S&T organization is provided funding for the projects approved in the 

POM and BES.

• Execution: S&T organizations carry out funded projects.

 

7

GAO, Weapon Systems: Prototyping Has Benefited Acquisition Programs, but More Can 

Be Done to Support Innovation Initiatives, GAO-17-309, (Washington, D.C., Jun. 27, 

2017).

8

Department of Defense Directive 7045.14, “The Planning, Programming, Budgeting, and 

Execution (PPBE) Process” (Jan. 25, 2013).

DOD’s Lengthy S&T 

Project Planning and 

Budgeting Cycle Can Slow 

Innovation While Existing 

Flexibilities to Expedite 

Have Been Underused
Page 31 GAO-17-499 Innovation Best Practices 

Figure 15 illustrates the notional time frames for DOD’s PPBE process.

Figure 15: Notional Timeframes for the Department of Defense’s (DOD) Planning, Programming, Budgeting, and Execution 

Process

In total, it can take almost 2 years from the time a project is proposed in 

the POM to the time it is funded. In contrast, the companies we reviewed 

reported that they planned projects in the same year they were executed, 

which helped them quickly respond to leaps in technology development. 

S&T officials we met with stated that the 2-year project planning process 

reduces their ability to be as nimble as the companies with whom we met. 

For example, if an unexpected technology breakthrough is identified 

through 6.1 or 6.2 research, the labs may have to wait up to 2 years 

before they may begin work on a follow-on project. DOD S&T executives 

expressed the need for greater flexibility with initiating new projects 

because the pace of technology development can be rapid and planning 

for S&T spending 2 years in advance can hinder innovation. They stated, 

however, that the PPBE process provides Congress with the information 
Page 32 GAO-17-499 Innovation Best Practices 

it needs to maintain oversight and ensure DOD meets its fiduciary 

responsibilities to the taxpayer.

We found that laboratory and research center directors in the military 

departments have less authority to initiate S&T work that is not directly 

linked to defined near- or far-term capability needs as compared to the 

leading companies we reviewed. While the Director of DARPA approves 

every project the agency undertakes and is not beholden to address 

defined requirements, the military departments’ labs and centers do not 

control all of the S&T-related R&D work they perform annually. These 

labs and centers regularly undertake work on behalf of acquisition 

community customers, such as a major defense acquisition program, who 

provide funding in support of the project. This work comes in addition to 

“direct funded” projects approved and funded to the lab or center through 

PPBE that are intended to address S&T requirements outlined in strategy. 

As a result, both the direct funded and customer-funded projects compete 

for lab resources, such as staff, and must be balanced. In fiscal year 

2015, for example, direct funded projects accounted for 19 percent of 

Naval Research Laboratory’s (NRL) $1.2 billion of funding. The other 81 

percent was customer-funded work from Navy, other DOD, or other 

governmental sources. Despite having direct funding, the POM review 

and approval process may constrain which projects are ultimately funded. 

For example, we found that the projects a lab or center proposes in its 

POM submission may be reviewed by as many as four different 

organizations before it is submitted to OSD. It is during this review 

process that lab officials explained that the culture within DOD is, at 

times, to focus on near-term, customer-driven projects, at the expense of 

far-term disruptive projects.

Regardless of the source of the funding, a senior ASD(R&E) official

explained that S&T investments are intended to address some defined 

capability need. This means that the military department’s disruptive 

technology projects are roadmapped to requirements, which differs from 

the practice at the leading companies we reviewed. This may limit the 

labs’ ability to address undefined customer needs through other 

potentially disruptive technologies.

Section 219 of the Duncan Hunter National Defense Authorization Act for 

Fiscal Year 2009, as implemented, has provided defense lab directors 

with some limited flexibility to initiate S&T projects, including those that 

are not roadmapped to defined requirements, outside of the normal 2-
Page 33 GAO-17-499 Innovation Best Practices 

year planning process.9 Specifically, as amended, the law directs the 

Secretary of Defense, in consultation with the Secretaries of the military 

departments, to establish mechanisms under which the director of a 

defense laboratory may use an amount of funds equal to a certain 

percentage of all funds available to the laboratory for the following 

purposes:

• innovative basic and applied research that is conducted at the 

defense laboratory and supports military missions;

• developing programs that support the transition of technologies 

developed by the defense laboratory into operational use;

• workforce development activities that improve the capacity of the 

defense laboratory to recruit and retain personnel with needed 

scientific and engineering expertise; and

• revitalization, recapitalization, or minor military construction 

of laboratory infrastructure.

While this authority directed the creation of a mechanism that may 

provide lab directors with the means to fund projects they consider to be a 

priority, the military departments have not maximized their use of these 

authorities. Until the passage of the National Defense Authorization Act 

for Fiscal Year 2017, the director of a defense lab could use funds equal 

to not more than 3 percent of all funds available to the defense laboratory 

for S&T activities. Each of the services has unique strategies for 

executing section 219 authorities but DOD reported that the full 3 percent 

of funds available to the labs has not been used. DOD officials told us 

that the full 3 percent available to each defense laboratory has not been 

used for a number of reasons, including due to competing S&T funding 

priorities. Additionally, DOD officials indicated that labs had concerns 

about charging customers a fee to fund such S&T activities, which was a 

factor in different amounts of funds available for section 219 purposes, as 

shown in figure 16. 

 

9

Pub. L. No. 110-417 § 219 (2008), as amended at 10 U.S.C. § 2358 note. 
Page 34 GAO-17-499 Innovation Best Practices 

Figure 16: Fiscal Year 2015 Section 219 Funds by Each Military Department (dollars 

in millions)

During our review, Congress amended the Section 219 authority in the 

National Defense Authorization Act for Fiscal Year 2017 to permit lab 

directors to use an amount of funds not less than 2 percent but not more 

than 4 percent of all funding available to the lab.10 As a result of the 

change, the military departments can increase the amount of section 219 

funds that the labs may obtain. The extent to which they choose to fund 

new projects could help them to initiate projects, including those which 

may be “off-roadmap,” faster than through the PPBE process.

 

10The National Defense Authorization Act for Fiscal Year 2017, further authorized defense 

lab directors to charge customer activities a fixed percentage fee, in addition to normal 

costs of performance, of up to four percent , in order to obtain funds to carry out activities 

authorized under Sec. 219 authority. Pub. L. No. 114-328 § 212 (2016).
Page 35 GAO-17-499 Innovation Best Practices 

While the selected leading companies we reviewed ensure close 

collaboration between stakeholders in technology and product 

development, cultural barriers have limited such collaboration within 

DOD. DOD’s funding policies reflect cultural barriers to collaboration 

between the S&T community and its product development stakeholders. 

Under DOD’s funding model, the labs are responsible for technology 

development associated with BAs 6.1 through 6.3, while the stakeholders 

in the acquisition community are traditionally responsible for product 

development, which begins with prototype activities under BA 6.4.11

Although we found that S&T projects funded by the acquisition community 

obtain collaborative input from those same eventual customers, this was 

not the case for direct-funded projects—those initiated by the lab. Lab 

officials explained that for direct-funded projects, they may consult with 

potential customers in the acquisition community to gauge interest before 

starting a project or to present results after it is complete, but those 

customers are not part of the development team. This approach is, in 

part, attributable to these organizations being separate—both in mission 

and in the type of funding they receive. For example, unlike the 

companies we reviewed, S&T officials explained that they do not 

transition scientists and engineers along with the technologies they 

developed to the acquisition programs. The companies we reviewed 

reported that they set the expectation that both technology and product 

development staff work together. DOD, however, has not established a 

formal policy on how these two communities should collaborate on 

projects to overcome these cultural barriers.

DOD has processes to help its research labs and centers collaborate on 

S&T work, but these processes do not emphasize collaboration between 

the S&T and acquisition communities. In 2014, ASD(R&E) revitalized its 

“Reliance 21” framework—a joint planning and coordination process that 

is intended to ensure DOD’s S&T community provides solutions and 

advice to the departments’ senior-level decision makers, warfighter, 

Congress, and other stakeholders. This is to be accomplished, in part, 

through groups of technical experts organized around 17 technical areas 

referred to as Communities of Interest (COI). Originally formed in 2009, 

COIs provide DOD with a mechanism for experts in technical areas, such 

as cyber or space, to coordinate and communicate what S&T-related 

R&D each military department is working on and identify areas for 

 

11BA 1 through 3 includes S&T R&D for basic research, applied research, and advanced 

technology development.

Cultural Barriers That Limit 

Collaboration between 

DOD’s S&T and 

Acquisition Communities 

Are Not Insurmountable
Page 36 GAO-17-499 Innovation Best Practices 

collaboration. Each COI is supposed to conduct a portfolio review every 2 

to 3 years—depending on the technical area—to assess those gaps and 

their impacts and to make recommendations to the S&T Executive 

Committee. We found that each COI has documented at least one such 

assessment since 2014. According to an ASD(R&E) official, each COI 

also provides updates to the S&T Executive Committee during its annual 

S&T strategy meeting.

DOD’s S&T executives stated that there is a need to improve how and 

when the acquisition community is brought in to contribute. One S&T 

executive pointed out that bringing in external stakeholders earlier in the 

process is a way to facilitate disruptive innovation. They described one 

instance in which the Air Force overcame existing cultural barriers to 

collaboration by funding technology maturation efforts with both S&T and 

acquisition community support, as described in figure 17. This different 

approach was the result of a deliberate decision by the Air Force to foster 

more collaboration between the S&T community and other stakeholders 

to increase the odds that new technologies end up in the hands of the 

warfighter, according to officials.
Page 37 GAO-17-499 Innovation Best Practices 

Figure 17: Close Collaboration Led to Successful Development of the Air Force’s Adaptive Engine Technologies

While selected leading companies we reviewed provided funding for 

prototypes during technology development, DOD has only recently begun 

to fund advanced prototyping efforts within the labs. The acquisition 

community, as opposed to the S&T labs, traditionally bears the 

responsibility of maturing technology through advanced prototyping, 

which is in contrast to how the companies with whom we met operate. 

DOD’s S&T labs and centers typically do not control the BA 6.4 funding 

that would allow them to conduct such prototyping. We found that the 

S&T community typically matures technologies to, at most, a prototype 

that is close to final form, fit, and function and tested in a relevant 

environment. This creates strong incentives for S&T project teams to 

identify technology transition partners in the form of major acquisition 

programs early in development, which may ultimately restrict disruptive 

innovation and push S&T projects to be more incremental to satisfy 

potential customers’ near-term needs. For example, applied research 

Military Department Labs 

Traditionally Funded 

Technology Development 

through Initial Prototypes, 

but Recent Initiatives Have 

Begun to Fund Advanced 

Prototyping
Page 38 GAO-17-499 Innovation Best Practices 

project proposals at the Naval Research Lab and the Army’s Engineer 

Research and Development Center both identify and consider potential 

transition partners as part of the selection criteria, regardless of whether it 

was for incremental or disruptive technologies. As we previously stated, it 

is difficult to transition a technology or identify a partner when the 

technology is disruptive. Companies recognized this and funded 

disruptive technology development projects through demonstration to 

help obtain a customer.

In June 2017, we reported that DOD’s approach to prototyping 

contributes, in part, to DOD’s broad challenges with transitioning 

technology from the labs into the hands of the warfighter.12 We further 

reported that prototyping that is not directly tied to acquisition programs 

can be seen as a way to “test the waters” because it does not require the 

level of commitment associated with starting acquisitions. 

Each military department has recently undertaken efforts to fund more 

advanced prototypes for incremental and disruptive technologies in S&T 

labs. For example, since 2012, the Army has used funding typically 

associated with acquisition programs to conduct higher-fidelity prototyping 

and further mature technology outside of those programs through its 

Technology Maturation Initiative. In the President’s fiscal year 2017

budget submission, the Army requested approximately $70 million for 

these efforts.

In May 2016, the Air Force established the Strategic Development 

Planning and Experimentation Office, in part, to run Air Force 

experimentation initiatives to achieve specific technology development 

objectives. Air Force officials explained that the intent of these initiatives 

is to develop more agile approaches to innovation by creating a learning 

organization that

• can rapidly take new innovative approaches,

• can quickly initiate new projects and is not hampered by the traditional 

2-year planning process,

• is composed of acquisition and S&T representatives to promote 

collaboration, and

 

12GAO-17-309. 
Page 39 GAO-17-499 Innovation Best Practices 

• conducts prototyping and demonstrations without a direct requirement 

from an acquisition community stakeholder to reduce risk and mature 

technologies.

In the President’s fiscal year 2017 budget request, the Air Force 

requested $62 million to fund these experimentation efforts. Currently, 

two technology development areas are addressed through 

experimentation initiatives and two more are planned. According to Air 

Force officials, these initiatives reflect its leadership’s desire to embrace a 

culture of encouraging and formulating innovative strategic choices 

independent of major weapon system acquisition programs.

The Department of Navy is pursuing similar efforts through its Rapid 

Prototyping, Experimentation, and Demonstration (RPED) initiative. 

RPED projects use prototyping to rapidly develop and assess new 

technologies and engineering innovations to address priority naval 

warfighting needs. The Navy expects that RPED projects will assist in

• developing new capability concepts,

• informing and refining requirements,

• addressing priority needs by demonstrating, and

• enabling quicker transition of technologies to naval programs.

The Navy developed its policy for RPED projects in December 2016 and 

requested $40 million to fund projects in the President’s fiscal year 2017 

budget submission.13

In our June 2017 report, we recommended that DOD develop a strategy

to better coordinate and communicate the goals of these and other 

prototyping efforts to ensure these efforts gain traction and achieve 

success.14

An October 2013 Defense Science Board report reinforces the military 

departments’ focus on experimentation as an innovation enabler.15 The 

 

13Secretary of the Navy Instruction 5000.42, “Department of the Navy Accelerated 

Acquisition for the Rapid Development, Demonstration and Fielding of Capability” (Dec. 

22, 2016).

14GAO-17-309. 

15Defense Science Board, Technology and Innovation Enablers for Superiority in 2030

(Washington, D.C.: Oct. 2013).
Page 40 GAO-17-499 Innovation Best Practices 

Defense Science Board found that DOD cannot continue to rely on 

technological superiority unless it adopts methods that allow it to 

anticipate, assess, and gain experience with new technological 

capabilities before its potential adversaries do. 

DOD’s organizational structure and incentives contribute to why it does 

not fully implement the S&T management practices that the selected 

leading companies we reviewed follow. This includes DOD’s budget 

environment, funding model, and the manner in which DOD is organized 

to execute technology development. As we have previously reported, the 

critical differences between the environments and cultures of private 

companies and DOD must be recognized before tangible progress can be 

made in establishing more efficient practices in S&T management. 

Further, we concluded that changing the mechanics of the processes, 

without changing the environment that determines incentives, may not 

produce better outcomes.16 Specifically:

• Companies operate in an environment where profitability is a constant 

business imperative. As such, leading companies we reviewed 

devoted a portion of their R&D investments toward futuristic concepts, 

which are intended to keep them competitive in the long-term, instead 

of just near-term products. Disruptive R&D includes significant 

technology development efforts addressing the anticipated customer 

needs of the future, potentially leading to products that render 

competitive products irrelevant. By separately organizing disruptive 

R&D from incremental, companies are able to protect funds from the 

near-term focused business unit managers. In the DOD environment, 

budget pressures and urgent requirements often drive military 

departments to focus on near-term needs over long-term innovation. 

For instance, a 2016 Air Force Studies Board report found that in 

much of the Air Force, little or no space for innovation exists.17

Because innovation is focused on future needs, the report found that 

Air Force organizations decide they can wait on addressing the needs 

of tomorrow. The report found that across the Air Force as a whole, 

insufficient processes existed to support “rapid-cycle” innovation with 

the same intensity and pace Air Force personnel regularly bring to 

bear to fulfill other missions.

 

16GAO-06-883. 

17Air Force Studies Board, The Role of Experimentation Campaigns in the Air Force 

Innovation Life Cycle (Washington, D.C.: 2016).

DOD and Commercial 

Firms Operate under 

Different Incentives and 

Organizational Structures
Page 41 GAO-17-499 Innovation Best Practices 

• Companies fiercely compete with one another for customers and in 

ever-changing market conditions. This environment requires agility in 

how they direct their technology and product investments, which 

includes rapidly initiating new projects and truncating underperforming 

ones. DOD, on the other hand, operates under different conditions. Its 

budget environment may incentivize starting and sustaining programs 

rather than discontinuing underperforming ones. In April 2014, we 

found that budgets to support major acquisition program commitments 

must be approved well ahead of when the information needs to 

support the decision is available.18 DOD’s S&T community operates 

under similar pressures and incentives as its acquisition community. 

In this environment, we found that it is easier to sustain a program 

until its funding expires, even if technical performance is lacking. 

According to DOD S&T officials, current budgeting and funding 

processes restrict, rather than encourage, innovation. DOD S&T 

executives told us that they want more flexibility outside of the 

cumbersome 2-year PPBE process to initiate and discontinue 

projects.

• Companies set their own budgets internally for various activities, 

including R&D. Conversely, as a government agency, DOD can 

influence, but not set, its annual budget. Ultimately, Congress 

determines what level of funding to appropriate DOD, including for 

S&T-related activities.

Overcoming many of these challenges may ultimately be the 

responsibility of the yet-to-be-created Office of the Under Secretary of 

Defense for Research and Engineering (USD(R&E)). The National 

Defense Authorization Act (NDAA) for Fiscal Year 2017 calls for the 

establishment of the position of USD(R&E) to serve as the CTO and 

elevate and enhance the mission of defense technological innovation.19

This office will have greater responsibilities than the current ASD(R&E) 

and will focus on innovation, oversight, and policy for defense research 

and engineering, technology development and transition, prototyping and 

experimentation, and testing activities. Specifically, where ASD(R&E) has 

taken a more hands-off approach to developing S&T policy, Congress 

legislated that the new office take a larger role in establishing policies to 

 

18GAO, Defense Acquisitions: Addressing Incentives is Key to Further Reform Efforts, 

GAO-14-563T (Washington, D.C.: Apr. 30, 2014).

19Pub. L. No. 114-328 § 901(a), which amends 10 U.S.C. by striking § 133 and adding 

new section 133a. See also, H. R. Conf. No. 114-840 at1129-1131 (2016).

New Office to Be 

Responsible for Promoting 

Innovation and 

Establishing S&T Policies
Page 42 GAO-17-499 Innovation Best Practices 

overcome the challenges DOD currently faces with promoting innovation. 

The USD(R&E) will also be responsible for the allocation of resources for 

defense research and engineering, and unifying these efforts across 

DOD. The fiscal year 2017 NDAA requires this position to be created by 

February 2018. In March 2017, DOD reported that it would submit final 

plans for creating this position to Congress no later than August 1, 2017, 

as required by law.

DOD’s S&T investments are key to maintaining our nation’s technological 

superiority over our adversaries. Congress has raised questions about 

DOD being innovative enough to maintain future technology superiority. 

The business imperatives that world-class technology companies must 

operate under force them to manage their S&T portfolios and projects to 

produce better outcomes for evolving current products, as well as well as 

develop disruptive technologies for the future. Leading companies have 

shown they do this by organizing and funding R&D to avoid the pressures 

to focus on incremental innovation at the expense of maintaining their 

technological edge in the future. With its focus on meeting warfighter 

needs, DOD does not operate under similar business imperatives; it has 

not, at a department-wide level, emphasized the need to invest in 

disruptive technologies. Instead, each military department’s S&T 

organizational construct and funding processes increase emphasis on 

investing in technologies that will support the near-term requirements of a 

major weapon system acquisition program at the expense of investing in 

innovative technologies that are not linked to a requirement.

As DOD determines the roles and responsibilities for its new Under 

Secretary of Defense for Research and Engineering, it is uniquely 

positioned to rethink its policies that govern technology development. 

While it may not be practical for each military department to organize its 

technology development as leading companies do, there are pockets 

within each department that are implementing some aspects of leading 

company practices. However, more needs to be done to facilitate more 

systematic adoption of these practices across DOD. Doing so can 

position DOD to develop more innovative, disruptive technologies. By not 

taking steps to ensure the right balance of incremental and disruptive 

technology investments, DOD lacks visibility into whether the 

technologies it is developing will provide superior capabilities to counter 

future and emerging adversarial threats. Additionally, the limited 

collaboration with product developers, limited use of existing flexible 

approaches to fund S&T projects outside of the 2-year planning process, 

Conclusions
Page 43 GAO-17-499 Innovation Best Practices 

and limited advanced prototyping of new technologies by the labs creates 

added barriers to innovation.

To ensure that DOD is positioned to counter both near and far term 

threats, consistent with its S&T framework, we recommend that the 

Secretary of Defense direct the new Under Secretary of Defense for 

Research and Engineering annually take the following two actions:

• define the mix of incremental and disruptive innovation investments 

for each military department, and

• assess whether that mix is achieved.

To ensure that DOD is positioned to more comprehensively implement 

leading practices for managing science and technology programs, we 

recommend that the Secretary of Defense direct the new Under Secretary 

of Defense for Research and Engineering to define, in policy or guidance, 

an S&T management framework that includes the three following actions:

• emphasizes greater use of existing flexibilities to more quickly initiate 

and discontinue projects to respond to the rapid pace of innovation;

• incorporates acquisition stakeholders into technology development 

programs to ensure they are relevant to customers; and

• promotes advanced prototyping of disruptive technologies within the 

labs so the S&T community can prove these technologies work to 

generate demand from future acquisition programs.

We provided a draft of this report to the DOD for review and comment. 

DOD’s written comments are reprinted in appendix III of this report and 

summarized below.

In its comments, DOD did not concur with each of our recommendations, 

citing that it is premature to get ahead of the Secretary of Defense’s final 

decisions on the role of the new Under Secretary of Defense for 

Research and Engineering (USD(R&E)) until that position is established, 

which is required by no later than February 1, 2018. We believe, however, 

as the roles and responsibilities of the USD(R&E) are in the process of 

being deliberated, that it is appropriate and timely for the Secretary of 

Defense to ensure that the USD(R&E) be responsible for implementing 

our recommendations. Although it did not concur, DOD identified actions 

that it could take that are generally responsive to our recommendations. 

Recommendations for 

Executive Action

Agency Comments 

and Our Evaluation
Page 44 GAO-17-499 Innovation Best Practices 

Specifically, in response to our recommendations that the Secretary of 

Defense direct the USD(R&E) to define and assess the mix of 

incremental and disruptive innovation investments for each military 

department, DOD stated that it would need to coordinate with each 

military department to establish appropriate goals for those investments. 

DOD further noted that it could assess whether that mix is achieved 

during its annual S&T Strategic Overview meeting. We continue to 

believe that such actions are necessary to ensure that DOD is positioned 

to counter both near and far term threats.

In response to our recommendation that the USD(R&E) define an S&T 

framework that emphasizes greater use of flexibilities to more quickly 

initiate and discontinue projects to respond to the rapid pace of 

innovation, DOD identified the Laboratory Quality Enhancement Program

as an activity to leverage existing flexibilities. This program—which DOD 

implemented in response to the National Defense Authorization Act for 

Fiscal Year 2017—requires DOD to create panels of experts to make 

recommendations to the Secretary of Defense on matters related to S&T 

policy and practices. DOD, however, did not explain how this program 

would help the labs make greater use of existing flexibilities to initiate 

projects, such as those granted under Section 219 of the Duncan Hunter 

National Defense Authorization Act for Fiscal Year 2009. We continue to 

believe that greater use of existing authorities, such as those provided 

under Section 219, could help labs to more quickly initiate projects 

outside of the normal planning cycle, which can take nearly two years for 

a project to be funded.

In response to our recommendation that the USD(R&E) define an S&T 

framework that incorporates acquisition stakeholders into technology 

development programs, DOD identified that it expects the USD(R&E) to 

provide policy and guidance that will include increased engagement with 

acquisition stakeholders. We continue to believe that enhancing 

collaboration between the S&T and acquisition communities is critical to 

ensuring that technologies in development will be relevant to potential 

customers. 

In response to our recommendation that the USD(R&E) define an S&T 

framework that promotes advanced prototyping of disruptive technologies 

within the labs, DOD noted the benefits of prototyping and that it is a 

critical piece of the larger research and engineering strategy. It did not, 

however, identify if any such strategy would be revised to promote earlier 

prototyping so the S&T community can prove technologies work and 

generate demand from future acquisition programs. We continue to 
Page 45 GAO-17-499 Innovation Best Practices 

believe that establishing an S&T framework that emphasizes prototyping 

outside of acquisition programs is needed.

Additionally, in response to our recommendations that the Secretary of 

Defense direct the USD(R&E) to define the three elements above in an 

S&T management framework, DOD also noted that Reliance 21 is 

expected to continue serving as the overarching framework for the S&T 

joint planning and coordination process. We continue to believe, however, 

that this framework does not fully address our recommendations and that 

further actions, such as those they outlined above, are necessary for 

DOD to ensure it is positioned to more comprehensively implement 

leading practices for managing S&T.

We are sending copies of the report to the appropriate congressional 

committees; the Secretary of Defense; the Under Secretary of Defense 

for Acquisition, Technology and Logistics; and the Secretaries of the 

Army, Navy, and Air Force; and to the eight leading companies we 

interviewed about their practices for this report. In addition, the report will 

be available at no charge on GAO’s website at http://www.gao.gov. 

If you or your staff have any questions about this report, please contact 

me at 202-512-4841 or sullivanm@gao.gov. Contact points for our Offices 

of Congressional Relations and Public Affairs may be found on the last 

page of this report. GAO staff who made major contributions to this report 

are listed in appendix IV.

Michael J. Sullivan

Director, Acquisition and Sourcing Management
Page 46 GAO-17-499 Innovation Best Practices 

List of Committees

The Honorable John McCain

Chairman

The Honorable Jack Reed

Ranking Member

Committee on Armed Services

United States Senate

The Honorable Thad Cochran

Chairman

The Honorable Richard J. Durbin

Ranking Member

Subcommittee on Defense

Committee on Appropriations

United States Senate

The Honorable Mac Thornberry

Chairman

The Honorable Adam Smith

Ranking Member

Committee on Armed Services

House of Representatives

The Honorable Kay Granger

Chairwoman

The Honorable Pete Visclosky

Ranking Member

Subcommittee on Defense

Committee on Appropriations

House of Representatives
Appendix I: Objectives, Scope, and 

Methodology

Page 47 GAO-17-499 Innovation Best Practices 

We used a case study approach to identify leading commercial 

companies’ research and development (R&D) practices. We selected and 

visited eight large (Fortune 500-listed) companies that were U.S. 

companies or equivalent foreign companies, or were owned by those 

companies. Our primary goal was to select large companies—those more 

comparable with the size of the Department of Defense (DOD) than 

smaller ones—from a range of different industries, that were profitable 

and that received two or more industry awards or other recognition for 

technology innovation since 2014.

We used the following sources to identify companies that have received 

awards or other recognition: Boston Consulting Group, PwC, MIT 

Technology Review, American Business Awards Gold Stevie awards, and 

Thomson-Reuters. These organizations either provide annual lists of 

leading innovator companies or select top innovator companies for 

awards on an annual basis. These groups are positioned to be 

knowledgeable regarding who the “leading companies” are for the 

purposes of the case study selection for our review. We used available 

corporate stock information at Morningstar.com to determine whether a 

company has been profitable, which is an indicator of their degree of 

success in their science and technology development efforts. Below are 

descriptions of the eight companies featured in this report.

• Amazon.com sells consumer electronics, operates retail websites 

serving over 100 countries, and provides cloud-computing services to 

hundreds of thousands of organizations in 190 countries around the 

world. Lab126 is Amazon’s inventive research and development 

company that designs and engineers high-profile consumer 

electronics, including Kindle Fire tablets, Fire TV and Amazon Echo. 

Amazon’s recent recognitions include being included among the 

Boston Consulting Group’s (BCG) “Most Innovative Companies,” 

PwC’s “10 Most Innovative Companies,” Thomson-Reuters’s “Top 100 

Global Innovators” and MIT Technology Review’s “50 Smartest 

Companies.”

• Ashland Global Holdings, Inc. is a global chemicals company serving 

customers in a wide range of consumer and industrial markets, 

including adhesives, architectural coatings, automotive, construction, 

energy, food and beverage, personal care, and pharmaceutical. 

Valvoline Inc., a leading producer and retailer of automotive lubricants 

that made the first trademarked American motor oil, was an Ashland 

subsidiary at the time we met with Valvoline company representatives. 

Ashland currently owns a controlling interest in Valvoline after it 

became a separate public company in 2016. Ashland’s recent 

Appendix I: Objectives, Scope, and 

Methodology
Appendix I: Objectives, Scope, and 

Methodology

Page 48 GAO-17-499 Innovation Best Practices 

recognitions include a Bronze Innovation Zone Functional Ingredient 

Award and Ringier Coatings Technology Innovation award and a 

Composites and Advanced Materials Exposition award for 

“Unsurpassed Innovation.”

• The Dow Chemical Company delivers a broad range of technologybased products and solutions to customers in 175 countries. Dow 

drives innovations that extract value from material, polymer, chemical 

and biological science to help address many of the world’s most 

challenging problems, such as the need for fresh food, safer and more 

sustainable transportation, clean water, energy efficiency, more 

durable infrastructure, and increasing agricultural productivity. Dow’s 

recent recognitions include receiving multiple R&D 100 awards, and 

being included among BCG’s “Most Innovative Companies” and 

Clarivate Analytics (formerly Thomson-Reuters’s) “Top 100 Global 

Innovators.”

• Honeywell International, Inc. invents and commercializes technologies 

that address some of the world’s most critical challenges around 

energy, safety, security, productivity and global urbanization. 

Honeywell’s Aerospace division, which is discussed in this report, is a 

leading provider of aircraft engines, integrated avionics, systems and 

service solutions, and related products and services for aircraft 

manufacturers, and turbochargers to improve the performance and 

efficiency of passenger cars and commercial vehicles. Honeywell’s 

recent recognitions include receiving American Business Awards for 

“Most Innovative Company” and “Most Innovative Technology 

Company,” and being included among Thomson-Reuters’s “Top 100 

Global Innovators.”

• General Motors Co. and its partners produce vehicles in 30 countries, 

including the Chevrolet, Cadillac, Baojun, Buick, GMC, Holden, 

Jiefang, Opel, Vauxhall and Wuling brands. GM develops innovative 

new technologies offering vehicle electrification, autonomous driving, 

vehicle health management, and alternative fuel usage. GM’s recent 

recognitions include an Edison Award for Automotive Computing, an 

Automotive News “Pace Award,” and being included among Fast 

Company’s “World’s 10 Most Innovative Companies of Automotive.”

• International Business Machines Corporation (IBM) develops and 

markets cognitive systems, or computers that learn through 

interactions with people and data, as well as enterprise systems and 

software. IBM also provides cloud computing, consulting and 

information technology implementation services. IBM’s recent 

recognitions include receiving an R&D 100 award, and being included 
Appendix I: Objectives, Scope, and 

Methodology

Page 49 GAO-17-499 Innovation Best Practices 

among BCG’s “Most Innovative Companies” and MIT Technology 

Review’s “50 Smartest Companies.”

• Qualcomm is a leader in the commercialization of digital 

communication technologies, including Code Division Multiple Access 

(CDMA), and Long Term Evolution (LTE), for cellular wireless 

communication applications. They also develop and commercialize 

numerous technologies used in handsets and tablets. They also own 

intellectual property contributing to other commercial technologies like 

wireless local area network, global positioning system, near field 

communication, and Bluetooth. Qualcomm’s recent recognitions 

include receiving an R&D 100 award, and being included among MIT 

Technology Review’s “50 Smartest Companies” and ThomsonReuters’s “Top 100 Global Innovators.”

• Siemens AG is one of the world’s largest producers of electrification, 

automation, and digitalization technologies. Siemens’s products 

include gas, steam, and wind turbines, integrated power plant 

solutions, power grid systems, building technologies, rail technologies, 

medical imaging and diagnostics, and other systems for industrial use. 

The company’s recent recognitions include receiving two R&D 100 

awards and being included among BCG’s “Most Innovative 

Companies.” Siemens is also category leader in the Dow Jones 

Sustainability Index ranking, with 100 out of 100 points for innovation 

management.

For each of the companies, we conducted semi-structured interviews with 

senior management officials and other company representatives 

knowledgeable about research and development activities to gather 

consistent information about processes and practices companies use to 

manage technology development. In particular, we discussed their (1) 

organizational structure and management culture, (2) R&D portfolio 

management and investment strategy, (3) R&D project management 

practices, and (4) their technology transition process, including when 

transition occurs, the organizations involved, and how technology is 

funded throughout the transition phase. We synthesized each company’s 

processes and created summary documents, which the companies then 

reviewed for accuracy and completeness to validate our assessment of 

their specific practices. Using this validated information, we identified the 

practices that were consistent among the selected companies and which 

company representatives considered key to promoting innovation. We 

also presented our analysis of these leading practices to senior DOD 

Science and Technology (S&T) executives within the Office of the 

Secretary of Defense, the military services, and other defense research 

organizations to obtain their views on the practices.
Appendix I: Objectives, Scope, and 

Methodology

Page 50 GAO-17-499 Innovation Best Practices 

To identify the extent to which DOD can employ these leading 

commercial practices, we interviewed officials responsible for the 

management, execution, and oversight of DOD’s S&T enterprise. At the 

Office of the Secretary of Defense and military department headquarters 

level, those responsible for the management and oversight of S&T 

activities, we met with officials from the

• Office of the Assistant Secretary of Defense for Research and 

Engineering;

• Office of the Deputy Assistant Secretary of the Army for Research and 

Technology;

• Office of the Deputy Assistant Secretary of the Air Force for Science, 

Technology, and Engineering;

• Office of the Deputy Assistant Secretary of the Navy for Research, 

Development, Test, and Evaluation; and

• Office of Naval Research.

We also met with military department laboratory officials responsible for 

the management and execution of S&T activities from the

• Army Research Laboratory;

• Army Armament Research, Development, and Engineering Center;

• Army Engineer Research and Development Center;

• Air Force Research Laboratory;

• Naval Research Laboratory; and

• Naval Undersea Warfare Center—Division Newport.

Finally, we met with officials from the Defense Advanced Research 

Projects Agency (DARPA) responsible for the planning and oversight of 

their S&T activities.

We conducted semi-structured interviews at each laboratory and DARPA 

to gather consistent information about processes and practices these 

organization used to manage S&T activities. In particular, we discussed 

their (1) organizational structure and management culture, (2) S&T 

portfolio management and investment strategy, (3) S&T project 

management practices, and (4) their technology transition process. We 

compared and contrasted those practices with the practices identified 

through our meetings with leading commercial companies to determine 

the extent to which DOD is employing these practices. Where 
Appendix I: Objectives, Scope, and 

Methodology

Page 51 GAO-17-499 Innovation Best Practices 

appropriate, we reviewed relevant regulations, policies, and guidance that 

establish the framework for how DOD S&T organizations plan, budget 

and execute S&T activities, including the Assistant Secretary of Defense 

for Research and Engineering’s Reliance 21 Operating Principles and 

DOD’s Financial Management Regulation. To further our understanding 

of the S&T management practices being used at the military department 

labs, we reviewed at least two recent S&T projects by each of these labs. 

These projects were identified by lab officials and included projects 

deemed successful as well as ones identified as unsuccessful.

Finally, we hosted a forum of DOD S&T executives in December 2016 to 

identify potential opportunities for DOD to adopt leading commercial 

practices in S&T management, as well as any barriers to adopting these 

practices. Forum participants included the following:

• Ms. Mary Miller, Principal Deputy to the Assistant Secretary of 

Defense for Research and Engineering;

• Dr. Melissa Flagg, Deputy Assistant Secretary of Defense for 

Research

• Mr. Michael Holthe, Acting Director of Technology, Office of the 

Deputy Assistant Secretary of the Army for Research and 

Technology;

• Dr. David Walker, Deputy Assistant Secretary of the Air Force for 

Science, Technology, and Engineering;

• Dr. Phil Perconti, Acting Director, Army Research Laboratory;

• Dr. Jeff Holland, Director, Army Engineer Research and Development 

Center;

• Mr. Jyuji Hewitt, Executive Deputy, Army Research, Development, 

and Engineering Command;

• Mr. John Uscilowicz, Director Plans, Programs, Analysis, and 

Evaluation, Army Medical Research and Materiel Command;

• Dr. Morley Stone, Chief Technology Officer, Air Force Research 

Laboratory;

• Dr. Edward Franchi, Acting Director, Naval Research Laboratory;

• Dr. Stephen Russell, Director of Science and Technology, Space and 

Naval Warfare Systems Command; and

• Mr. Ellison Urban, Special Assistant to the Director, Defense 

Advanced Research Projects Agency.
Appendix II: Department of Defense Research, 

Development, Test, and Evaluation (RDT&E) 

Budget Activities

Page 52 GAO-17-499 Innovation Best Practices 

Table 1: Department of Defense (DOD) Research, Development, Test, and Evaluation (RDT&E) Budget Activities

DOD RDT&E Budget Activity Description

Science and 

technology funding

Basic research (6.1) Scientific study and experimentation focusing on increasing fundamental 

knowledge, which may address long-term national security needs. Includes preMilestone A efforts.

Applied research (6.2) Research focuses on the expansion and application of knowledge and is 

directed toward general military needs to determine the initial feasibility and 

practicality of proposed solutions. Includes pre-Milestone B efforts.

Advanced technology 

development (6.3)

Concept and technology demonstrations that assess the technological 

feasibility, operability, and producibility of components, subsystems, or system 

models. Demonstrations evaluate general military utility or cost reduction 

potential of the technology. Projects in this category should have the goal of 

moving out of S&T and into the acquisition process within 5 years. Includes preMilestone B efforts and technologies generally have a Technology Readiness 

Level (TRL) of 4, 5, or 6.

Acquisition-based 

funding

Advanced component 

development & prototypes 

(6.4)

System specific evaluations of integrated technologies, representative models, 

or prototype systems in a realistic operating environment. Focuses on proving 

component and subsystem maturity prior to integration into major systems. 

Includes pre-Milestone B efforts and TRL 6 or 7 should be achieved. 

System development & 

demonstration (6.5)

Engineering and manufacturing development tasks aimed at meeting 

requirements prior to full-rate production. Prototype performance is near or at 

planned operational system levels. Conduct live fire and initial operational test 

and evaluation. Includes post-Milestone B efforts to support Milestone C 

decisions. 

RDT&E management support 

(6.6)

Efforts to sustain and/or modernize installations or operations required for 

RDT&E such as test ranges, military construction, and studies and analyses in 

support of RDT&E. 

Operational system 

development (6.7)

Efforts to upgrade systems that have been fielded or will soon enter full rate 

production. Includes post-Milestone C efforts.

Source: GAO summary of DOD regulations. | GAO-17-499

Note: Technology Readiness Levels (TRL) are a tool that DOD, among others, uses to assess 

technology maturity. TRLs are measured on a scale from 1 to 9, beginning with paper studies of a 

technology’s feasibility and culminating with a technology fully integrated into a completed product. 

Appendix II: Department of Defense 

Research, Development, Test, and 

Evaluation (RDT&E) Budget Activities
Appendix III: Comments from the Department 

of Defense

Page 53 GAO-17-499 Innovation Best Practices 

Appendix III: Comments from the 

Department of Defense 
Appendix III: Comments from the Department 

of Defense

Page 54 GAO-17-499 Innovation Best Practices 
Appendix III: Comments from the Department 

of Defense

Page 55 GAO-17-499 Innovation Best Practices 
Appendix IV: GAO Contact and Staff 

Acknowledgments

Page 56 GAO-17-499 Innovation Best Practices 

Michael J. Sullivan, (202) 512-4841, sullivanm@gao.gov

In addition to the contact named above, Christopher R. Durbin, Assistant 

Director; Marie Ahearn; Emily Bond; Jared Dmello; Lorraine Ettaro; Rich 

Hung; Justin Jaynes; Ron La Due Lake; Sean Seales; Brian Smith; and 

Robin Wilson made significant contributions to this report.

Appendix IV: GAO Contact and Staff 

Acknowledgments

GAO Contact

Staff 

Acknowledgments

(100656)
The Government Accountability Office, the audit, evaluation, and investigative 

arm of Congress, exists to support Congress in meeting its constitutional 

responsibilities and to help improve the performance and accountability of the 

federal government for the American people. GAO examines the use of public 

funds; evaluates federal programs and policies; and provides analyses, 

recommendations, and other assistance to help Congress make informed 

oversight, policy, and funding decisions. GAO’s commitment to good government 

is reflected in its core values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through GAO’s website (http://www.gao.gov). Each weekday afternoon, GAO 

posts on its website newly released reports, testimony, and correspondence. To 

have GAO e-mail you a list of newly posted products, go to http://www.gao.gov

and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of production and 

distribution and depends on the number of pages in the publication and whether 

the publication is printed in color or black and white. Pricing and ordering 

information is posted on GAO’s website, http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, MasterCard, 

Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, LinkedIn, Twitter, and YouTube. 

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. 

Visit GAO on the web at www.gao.gov and read The Watchblog. 

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400, 

U.S. Government Accountability Office, 441 G Street NW, Room 7125, 

Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707

U.S. Government Accountability Office, 441 G Street NW, Room 7814, 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Strategic Planning and 

External Liaison

Please Print on Recycled Paper.