Defense Science and Technology: Adopting Best Practices Can Improve Innovation Investments and Management
Published: 2017-06-29T00:00:00

Fast Facts:
DOD plans to invest $12.5 billion on innovative technology in fiscal year 2017. We compared DOD's approach with how leading technology companies like Amazon and Dow manage their science and technology investments.
We found that DOD's ability to innovate is limited by its funding policies, as projects are planned up to 2 years in advance, which can slow innovation and limit lab directors’ autonomy. Additionally, DOD’s culture prioritizes developing technologies only for programs that already have the needed funding.
We recommended that DOD define a science and technology management framework that uses leading commercial practices.

Highlights:
section_title: Why GAO Did This Study
paragraphs: DOD relies on innovative technologies to ensure the superiority of its weapon systems and planned to invest about $12.5 billion in fiscal year 2017 to achieve this aim. Recently, DOD's leadership role in fostering innovation has been supplanted by the commercial sector. This has changed DOD's approach to technology development by relying more on commercial innovation.
Conference Report 112-329 included a provision for GAO to review DOD's S&T enterprise. This report assesses (1) the practices leading companies employ to manage technology development and (2) the extent to which DOD can incorporate these practices into its own. GAO interviewed eight large, profitable, leading technology companies (Amazon, Dow Chemical, Honeywell, General Motors, IBM, Qualcomm, Siemens AG, and Valvoline) to identify practices they used to manage, prioritize, and assess their technology portfolios. GAO also met with DOD organizations that manage and execute S&T funds to identify their practices.
section_title: What GAO Found
paragraphs: The eight leading companies whose practices GAO assessed take a disciplined approach to organizing and executing their technology development activities by grouping them into two portfolios: incremental and disruptive, as shown in the figure. Incremental development improves product lines whereas disruptive development is for riskier innovative and potentially market-shifting technologies.
By separating these two portfolios, companies reported that they could promote existing product lines in the short term while exploring opportunities to remain competitive in the long term, and mitigate the financial risk associated with disruptive technology development. Moreover, GAO found that leading companies also ensure technologies will be relevant in the marketplace by engaging a wide range of internal stakeholders. These companies also reported that they gain leadership buy-in by prototyping technologies before committing to further development and product integration.
While some Department of Defense (DOD) practices closely mirror those of the companies GAO reviewed, DOD's ability to adopt leading commercial practices in its approach to managing science and technology (S&T) investments is limited by its funding policies and culture. Unlike the companies GAO reviewed, DOD leadership does not provide guidance on or assess the mix of incremental and disruptive innovation. As a result, officials reported that DOD labs struggle to find the right balance between these investment areas. Under DOD's budget policy, projects are planned up to 2 years in advance, which can slow innovation and limit lab directors' autonomy as compared to companies. Congress has provided a means for lab directors to initiate work outside of this lengthy process, but it has not been fully utilized. Additionally, responsibilities for technology versus product development also contribute to a culture that discourages collaboration and limits labs' ability to prototype. Yet these issues are not insurmountable, as pockets of each military department have demonstrated, such as through recent efforts to expand advanced prototyping in the labs. Further, Congress has required that by February 2018 DOD create a new Under Secretary of Defense for Research and Engineering (USD(R&E)), which will be charged with developing policies to improve innovation. This position creates an opportunity to develop policies that further promote adoption of leading commercial practices.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD annually define and assess the mix of innovation investments and define, in policy or guidance, an S&T management framework that comprehensively employs leading commercial practices. DOD did not agree with the recommendations, citing its ongoing deliberations on the new USD R&E's role, but did identify some planned actions. GAO believes its recommendations are valid as discussed in the report.

section_title: Letter
paragraphs: The Department of Defense (DOD) relies on the technological superiority  of its weapon systems and armed forces to protect U.S. interests at home  and abroad. This technological superiority is being challenged by  potential adversaries in ways not seen since the Cold War. To counter  these challenges, DOD’s science and technology (S&T) community is  charged with identifying and maturing new technologies and making them  available to DOD’s acquisition community for integration into weapon  systems. In the President’s fiscal year 2017 budget submission, DOD  requested approximately $12.5 billion for S&T activities aimed at  developing technologies that meet both the short-term and long-term  needs of current and future warfighters. In recent years, DOD has  reported that its leadership role in fostering innovation has been  supplanted by the commercial sector, which has changed the paradigm  for DOD S&T. DOD now relies increasingly on the innovations  commercial industry produces to guide its own S&T investments.
The conference report 112-329 accompanying the National Defense  Authorization Act for Fiscal Year 2012 included a provision for us to  review DOD’s S&T enterprise, including its interactions with industry,  investment strategies, technology development methods, and transition  activities—the process of migrating new technologies from the research  environment to military users. We previously issued a report in March  2013 to address the technology transition component of this provision. In  this report, we analyze DOD’s approach to managing technology  development as compared to best practices that leading companies use.  Specifically, we identified (1) practices that selected leading companies  use to manage their S&T portfolios and programs, and (2) the extent to  which DOD can employ these practices within its S&T enterprise,  including any barriers to adopting such practices.
To identify the practices that leading companies use to manage their S&T  portfolios and programs, we selected eight companies to include as case  studies in our review. These companies were
Dow Chemical Company,
Honeywell International, Inc.,
General Motors Co.,  International Business Machines (IBM) Corporation,
Siemens AG, and
Valvoline.
We selected these companies primarily on the basis of their having  received awards or other recognition for technology innovation since  2014, as well as factors such as profitability and industry type. All of the  companies we selected were profitable, large Fortune 500-listed  companies, or were owned by those companies. For each of the  companies, we interviewed senior management officials and other  representatives knowledgeable about research and development (R&D),  which are technology development activities equivalent to S&T  development activities at DOD. We collected documentation, when  available, and conducted semi-structured interviews to gather consistent  information about processes and practices these selected companies  used to manage technology development. In particular, we obtained  information on their (1) organizational structures and management  cultures, (2) R&D portfolio management and investment strategies, (3)  R&D project management practices, and (4) their technology transition  processes, including when transitions occur, what organizations are  involved, and how technologies are further funded. We synthesized each  company’s processes and created summary documents, which the  companies then reviewed for accuracy and completeness to validate our  assessment of their specific practices. Using this validated information,  we identified the practices that were consistent among the selected  companies and which company representatives considered key to  promoting innovation. We also presented our analysis of these leading  practices to senior DOD S&T executives within the Office of the Secretary  of Defense, the military services, and other defense research  organizations to obtain their views on the practices.
To identify the extent to which DOD can employ practices that leading  companies use to manage S&T investments, we met with officials from  across the DOD S&T enterprise, including the Office of the Assistant  Secretary of Defense for Research and Engineering (ASD(R&E)), the  three military departments’ laboratories, and other defense research  organizations that execute S&T funds. We conducted semi-structured  interviews to gather consistent information about processes and practices  these organization used to manage S&T development. In particular, we  discussed their (1) organizational structure and management culture, (2)  S&T portfolio management and investment strategy, (3) S&T project  management practices, and (4) their technology transition process. From  these, we identified the existing practices in S&T management being  used throughout DOD. We compared and contrasted those practices with  those identified through our meetings with selected leading companies to  determine the extent to which DOD is employing commercial practices.  We reviewed relevant policies and regulations used by DOD S&T  organizations to manage the S&T enterprise. To further our  understanding of the S&T management practices being used at the  military department labs, we reviewed at least two recent S&T projects by  each of these labs. These projects were identified by lab officials and  included projects deemed successful as well as ones identified as  unsuccessful. Finally, we hosted a forum in December 2016 that included  12 DOD S&T executives, including those associated with the  organizations we met with during our review, to identify what opportunities  exist within DOD to adopt selected leading companies’ practices, as well  as any obstacles that may preclude DOD from fully adopting these  practices. Appendix I provides additional detail on our objectives, scope,  and methodology.
We conducted this performance audit from February 2016 to June 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DOD’s S&T community—including research laboratories, test facilities,  industry, and academia—conducts initial research, development, and  testing of new technologies to improve military operations and ensure  technological superiority over potential adversaries. Key expectations  DOD places on its S&T community include the following: expand scientific knowledge and investigate technologies that may  provide new warfighting capabilities, anticipate technological needs for an uncertain future, and produce relevant and feasible technologies that can transition into  weapon system programs or go directly to the warfighter in the field.
As a result, some investments focus on conducting research to generate  scientific knowledge, exploring new technologies, demonstrating the  feasibility of a technology concept, and pursuing other science and  technology endeavors. We have previously reported that the challenge is  finding the right balance between developing breakthrough or “disruptive”  technologies—those considered to be innovative—and investing in  moderate, “incremental” technology enhancements.
Figure 1 below provides a notional picture of how DOD’s S&T community  manages technology investment, development, and transition to a user.
Following technology development, DOD’s acquisition community  manages the next phase, product development, in which technologies are  further advanced and system development begins. DOD has long  reported the existence of a chasm between its S&T community and the  acquisition community, which often precludes effective transitioning of  technologies out of the S&T environment into weapon systems. In a  series of reports, we found that technologies may not leave the lab  because their potential has not been adequately demonstrated or  recognized, acquisition programs may be unwilling to fund final stages of  development, or private industry chooses to develop the technologies  itself. Further, we found that the acquisition community frequently  integrates technologies too early and takes on the task of maturing  technologies—an activity that is the primary responsibility of the S&T  community—at the start of an acquisition program. These challenges, in  part, contribute to cost growth, schedule delays, and performance  shortfalls that we have frequently found and reported on in DOD weapon  programs.
DOD funds technology and product development activities under its  research, development, test, and evaluation (RDT&E) budget, which DOD  groups into seven budget activity categories for its annual budget  estimates. The categories follow a mostly sequential path for developing  technologies from basic research to operational system development, as  is shown in figure 2. The first three budget activity categories generally  represent activities undertaken by DOD’s S&T enterprise to advance  research and develop technology, while the remaining budget activity  categories are typically associated with product development for  acquisition programs. See Appendix II for a description of each budget  activity.
section_title: Leading Companies Foster Innovation by Investing in Both Near- and Long-Term R&D and Taking Steps to Prove New Technologies Work
paragraphs: Selected leading companies that we reviewed follow six key practices that  together reflect a disciplined approach to managing their R&D activities— those akin to DOD’s S&T activities. First, they define their corporate  strategy by identifying desired markets. Next, they invest in technology  programs to penetrate those desired markets. Effective management of  these portfolios requires balancing investments between two types of  R&D efforts: incremental R&D, which is tied to near-term products; and  disruptive R&D, which is intended to deliver innovative technologies that  can provide longer-term growth. According to company representatives,  this balance is driven by the business imperative of sustaining current  markets while also developing future ones. Leading companies align their  goals for incremental technology development with product development,  while also providing independent paths for developing disruptive  technologies not tied to product development. In addition, these  companies identify stakeholders outside the scientific realm and  collaborate extensively with them to ensure that technologies are relevant  and can be efficiently integrated into marketable products. Among the key  R&D stakeholders are representatives from the business units who are  responsible for identifying customer needs and getting products to  market. They also scale the rigor in project oversight based on the  amounts of time and money invested. Nonetheless, leading companies  expect all R&D projects to include prototyping or other demonstrations to  prove out the technology before it is integrated into a product for the  company to sell. Figure 3 below summarizes the general management  process these leading companies use to plan and execute their R&D  investments.
subsections: section_title: Leading Companies Use Corporate Strategies to Guide Investments in Portfolios of Incremental and Disruptive Technologies
paragraphs: Among the eight leading companies we reviewed, each manages R&D  investments that are underpinned by defined strategies, markets, and  financial goals. In addition, each company sets aside a percentage of  company revenues to fund R&D. The company’s strategic direction is set  by the Chief Executive Officer (CEO), in coordination with the company’s  top executives, including the Chief Technology Officer (CTO) or other  senior R&D executives.
These corporate strategies balance near-term profitability with long-term  growth potential and market expansion. Companies stay competitive by  dividing their collection of R&D projects, also known as their R&D  portfolios, into two categories:  1.  Incremental R&D: lower-risk projects to be integrated quickly into  near-term products.  2.  Disruptive R&D: projects that carry a higher risk of failure, but offer  significant rewards for the company in the long-term. These  investments may lead to non-incremental innovations that become an  important piece of their portfolio. In some cases, these technologies  render competing products obsolete by creating new markets or  displacing existing product lines.
According to representatives of leading companies, around 80 percent of  R&D funding is spent on incremental development, while the balance is  spent on disruptive projects. Corporate leadership determines this  percentage based on tolerance for risk and the company’s financial  standing.
In addition to these two portfolio types, leading companies provide  scientists with the flexibility to work on lower-cost exploratory projects.  Such work is conducted by a few scientists or researchers and is not part  of the annual process for approving projects within each portfolio.  However, the work derived from these efforts could eventually become  part of incremental or disruptive R&D portfolios.
subsections: section_title: Setting Company Strategy
paragraphs: In determining an appropriate balance between incremental and  disruptive R&D investments, company leaders consider long-term  scenarios based on current trends and technologies in the market. For  example, R&D leaders at Siemens reported that they conduct an annual  “Innovation Review” of the company’s entire technology development  portfolio for the purpose of informing top leadership’s strategic decisions.  These reviews evaluate Siemens’ technological competitiveness,  strategic resource allocation, and long-term corporate strategy. During  this review, Siemens asks a number of questions, including the following:
Are the overall resource allocations for R&D investments appropriate?
Is the business unit’s technology position competitive, and will  planned investments safeguard the business unit’s technological  competitiveness?
Is there a convincing long-term strategy for how to translate these  investments into sustainable business success?
Is there an adequate strategy for translating new technologies into  winning offerings?
Siemens executives reported that they understand that these factors  directly impact their ability to grow and profit as a company, which is why  the desire to remain technologically competitive drives corporate strategy  decisions.
Siemens also creates forecasting tools called “Pictures of the Future” that  provide graphical representations of how future technologies could be  used by customers 7 to 15 years in the future. Figure 4 provides an  example of the elements that Siemens includes in a Picture of the Future.
According to Siemens, it uses Pictures of the Future to assess societal, technological, and other trends to guide visionary  concepts for potential new markets and customer needs;  consider existing product lines, technologies, and customer needs; analyze the opportunities and risks for the company’s core business;  identify what is required to allow the company to act upon potential  future scenarios.
Most importantly, company representatives stated that these pictures help  develop consensus within Siemens regarding the technologies the  company needs to develop to drive innovation and remain a market  leader.
After leading companies settle upon their corporate strategies for R&D  investment, different units are charged with sponsoring—approving and  funding—incremental and disruptive R&D projects. Incremental R&D  projects are typically sponsored by business units, who are also  responsible for product development. Disruptive R&D is often sponsored  by a corporate research organization, which makes project investment  decisions independently from the business units. Figure 5 shows this  division of R&D that we observed in the private sector.
section_title: Leading Companies Align Incremental Technology Investments with Product Development Goals
paragraphs: Selected leading companies we reviewed align plans for developing new  incremental technologies with plans for developing future products, which  companies sometimes refer to as roadmapping. Individual business units  are responsible for product sales in a company and have their own  executive management teams that are charged with generating profits  from their product lines. Business units sponsor incremental development  projects intended to yield technologies to meet identified customer needs.  Incremental R&D generally adds new capabilities to current products or  next generation versions of existing products; therefore companies expect  these to have lower risk of failure. Depending on the industry, business  units generally do not look beyond a 5-year timeframe when making  decisions about these new technologies due to the unpredictable nature  of the markets in which they operate.
These leading companies document planned future products in product  roadmaps, while the technologies that are to be integrated into those  products are documented in technology roadmaps. By aligning these  plans, business units can better identify and prioritize technology  development investments. To develop these plans, companies solicit  ideas and information from people across the organization to determine  the composition of incremental R&D portfolios. Ultimately, however,  technology development decisions come down to the management’s  qualitative judgements regarding the merits of individual R&D projects, as  well as on quantitative metrics like potential return on investment. Once a  project is approved, it may be immediately funded and executed.
The process and the number of people involved in these R&D investment  decisions vary depending on the company. In general, these leading  companies solicit input from  top leadership responsible for setting the company’s overall strategy  and funding, including the CEO, CTO, and other corporate leaders;  representatives from business units responsible for getting relevant  products to market; and  scientists and technologists who plan for future technology  development and identify when technologies are ready for integration  into products.
Technology and product development teams at Honeywell Aerospace, for  example, complete an annual roadmapping process to align incremental  technology development activities with the company’s product plans. As  part of this process, business units identify customer needs; marketing and product management staff review market trend  information, and determine future products and when they must be  completed; and
Honeywell’s corporate research organization reviews external  technology trends and creates technology roadmaps.
Honeywell Aerospace’s roadmapping process is illustrated in figure 6.
Honeywell’s roadmapping process focuses on needs and trends  projected for the next 10 years, with less emphasis on the more distant  future, due to the difficulty in making reliable predictions that far in  advance. In general, the leading companies we met with consider this  timeframe to be realistic and manageable for planning incremental R&D  investments. To assist with management decisions regarding what  incremental R&D projects to start or change, Honeywell considers  potential revenues in the next 5 years. However, certain technologies,  such as those associated with jet propulsion engines, require longer-term  plans because technology development takes many years to complete.  Although the company’s technology development plans do not extend  beyond the next 10 years, company officials reported they do review  industry trends potentially leading to new developments further in the  future and develop concept ideas for new technologies based on these  trends.
Honeywell obtains input from a variety of sources to ensure technologies  will be feasible for integration into future products and relevant to  customer needs. Technology development plans, or roadmaps, are  periodically revised based on changes in customer needs, prototyping  results not meeting expectations, and other changes in circumstances  requiring additional consideration. According to company representatives,  Honeywell stays in close communication with customers to ensure the  company’s understanding of market needs remains accurate, which helps  them avoid wasting time and money on projects that have lost relevancy.
section_title: Leading Companies Make Decisions to Fund Disruptive R&D Independently from Product Development Plans
paragraphs: Selected leading companies we reviewed also ensure that a portion of  their R&D is independently focused on futuristic concepts, which are  intended to keep the companies competitive in the long term. Disruptive  R&D includes significant technology development efforts addressing the  anticipated customer needs of the future, potentially leading to products  that render the competition’s products irrelevant in the marketplace.
The disruptive R&D portfolio is initiated separately from incremental  portfolios and often managed by a corporate research organization.  Corporate research looks for solutions that provide customers with  capabilities they may not realize they need or want. By separately  organizing disruptive R&D from incremental, companies are able to  protect funds from near-term-focused business unit managers. Generally  speaking, companies in our review ensure that disruptive R&D is planned  and executed by management not averse to taking risks when significant  long-term rewards are possible. Due to their near-term focus, business  unit managers usually begin having significant influence over disruptive  technologies only after they have been demonstrated and are ready to  begin transitioning into products. Allowing exploratory and disruptive  technology development to occur without requiring product development  approval helps prevent the company’s products from becoming obsolete  and gives potential to capture new markets.
These companies use various approaches to leverage the ideas of their  own staff and external partners to innovate for futuristic technologies that  look beyond product roadmaps. These approaches include  challenging R&D staff to come up with feasible ideas to create  disruptive technologies leading to entirely new product lines, and  investing in external startup companies or leveraging externally  developed technologies.
subsections: section_title: Companies Challenge Staff to Develop Ideas for Disruptive Innovation
paragraphs: Companies seek to ensure longer-term competiveness by challenging  their R&D staff to make scientific advances that will make existing  products irrelevant in the future. IBM, for example, issues grand  challenges to R&D staff to develop these kinds of technologies. Company  representatives stated that about 15 to 20 percent of IBM’s R&D funding  goes toward development of disruptive new technologies not aligned with  known customer needs. While all of IBM’s disruptive technology  development is executed by its corporate research organization, IBM  business units provide funding to sponsor these efforts. To generate  ideas for disruptive R&D projects, IBM management issues “Grand  Challenges” asking for project proposals. Figure 7 below provides  information on how IBM’s corporate R&D organization independently  developed the IBM Watson supercomputer as a result of one such Grand  Challenge.
Amazon also provides a number of avenues for individual staff members  to submit innovative R&D project proposals that are not necessarily tied  to defined customer needs or market trends. For example, company  representatives explained that Amazon holds week-long innovation  forums, where R&D staff collaborates to develop new project ideas. R&D  staff then vote on the best ideas, which are submitted to top company  leadership for approval. Individual scientists or technicians can also  propose new projects or ideas directly to their supervisors, who may help  them develop formal proposals.
GM initiates disruptive projects through their corporate research  organization by funding “Internal Startup” projects. These are disruptive  technology development projects initiated solely at the discretion of GM’s  CTO and the director of GM’s R&D labs. Aside from the project teams,  these are the only GM employees that know the details of these projects.  This level of confidentiality allows GM to take risks. GM expects these  projects to have features that develop phenomenal value to customers, make existing technology obsolete, and make the competition irrelevant.
Any R&D staff member can propose this kind of project, although its  potential value to the company must be defined in the proposal. GM also  provides these projects with more funding than typical projects so that  R&D work can progress about three times faster than usual. As these  projects are generally high in technical risk, GM officials stated that they  have about a 50 percent success rate, which they considered acceptable  given the value associated with successful projects.
Leading companies also encourage their scientists to explore and initiate  low-cost research projects, either as unfunded side-projects or using  limited resources following approval from a supervisor. Providing this  flexibility allows scientists to be creative, while also affording them access  to the company’s laboratory or other resources. These less intensive and  inexpensive projects are not required to be approved by senior  management to avoid unnecessary administrative burden with scientists  during exploratory development. If scientists and management deem a  technology to be feasible, it will be referred to the company’s R&D project  approval process for additional funding.
section_title: Companies Maximize R&D Value through External Partnerships
paragraphs: Sometimes leading companies develop technologies that may be useful  in products beyond their preferred markets. In such instances, these  companies seek to maximize the value of their R&D investments through  external partnerships. This may occur through investments in startup  companies or licensing arrangements. For example, Siemens provides  alternate paths for innovative technologies to move into products outside  of their own product lines by co-founding start-up companies that could  turn them into products, thereby allowing Siemens to benefit financially  when technologies they developed are used in other companies’  products. Siemens also licenses some of its technologies to other  companies so they can be used in their products.
Conversely, sometimes leading companies seek technologies from  outside firms that show promise for use in the company’s own products.  The Dow Chemical Company uses a Corporate Venture Fund to make  investments in companies that have formed to commercialize new  technologies. In many cases, Dow assumes a minority position, although  in some cases, Dow may elect to acquire the company or partner without  financial investment to mutually develop technologies. Dow scientists may  work alongside R&D staff from these companies or Dow may just choose  to be an investor. Dow employs technology scouts around the world  responsible for finding opportunities to bring innovative technologies into  Dow. These scouts are responsible for learning everything they can about  a company before Dow proceeds with the investment or partnership.
section_title: R&D Project Teams at Leading Companies Collaborate Closely with Stakeholders to Produce Relevant and Feasible Technologies
paragraphs: At the selected leading companies we reviewed, once an R&D project is  initiated, the research team in charge of the project actively collaborates  with stakeholders outside the R&D office to help assist and inform project  execution efforts. These stakeholders typically include product development staff and engineers who understand technical  requirements for technologies to eventually transition beyond R&D, marketing staff familiar with how products might fit into the outside business unit staff who interface directly with customers, and potential users of the technology.
The level and timing of stakeholder involvement can vary based on the  type of project. Stakeholders, such as product development staff, typically  become involved early in incremental R&D projects. On the other hand,  those same stakeholders might not get involved in disruptive projects until  later phases of development. Collaboration between stakeholders  continues even after the technology development effort concludes and a  business unit begins product development. R&D staff may continue to  assist product development efforts after technology development is  completed as products are customized for different types of customers.  For example, figure 8 describes how a cross-functional team at Dow  Chemical collaborated to develop a new polymer.
Leading companies also look outside the company when undertaking an  R&D project to gain insights from potential customers. These customers  provide input and perspectives that help inform refinements to  technologies. Figure 9 details the process Honeywell Aerospace used to  seek customer input when developing its Synthetic Vision System.
Company representatives explained that input from both internal  stakeholders and potential customers helps R&D staff to transition  emerging technologies into product roadmaps. This collaboration also  helps the R&D project team obtain the requisite information and  resources it needs to develop a technology that is feasible for use as part  of future products, while also being relevant to future customer needs so  it is accepted in the marketplace.
section_title: Leading Companies Tailor R&D Project Review Processes to the Scope of Investment
paragraphs: At the selected leading companies we reviewed, R&D projects are more  rigorously reviewed by higher levels of management as their needs for  staff and funding grow. This leads to a subset of these projects continuing  into later stages of development while others are ended. The Dow  Chemical Company, for example, uses a stage-gate process to oversee  R&D projects requiring significant investments. At each stage of  development the project’s funding increases and the hurdles for moving  forward become greater. Dow requires only minimal oversight for low-cost  exploratory research. Once a scientist proves a technology’s feasibility,  the project enters Dow’s normal processes for integrated project oversight  and portfolio management involving all of the relevant Dow stakeholders.  Later stage development projects generally have higher budgets and are  more closely monitored to ensure they meet specific technical criteria and  time-based milestones, according to company representatives. While  specific review processes vary among leading companies, figure 10  outlines the key principles that all these different processes embody.
IBM also emphasizes relevant stakeholder participation in reviews for  R&D projects progressing beyond exploratory or disruptive corporate  research. After projects are initiated, IBM leadership reviews R&D  projects at least quarterly during business reviews, although company  representatives noted they may be reviewed more frequently in some  cases when more attention is warranted. IBM leadership believes— according to representatives we interviewed—that it must have agile  review processes that facilitate timely adjustments or, in certain cases,  terminations to projects.
section_title: Leading Companies Prove Technologies Work before Integrating Them into Products
paragraphs: The selected leading companies we reviewed consider technology  demonstrations, or prototyping, an inherent part of R&D and use  demonstrations for a variety of purposes, including to  create demand by convincing stakeholders or customers of the  potential value of a future technology; and obtain feedback from potential end-users to add knowledge and  improve technologies.
Both incremental and disruptive R&D projects receive funds to  demonstrate technologies. Figure 11 depicts the technology  demonstration process of the leading companies.
Leading companies demonstrate concepts for new disruptive  technologies to stakeholders to generate demand so business units will  contribute to technology development. Specifically, once corporate  research organizations believe technology components are sufficiently  mature for product development, they offer demonstrations to show their  potential value to stakeholders in the business units. For example,  Siemens use prototypes once a technology reaches the point that its  components can be validated in a laboratory or relevant environment.
Siemens representatives explained that should the demonstration prove  successful, a business unit assumes development responsibilities.
Leading companies also demonstrate these concepts for disruptive  technologies to potential customers if business units are hesitant to invest  in further development. For example, Qualcomm corporate research  representatives told us they must sometimes overcome internal  resistance to accepting new technologies by demonstrating their value to  both internal and external customers. Barriers to transition of disruptive  technologies may be even more prevalent when developing technologies  that could lead to dropping an existing product or feature. In their role as  a R&D component, not a business unit focused on product lines,  corporate research works with both internal and external customers  without committing to future products but with a clear pathway to adoption  if successful. Qualcomm representatives explained that if these early  demonstrations and advocacy for disruptive technologies proves  successful, then customers may ask product developers to use them in  future products.
subsections: section_title: Feedback from Demonstrations Add Knowledge and Improve Technologies
paragraphs: Once a concept is proven, leading companies use technology  demonstrations to inform developers how a technology needs to be  improved. Company representatives explained that rapidly developing  and demonstrating a series of iterations of a new technology provides  early opportunities for improving the technology, rather than taking longer  amounts of time to develop technologies without testing them. Figure 12  depicts the iterative technology development and demonstration process  that leading companies use.
When Amazon pursues a product or technology, the company already  has an idea of its customers’ needs, but does not consider this  information to be complete in the absence of user feedback from  prototype demonstrations. To ensure technologies address these needs,  Amazon representatives stated that the company builds iterations of  prototypes during technology development in a facility specifically  designed for doing so quickly and at minimal cost. Figure 13 illustrates  how these early prototypes are generally used by Amazon employees in  real world settings to help inform technology development.
Similar to Amazon, IBM’s development method uses prototyping during  technology and product development. IBM developers show early  iterations of new technology to customers, improve the design based on  feedback, and then produce another prototype after that. This process  repeats until a releasable product is completed. IBM representatives  found that development models with longer sequential steps and a single  deliverable are less useful than faster paced and smaller deliverables  under shorter timeframes.
Companies may also use external demonstrations of maturing  technologies to prove they work in realistic environments. For example,  Valvoline produces and tests small volumes of new motor oil formulas in  its own laboratory or at a few select customers’ facilities. To obtain  customer insights from prototyping, Valvoline develops data and uses cameras on test engines to demonstrate the uses customer test engines to conduct tests and provide specific data  to customers on how new formulas perform in their engines; and provides test formulas to outside customers and obtains feedback  through sales staff and focus groups.
Offering external customers an opportunity to test new technologies  provides important feedback for companies such as Valvoline. By taking  steps like these to involve outside customers in prototype testing,  representatives of leading companies stated they are able to identify a  new technology’s tangible benefits, while also encouraging eventual  customer acceptance of new products.
section_title: DOD Implements Some Practices Used by Leading Companies, but Policy Limitations and Culture Hinder Adopting Others
paragraphs: While some DOD S&T practices closely mirror those of the selected  leading companies we reviewed, DOD’s funding policies and culture limit  its ability to adopt other practices for managing its S&T investments.  Unlike the companies we reviewed, DOD does not organize and fund  incremental and disruptive innovation separately. Nor does its leadership  provide guidance on or assess how these innovation investments should  be or are mixed. Instead, S&T officials explained that DOD labs face  pressures to prioritize near-term requirements at the expense of  potentially disruptive technologies. As a result of DOD funding policies,  projects are planned 2 years in advance, which can slow innovation and  limit lab directors’ autonomy to initiate work. While Congress has provided  authority that, as implemented, has enabled the military department lab  directors to initiate work outside of the normal lengthy process, DOD has  not fully utilized these flexibilities. Additionally, we found that divided  responsibilities for technology versus product development contribute to a  culture that does not encourage collaboration between DOD’s S&T and  acquisition communities and limits the S&T community’s ability to conduct  advanced prototyping. These issues are not insurmountable, however, as  demonstrated in pockets of each military department. In recognition of  these and other issues, Congress has required that DOD create a new  Under Secretary of Defense for Research and Engineering charged with  advancing defense technology and innovation and establishing policies  on technology development, prototyping, and experimentation, among  other responsibilities, by February 2018.
subsections: section_title: Some DOD Practices for Managing and Executing S&T Investments Closely Resemble Practices Used by Leading Companies
paragraphs: Some of DOD’s practices for managing and executing S&T investments  closely resemble those employed by the selected leading companies we  reviewed. DOD has a corporate research organization for disruptive  innovation and its leadership defines S&T strategies to guide investments  which are consistent with elements that the companies we met with used  to manage investments in incremental and disruptive portfolios. DOD  project oversight is scaled based on the scope of investment, which  aligns closely with leading company practices.
While differences exist between how these practices are implemented in  DOD and at the companies we reviewed, we found that the outcomes are  the same.
The Defense Advanced Research Projects Agency (DARPA), for  example, closely resembles the corporate research organization that  many leading companies employ to foster disruptive innovation. In the  President’s budget submission for fiscal year 2017, DARPA requested  $2.9 billion, about 23 percent of DOD’s S&T budget request. Similar to  a company’s corporate research organization, DARPA’s projects are  generally not tied to existing DOD weapon systems or a specific  military department requirement. Instead, their mission is to produce  disruptive innovation that could support any military department. The  DARPA Director makes all funding decisions and project  prioritizations.
DOD’s market research, which informs its S&T strategy, is based on  near and far-term adversarial threats, capability needs, and warfighter  requirements. While these inputs may differ from the companies we  reviewed, they are likewise used to prioritize S&T projects.
DOD does not necessarily use the same metrics as companies to  evaluate projects, but its labs similarly scale the scope of their project  reviews based on the maturity of the technology and scope of  investment. For example, a lab typically reviews basic research  projects once a year, while officials said more mature, larger  investments are reviewed multiple times per year by the lab, its  customers, and military department leadership. Similarly, both DOD  and the companies we reviewed assess projects based on their cost,  schedule, and technical performance requirements. One inherent  difference is that leading companies are concerned with potential  financial returns on investment, whereas DOD prioritizes for other  reasons, such as whether the technology carries potential to reduce  risk to the warfighter.
section_title: Military Departments Do Not Define in Strategy or Assess the Mix of Investments in Incremental and Disruptive Innovation
paragraphs: Although our review of selected leading companies found that they define  in strategy their annual mix of investments in incremental and disruptive  innovation, the military departments do not do this, nor do they assess  such a mix. The office of the ASD(R&E)—the Office of the Secretary of  Defense (OSD) organization responsible for establishing DOD S&T policy  and guidance—does not provide guidance to the military departments on  the mix of incremental and disruptive S&T investments. The military  departments are responsible for defining their own S&T strategies,  formulating and managing budgets, and developing technologies. Their  S&T strategies, however, do not define the mix of incremental and  disruptive investments each department should make annually. Instead,  DOD S&T investments are organized and funded based on budget  activities (BA) that reflect stages of technology maturity. DOD uses this  approach, in part, because the Financial Management Regulation dictates  how R&D activities are identified for the purposes of budgeting. We found  that one limitation of funding under BAs is a lack of visibility into whether  individual projects that labs and research centers invest in are geared  toward disruptive or incremental innovation. The Financial Management  Regulation, however, does not preclude DOD from developing investment  targets for both incremental and disruptive R&D.
Military department lab and center officials we interviewed, however,  identified certain projects they were working on that could lead to  disruptive technologies. Officials from these labs and centers  acknowledged that they struggle to determine the right balance between  disruptive and incremental innovation projects. They expressed concern  that military department leadership responsible for setting requirements  for and approving S&T spending, at times, are more focused on near- term, less risky, more incremental types of innovation investments at the  expense of long-term, disruptive innovation.
The Navy is one military department that has taken steps to ensure  funding for some investments in disruptive innovation. The Navy  organizes S&T investments around “strategic buckets” to ensure it  maintains investments in both near- and long-term projects and protects  funding for potentially disruptive projects. The distribution of resources is  determined by senior leadership based on the Navy’s S&T strategy. The  strategy maps out roughly the minimum percentage of funding that the  Navy plans to request for high-priority, disruptive projects within its S&T  portfolio, as reflected in figure 14 below. In fiscal year 2017, the Navy  plans to invest more than $313 million for Leap Ahead Innovations which  are intended to be disruptive technologies and deliver transformational  warfighting capabilities. These are in addition to investments in other  disruptive technologies that are categorized, but not quantified, under its  other strategic buckets.
In a June 2017 report, we recommended that DOD take steps—such as  the Navy’s—to help ensure adequate investments in innovation that align  with DOD-wide strategy to overcome the department’s risk-averse culture  and pressures to focus on near-term projects.
section_title: DOD’s Lengthy S&T Project Planning and Budgeting Cycle Can Slow Innovation While Existing Flexibilities to Expedite Have Been Underused
paragraphs: In comparison to the practice at selected leading companies we reviewed,  which annually align their investments to product goals, DOD’s process  for prioritizing and funding projects takes longer—almost 2 years to  complete—which we found can slow innovation. Like every other good  and service DOD acquires, all S&T investments must follow DOD’s  planning and budgeting policy. This policy is underpinned by DOD’s  Planning, Programming, Budgeting, and Execution (PPBE) process. The  PPBE process for S&T investments includes the following stages:
Planning: DOD leadership, in guidance and planning documents,  identifies strategic priorities, weapon system requirements, and  adversarial threats. Collectively, these serve as DOD’s broad  requirements for technology development.
Programming: S&T organizations give consideration to those  requirements and propose technology development projects to  address them. Proposed projects and associated costs are  documented in Program Objective Memorandums (POM). Each  organization is tasked with determining which projects to propose in  the POM, while maintaining balance across their portfolios of  investment, as well as maintaining an appropriate mix of funding  based on BA. POM documents are reviewed by senior officials across  DOD—including those responsible for setting requirements and the  budget—who also have a role in prioritizing S&T investments.
Budgeting: Each S&T organization’s POM is used to formulate their  respective military department’s Budget Estimate Submission (BES),  which outlines the total funding needed, including how much it will  need by budget activity. After the President’s budget is submitted,  Congress enacts an appropriation. Once funds are appropriated, each  S&T organization is provided funding for the projects approved in the  POM and BES.
Execution: S&T organizations carry out funded projects.
Figure 15 illustrates the notional time frames for DOD’s PPBE process.
In total, it can take almost 2 years from the time a project is proposed in  the POM to the time it is funded. In contrast, the companies we reviewed  reported that they planned projects in the same year they were executed,  which helped them quickly respond to leaps in technology development.  S&T officials we met with stated that the 2-year project planning process  reduces their ability to be as nimble as the companies with whom we met.  For example, if an unexpected technology breakthrough is identified  through 6.1 or 6.2 research, the labs may have to wait up to 2 years  before they may begin work on a follow-on project. DOD S&T executives  expressed the need for greater flexibility with initiating new projects  because the pace of technology development can be rapid and planning  for S&T spending 2 years in advance can hinder innovation. They stated,  however, that the PPBE process provides Congress with the information  it needs to maintain oversight and ensure DOD meets its fiduciary  responsibilities to the taxpayer.
We found that laboratory and research center directors in the military  departments have less authority to initiate S&T work that is not directly  linked to defined near- or far-term capability needs as compared to the  leading companies we reviewed. While the Director of DARPA approves  every project the agency undertakes and is not beholden to address  defined requirements, the military departments’ labs and centers do not  control all of the S&T-related R&D work they perform annually. These  labs and centers regularly undertake work on behalf of acquisition  community customers, such as a major defense acquisition program, who  provide funding in support of the project. This work comes in addition to  “direct funded” projects approved and funded to the lab or center through  PPBE that are intended to address S&T requirements outlined in strategy.  As a result, both the direct funded and customer-funded projects compete  for lab resources, such as staff, and must be balanced. In fiscal year  2015, for example, direct funded projects accounted for 19 percent of  Naval Research Laboratory’s (NRL) $1.2 billion of funding. The other 81  percent was customer-funded work from Navy, other DOD, or other  governmental sources. Despite having direct funding, the POM review  and approval process may constrain which projects are ultimately funded.  For example, we found that the projects a lab or center proposes in its  POM submission may be reviewed by as many as four different  organizations before it is submitted to OSD. It is during this review  process that lab officials explained that the culture within DOD is, at  times, to focus on near-term, customer-driven projects, at the expense of  far-term disruptive projects.
Regardless of the source of the funding, a senior ASD(R&E) official  explained that S&T investments are intended to address some defined  capability need. This means that the military department’s disruptive  technology projects are roadmapped to requirements, which differs from  the practice at the leading companies we reviewed. This may limit the  labs’ ability to address undefined customer needs through other  potentially disruptive technologies.
Section 219 of the Duncan Hunter National Defense Authorization Act for  Fiscal Year 2009, as implemented, has provided defense lab directors  with some limited flexibility to initiate S&T projects, including those that  are not roadmapped to defined requirements, outside of the normal 2- year planning process. Specifically, as amended, the law directs the  Secretary of Defense, in consultation with the Secretaries of the military  departments, to establish mechanisms under which the director of a  defense laboratory may use an amount of funds equal to a certain  percentage of all funds available to the laboratory for the following  purposes:  innovative basic and applied research that is conducted at the  defense laboratory and supports military missions; developing programs that support the transition of technologies  developed by the defense laboratory into operational use; workforce development activities that improve the capacity of the  defense laboratory to recruit and retain personnel with needed  scientific and engineering expertise; and  revitalization, recapitalization, or minor military construction  of laboratory infrastructure.
While this authority directed the creation of a mechanism that may  provide lab directors with the means to fund projects they consider to be a  priority, the military departments have not maximized their use of these  authorities. Until the passage of the National Defense Authorization Act  for Fiscal Year 2017, the director of a defense lab could use funds equal  to not more than 3 percent of all funds available to the defense laboratory  for S&T activities. Each of the services has unique strategies for  executing section 219 authorities but DOD reported that the full 3 percent  of funds available to the labs has not been used. DOD officials told us  that the full 3 percent available to each defense laboratory has not been  used for a number of reasons, including due to competing S&T funding  priorities. Additionally, DOD officials indicated that labs had concerns  about charging customers a fee to fund such S&T activities, which was a  factor in different amounts of funds available for section 219 purposes, as  shown in figure 16.
During our review, Congress amended the Section 219 authority in the  National Defense Authorization Act for Fiscal Year 2017 to permit lab  directors to use an amount of funds not less than 2 percent but not more  than 4 percent of all funding available to the lab. As a result of the  change, the military departments can increase the amount of section 219  funds that the labs may obtain. The extent to which they choose to fund  new projects could help them to initiate projects, including those which  may be “off-roadmap,” faster than through the PPBE process.
section_title: Cultural Barriers That Limit Collaboration between DOD’s S&T and Acquisition Communities Are Not Insurmountable
paragraphs: While the selected leading companies we reviewed ensure close  collaboration between stakeholders in technology and product  development, cultural barriers have limited such collaboration within  DOD. DOD’s funding policies reflect cultural barriers to collaboration  between the S&T community and its product development stakeholders.  Under DOD’s funding model, the labs are responsible for technology  development associated with BAs 6.1 through 6.3, while the stakeholders  in the acquisition community are traditionally responsible for product  development, which begins with prototype activities under BA 6.4.  Although we found that S&T projects funded by the acquisition community  obtain collaborative input from those same eventual customers, this was  not the case for direct-funded projects—those initiated by the lab. Lab  officials explained that for direct-funded projects, they may consult with  potential customers in the acquisition community to gauge interest before  starting a project or to present results after it is complete, but those  customers are not part of the development team. This approach is, in  part, attributable to these organizations being separate—both in mission  and in the type of funding they receive. For example, unlike the  companies we reviewed, S&T officials explained that they do not  transition scientists and engineers along with the technologies they  developed to the acquisition programs. The companies we reviewed  reported that they set the expectation that both technology and product  development staff work together. DOD, however, has not established a  formal policy on how these two communities should collaborate on  projects to overcome these cultural barriers.
DOD has processes to help its research labs and centers collaborate on  S&T work, but these processes do not emphasize collaboration between  the S&T and acquisition communities. In 2014, ASD(R&E) revitalized its  “Reliance 21” framework—a joint planning and coordination process that  is intended to ensure DOD’s S&T community provides solutions and  advice to the departments’ senior-level decision makers, warfighter,  Congress, and other stakeholders. This is to be accomplished, in part,  through groups of technical experts organized around 17 technical areas  referred to as Communities of Interest (COI). Originally formed in 2009,  COIs provide DOD with a mechanism for experts in technical areas, such  as cyber or space, to coordinate and communicate what S&T-related  R&D each military department is working on and identify areas for  collaboration. Each COI is supposed to conduct a portfolio review every 2  to 3 years—depending on the technical area—to assess those gaps and  their impacts and to make recommendations to the S&T Executive  Committee. We found that each COI has documented at least one such  assessment since 2014. According to an ASD(R&E) official, each COI  also provides updates to the S&T Executive Committee during its annual  S&T strategy meeting.
DOD’s S&T executives stated that there is a need to improve how and  when the acquisition community is brought in to contribute. One S&T  executive pointed out that bringing in external stakeholders earlier in the  process is a way to facilitate disruptive innovation. They described one  instance in which the Air Force overcame existing cultural barriers to  collaboration by funding technology maturation efforts with both S&T and  acquisition community support, as described in figure 17. This different  approach was the result of a deliberate decision by the Air Force to foster  more collaboration between the S&T community and other stakeholders  to increase the odds that new technologies end up in the hands of the  warfighter, according to officials.
section_title: Military Department Labs Traditionally Funded Technology Development through Initial Prototypes, but Recent Initiatives Have Begun to Fund Advanced Prototyping
paragraphs: While selected leading companies we reviewed provided funding for  prototypes during technology development, DOD has only recently begun  to fund advanced prototyping efforts within the labs. The acquisition  community, as opposed to the S&T labs, traditionally bears the  responsibility of maturing technology through advanced prototyping,  which is in contrast to how the companies with whom we met operate.  DOD’s S&T labs and centers typically do not control the BA 6.4 funding  that would allow them to conduct such prototyping. We found that the  S&T community typically matures technologies to, at most, a prototype  that is close to final form, fit, and function and tested in a relevant  environment. This creates strong incentives for S&T project teams to  identify technology transition partners in the form of major acquisition  programs early in development, which may ultimately restrict disruptive  innovation and push S&T projects to be more incremental to satisfy  potential customers’ near-term needs. For example, applied research  project proposals at the Naval Research Lab and the Army’s Engineer  Research and Development Center both identify and consider potential  transition partners as part of the selection criteria, regardless of whether it  was for incremental or disruptive technologies. As we previously stated, it  is difficult to transition a technology or identify a partner when the  technology is disruptive. Companies recognized this and funded  disruptive technology development projects through demonstration to  help obtain a customer.
In June 2017, we reported that DOD’s approach to prototyping  contributes, in part, to DOD’s broad challenges with transitioning  technology from the labs into the hands of the warfighter. We further  reported that prototyping that is not directly tied to acquisition programs  can be seen as a way to “test the waters” because it does not require the  level of commitment associated with starting acquisitions.
Each military department has recently undertaken efforts to fund more  advanced prototypes for incremental and disruptive technologies in S&T  labs. For example, since 2012, the Army has used funding typically  associated with acquisition programs to conduct higher-fidelity prototyping  and further mature technology outside of those programs through its  Technology Maturation Initiative. In the President’s fiscal year 2017  budget submission, the Army requested approximately $70 million for  these efforts.
In May 2016, the Air Force established the Strategic Development  Planning and Experimentation Office, in part, to run Air Force  experimentation initiatives to achieve specific technology development  objectives. Air Force officials explained that the intent of these initiatives  is to develop more agile approaches to innovation by creating a learning  organization that  can rapidly take new innovative approaches,  can quickly initiate new projects and is not hampered by the traditional  2-year planning process,  is composed of acquisition and S&T representatives to promote  collaboration, and  conducts prototyping and demonstrations without a direct requirement  from an acquisition community stakeholder to reduce risk and mature  technologies.
In the President’s fiscal year 2017 budget request, the Air Force  requested $62 million to fund these experimentation efforts. Currently,  two technology development areas are addressed through  experimentation initiatives and two more are planned. According to Air  Force officials, these initiatives reflect its leadership’s desire to embrace a  culture of encouraging and formulating innovative strategic choices  independent of major weapon system acquisition programs.
The Department of Navy is pursuing similar efforts through its Rapid  Prototyping, Experimentation, and Demonstration (RPED) initiative.  RPED projects use prototyping to rapidly develop and assess new  technologies and engineering innovations to address priority naval  warfighting needs. The Navy expects that RPED projects will assist in developing new capability concepts,  informing and refining requirements, addressing priority needs by demonstrating, and enabling quicker transition of technologies to naval programs.
The Navy developed its policy for RPED projects in December 2016 and  requested $40 million to fund projects in the President’s fiscal year 2017  budget submission.
In our June 2017 report, we recommended that DOD develop a strategy  to better coordinate and communicate the goals of these and other  prototyping efforts to ensure these efforts gain traction and achieve  success.
An October 2013 Defense Science Board report reinforces the military  departments’ focus on experimentation as an innovation enabler. The  Defense Science Board found that DOD cannot continue to rely on  technological superiority unless it adopts methods that allow it to  anticipate, assess, and gain experience with new technological  capabilities before its potential adversaries do.
section_title: DOD and Commercial Firms Operate under Different Incentives and Organizational Structures
paragraphs: DOD’s organizational structure and incentives contribute to why it does  not fully implement the S&T management practices that the selected  leading companies we reviewed follow. This includes DOD’s budget  environment, funding model, and the manner in which DOD is organized  to execute technology development. As we have previously reported, the  critical differences between the environments and cultures of private  companies and DOD must be recognized before tangible progress can be  made in establishing more efficient practices in S&T management.  Further, we concluded that changing the mechanics of the processes,  without changing the environment that determines incentives, may not  produce better outcomes. Specifically:
Companies operate in an environment where profitability is a constant  business imperative. As such, leading companies we reviewed  devoted a portion of their R&D investments toward futuristic concepts,  which are intended to keep them competitive in the long-term, instead  of just near-term products. Disruptive R&D includes significant  technology development efforts addressing the anticipated customer  needs of the future, potentially leading to products that render  competitive products irrelevant. By separately organizing disruptive  R&D from incremental, companies are able to protect funds from the  near-term focused business unit managers. In the DOD environment,  budget pressures and urgent requirements often drive military  departments to focus on near-term needs over long-term innovation.  For instance, a 2016 Air Force Studies Board report found that in  much of the Air Force, little or no space for innovation exists.  Because innovation is focused on future needs, the report found that  Air Force organizations decide they can wait on addressing the needs  of tomorrow. The report found that across the Air Force as a whole,  insufficient processes existed to support “rapid-cycle” innovation with  the same intensity and pace Air Force personnel regularly bring to  bear to fulfill other missions.
Companies fiercely compete with one another for customers and in  ever-changing market conditions. This environment requires agility in  how they direct their technology and product investments, which  includes rapidly initiating new projects and truncating underperforming  ones. DOD, on the other hand, operates under different conditions. Its  budget environment may incentivize starting and sustaining programs  rather than discontinuing underperforming ones. In April 2014, we  found that budgets to support major acquisition program commitments  must be approved well ahead of when the information needs to  support the decision is available. DOD’s S&T community operates  under similar pressures and incentives as its acquisition community.  In this environment, we found that it is easier to sustain a program  until its funding expires, even if technical performance is lacking.  According to DOD S&T officials, current budgeting and funding  processes restrict, rather than encourage, innovation. DOD S&T  executives told us that they want more flexibility outside of the  cumbersome 2-year PPBE process to initiate and discontinue  projects.
Companies set their own budgets internally for various activities,  including R&D. Conversely, as a government agency, DOD can  influence, but not set, its annual budget. Ultimately, Congress  determines what level of funding to appropriate DOD, including for  S&T-related activities.
section_title: New Office to Be Responsible for Promoting Innovation and Establishing S&T Policies
paragraphs: Overcoming many of these challenges may ultimately be the  responsibility of the yet-to-be-created Office of the Under Secretary of  Defense for Research and Engineering (USD(R&E)). The National  Defense Authorization Act (NDAA) for Fiscal Year 2017 calls for the  establishment of the position of USD(R&E) to serve as the CTO and  elevate and enhance the mission of defense technological innovation.  This office will have greater responsibilities than the current ASD(R&E)  and will focus on innovation, oversight, and policy for defense research  and engineering, technology development and transition, prototyping and  experimentation, and testing activities. Specifically, where ASD(R&E) has  taken a more hands-off approach to developing S&T policy, Congress  legislated that the new office take a larger role in establishing policies to  overcome the challenges DOD currently faces with promoting innovation.  The USD(R&E) will also be responsible for the allocation of resources for  defense research and engineering, and unifying these efforts across  DOD. The fiscal year 2017 NDAA requires this position to be created by  February 2018. In March 2017, DOD reported that it would submit final  plans for creating this position to Congress no later than August 1, 2017,  as required by law.
section_title: Conclusions
paragraphs: DOD’s S&T investments are key to maintaining our nation’s technological  superiority over our adversaries. Congress has raised questions about  DOD being innovative enough to maintain future technology superiority.  The business imperatives that world-class technology companies must  operate under force them to manage their S&T portfolios and projects to  produce better outcomes for evolving current products, as well as well as  develop disruptive technologies for the future. Leading companies have  shown they do this by organizing and funding R&D to avoid the pressures  to focus on incremental innovation at the expense of maintaining their  technological edge in the future. With its focus on meeting warfighter  needs, DOD does not operate under similar business imperatives; it has  not, at a department-wide level, emphasized the need to invest in  disruptive technologies. Instead, each military department’s S&T  organizational construct and funding processes increase emphasis on  investing in technologies that will support the near-term requirements of a  major weapon system acquisition program at the expense of investing in  innovative technologies that are not linked to a requirement.
As DOD determines the roles and responsibilities for its new Under  Secretary of Defense for Research and Engineering, it is uniquely  positioned to rethink its policies that govern technology development.  While it may not be practical for each military department to organize its  technology development as leading companies do, there are pockets  within each department that are implementing some aspects of leading  company practices. However, more needs to be done to facilitate more  systematic adoption of these practices across DOD. Doing so can  position DOD to develop more innovative, disruptive technologies. By not  taking steps to ensure the right balance of incremental and disruptive  technology investments, DOD lacks visibility into whether the  technologies it is developing will provide superior capabilities to counter  future and emerging adversarial threats. Additionally, the limited  collaboration with product developers, limited use of existing flexible  approaches to fund S&T projects outside of the 2-year planning process,  and limited advanced prototyping of new technologies by the labs creates  added barriers to innovation.
section_title: Recommendations for Executive Action
paragraphs: To ensure that DOD is positioned to counter both near and far term  threats, consistent with its S&T framework, we recommend that the  Secretary of Defense direct the new Under Secretary of Defense for  Research and Engineering annually take the following two actions: define the mix of incremental and disruptive innovation investments  for each military department, and assess whether that mix is achieved.
To ensure that DOD is positioned to more comprehensively implement  leading practices for managing science and technology programs, we  recommend that the Secretary of Defense direct the new Under Secretary  of Defense for Research and Engineering to define, in policy or guidance,  an S&T management framework that includes the three following actions: emphasizes greater use of existing flexibilities to more quickly initiate  and discontinue projects to respond to the rapid pace of innovation;  incorporates acquisition stakeholders into technology development  programs to ensure they are relevant to customers; and promotes advanced prototyping of disruptive technologies within the  labs so the S&T community can prove these technologies work to  generate demand from future acquisition programs.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the DOD for review and comment.  DOD’s written comments are reprinted in appendix III of this report and  summarized below.
In its comments, DOD did not concur with each of our recommendations,  citing that it is premature to get ahead of the Secretary of Defense’s final  decisions on the role of the new Under Secretary of Defense for  Research and Engineering (USD(R&E)) until that position is established,  which is required by no later than February 1, 2018. We believe, however,  as the roles and responsibilities of the USD(R&E) are in the process of  being deliberated, that it is appropriate and timely for the Secretary of  Defense to ensure that the USD(R&E) be responsible for implementing  our recommendations. Although it did not concur, DOD identified actions  that it could take that are generally responsive to our recommendations.
Specifically, in response to our recommendations that the Secretary of  Defense direct the USD(R&E) to define and assess the mix of  incremental and disruptive innovation investments for each military  department, DOD stated that it would need to coordinate with each  military department to establish appropriate goals for those investments.  DOD further noted that it could assess whether that mix is achieved  during its annual S&T Strategic Overview meeting. We continue to  believe that such actions are necessary to ensure that DOD is positioned  to counter both near and far term threats.
In response to our recommendation that the USD(R&E) define an S&T  framework that emphasizes greater use of flexibilities to more quickly  initiate and discontinue projects to respond to the rapid pace of  innovation, DOD identified the Laboratory Quality Enhancement Program  as an activity to leverage existing flexibilities. This program—which DOD  implemented in response to the National Defense Authorization Act for  Fiscal Year 2017—requires DOD to create panels of experts to make  recommendations to the Secretary of Defense on matters related to S&T  policy and practices. DOD, however, did not explain how this program  would help the labs make greater use of existing flexibilities to initiate  projects, such as those granted under Section 219 of the Duncan Hunter  National Defense Authorization Act for Fiscal Year 2009. We continue to  believe that greater use of existing authorities, such as those provided  under Section 219, could help labs to more quickly initiate projects  outside of the normal planning cycle, which can take nearly two years for  a project to be funded.
In response to our recommendation that the USD(R&E) define an S&T  framework that incorporates acquisition stakeholders into technology  development programs, DOD identified that it expects the USD(R&E) to  provide policy and guidance that will include increased engagement with  acquisition stakeholders. We continue to believe that enhancing  collaboration between the S&T and acquisition communities is critical to  ensuring that technologies in development will be relevant to potential  customers.
In response to our recommendation that the USD(R&E) define an S&T  framework that promotes advanced prototyping of disruptive technologies  within the labs, DOD noted the benefits of prototyping and that it is a  critical piece of the larger research and engineering strategy. It did not,  however, identify if any such strategy would be revised to promote earlier  prototyping so the S&T community can prove technologies work and  generate demand from future acquisition programs. We continue to  believe that establishing an S&T framework that emphasizes prototyping  outside of acquisition programs is needed.
Additionally, in response to our recommendations that the Secretary of  Defense direct the USD(R&E) to define the three elements above in an  S&T management framework, DOD also noted that Reliance 21 is  expected to continue serving as the overarching framework for the S&T  joint planning and coordination process. We continue to believe, however,  that this framework does not fully address our recommendations and that  further actions, such as those they outlined above, are necessary for  DOD to ensure it is positioned to more comprehensively implement  leading practices for managing S&T.
We are sending copies of the report to the appropriate congressional  committees; the Secretary of Defense; the Under Secretary of Defense  for Acquisition, Technology and Logistics; and the Secretaries of the  Army, Navy, and Air Force; and to the eight leading companies we  interviewed about their practices for this report. In addition, the report will  be available at no charge on GAO’s website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at 202-512-4841 or sullivanm@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: We used a case study approach to identify leading commercial  companies’ research and development (R&D) practices. We selected and  visited eight large (Fortune 500-listed) companies that were U.S.  companies or equivalent foreign companies, or were owned by those  companies. Our primary goal was to select large companies—those more  comparable with the size of the Department of Defense (DOD) than  smaller ones—from a range of different industries, that were profitable  and that received two or more industry awards or other recognition for  technology innovation since 2014.
We used the following sources to identify companies that have received  awards or other recognition: Boston Consulting Group, PwC, MIT  Technology Review, American Business Awards Gold Stevie awards, and  Thomson-Reuters. These organizations either provide annual lists of  leading innovator companies or select top innovator companies for  awards on an annual basis. These groups are positioned to be  knowledgeable regarding who the “leading companies” are for the  purposes of the case study selection for our review. We used available  corporate stock information at Morningstar.com to determine whether a  company has been profitable, which is an indicator of their degree of  success in their science and technology development efforts. Below are  descriptions of the eight companies featured in this report.
Amazon.com sells consumer electronics, operates retail websites  serving over 100 countries, and provides cloud-computing services to  hundreds of thousands of organizations in 190 countries around the  world. Lab126 is Amazon’s inventive research and development  company that designs and engineers high-profile consumer  electronics, including Kindle Fire tablets, Fire TV and Amazon Echo.  Amazon’s recent recognitions include being included among the  Boston Consulting Group’s (BCG) “Most Innovative Companies,”  PwC’s “10 Most Innovative Companies,” Thomson-Reuters’s “Top 100  Global Innovators” and MIT Technology Review’s “50 Smartest  Companies.”
Ashland Global Holdings, Inc. is a global chemicals company serving  customers in a wide range of consumer and industrial markets,  including adhesives, architectural coatings, automotive, construction,  energy, food and beverage, personal care, and pharmaceutical.  Valvoline Inc., a leading producer and retailer of automotive lubricants  that made the first trademarked American motor oil, was an Ashland  subsidiary at the time we met with Valvoline company representatives.  Ashland currently owns a controlling interest in Valvoline after it  became a separate public company in 2016. Ashland’s recent  recognitions include a Bronze Innovation Zone Functional Ingredient  Award and Ringier Coatings Technology Innovation award and a  Composites and Advanced Materials Exposition award for  “Unsurpassed Innovation.”
The Dow Chemical Company delivers a broad range of technology- based products and solutions to customers in 175 countries. Dow  drives innovations that extract value from material, polymer, chemical  and biological science to help address many of the world’s most  challenging problems, such as the need for fresh food, safer and more  sustainable transportation, clean water, energy efficiency, more  durable infrastructure, and increasing agricultural productivity. Dow’s  recent recognitions include receiving multiple R&D 100 awards, and  being included among BCG’s “Most Innovative Companies” and  Clarivate Analytics (formerly Thomson-Reuters’s) “Top 100 Global  Innovators.”
Honeywell International, Inc. invents and commercializes technologies  that address some of the world’s most critical challenges around  energy, safety, security, productivity and global urbanization.  Honeywell’s Aerospace division, which is discussed in this report, is a  leading provider of aircraft engines, integrated avionics, systems and  service solutions, and related products and services for aircraft  manufacturers, and turbochargers to improve the performance and  efficiency of passenger cars and commercial vehicles. Honeywell’s  recent recognitions include receiving American Business Awards for  “Most Innovative Company” and “Most Innovative Technology  Company,” and being included among Thomson-Reuters’s “Top 100  Global Innovators.”
General Motors Co. and its partners produce vehicles in 30 countries,  including the Chevrolet, Cadillac, Baojun, Buick, GMC, Holden,  Jiefang, Opel, Vauxhall and Wuling brands. GM develops innovative  new technologies offering vehicle electrification, autonomous driving,  vehicle health management, and alternative fuel usage. GM’s recent  recognitions include an Edison Award for Automotive Computing, an  Automotive News “Pace Award,” and being included among Fast  Company’s “World’s 10 Most Innovative Companies of Automotive.”
International Business Machines Corporation (IBM) develops and  markets cognitive systems, or computers that learn through  interactions with people and data, as well as enterprise systems and  software. IBM also provides cloud computing, consulting and  information technology implementation services. IBM’s recent  recognitions include receiving an R&D 100 award, and being included  among BCG’s “Most Innovative Companies” and MIT Technology  Review’s “50 Smartest Companies.”
Qualcomm is a leader in the commercialization of digital  communication technologies, including Code Division Multiple Access  (CDMA), and Long Term Evolution (LTE), for cellular wireless  communication applications. They also develop and commercialize  numerous technologies used in handsets and tablets. They also own  intellectual property contributing to other commercial technologies like  wireless local area network, global positioning system, near field  communication, and Bluetooth. Qualcomm’s recent recognitions  include receiving an R&D 100 award, and being included among MIT  Technology Review’s “50 Smartest Companies” and Thomson- Reuters’s “Top 100 Global Innovators.”
Siemens AG is one of the world’s largest producers of electrification,  automation, and digitalization technologies. Siemens’s products  include gas, steam, and wind turbines, integrated power plant  solutions, power grid systems, building technologies, rail technologies,  medical imaging and diagnostics, and other systems for industrial use.  The company’s recent recognitions include receiving two R&D 100  awards and being included among BCG’s “Most Innovative  Companies.” Siemens is also category leader in the Dow Jones  Sustainability Index ranking, with 100 out of 100 points for innovation  management.
For each of the companies, we conducted semi-structured interviews with  senior management officials and other company representatives  knowledgeable about research and development activities to gather  consistent information about processes and practices companies use to  manage technology development. In particular, we discussed their (1)  organizational structure and management culture, (2) R&D portfolio  management and investment strategy, (3) R&D project management  practices, and (4) their technology transition process, including when  transition occurs, the organizations involved, and how technology is  funded throughout the transition phase. We synthesized each company’s  processes and created summary documents, which the companies then  reviewed for accuracy and completeness to validate our assessment of  their specific practices. Using this validated information, we identified the  practices that were consistent among the selected companies and which  company representatives considered key to promoting innovation. We  also presented our analysis of these leading practices to senior DOD  Science and Technology (S&T) executives within the Office of the  Secretary of Defense, the military services, and other defense research  organizations to obtain their views on the practices.
To identify the extent to which DOD can employ these leading  commercial practices, we interviewed officials responsible for the  management, execution, and oversight of DOD’s S&T enterprise. At the  Office of the Secretary of Defense and military department headquarters  level, those responsible for the management and oversight of S&T  activities, we met with officials from the
Office of the Assistant Secretary of Defense for Research and
Office of the Deputy Assistant Secretary of the Army for Research and
Office of the Deputy Assistant Secretary of the Air Force for Science,  Technology, and Engineering;
Office of the Deputy Assistant Secretary of the Navy for Research,  Development, Test, and Evaluation; and
Office of Naval Research.
We also met with military department laboratory officials responsible for  the management and execution of S&T activities from the
Army Research Laboratory;
Army Armament Research, Development, and Engineering Center;
Army Engineer Research and Development Center;
Air Force Research Laboratory;
Naval Research Laboratory; and
Naval Undersea Warfare Center—Division Newport.
Finally, we met with officials from the Defense Advanced Research  Projects Agency (DARPA) responsible for the planning and oversight of  their S&T activities.
We conducted semi-structured interviews at each laboratory and DARPA  to gather consistent information about processes and practices these  organization used to manage S&T activities. In particular, we discussed  their (1) organizational structure and management culture, (2) S&T  portfolio management and investment strategy, (3) S&T project  management practices, and (4) their technology transition process. We  compared and contrasted those practices with the practices identified  through our meetings with leading commercial companies to determine  the extent to which DOD is employing these practices. Where  appropriate, we reviewed relevant regulations, policies, and guidance that  establish the framework for how DOD S&T organizations plan, budget  and execute S&T activities, including the Assistant Secretary of Defense  for Research and Engineering’s Reliance 21 Operating Principles and  DOD’s Financial Management Regulation. To further our understanding  of the S&T management practices being used at the military department  labs, we reviewed at least two recent S&T projects by each of these labs.  These projects were identified by lab officials and included projects  deemed successful as well as ones identified as unsuccessful.
Finally, we hosted a forum of DOD S&T executives in December 2016 to  identify potential opportunities for DOD to adopt leading commercial  practices in S&T management, as well as any barriers to adopting these  practices. Forum participants included the following:
Ms. Mary Miller, Principal Deputy to the Assistant Secretary of  Defense for Research and Engineering;
Dr. Melissa Flagg, Deputy Assistant Secretary of Defense for
Mr. Michael Holthe, Acting Director of Technology, Office of the  Deputy Assistant Secretary of the Army for Research and  Technology;
Dr. David Walker, Deputy Assistant Secretary of the Air Force for  Science, Technology, and Engineering;
Dr. Phil Perconti, Acting Director, Army Research Laboratory;
Dr. Jeff Holland, Director, Army Engineer Research and Development
Mr. Jyuji Hewitt, Executive Deputy, Army Research, Development,
Mr. John Uscilowicz, Director Plans, Programs, Analysis, and  Evaluation, Army Medical Research and Materiel Command;
Dr. Morley Stone, Chief Technology Officer, Air Force Research
Dr. Edward Franchi, Acting Director, Naval Research Laboratory;
Dr. Stephen Russell, Director of Science and Technology, Space and  Naval Warfare Systems Command; and
Mr. Ellison Urban, Special Assistant to the Director, Defense  Advanced Research Projects Agency.
section_title: Appendix II: Department of Defense Research, Development, Test, and Evaluation (RDT&E) Budget Activities
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Christopher R. Durbin, Assistant  Director; Marie Ahearn; Emily Bond; Jared Dmello; Lorraine Ettaro; Rich  Hung; Justin Jaynes; Ron La Due Lake; Sean Seales; Brian Smith; and  Robin Wilson made significant contributions to this report.