Defense Science and Technology: Adopting Best Practices Can Improve Innovation Investments and Management
Released: 2017-06-29T00:00:00

Letter

Background

DOD’s S&T community—including research laboratories, test facilities, industry, and academia—conducts initial research, development, and testing of new technologies to improve military operations and ensure technological superiority over potential adversaries. Key expectations DOD places on its S&T community include the following: expand scientific knowledge and investigate technologies that may provide new warfighting capabilities, anticipate technological needs for an uncertain future, and produce relevant and feasible technologies that can transition into weapon system programs or go directly to the warfighter in the field.

As a result, some investments focus on conducting research to generate scientific knowledge, exploring new technologies, demonstrating the feasibility of a technology concept, and pursuing other science and technology endeavors. We have previously reported that the challenge is finding the right balance between developing breakthrough or “disruptive” technologies—those considered to be innovative—and investing in moderate, “incremental” technology enhancements.

Figure 1 below provides a notional picture of how DOD’s S&T community manages technology investment, development, and transition to a user.

Following technology development, DOD’s acquisition community manages the next phase, product development, in which technologies are further advanced and system development begins. DOD has long reported the existence of a chasm between its S&T community and the acquisition community, which often precludes effective transitioning of technologies out of the S&T environment into weapon systems. In a series of reports, we found that technologies may not leave the lab because their potential has not been adequately demonstrated or recognized, acquisition programs may be unwilling to fund final stages of development, or private industry chooses to develop the technologies itself. Further, we found that the acquisition community frequently integrates technologies too early and takes on the task of maturing technologies—an activity that is the primary responsibility of the S&T community—at the start of an acquisition program. These challenges, in part, contribute to cost growth, schedule delays, and performance shortfalls that we have frequently found and reported on in DOD weapon programs.

DOD funds technology and product development activities under its research, development, test, and evaluation (RDT&E) budget, which DOD groups into seven budget activity categories for its annual budget estimates. The categories follow a mostly sequential path for developing technologies from basic research to operational system development, as is shown in figure 2. The first three budget activity categories generally represent activities undertaken by DOD’s S&T enterprise to advance research and develop technology, while the remaining budget activity categories are typically associated with product development for acquisition programs. See Appendix II for a description of each budget activity.

Leading Companies Foster Innovation by Investing in Both Near- and Long-Term R&D and Taking Steps to Prove New Technologies Work

Selected leading companies that we reviewed follow six key practices that together reflect a disciplined approach to managing their R&D activities— those akin to DOD’s S&T activities. First, they define their corporate strategy by identifying desired markets. Next, they invest in technology programs to penetrate those desired markets. Effective management of these portfolios requires balancing investments between two types of R&D efforts: incremental R&D, which is tied to near-term products; and disruptive R&D, which is intended to deliver innovative technologies that can provide longer-term growth. According to company representatives, this balance is driven by the business imperative of sustaining current markets while also developing future ones. Leading companies align their goals for incremental technology development with product development, while also providing independent paths for developing disruptive technologies not tied to product development. In addition, these companies identify stakeholders outside the scientific realm and collaborate extensively with them to ensure that technologies are relevant and can be efficiently integrated into marketable products. Among the key R&D stakeholders are representatives from the business units who are responsible for identifying customer needs and getting products to market. They also scale the rigor in project oversight based on the amounts of time and money invested. Nonetheless, leading companies expect all R&D projects to include prototyping or other demonstrations to prove out the technology before it is integrated into a product for the company to sell. Figure 3 below summarizes the general management process these leading companies use to plan and execute their R&D investments.

Leading Companies Use Corporate Strategies to Guide Investments in Portfolios of Incremental and Disruptive Technologies

Among the eight leading companies we reviewed, each manages R&D investments that are underpinned by defined strategies, markets, and financial goals. In addition, each company sets aside a percentage of company revenues to fund R&D. The company’s strategic direction is set by the Chief Executive Officer (CEO), in coordination with the company’s top executives, including the Chief Technology Officer (CTO) or other senior R&D executives.

These corporate strategies balance near-term profitability with long-term growth potential and market expansion. Companies stay competitive by dividing their collection of R&D projects, also known as their R&D portfolios, into two categories: 1. Incremental R&D: lower-risk projects to be integrated quickly into near-term products. 2. Disruptive R&D: projects that carry a higher risk of failure, but offer significant rewards for the company in the long-term. These investments may lead to non-incremental innovations that become an important piece of their portfolio. In some cases, these technologies render competing products obsolete by creating new markets or displacing existing product lines.

According to representatives of leading companies, around 80 percent of R&D funding is spent on incremental development, while the balance is spent on disruptive projects. Corporate leadership determines this percentage based on tolerance for risk and the company’s financial standing.

In addition to these two portfolio types, leading companies provide scientists with the flexibility to work on lower-cost exploratory projects. Such work is conducted by a few scientists or researchers and is not part of the annual process for approving projects within each portfolio. However, the work derived from these efforts could eventually become part of incremental or disruptive R&D portfolios.

Setting Company Strategy

In determining an appropriate balance between incremental and disruptive R&D investments, company leaders consider long-term scenarios based on current trends and technologies in the market. For example, R&D leaders at Siemens reported that they conduct an annual “Innovation Review” of the company’s entire technology development portfolio for the purpose of informing top leadership’s strategic decisions. These reviews evaluate Siemens’ technological competitiveness, strategic resource allocation, and long-term corporate strategy. During this review, Siemens asks a number of questions, including the following:

Are the overall resource allocations for R&D investments appropriate?

Is the business unit’s technology position competitive, and will planned investments safeguard the business unit’s technological competitiveness?

Is there a convincing long-term strategy for how to translate these investments into sustainable business success?

Is there an adequate strategy for translating new technologies into winning offerings?

Siemens executives reported that they understand that these factors directly impact their ability to grow and profit as a company, which is why the desire to remain technologically competitive drives corporate strategy decisions.

Siemens also creates forecasting tools called “Pictures of the Future” that provide graphical representations of how future technologies could be used by customers 7 to 15 years in the future. Figure 4 provides an example of the elements that Siemens includes in a Picture of the Future.

According to Siemens, it uses Pictures of the Future to assess societal, technological, and other trends to guide visionary concepts for potential new markets and customer needs; consider existing product lines, technologies, and customer needs; analyze the opportunities and risks for the company’s core business; identify what is required to allow the company to act upon potential future scenarios.

Most importantly, company representatives stated that these pictures help develop consensus within Siemens regarding the technologies the company needs to develop to drive innovation and remain a market leader.

After leading companies settle upon their corporate strategies for R&D investment, different units are charged with sponsoring—approving and funding—incremental and disruptive R&D projects. Incremental R&D projects are typically sponsored by business units, who are also responsible for product development. Disruptive R&D is often sponsored by a corporate research organization, which makes project investment decisions independently from the business units. Figure 5 shows this division of R&D that we observed in the private sector.

Leading Companies Align Incremental Technology Investments with Product Development Goals

Selected leading companies we reviewed align plans for developing new incremental technologies with plans for developing future products, which companies sometimes refer to as roadmapping. Individual business units are responsible for product sales in a company and have their own executive management teams that are charged with generating profits from their product lines. Business units sponsor incremental development projects intended to yield technologies to meet identified customer needs. Incremental R&D generally adds new capabilities to current products or next generation versions of existing products; therefore companies expect these to have lower risk of failure. Depending on the industry, business units generally do not look beyond a 5-year timeframe when making decisions about these new technologies due to the unpredictable nature of the markets in which they operate.

These leading companies document planned future products in product roadmaps, while the technologies that are to be integrated into those products are documented in technology roadmaps. By aligning these plans, business units can better identify and prioritize technology development investments. To develop these plans, companies solicit ideas and information from people across the organization to determine the composition of incremental R&D portfolios. Ultimately, however, technology development decisions come down to the management’s qualitative judgements regarding the merits of individual R&D projects, as well as on quantitative metrics like potential return on investment. Once a project is approved, it may be immediately funded and executed.

The process and the number of people involved in these R&D investment decisions vary depending on the company. In general, these leading companies solicit input from top leadership responsible for setting the company’s overall strategy and funding, including the CEO, CTO, and other corporate leaders; representatives from business units responsible for getting relevant products to market; and scientists and technologists who plan for future technology development and identify when technologies are ready for integration into products.

Technology and product development teams at Honeywell Aerospace, for example, complete an annual roadmapping process to align incremental technology development activities with the company’s product plans. As part of this process, business units identify customer needs; marketing and product management staff review market trend information, and determine future products and when they must be completed; and

Honeywell’s corporate research organization reviews external technology trends and creates technology roadmaps.

Honeywell Aerospace’s roadmapping process is illustrated in figure 6.

Honeywell’s roadmapping process focuses on needs and trends projected for the next 10 years, with less emphasis on the more distant future, due to the difficulty in making reliable predictions that far in advance. In general, the leading companies we met with consider this timeframe to be realistic and manageable for planning incremental R&D investments. To assist with management decisions regarding what incremental R&D projects to start or change, Honeywell considers potential revenues in the next 5 years. However, certain technologies, such as those associated with jet propulsion engines, require longer-term plans because technology development takes many years to complete. Although the company’s technology development plans do not extend beyond the next 10 years, company officials reported they do review industry trends potentially leading to new developments further in the future and develop concept ideas for new technologies based on these trends.

Honeywell obtains input from a variety of sources to ensure technologies will be feasible for integration into future products and relevant to customer needs. Technology development plans, or roadmaps, are periodically revised based on changes in customer needs, prototyping results not meeting expectations, and other changes in circumstances requiring additional consideration. According to company representatives, Honeywell stays in close communication with customers to ensure the company’s understanding of market needs remains accurate, which helps them avoid wasting time and money on projects that have lost relevancy.

Leading Companies Make Decisions to Fund Disruptive R&D Independently from Product Development Plans

Selected leading companies we reviewed also ensure that a portion of their R&D is independently focused on futuristic concepts, which are intended to keep the companies competitive in the long term. Disruptive R&D includes significant technology development efforts addressing the anticipated customer needs of the future, potentially leading to products that render the competition’s products irrelevant in the marketplace.

The disruptive R&D portfolio is initiated separately from incremental portfolios and often managed by a corporate research organization. Corporate research looks for solutions that provide customers with capabilities they may not realize they need or want. By separately organizing disruptive R&D from incremental, companies are able to protect funds from near-term-focused business unit managers. Generally speaking, companies in our review ensure that disruptive R&D is planned and executed by management not averse to taking risks when significant long-term rewards are possible. Due to their near-term focus, business unit managers usually begin having significant influence over disruptive technologies only after they have been demonstrated and are ready to begin transitioning into products. Allowing exploratory and disruptive technology development to occur without requiring product development approval helps prevent the company’s products from becoming obsolete and gives potential to capture new markets.

These companies use various approaches to leverage the ideas of their own staff and external partners to innovate for futuristic technologies that look beyond product roadmaps. These approaches include challenging R&D staff to come up with feasible ideas to create disruptive technologies leading to entirely new product lines, and investing in external startup companies or leveraging externally developed technologies.

Companies Challenge Staff to Develop Ideas for Disruptive Innovation

Companies seek to ensure longer-term competiveness by challenging their R&D staff to make scientific advances that will make existing products irrelevant in the future. IBM, for example, issues grand challenges to R&D staff to develop these kinds of technologies. Company representatives stated that about 15 to 20 percent of IBM’s R&D funding goes toward development of disruptive new technologies not aligned with known customer needs. While all of IBM’s disruptive technology development is executed by its corporate research organization, IBM business units provide funding to sponsor these efforts. To generate ideas for disruptive R&D projects, IBM management issues “Grand Challenges” asking for project proposals. Figure 7 below provides information on how IBM’s corporate R&D organization independently developed the IBM Watson supercomputer as a result of one such Grand Challenge.

Amazon also provides a number of avenues for individual staff members to submit innovative R&D project proposals that are not necessarily tied to defined customer needs or market trends. For example, company representatives explained that Amazon holds week-long innovation forums, where R&D staff collaborates to develop new project ideas. R&D staff then vote on the best ideas, which are submitted to top company leadership for approval. Individual scientists or technicians can also propose new projects or ideas directly to their supervisors, who may help them develop formal proposals.

GM initiates disruptive projects through their corporate research organization by funding “Internal Startup” projects. These are disruptive technology development projects initiated solely at the discretion of GM’s CTO and the director of GM’s R&D labs. Aside from the project teams, these are the only GM employees that know the details of these projects. This level of confidentiality allows GM to take risks. GM expects these projects to have features that develop phenomenal value to customers, make existing technology obsolete, and make the competition irrelevant.

Any R&D staff member can propose this kind of project, although its potential value to the company must be defined in the proposal. GM also provides these projects with more funding than typical projects so that R&D work can progress about three times faster than usual. As these projects are generally high in technical risk, GM officials stated that they have about a 50 percent success rate, which they considered acceptable given the value associated with successful projects.

Leading companies also encourage their scientists to explore and initiate low-cost research projects, either as unfunded side-projects or using limited resources following approval from a supervisor. Providing this flexibility allows scientists to be creative, while also affording them access to the company’s laboratory or other resources. These less intensive and inexpensive projects are not required to be approved by senior management to avoid unnecessary administrative burden with scientists during exploratory development. If scientists and management deem a technology to be feasible, it will be referred to the company’s R&D project approval process for additional funding.

Companies Maximize R&D Value through External Partnerships

Sometimes leading companies develop technologies that may be useful in products beyond their preferred markets. In such instances, these companies seek to maximize the value of their R&D investments through external partnerships. This may occur through investments in startup companies or licensing arrangements. For example, Siemens provides alternate paths for innovative technologies to move into products outside of their own product lines by co-founding start-up companies that could turn them into products, thereby allowing Siemens to benefit financially when technologies they developed are used in other companies’ products. Siemens also licenses some of its technologies to other companies so they can be used in their products.

Conversely, sometimes leading companies seek technologies from outside firms that show promise for use in the company’s own products. The Dow Chemical Company uses a Corporate Venture Fund to make investments in companies that have formed to commercialize new technologies. In many cases, Dow assumes a minority position, although in some cases, Dow may elect to acquire the company or partner without financial investment to mutually develop technologies. Dow scientists may work alongside R&D staff from these companies or Dow may just choose to be an investor. Dow employs technology scouts around the world responsible for finding opportunities to bring innovative technologies into Dow. These scouts are responsible for learning everything they can about a company before Dow proceeds with the investment or partnership.

R&D Project Teams at Leading Companies Collaborate Closely with Stakeholders to Produce Relevant and Feasible Technologies

At the selected leading companies we reviewed, once an R&D project is initiated, the research team in charge of the project actively collaborates with stakeholders outside the R&D office to help assist and inform project execution efforts. These stakeholders typically include product development staff and engineers who understand technical requirements for technologies to eventually transition beyond R&D, marketing staff familiar with how products might fit into the outside business unit staff who interface directly with customers, and potential users of the technology.

The level and timing of stakeholder involvement can vary based on the type of project. Stakeholders, such as product development staff, typically become involved early in incremental R&D projects. On the other hand, those same stakeholders might not get involved in disruptive projects until later phases of development. Collaboration between stakeholders continues even after the technology development effort concludes and a business unit begins product development. R&D staff may continue to assist product development efforts after technology development is completed as products are customized for different types of customers. For example, figure 8 describes how a cross-functional team at Dow Chemical collaborated to develop a new polymer.

Leading companies also look outside the company when undertaking an R&D project to gain insights from potential customers. These customers provide input and perspectives that help inform refinements to technologies. Figure 9 details the process Honeywell Aerospace used to seek customer input when developing its Synthetic Vision System.

Company representatives explained that input from both internal stakeholders and potential customers helps R&D staff to transition emerging technologies into product roadmaps. This collaboration also helps the R&D project team obtain the requisite information and resources it needs to develop a technology that is feasible for use as part of future products, while also being relevant to future customer needs so it is accepted in the marketplace.

Leading Companies Tailor R&D Project Review Processes to the Scope of Investment

At the selected leading companies we reviewed, R&D projects are more rigorously reviewed by higher levels of management as their needs for staff and funding grow. This leads to a subset of these projects continuing into later stages of development while others are ended. The Dow Chemical Company, for example, uses a stage-gate process to oversee R&D projects requiring significant investments. At each stage of development the project’s funding increases and the hurdles for moving forward become greater. Dow requires only minimal oversight for low-cost exploratory research. Once a scientist proves a technology’s feasibility, the project enters Dow’s normal processes for integrated project oversight and portfolio management involving all of the relevant Dow stakeholders. Later stage development projects generally have higher budgets and are more closely monitored to ensure they meet specific technical criteria and time-based milestones, according to company representatives. While specific review processes vary among leading companies, figure 10 outlines the key principles that all these different processes embody.

IBM also emphasizes relevant stakeholder participation in reviews for R&D projects progressing beyond exploratory or disruptive corporate research. After projects are initiated, IBM leadership reviews R&D projects at least quarterly during business reviews, although company representatives noted they may be reviewed more frequently in some cases when more attention is warranted. IBM leadership believes— according to representatives we interviewed—that it must have agile review processes that facilitate timely adjustments or, in certain cases, terminations to projects.

Leading Companies Prove Technologies Work before Integrating Them into Products

The selected leading companies we reviewed consider technology demonstrations, or prototyping, an inherent part of R&D and use demonstrations for a variety of purposes, including to create demand by convincing stakeholders or customers of the potential value of a future technology; and obtain feedback from potential end-users to add knowledge and improve technologies.

Both incremental and disruptive R&D projects receive funds to demonstrate technologies. Figure 11 depicts the technology demonstration process of the leading companies.

Leading companies demonstrate concepts for new disruptive technologies to stakeholders to generate demand so business units will contribute to technology development. Specifically, once corporate research organizations believe technology components are sufficiently mature for product development, they offer demonstrations to show their potential value to stakeholders in the business units. For example, Siemens use prototypes once a technology reaches the point that its components can be validated in a laboratory or relevant environment.

Siemens representatives explained that should the demonstration prove successful, a business unit assumes development responsibilities.

Leading companies also demonstrate these concepts for disruptive technologies to potential customers if business units are hesitant to invest in further development. For example, Qualcomm corporate research representatives told us they must sometimes overcome internal resistance to accepting new technologies by demonstrating their value to both internal and external customers. Barriers to transition of disruptive technologies may be even more prevalent when developing technologies that could lead to dropping an existing product or feature. In their role as a R&D component, not a business unit focused on product lines, corporate research works with both internal and external customers without committing to future products but with a clear pathway to adoption if successful. Qualcomm representatives explained that if these early demonstrations and advocacy for disruptive technologies proves successful, then customers may ask product developers to use them in future products.

Feedback from Demonstrations Add Knowledge and Improve Technologies

Once a concept is proven, leading companies use technology demonstrations to inform developers how a technology needs to be improved. Company representatives explained that rapidly developing and demonstrating a series of iterations of a new technology provides early opportunities for improving the technology, rather than taking longer amounts of time to develop technologies without testing them. Figure 12 depicts the iterative technology development and demonstration process that leading companies use.

When Amazon pursues a product or technology, the company already has an idea of its customers’ needs, but does not consider this information to be complete in the absence of user feedback from prototype demonstrations. To ensure technologies address these needs, Amazon representatives stated that the company builds iterations of prototypes during technology development in a facility specifically designed for doing so quickly and at minimal cost. Figure 13 illustrates how these early prototypes are generally used by Amazon employees in real world settings to help inform technology development.

Similar to Amazon, IBM’s development method uses prototyping during technology and product development. IBM developers show early iterations of new technology to customers, improve the design based on feedback, and then produce another prototype after that. This process repeats until a releasable product is completed. IBM representatives found that development models with longer sequential steps and a single deliverable are less useful than faster paced and smaller deliverables under shorter timeframes.

Companies may also use external demonstrations of maturing technologies to prove they work in realistic environments. For example, Valvoline produces and tests small volumes of new motor oil formulas in its own laboratory or at a few select customers’ facilities. To obtain customer insights from prototyping, Valvoline develops data and uses cameras on test engines to demonstrate the uses customer test engines to conduct tests and provide specific data to customers on how new formulas perform in their engines; and provides test formulas to outside customers and obtains feedback through sales staff and focus groups.

Offering external customers an opportunity to test new technologies provides important feedback for companies such as Valvoline. By taking steps like these to involve outside customers in prototype testing, representatives of leading companies stated they are able to identify a new technology’s tangible benefits, while also encouraging eventual customer acceptance of new products.

DOD Implements Some Practices Used by Leading Companies, but Policy Limitations and Culture Hinder Adopting Others

While some DOD S&T practices closely mirror those of the selected leading companies we reviewed, DOD’s funding policies and culture limit its ability to adopt other practices for managing its S&T investments. Unlike the companies we reviewed, DOD does not organize and fund incremental and disruptive innovation separately. Nor does its leadership provide guidance on or assess how these innovation investments should be or are mixed. Instead, S&T officials explained that DOD labs face pressures to prioritize near-term requirements at the expense of potentially disruptive technologies. As a result of DOD funding policies, projects are planned 2 years in advance, which can slow innovation and limit lab directors’ autonomy to initiate work. While Congress has provided authority that, as implemented, has enabled the military department lab directors to initiate work outside of the normal lengthy process, DOD has not fully utilized these flexibilities. Additionally, we found that divided responsibilities for technology versus product development contribute to a culture that does not encourage collaboration between DOD’s S&T and acquisition communities and limits the S&T community’s ability to conduct advanced prototyping. These issues are not insurmountable, however, as demonstrated in pockets of each military department. In recognition of these and other issues, Congress has required that DOD create a new Under Secretary of Defense for Research and Engineering charged with advancing defense technology and innovation and establishing policies on technology development, prototyping, and experimentation, among other responsibilities, by February 2018.

Some DOD Practices for Managing and Executing S&T Investments Closely Resemble Practices Used by Leading Companies

Some of DOD’s practices for managing and executing S&T investments closely resemble those employed by the selected leading companies we reviewed. DOD has a corporate research organization for disruptive innovation and its leadership defines S&T strategies to guide investments which are consistent with elements that the companies we met with used to manage investments in incremental and disruptive portfolios. DOD project oversight is scaled based on the scope of investment, which aligns closely with leading company practices.

While differences exist between how these practices are implemented in DOD and at the companies we reviewed, we found that the outcomes are the same.

The Defense Advanced Research Projects Agency (DARPA), for example, closely resembles the corporate research organization that many leading companies employ to foster disruptive innovation. In the President’s budget submission for fiscal year 2017, DARPA requested $2.9 billion, about 23 percent of DOD’s S&T budget request. Similar to a company’s corporate research organization, DARPA’s projects are generally not tied to existing DOD weapon systems or a specific military department requirement. Instead, their mission is to produce disruptive innovation that could support any military department. The DARPA Director makes all funding decisions and project prioritizations.

DOD’s market research, which informs its S&T strategy, is based on near and far-term adversarial threats, capability needs, and warfighter requirements. While these inputs may differ from the companies we reviewed, they are likewise used to prioritize S&T projects.

DOD does not necessarily use the same metrics as companies to evaluate projects, but its labs similarly scale the scope of their project reviews based on the maturity of the technology and scope of investment. For example, a lab typically reviews basic research projects once a year, while officials said more mature, larger investments are reviewed multiple times per year by the lab, its customers, and military department leadership. Similarly, both DOD and the companies we reviewed assess projects based on their cost, schedule, and technical performance requirements. One inherent difference is that leading companies are concerned with potential financial returns on investment, whereas DOD prioritizes for other reasons, such as whether the technology carries potential to reduce risk to the warfighter.

Military Departments Do Not Define in Strategy or Assess the Mix of Investments in Incremental and Disruptive Innovation

Although our review of selected leading companies found that they define in strategy their annual mix of investments in incremental and disruptive innovation, the military departments do not do this, nor do they assess such a mix. The office of the ASD(R&E)—the Office of the Secretary of Defense (OSD) organization responsible for establishing DOD S&T policy and guidance—does not provide guidance to the military departments on the mix of incremental and disruptive S&T investments. The military departments are responsible for defining their own S&T strategies, formulating and managing budgets, and developing technologies. Their S&T strategies, however, do not define the mix of incremental and disruptive investments each department should make annually. Instead, DOD S&T investments are organized and funded based on budget activities (BA) that reflect stages of technology maturity. DOD uses this approach, in part, because the Financial Management Regulation dictates how R&D activities are identified for the purposes of budgeting. We found that one limitation of funding under BAs is a lack of visibility into whether individual projects that labs and research centers invest in are geared toward disruptive or incremental innovation. The Financial Management Regulation, however, does not preclude DOD from developing investment targets for both incremental and disruptive R&D.

Military department lab and center officials we interviewed, however, identified certain projects they were working on that could lead to disruptive technologies. Officials from these labs and centers acknowledged that they struggle to determine the right balance between disruptive and incremental innovation projects. They expressed concern that military department leadership responsible for setting requirements for and approving S&T spending, at times, are more focused on near- term, less risky, more incremental types of innovation investments at the expense of long-term, disruptive innovation.

The Navy is one military department that has taken steps to ensure funding for some investments in disruptive innovation. The Navy organizes S&T investments around “strategic buckets” to ensure it maintains investments in both near- and long-term projects and protects funding for potentially disruptive projects. The distribution of resources is determined by senior leadership based on the Navy’s S&T strategy. The strategy maps out roughly the minimum percentage of funding that the Navy plans to request for high-priority, disruptive projects within its S&T portfolio, as reflected in figure 14 below. In fiscal year 2017, the Navy plans to invest more than $313 million for Leap Ahead Innovations which are intended to be disruptive technologies and deliver transformational warfighting capabilities. These are in addition to investments in other disruptive technologies that are categorized, but not quantified, under its other strategic buckets.

In a June 2017 report, we recommended that DOD take steps—such as the Navy’s—to help ensure adequate investments in innovation that align with DOD-wide strategy to overcome the department’s risk-averse culture and pressures to focus on near-term projects.

DOD’s Lengthy S&T Project Planning and Budgeting Cycle Can Slow Innovation While Existing Flexibilities to Expedite Have Been Underused

In comparison to the practice at selected leading companies we reviewed, which annually align their investments to product goals, DOD’s process for prioritizing and funding projects takes longer—almost 2 years to complete—which we found can slow innovation. Like every other good and service DOD acquires, all S&T investments must follow DOD’s planning and budgeting policy. This policy is underpinned by DOD’s Planning, Programming, Budgeting, and Execution (PPBE) process. The PPBE process for S&T investments includes the following stages:

Planning: DOD leadership, in guidance and planning documents, identifies strategic priorities, weapon system requirements, and adversarial threats. Collectively, these serve as DOD’s broad requirements for technology development.

Programming: S&T organizations give consideration to those requirements and propose technology development projects to address them. Proposed projects and associated costs are documented in Program Objective Memorandums (POM). Each organization is tasked with determining which projects to propose in the POM, while maintaining balance across their portfolios of investment, as well as maintaining an appropriate mix of funding based on BA. POM documents are reviewed by senior officials across DOD—including those responsible for setting requirements and the budget—who also have a role in prioritizing S&T investments.

Budgeting: Each S&T organization’s POM is used to formulate their respective military department’s Budget Estimate Submission (BES), which outlines the total funding needed, including how much it will need by budget activity. After the President’s budget is submitted, Congress enacts an appropriation. Once funds are appropriated, each S&T organization is provided funding for the projects approved in the POM and BES.

Execution: S&T organizations carry out funded projects.

Figure 15 illustrates the notional time frames for DOD’s PPBE process.

In total, it can take almost 2 years from the time a project is proposed in the POM to the time it is funded. In contrast, the companies we reviewed reported that they planned projects in the same year they were executed, which helped them quickly respond to leaps in technology development. S&T officials we met with stated that the 2-year project planning process reduces their ability to be as nimble as the companies with whom we met. For example, if an unexpected technology breakthrough is identified through 6.1 or 6.2 research, the labs may have to wait up to 2 years before they may begin work on a follow-on project. DOD S&T executives expressed the need for greater flexibility with initiating new projects because the pace of technology development can be rapid and planning for S&T spending 2 years in advance can hinder innovation. They stated, however, that the PPBE process provides Congress with the information it needs to maintain oversight and ensure DOD meets its fiduciary responsibilities to the taxpayer.

We found that laboratory and research center directors in the military departments have less authority to initiate S&T work that is not directly linked to defined near- or far-term capability needs as compared to the leading companies we reviewed. While the Director of DARPA approves every project the agency undertakes and is not beholden to address defined requirements, the military departments’ labs and centers do not control all of the S&T-related R&D work they perform annually. These labs and centers regularly undertake work on behalf of acquisition community customers, such as a major defense acquisition program, who provide funding in support of the project. This work comes in addition to “direct funded” projects approved and funded to the lab or center through PPBE that are intended to address S&T requirements outlined in strategy. As a result, both the direct funded and customer-funded projects compete for lab resources, such as staff, and must be balanced. In fiscal year 2015, for example, direct funded projects accounted for 19 percent of Naval Research Laboratory’s (NRL) $1.2 billion of funding. The other 81 percent was customer-funded work from Navy, other DOD, or other governmental sources. Despite having direct funding, the POM review and approval process may constrain which projects are ultimately funded. For example, we found that the projects a lab or center proposes in its POM submission may be reviewed by as many as four different organizations before it is submitted to OSD. It is during this review process that lab officials explained that the culture within DOD is, at times, to focus on near-term, customer-driven projects, at the expense of far-term disruptive projects.

Regardless of the source of the funding, a senior ASD(R&E) official explained that S&T investments are intended to address some defined capability need. This means that the military department’s disruptive technology projects are roadmapped to requirements, which differs from the practice at the leading companies we reviewed. This may limit the labs’ ability to address undefined customer needs through other potentially disruptive technologies.

Section 219 of the Duncan Hunter National Defense Authorization Act for Fiscal Year 2009, as implemented, has provided defense lab directors with some limited flexibility to initiate S&T projects, including those that are not roadmapped to defined requirements, outside of the normal 2- year planning process. Specifically, as amended, the law directs the Secretary of Defense, in consultation with the Secretaries of the military departments, to establish mechanisms under which the director of a defense laboratory may use an amount of funds equal to a certain percentage of all funds available to the laboratory for the following purposes: innovative basic and applied research that is conducted at the defense laboratory and supports military missions; developing programs that support the transition of technologies developed by the defense laboratory into operational use; workforce development activities that improve the capacity of the defense laboratory to recruit and retain personnel with needed scientific and engineering expertise; and revitalization, recapitalization, or minor military construction of laboratory infrastructure.

While this authority directed the creation of a mechanism that may provide lab directors with the means to fund projects they consider to be a priority, the military departments have not maximized their use of these authorities. Until the passage of the National Defense Authorization Act for Fiscal Year 2017, the director of a defense lab could use funds equal to not more than 3 percent of all funds available to the defense laboratory for S&T activities. Each of the services has unique strategies for executing section 219 authorities but DOD reported that the full 3 percent of funds available to the labs has not been used. DOD officials told us that the full 3 percent available to each defense laboratory has not been used for a number of reasons, including due to competing S&T funding priorities. Additionally, DOD officials indicated that labs had concerns about charging customers a fee to fund such S&T activities, which was a factor in different amounts of funds available for section 219 purposes, as shown in figure 16.

During our review, Congress amended the Section 219 authority in the National Defense Authorization Act for Fiscal Year 2017 to permit lab directors to use an amount of funds not less than 2 percent but not more than 4 percent of all funding available to the lab. As a result of the change, the military departments can increase the amount of section 219 funds that the labs may obtain. The extent to which they choose to fund new projects could help them to initiate projects, including those which may be “off-roadmap,” faster than through the PPBE process.

Cultural Barriers That Limit Collaboration between DOD’s S&T and Acquisition Communities Are Not Insurmountable

While the selected leading companies we reviewed ensure close collaboration between stakeholders in technology and product development, cultural barriers have limited such collaboration within DOD. DOD’s funding policies reflect cultural barriers to collaboration between the S&T community and its product development stakeholders. Under DOD’s funding model, the labs are responsible for technology development associated with BAs 6.1 through 6.3, while the stakeholders in the acquisition community are traditionally responsible for product development, which begins with prototype activities under BA 6.4. Although we found that S&T projects funded by the acquisition community obtain collaborative input from those same eventual customers, this was not the case for direct-funded projects—those initiated by the lab. Lab officials explained that for direct-funded projects, they may consult with potential customers in the acquisition community to gauge interest before starting a project or to present results after it is complete, but those customers are not part of the development team. This approach is, in part, attributable to these organizations being separate—both in mission and in the type of funding they receive. For example, unlike the companies we reviewed, S&T officials explained that they do not transition scientists and engineers along with the technologies they developed to the acquisition programs. The companies we reviewed reported that they set the expectation that both technology and product development staff work together. DOD, however, has not established a formal policy on how these two communities should collaborate on projects to overcome these cultural barriers.

DOD has processes to help its research labs and centers collaborate on S&T work, but these processes do not emphasize collaboration between the S&T and acquisition communities. In 2014, ASD(R&E) revitalized its “Reliance 21” framework—a joint planning and coordination process that is intended to ensure DOD’s S&T community provides solutions and advice to the departments’ senior-level decision makers, warfighter, Congress, and other stakeholders. This is to be accomplished, in part, through groups of technical experts organized around 17 technical areas referred to as Communities of Interest (COI). Originally formed in 2009, COIs provide DOD with a mechanism for experts in technical areas, such as cyber or space, to coordinate and communicate what S&T-related R&D each military department is working on and identify areas for collaboration. Each COI is supposed to conduct a portfolio review every 2 to 3 years—depending on the technical area—to assess those gaps and their impacts and to make recommendations to the S&T Executive Committee. We found that each COI has documented at least one such assessment since 2014. According to an ASD(R&E) official, each COI also provides updates to the S&T Executive Committee during its annual S&T strategy meeting.

DOD’s S&T executives stated that there is a need to improve how and when the acquisition community is brought in to contribute. One S&T executive pointed out that bringing in external stakeholders earlier in the process is a way to facilitate disruptive innovation. They described one instance in which the Air Force overcame existing cultural barriers to collaboration by funding technology maturation efforts with both S&T and acquisition community support, as described in figure 17. This different approach was the result of a deliberate decision by the Air Force to foster more collaboration between the S&T community and other stakeholders to increase the odds that new technologies end up in the hands of the warfighter, according to officials.

Military Department Labs Traditionally Funded Technology Development through Initial Prototypes, but Recent Initiatives Have Begun to Fund Advanced Prototyping

While selected leading companies we reviewed provided funding for prototypes during technology development, DOD has only recently begun to fund advanced prototyping efforts within the labs. The acquisition community, as opposed to the S&T labs, traditionally bears the responsibility of maturing technology through advanced prototyping, which is in contrast to how the companies with whom we met operate. DOD’s S&T labs and centers typically do not control the BA 6.4 funding that would allow them to conduct such prototyping. We found that the S&T community typically matures technologies to, at most, a prototype that is close to final form, fit, and function and tested in a relevant environment. This creates strong incentives for S&T project teams to identify technology transition partners in the form of major acquisition programs early in development, which may ultimately restrict disruptive innovation and push S&T projects to be more incremental to satisfy potential customers’ near-term needs. For example, applied research project proposals at the Naval Research Lab and the Army’s Engineer Research and Development Center both identify and consider potential transition partners as part of the selection criteria, regardless of whether it was for incremental or disruptive technologies. As we previously stated, it is difficult to transition a technology or identify a partner when the technology is disruptive. Companies recognized this and funded disruptive technology development projects through demonstration to help obtain a customer.

In June 2017, we reported that DOD’s approach to prototyping contributes, in part, to DOD’s broad challenges with transitioning technology from the labs into the hands of the warfighter. We further reported that prototyping that is not directly tied to acquisition programs can be seen as a way to “test the waters” because it does not require the level of commitment associated with starting acquisitions.

Each military department has recently undertaken efforts to fund more advanced prototypes for incremental and disruptive technologies in S&T labs. For example, since 2012, the Army has used funding typically associated with acquisition programs to conduct higher-fidelity prototyping and further mature technology outside of those programs through its Technology Maturation Initiative. In the President’s fiscal year 2017 budget submission, the Army requested approximately $70 million for these efforts.

In May 2016, the Air Force established the Strategic Development Planning and Experimentation Office, in part, to run Air Force experimentation initiatives to achieve specific technology development objectives. Air Force officials explained that the intent of these initiatives is to develop more agile approaches to innovation by creating a learning organization that can rapidly take new innovative approaches, can quickly initiate new projects and is not hampered by the traditional 2-year planning process, is composed of acquisition and S&T representatives to promote collaboration, and conducts prototyping and demonstrations without a direct requirement from an acquisition community stakeholder to reduce risk and mature technologies.

In the President’s fiscal year 2017 budget request, the Air Force requested $62 million to fund these experimentation efforts. Currently, two technology development areas are addressed through experimentation initiatives and two more are planned. According to Air Force officials, these initiatives reflect its leadership’s desire to embrace a culture of encouraging and formulating innovative strategic choices independent of major weapon system acquisition programs.

The Department of Navy is pursuing similar efforts through its Rapid Prototyping, Experimentation, and Demonstration (RPED) initiative. RPED projects use prototyping to rapidly develop and assess new technologies and engineering innovations to address priority naval warfighting needs. The Navy expects that RPED projects will assist in developing new capability concepts, informing and refining requirements, addressing priority needs by demonstrating, and enabling quicker transition of technologies to naval programs.

The Navy developed its policy for RPED projects in December 2016 and requested $40 million to fund projects in the President’s fiscal year 2017 budget submission.

In our June 2017 report, we recommended that DOD develop a strategy to better coordinate and communicate the goals of these and other prototyping efforts to ensure these efforts gain traction and achieve success.

An October 2013 Defense Science Board report reinforces the military departments’ focus on experimentation as an innovation enabler. The Defense Science Board found that DOD cannot continue to rely on technological superiority unless it adopts methods that allow it to anticipate, assess, and gain experience with new technological capabilities before its potential adversaries do.

DOD and Commercial Firms Operate under Different Incentives and Organizational Structures

DOD’s organizational structure and incentives contribute to why it does not fully implement the S&T management practices that the selected leading companies we reviewed follow. This includes DOD’s budget environment, funding model, and the manner in which DOD is organized to execute technology development. As we have previously reported, the critical differences between the environments and cultures of private companies and DOD must be recognized before tangible progress can be made in establishing more efficient practices in S&T management. Further, we concluded that changing the mechanics of the processes, without changing the environment that determines incentives, may not produce better outcomes. Specifically:

Companies operate in an environment where profitability is a constant business imperative. As such, leading companies we reviewed devoted a portion of their R&D investments toward futuristic concepts, which are intended to keep them competitive in the long-term, instead of just near-term products. Disruptive R&D includes significant technology development efforts addressing the anticipated customer needs of the future, potentially leading to products that render competitive products irrelevant. By separately organizing disruptive R&D from incremental, companies are able to protect funds from the near-term focused business unit managers. In the DOD environment, budget pressures and urgent requirements often drive military departments to focus on near-term needs over long-term innovation. For instance, a 2016 Air Force Studies Board report found that in much of the Air Force, little or no space for innovation exists. Because innovation is focused on future needs, the report found that Air Force organizations decide they can wait on addressing the needs of tomorrow. The report found that across the Air Force as a whole, insufficient processes existed to support “rapid-cycle” innovation with the same intensity and pace Air Force personnel regularly bring to bear to fulfill other missions.

Companies fiercely compete with one another for customers and in ever-changing market conditions. This environment requires agility in how they direct their technology and product investments, which includes rapidly initiating new projects and truncating underperforming ones. DOD, on the other hand, operates under different conditions. Its budget environment may incentivize starting and sustaining programs rather than discontinuing underperforming ones. In April 2014, we found that budgets to support major acquisition program commitments must be approved well ahead of when the information needs to support the decision is available. DOD’s S&T community operates under similar pressures and incentives as its acquisition community. In this environment, we found that it is easier to sustain a program until its funding expires, even if technical performance is lacking. According to DOD S&T officials, current budgeting and funding processes restrict, rather than encourage, innovation. DOD S&T executives told us that they want more flexibility outside of the cumbersome 2-year PPBE process to initiate and discontinue projects.

Companies set their own budgets internally for various activities, including R&D. Conversely, as a government agency, DOD can influence, but not set, its annual budget. Ultimately, Congress determines what level of funding to appropriate DOD, including for S&T-related activities.

New Office to Be Responsible for Promoting Innovation and Establishing S&T Policies

Overcoming many of these challenges may ultimately be the responsibility of the yet-to-be-created Office of the Under Secretary of Defense for Research and Engineering (USD(R&E)). The National Defense Authorization Act (NDAA) for Fiscal Year 2017 calls for the establishment of the position of USD(R&E) to serve as the CTO and elevate and enhance the mission of defense technological innovation. This office will have greater responsibilities than the current ASD(R&E) and will focus on innovation, oversight, and policy for defense research and engineering, technology development and transition, prototyping and experimentation, and testing activities. Specifically, where ASD(R&E) has taken a more hands-off approach to developing S&T policy, Congress legislated that the new office take a larger role in establishing policies to overcome the challenges DOD currently faces with promoting innovation. The USD(R&E) will also be responsible for the allocation of resources for defense research and engineering, and unifying these efforts across DOD. The fiscal year 2017 NDAA requires this position to be created by February 2018. In March 2017, DOD reported that it would submit final plans for creating this position to Congress no later than August 1, 2017, as required by law.

Conclusions

DOD’s S&T investments are key to maintaining our nation’s technological superiority over our adversaries. Congress has raised questions about DOD being innovative enough to maintain future technology superiority. The business imperatives that world-class technology companies must operate under force them to manage their S&T portfolios and projects to produce better outcomes for evolving current products, as well as well as develop disruptive technologies for the future. Leading companies have shown they do this by organizing and funding R&D to avoid the pressures to focus on incremental innovation at the expense of maintaining their technological edge in the future. With its focus on meeting warfighter needs, DOD does not operate under similar business imperatives; it has not, at a department-wide level, emphasized the need to invest in disruptive technologies. Instead, each military department’s S&T organizational construct and funding processes increase emphasis on investing in technologies that will support the near-term requirements of a major weapon system acquisition program at the expense of investing in innovative technologies that are not linked to a requirement.

As DOD determines the roles and responsibilities for its new Under Secretary of Defense for Research and Engineering, it is uniquely positioned to rethink its policies that govern technology development. While it may not be practical for each military department to organize its technology development as leading companies do, there are pockets within each department that are implementing some aspects of leading company practices. However, more needs to be done to facilitate more systematic adoption of these practices across DOD. Doing so can position DOD to develop more innovative, disruptive technologies. By not taking steps to ensure the right balance of incremental and disruptive technology investments, DOD lacks visibility into whether the technologies it is developing will provide superior capabilities to counter future and emerging adversarial threats. Additionally, the limited collaboration with product developers, limited use of existing flexible approaches to fund S&T projects outside of the 2-year planning process, and limited advanced prototyping of new technologies by the labs creates added barriers to innovation.

Recommendations for Executive Action

To ensure that DOD is positioned to counter both near and far term threats, consistent with its S&T framework, we recommend that the Secretary of Defense direct the new Under Secretary of Defense for Research and Engineering annually take the following two actions: define the mix of incremental and disruptive innovation investments for each military department, and assess whether that mix is achieved.

To ensure that DOD is positioned to more comprehensively implement leading practices for managing science and technology programs, we recommend that the Secretary of Defense direct the new Under Secretary of Defense for Research and Engineering to define, in policy or guidance, an S&T management framework that includes the three following actions: emphasizes greater use of existing flexibilities to more quickly initiate and discontinue projects to respond to the rapid pace of innovation; incorporates acquisition stakeholders into technology development programs to ensure they are relevant to customers; and promotes advanced prototyping of disruptive technologies within the labs so the S&T community can prove these technologies work to generate demand from future acquisition programs.

Agency Comments and Our Evaluation

We provided a draft of this report to the DOD for review and comment. DOD’s written comments are reprinted in appendix III of this report and summarized below.

In its comments, DOD did not concur with each of our recommendations, citing that it is premature to get ahead of the Secretary of Defense’s final decisions on the role of the new Under Secretary of Defense for Research and Engineering (USD(R&E)) until that position is established, which is required by no later than February 1, 2018. We believe, however, as the roles and responsibilities of the USD(R&E) are in the process of being deliberated, that it is appropriate and timely for the Secretary of Defense to ensure that the USD(R&E) be responsible for implementing our recommendations. Although it did not concur, DOD identified actions that it could take that are generally responsive to our recommendations.

Specifically, in response to our recommendations that the Secretary of Defense direct the USD(R&E) to define and assess the mix of incremental and disruptive innovation investments for each military department, DOD stated that it would need to coordinate with each military department to establish appropriate goals for those investments. DOD further noted that it could assess whether that mix is achieved during its annual S&T Strategic Overview meeting. We continue to believe that such actions are necessary to ensure that DOD is positioned to counter both near and far term threats.

In response to our recommendation that the USD(R&E) define an S&T framework that emphasizes greater use of flexibilities to more quickly initiate and discontinue projects to respond to the rapid pace of innovation, DOD identified the Laboratory Quality Enhancement Program as an activity to leverage existing flexibilities. This program—which DOD implemented in response to the National Defense Authorization Act for Fiscal Year 2017—requires DOD to create panels of experts to make recommendations to the Secretary of Defense on matters related to S&T policy and practices. DOD, however, did not explain how this program would help the labs make greater use of existing flexibilities to initiate projects, such as those granted under Section 219 of the Duncan Hunter National Defense Authorization Act for Fiscal Year 2009. We continue to believe that greater use of existing authorities, such as those provided under Section 219, could help labs to more quickly initiate projects outside of the normal planning cycle, which can take nearly two years for a project to be funded.

In response to our recommendation that the USD(R&E) define an S&T framework that incorporates acquisition stakeholders into technology development programs, DOD identified that it expects the USD(R&E) to provide policy and guidance that will include increased engagement with acquisition stakeholders. We continue to believe that enhancing collaboration between the S&T and acquisition communities is critical to ensuring that technologies in development will be relevant to potential customers.

In response to our recommendation that the USD(R&E) define an S&T framework that promotes advanced prototyping of disruptive technologies within the labs, DOD noted the benefits of prototyping and that it is a critical piece of the larger research and engineering strategy. It did not, however, identify if any such strategy would be revised to promote earlier prototyping so the S&T community can prove technologies work and generate demand from future acquisition programs. We continue to believe that establishing an S&T framework that emphasizes prototyping outside of acquisition programs is needed.

Additionally, in response to our recommendations that the Secretary of Defense direct the USD(R&E) to define the three elements above in an S&T management framework, DOD also noted that Reliance 21 is expected to continue serving as the overarching framework for the S&T joint planning and coordination process. We continue to believe, however, that this framework does not fully address our recommendations and that further actions, such as those they outlined above, are necessary for DOD to ensure it is positioned to more comprehensively implement leading practices for managing S&T.

We are sending copies of the report to the appropriate congressional committees; the Secretary of Defense; the Under Secretary of Defense for Acquisition, Technology and Logistics; and the Secretaries of the Army, Navy, and Air Force; and to the eight leading companies we interviewed about their practices for this report. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at 202-512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

We used a case study approach to identify leading commercial companies’ research and development (R&D) practices. We selected and visited eight large (Fortune 500-listed) companies that were U.S. companies or equivalent foreign companies, or were owned by those companies. Our primary goal was to select large companies—those more comparable with the size of the Department of Defense (DOD) than smaller ones—from a range of different industries, that were profitable and that received two or more industry awards or other recognition for technology innovation since 2014.

We used the following sources to identify companies that have received awards or other recognition: Boston Consulting Group, PwC, MIT Technology Review, American Business Awards Gold Stevie awards, and Thomson-Reuters. These organizations either provide annual lists of leading innovator companies or select top innovator companies for awards on an annual basis. These groups are positioned to be knowledgeable regarding who the “leading companies” are for the purposes of the case study selection for our review. We used available corporate stock information at Morningstar.com to determine whether a company has been profitable, which is an indicator of their degree of success in their science and technology development efforts. Below are descriptions of the eight companies featured in this report.

Amazon.com sells consumer electronics, operates retail websites serving over 100 countries, and provides cloud-computing services to hundreds of thousands of organizations in 190 countries around the world. Lab126 is Amazon’s inventive research and development company that designs and engineers high-profile consumer electronics, including Kindle Fire tablets, Fire TV and Amazon Echo. Amazon’s recent recognitions include being included among the Boston Consulting Group’s (BCG) “Most Innovative Companies,” PwC’s “10 Most Innovative Companies,” Thomson-Reuters’s “Top 100 Global Innovators” and MIT Technology Review’s “50 Smartest Companies.”

Ashland Global Holdings, Inc. is a global chemicals company serving customers in a wide range of consumer and industrial markets, including adhesives, architectural coatings, automotive, construction, energy, food and beverage, personal care, and pharmaceutical. Valvoline Inc., a leading producer and retailer of automotive lubricants that made the first trademarked American motor oil, was an Ashland subsidiary at the time we met with Valvoline company representatives. Ashland currently owns a controlling interest in Valvoline after it became a separate public company in 2016. Ashland’s recent recognitions include a Bronze Innovation Zone Functional Ingredient Award and Ringier Coatings Technology Innovation award and a Composites and Advanced Materials Exposition award for “Unsurpassed Innovation.”

The Dow Chemical Company delivers a broad range of technology- based products and solutions to customers in 175 countries. Dow drives innovations that extract value from material, polymer, chemical and biological science to help address many of the world’s most challenging problems, such as the need for fresh food, safer and more sustainable transportation, clean water, energy efficiency, more durable infrastructure, and increasing agricultural productivity. Dow’s recent recognitions include receiving multiple R&D 100 awards, and being included among BCG’s “Most Innovative Companies” and Clarivate Analytics (formerly Thomson-Reuters’s) “Top 100 Global Innovators.”

Honeywell International, Inc. invents and commercializes technologies that address some of the world’s most critical challenges around energy, safety, security, productivity and global urbanization. Honeywell’s Aerospace division, which is discussed in this report, is a leading provider of aircraft engines, integrated avionics, systems and service solutions, and related products and services for aircraft manufacturers, and turbochargers to improve the performance and efficiency of passenger cars and commercial vehicles. Honeywell’s recent recognitions include receiving American Business Awards for “Most Innovative Company” and “Most Innovative Technology Company,” and being included among Thomson-Reuters’s “Top 100 Global Innovators.”

General Motors Co. and its partners produce vehicles in 30 countries, including the Chevrolet, Cadillac, Baojun, Buick, GMC, Holden, Jiefang, Opel, Vauxhall and Wuling brands. GM develops innovative new technologies offering vehicle electrification, autonomous driving, vehicle health management, and alternative fuel usage. GM’s recent recognitions include an Edison Award for Automotive Computing, an Automotive News “Pace Award,” and being included among Fast Company’s “World’s 10 Most Innovative Companies of Automotive.”

International Business Machines Corporation (IBM) develops and markets cognitive systems, or computers that learn through interactions with people and data, as well as enterprise systems and software. IBM also provides cloud computing, consulting and information technology implementation services. IBM’s recent recognitions include receiving an R&D 100 award, and being included among BCG’s “Most Innovative Companies” and MIT Technology Review’s “50 Smartest Companies.”

Qualcomm is a leader in the commercialization of digital communication technologies, including Code Division Multiple Access (CDMA), and Long Term Evolution (LTE), for cellular wireless communication applications. They also develop and commercialize numerous technologies used in handsets and tablets. They also own intellectual property contributing to other commercial technologies like wireless local area network, global positioning system, near field communication, and Bluetooth. Qualcomm’s recent recognitions include receiving an R&D 100 award, and being included among MIT Technology Review’s “50 Smartest Companies” and Thomson- Reuters’s “Top 100 Global Innovators.”

Siemens AG is one of the world’s largest producers of electrification, automation, and digitalization technologies. Siemens’s products include gas, steam, and wind turbines, integrated power plant solutions, power grid systems, building technologies, rail technologies, medical imaging and diagnostics, and other systems for industrial use. The company’s recent recognitions include receiving two R&D 100 awards and being included among BCG’s “Most Innovative Companies.” Siemens is also category leader in the Dow Jones Sustainability Index ranking, with 100 out of 100 points for innovation management.

For each of the companies, we conducted semi-structured interviews with senior management officials and other company representatives knowledgeable about research and development activities to gather consistent information about processes and practices companies use to manage technology development. In particular, we discussed their (1) organizational structure and management culture, (2) R&D portfolio management and investment strategy, (3) R&D project management practices, and (4) their technology transition process, including when transition occurs, the organizations involved, and how technology is funded throughout the transition phase. We synthesized each company’s processes and created summary documents, which the companies then reviewed for accuracy and completeness to validate our assessment of their specific practices. Using this validated information, we identified the practices that were consistent among the selected companies and which company representatives considered key to promoting innovation. We also presented our analysis of these leading practices to senior DOD Science and Technology (S&T) executives within the Office of the Secretary of Defense, the military services, and other defense research organizations to obtain their views on the practices.

To identify the extent to which DOD can employ these leading commercial practices, we interviewed officials responsible for the management, execution, and oversight of DOD’s S&T enterprise. At the Office of the Secretary of Defense and military department headquarters level, those responsible for the management and oversight of S&T activities, we met with officials from the

Office of the Assistant Secretary of Defense for Research and

Office of the Deputy Assistant Secretary of the Army for Research and

Office of the Deputy Assistant Secretary of the Air Force for Science, Technology, and Engineering;

Office of the Deputy Assistant Secretary of the Navy for Research, Development, Test, and Evaluation; and

Office of Naval Research.

We also met with military department laboratory officials responsible for the management and execution of S&T activities from the

Army Research Laboratory;

Army Armament Research, Development, and Engineering Center;

Army Engineer Research and Development Center;

Air Force Research Laboratory;

Naval Research Laboratory; and

Naval Undersea Warfare Center—Division Newport.

Finally, we met with officials from the Defense Advanced Research Projects Agency (DARPA) responsible for the planning and oversight of their S&T activities.

We conducted semi-structured interviews at each laboratory and DARPA to gather consistent information about processes and practices these organization used to manage S&T activities. In particular, we discussed their (1) organizational structure and management culture, (2) S&T portfolio management and investment strategy, (3) S&T project management practices, and (4) their technology transition process. We compared and contrasted those practices with the practices identified through our meetings with leading commercial companies to determine the extent to which DOD is employing these practices. Where appropriate, we reviewed relevant regulations, policies, and guidance that establish the framework for how DOD S&T organizations plan, budget and execute S&T activities, including the Assistant Secretary of Defense for Research and Engineering’s Reliance 21 Operating Principles and DOD’s Financial Management Regulation. To further our understanding of the S&T management practices being used at the military department labs, we reviewed at least two recent S&T projects by each of these labs. These projects were identified by lab officials and included projects deemed successful as well as ones identified as unsuccessful.

Finally, we hosted a forum of DOD S&T executives in December 2016 to identify potential opportunities for DOD to adopt leading commercial practices in S&T management, as well as any barriers to adopting these practices. Forum participants included the following:

Ms. Mary Miller, Principal Deputy to the Assistant Secretary of Defense for Research and Engineering;

Dr. Melissa Flagg, Deputy Assistant Secretary of Defense for

Mr. Michael Holthe, Acting Director of Technology, Office of the Deputy Assistant Secretary of the Army for Research and Technology;

Dr. David Walker, Deputy Assistant Secretary of the Air Force for Science, Technology, and Engineering;

Dr. Phil Perconti, Acting Director, Army Research Laboratory;

Dr. Jeff Holland, Director, Army Engineer Research and Development

Mr. Jyuji Hewitt, Executive Deputy, Army Research, Development,

Mr. John Uscilowicz, Director Plans, Programs, Analysis, and Evaluation, Army Medical Research and Materiel Command;

Dr. Morley Stone, Chief Technology Officer, Air Force Research

Dr. Edward Franchi, Acting Director, Naval Research Laboratory;

Dr. Stephen Russell, Director of Science and Technology, Space and Naval Warfare Systems Command; and

Mr. Ellison Urban, Special Assistant to the Director, Defense Advanced Research Projects Agency.

Appendix II: Department of Defense Research, Development, Test, and Evaluation (RDT&E) Budget Activities

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Christopher R. Durbin, Assistant Director; Marie Ahearn; Emily Bond; Jared Dmello; Lorraine Ettaro; Rich Hung; Justin Jaynes; Ron La Due Lake; Sean Seales; Brian Smith; and Robin Wilson made significant contributions to this report.