Defense Advanced Research Projects Agency: Key Factors Drive Transition of Technologies, but Better Training and Data Dissemination Can Increase Success
Released: 2015-11-18T00:00:00

Letter

Background

DOD Technology Management Process

DOD invests about $12 billion in funding to support its science and technology community, which it relies upon to identify, pursue, and develop new technologies to improve and enhance military capabilities. This community is comprised of DOD-wide research agencies, including DARPA, as well as military service research agencies and laboratories, test facilities, private industry, and academic institutions, and is overseen by the Office of the Assistant Secretary of Defense for Research and Engineering. The research and development activities these different components engage in are intended to produce mature technologies that DOD can integrate and deliver in systems that support its warfighters. This integration process, known as product development, represents the handover of breakthrough technologies from DOD’s science and technology community to its acquisition community. Although not precisely defined, technology transition generally occurs at the point when advanced technology development ends and this new product development begins. Figure 1 illustrates DOD’s technology management process.

DOD has long noted the existence of a chasm between its science and technology community and its acquisition community that impedes technology transition from consistently occurring. This chasm, often referred to by department insiders as “the valley of death,” exists because the acquisition community often requires a higher level of technology maturity than the science and technology community is willing to fund and develop. In 2007, DOD reported that this gap can only be bridged through cooperative efforts and investments from both communities, such as early and frequent collaboration among the developer, acquirer, and user.

We have also reported extensively on shortfalls across DOD’s technology management enterprise in transitioning technologies from development to acquisition and fielding. In June 2005, we found that DOD technology transition programs faced challenges selecting, managing, and overseeing projects, and assessing outcomes. In September 2006, we found that DOD lacked the key planning, processes, and metrics used by leading commercial companies to successfully develop and transition technologies. More recently, in March 2013, we found that the vast majority of DOD technology transition programs provide technologies to military users, but tracking of project outcomes and other benefits derived after transition remained limited.

DARPA Processes and Programs

DARPA’s scientific investigations run the gamut from laboratory efforts to the creation of full-scale technology demonstrations in the fields of biology, medicine, computer science, chemistry, physics, engineering, mathematics, material sciences, social sciences, neurosciences, and more. The agency solicits proposals for research work in support of its scientific endeavors through broad agency announcements. These solicitations seek thought leaders and technological pioneers that can leverage new ideas in science to advance the state of the art beyond the practical application of knowledge. Non-DARPA entities respond to broad agency announcements by submitting proposals for executing work to meet the agency’s stated needs. DARPA reviews those proposals based on technical merit, and entities receiving awards are thereafter referred to as performers.

To execute solicitations, awards, and program oversight, DARPA relies on approximately 220 government employees, including nearly 100 program managers. Program managers report to DARPA’s office directors and their deputies, who are responsible for charting the strategic directions of six technical offices. The technical staff is supported by experts in security, legal and contracting issues, finance, human resources, and communications. DARPA’s Director and Deputy Director approve new programs and lead scientific and technical reviews of ongoing programs, while setting agency-wide priorities and ensuring a balanced investment portfolio. Currently, DARPA has about 250 ongoing research and development programs in its portfolio.

The 10 recently completed programs that we reviewed for this report together spanned a broad range of research areas, including communications, navigation, and health and marine sciences. Table 1 highlights the research focuses of these 10 programs in more detail.

Overall Transition Rates Not Determinable, but Selected Programs Illustrate Factors for Transition Success

Since 2010, DARPA has had success in transitioning new technologies from the research environment to military users, including DOD acquisition programs and warfighters. DARPA maintains a portfolio-level database that identifies these outcomes by program. However, the agency’s process for tracking technology transition outcomes is not designed to capture transitions that occur after a program completes and does not provide DARPA with an effective means for updating its database. We used outputs from this database to select 10 case study programs, but later identified inconsistencies affecting three programs in how transition outcomes were reported in the portfolio-level database versus how they were reported in other program documentation. We then concluded that DARPA’s portfolio-level database was unreliable for assessing transition rates and outcomes since fiscal year 2010. Our analysis of the 10 selected programs did, however, identify four factors that contributed to transition successes, the most important of which were military or commercial demand for the planned technology and linkage to a research area where DARPA has sustained interest.

DARPA’s Approach to Developing Technology Contributes to a Broad Definition of Transition Success

DARPA’s technological approach focuses on radical innovation that addresses future warfighting needs, rather than developing technologies that address current warfighting needs. This approach shapes how the agency defines, pursues, and tracks technology transition. DARPA considers a successful transition to be one where its program, or a portion of its program, influences or introduces new knowledge. This knowledge is often passed through program performers, which DARPA relies on to execute technology development in its programs. Typical performers include commercial enterprises; other DOD entities, such as military service laboratories and research agencies; and academic institutions. Further, DARPA generally does not develop technologies to full maturity. Instead, the agency focuses on demonstrating the feasibility of new technologies, which includes verifying that the concepts behind the technologies have potential for real life applications. As a result, most DARPA technologies require additional development before they are ready for operational or commercial use. Therefore, follow-on development is the predominant path of technology transition at DARPA. Table 2 highlights the different technology transition paths that DARPA technologies can take.

DARPA’s definition of what constitutes technology transition reflects one of many in use within DOD. In June 2005, the Office of the Deputy Under Secretary of Defense for Advanced Systems and Concepts in collaboration with the Defense Acquisition University (DAU) published guidance defining technology transition as “the use of technology in military systems to create effective weapons and support systems—in the quantity and quality needed by the warfighter to carry out assigned missions at the ‘best value’ as measured by the warfighter.” However, DOD officials told us the 2005 guidance is outdated, does not constitute department policy, and should only be considered as a useful reference source. In the absence of current DOD policy, in a March 2013 report we identified three communities that DOD technologies typically transitioned to: acquisition programs; directly to the field for use by the warfighter; and to other users such as science and technology organizations, test and evaluation centers, or industry. The communities we identified in 2013 are similar to the transition outcomes listed in the 2005 guidance, which broadly lists commercialization, acquisition program, and follow-on development by the prime contractor as primary pathways of technology transition. In a subsequent report in December 2013, we found further differences among what the military services define as technology transition and additional confirmation that DOD itself lacks a formal definition for technology transition across the department. These variations, in tandem with the absence of a standard DOD-wide definition of technology transition, prevents the military services, DOD research agencies, and other DOD entities from consistently defining and tracking technology transition. This lack of a formal definition of technology transition means that DOD entities, such as DARPA, are free to define and categorize technology transition for themselves.

DARPA’s Tracking of Technology Transition Concludes at Program Completion, Which Limits Data Reliability and Precludes Evaluation of Transition Rates

Following a program’s completion, DARPA officials identify and record transition outcomes in accordance with the technology transition paths identified in table 2. DARPA collects this information within a portfolio- level database that spans all of its recently completed programs. The agency uses this database primarily to provide incoming program managers with training on potential transition opportunities. Figure 2 illustrates in more detail DARPA’s process for assessing technology transition outcomes in its programs.

DARPA’s process for tracking technology transition outcomes is not designed to capture transitions that occur after a program completes and the agency’s agreements with program performers have ended. After this point, however, program performers often continue to develop their technologies using non-DARPA sources of funding. According to DARPA officials, these efforts can result in later transitions of technologies to commercial products—including ones that are sold back to DOD for military use—without the agency’s knowledge.

This process for tracking technology transition outcomes also does not provide DARPA with an effective means for updating its portfolio-level database. We used outputs from this database to select 10 case study programs (5 that transitioned and 5 that did not transition), but later identified inconsistencies affecting three programs in how transition outcomes were reported in the portfolio-level database versus how they were reported in other program documentation that we reviewed. This confusion about ultimate transition outcomes persisted during our interviews with DARPA officials. As a result, we concluded that DARPA’s portfolio-level database was unreliable for assessing transition rates and outcomes since fiscal year 2010. Table 3 highlights the inconsistencies we found in our reviews.

The inconsistencies we identified suggest that DARPA’s current approach to tracking technology transitions can limit its understanding of transition outcomes. This may undermine its ability to craft transition plans for new programs based on the lessons learned from previous programs. We have previously identified technology transition tracking as a longstanding issue at DOD. For example, in September 2006, we found that tracking technology transitions and the effect of transitions, such as cost savings or deployment of the technology in a product, provided key feedback that can inform the future management of programs. However, in March 2013, we found that DOD stopped tracking transition outcomes in many programs once a program stopped receiving funding, which consequently limited visibility into the extent of successful transitions within the DOD portfolio.

Selected Programs Indicate That Several Factors Contribute to Successful Transitions

DARPA has undertaken efforts to understand the elements that contribute to or impede successful technology transitions. According to DARPA officials, a technology’s maturity level, availability of military service funding, alignment with military service requirements, and transition planning by the program manager influence whether or not a DARPA- developed technology successfully transitions. These characteristics align with the findings of a 2001 DARPA-funded study, which reported that mission, program manager turnover, timing, funding, and regulations, among other elements, affect transition success.

In our review of 10 case study programs, we found different, but related, factors for transition success as compared to the ones put forward by DARPA: (1) military or commercial demand for the technology, (2) linkage to a research area where DARPA has sustained interest, (3) active collaboration with potential transition partners, and (4) achievement of clearly defined technical goals. Based on our analyses, we identified two factors—military or commercial demand for the planned technology and linkage to a research area where DARPA has sustained interest—as factors that were generally evident at program initiation and were most important to transition. The remaining two factors—active collaboration with potential transition partners and achievement of clearly defined technical goals—sequentially follow the first two factors and become observable once a program is underway. Figure 3 highlights these four factors.

In reviewing the 10 programs, we found that the existence of the factors identified varied from program to program. We assessed the extent to which the four factors were present within the 10 programs we reviewed, and table 4 highlights these results.

We found that successful transitions were often underpinned by existing military or commercial demand for the technology. DARPA officials told us that all of the agency’s programs are linked to military and joint service needs at a high level, but through our analyses, we found that this commitment was exemplified when any of the following components were present in the program files:

Agreement between DARPA and (1) a military service, (2) a DOD research agency or laboratory, or (3) other warfighter representative that a related military capability gap or requirement exists; or

A private company identified a commercial demand for the technology or showed an interest in commercializing it.

For example, the Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) program addressed a known capability gap for speech translation technology within the Army. As a result, the Army developed the appropriate requirements documents that allowed the technology to successfully transition to an Army acquisition program of record. These documents identified desired performance attributes and system parameters, which served to better define and communicate the Army’s need for TRANSTAC. The Army’s decision to validate specific performance requirements provided TRANSTAC an opportunity to transition into an Army program of record.

Within our 10 case studies, we found that a military or commercial demand was fully present within four of the five programs that successfully transitioned. In two cases—TRANSTAC and Quint Networking Technology (QNT)—near-term military demand was a result of DOD’s ongoing involvement in warfighting operations. However, in the other two cases, an immediate military need for the technology was not as prevalent. A fifth program that transitioned, Advanced Wireless Networks for Soldier (AWNS), initially was in demand by the Army, but interest waned over time as other options for radio networking platforms emerged. In addition, several programs developed technologies that demonstrated military applicability but lacked a military or commercial demand, which precluded successful transition. For example, the Predicting Health and Disease (PHD) and Nastic Materials programs successfully demonstrated innovative research concepts that had potential military applications, but an immediate military/commercial demand simply did not exist without further maturation of technologies past the point of program completion.

Linkage to a Research Area Where DARPA Has Sustained Interest

We also found that a program’s linkage to a research area in which DARPA has sustained interest often facilitated successful transition. This interest was demonstrated by evidence that in the years preceding the program’s initiation, at least two related DARPA or other related DOD science and technology programs had been completed. Sustained interest is also exemplified by a program’s reuse of existing research facilities and data from related programs, among other things. Of our 10 case studies, all 5 programs that successfully transitioned were fully linked to sustained research interests, whereas 4 of the 5 non- transitioning programs did not have any such linkage.

DARPA’s program portfolio is currently organized around 10 research focus areas under four key research themes. DARPA officials report that the Hypersonics Capability focus area, for example, reflects an ongoing interest for the agency that dates back to the mid-1980s. The Falcon Combined-cycle Engine Technology (FaCET) is one of several recent DARPA programs within the Hypersonics Capability focus area. In addition, FaCET’s research was done in concert with other hypersonic programs within DOD. As a result of this sustained interest, FaCET technologies transitioned to other hypersonics programs, including DARPA’s Mode Transition program, the joint DARPA/Air Force Hypersonic Air-breathing Weapon Concept program, and the Air Force Research Laboratory’s Robust Scramjet and Enhanced Operability Scramjet Technology. Moreover, due to the National Aeronautics and Space Administration’s (NASA) sustained involvement in FaCET, technologies were also transitioned to NASA’s Glenn Research Center’s Combined-Cycle Engine Large Scale Inlet Mode Transition Experiment program.

Active Collaboration with Potential Transition Partners

We found that in all five cases where transition occurred, active collaboration with potential transition partners was fully present. This collaboration generally consisted of early program involvement by stakeholders within the government and commercial sectors, service requirements officials, and military liaison officers, among others. DARPA program managers were responsible for facilitating this early stakeholder involvement, including identifying the potential transition partners needed to assist with their programs. According to DARPA officials, achieving active collaboration with potential transition partners is highly dependent on the nature of the program and background of the program manager, which might be in academia, private industry, or military services. For example, a program manager with a military background might be familiar with DOD’s acquisition process and have connections with service officials who can facilitate transition. On the other hand, a program manager with an academic background might lack DOD service connections, in which case DARPA’s military liaison officers can be used to facilitate collaboration.

DARPA’s Architecture for Diode High Energy Laser Systems (ADHELS) program exemplifies how active collaboration with potential transition partners can facilitate successful technology transition. ADHELS development included several technological components, including volume bragg grating (VBG) technology. VBG is a transparent device made of refractive glass that when combined with a diode laser can control the laser output—such as by magnifying laser power, narrowing a laser beam, or controlling the beam quality of the laser diode. According to DARPA officials, the agency contracted with the foremost experts on VBG technology to develop ADHELS components, recognizing that adaptations of the VBG technology had potential applications within the commercial marketplace. As ADHELS development progressed, DARPA continued to engage its performers, who then licensed the VBG technology to an ADHELS subcontractor. This subcontractor formed the commercial entity Optigrate to further develop the VBG technology for commercial sale.

Conversely, the programs that lacked active collaboration with potential transition partners encountered challenges such as funding shortfalls, requirements uncertainties, and underperforming technologies. For example, early technical challenges prompted DARPA to restructure the Self-Regenerative Systems (SRS) program to focus exclusively on technology maturation, canceling initial plans to demonstrate and evaluate SRS technologies on a transition partner’s system. This decision constrained opportunities to identify potential transition partners and actively collaborate with them during the program.

Achievement of Clearly Defined Technical Goals

We found that defining and, ultimately, achieving clear technical goals helped facilitate technology transition. Of the five programs that successfully transitioned, this factor was fully present in three programs and partially present in the remaining two. Clearly defined technical goals often existed in the form of documented agreements among stakeholders that outlined technical specifications and desired capabilities, funding requirements, development schedule, and organizational responsibilities for technology development. These agreements allowed DARPA to share development, management, and funding responsibilities with its service partners, which facilitated shared understanding of technical goals and mutual commitments to the program’s success and transition. Equally important to this factor though was the degree to which a program achieved its stated technical goals. Most of the programs we reviewed identified clear technical goals, but fewer than half actually achieved the technical goals that were originally set.

DARPA’s QNT program represents one example where clearly defined technical goals were set and achieved. QNT was initiated with support from the Air Force and Navy, which helped DARPA craft clear technical goals including size, weight, robustness, transmission rates, and other performance attributes of the technology. Defining technical goals during the early stages of the program also secured each organization’s commitment to playing a role in managing, developing, funding, demonstrating, and testing QNT. As a result, stakeholders then worked together to test QNT technical performance at several military exercises and in theater, where the system performed to expectations and gained added exposure within DOD. Ultimately, QNT transitioned to the Army’s Intelligence, Surveillance, and Reconnaissance Network program, which fielded the system in Afghanistan in September 2011. QNT also transitioned to two Navy weapons programs and was also selected by the Air Force for use in its Battlefield Airborne Communications Node program, which hosts a data link communications system between aircraft and ground units.

In other cases, such as AWNS, ADHELS, and Nastic Materials, technical goals were clearly defined, but only partially met. These partial successes, nonetheless, produced substantive technological gains. In the cases of AWNS and ADHELS, these gains—coupled with the presence of other key factors—proved sufficient to promote technology transition. On the other hand, three programs lacked clearly defined goals—or did not substantively achieve those goals—which led to significant restructuring or development of technologies that did not align with the needs of a planned transition partner. For example, Marine Corps’ officials stated that the Tactical Underwater Navigation System relied on divers swimming at unsustainable speeds to calibrate its positioning, which was not responsive to their interests.

DARPA Prioritizes Innovation and Deemphasizes Technology Transition in Key Processes

DARPA’s investment of program funds and staff are primarily focused on the highest priority of its agency mission, which is creating radically innovative technologies that support DOD’s warfighting mission. Technology transition is a secondary priority at the agency. DARPA leadership conducts periodic reviews of agency programs, but these reviews are focused on scientific and technical aspects of the programs and do not assess technology transition strategies. Instead, the Director, DARPA, delegates responsibility for oversight and assessment of technology transition strategies to a subordinate office. DARPA also provides limited training to program managers related to technology transition, instead relying on others within the agency to assist program managers with this activity, as needed. In addition, although DARPA disseminates information on its past programs, it does not take full advantage of available, government-sponsored resources for sharing technical data. DARPA has also elected not to participate in most DOD programs intended to facilitate technology transition, with the exception of mandated small business programs, citing the challenges it perceives in meeting the process and reporting requirements of these DOD programs within DARPA’s typical timeframes for executing its research initiatives.

DARPA Investments in People and Programs Drive a Culture of Innovation

At DARPA, the desire for innovation drives investment, both in terms of recruitment and programs. DARPA hires world-class scientists and engineers from private industry, universities, government laboratories, and research centers to serve as program managers. According to DARPA officials, program managers are given great flexibility in leading their programs, building their teams, and allocating funds to achieve their programs’ objectives, including technology transition. DARPA officials stated that these expectations are outlined to program managers during new hire orientations, but are not codified in any agency-wide policy or guidance. To ensure that new ideas for advanced technologies are continuously coming into DARPA, the agency usually limits the tenure of its program managers, as well as the duration of its programs, to 3 to 5 years. In this environment, program managers prioritize achieving programs’ technical objectives, which can require the overwhelming majority of their available time.

This focus on innovation, which corresponds with undertaking bold, ambitious programs, makes the pursuit of technology transition a secondary priority for the agency. Consequently, programs generally seek to prove the art of “what is possible” rather than refining, producing, and delivering tactical equipment to warfighters. According to DARPA officials, the agency views these latter processes as the responsibility of military service research agencies, laboratories, and acquisition programs of record. However, DARPA officials report that potential transition partners in the acquisition community are often unwilling to commit to incorporating new technology into their programs of record without additional maturation, and service research agencies and laboratories both have their own programs and priorities to pursue. Consequently, the additional maturation work needed to position DARPA programs for effective transitions can go unfunded. According to DARPA officials, this dynamic has proven to be a major impediment for the agency in transitioning technology.

In addition, the introduction of DARPA’s radically innovative technologies can disrupt the status quo for military programs, budgets, and warfighting doctrine, which can drive cultural opposition within the military services. DARPA officials stated that the agency’s research sometimes leads to the identification of technologies and capabilities that military service officials do not initially want or think their services will need, although these technologies can eventually provide important military capabilities. For example, DARPA officials said that the Air Force was initially highly resistant to investments in stealth technologies for aircraft. Despite this resistance, DARPA proceeded with the development of stealth technologies, and today they are in use on multiple DOD weapon systems, including the F-22 Raptor and F-35 Lightning II fighter aircraft.

DARPA’s secondary emphasis on transition is a long-standing characteristic of the agency’s culture, as evidenced in studies commissioned by DARPA in 1985 and 2001, which found that the agency does not place enough emphasis on technology transition. The 1985 report recommended that DARPA designate full-time technology transition facilitators, due to problems that were identified in the transition of technologies to the military services. The 2001 report recommended matching program manager tenure to the expected length of the programs to which they are assigned—rather than setting arbitrary dates of departure—and defining additional training and incentives for technology transition.

According to DARPA officials, the Director, DARPA, has undertaken several initiatives to improve the agency’s emphasis on technology transition, including transition-focused quarterly meetings with each of the military service chiefs or their deputies and establishment of the Adaptive Execution Office in 2013, which was chartered to accelerate the transition of game-changing DARPA technologies into DOD capabilities. In addition, DARPA officials stated that the Director has shifted the role of the agency’s military service liaisons to focus exclusively on assisting program managers with military service engagement and transition of DARPA technologies. DARPA officials report that these actions have elevated the priority of and resources devoted to technology transition within the agency.

DARPA Leadership Forgoes Opportunities to Assess, and Thus Potentially Improve, Technology Transition Strategies

The Director, DARPA, conducts oversight of programs through periodic milestone reviews. These reviews assess a program’s scientific and technical merit, and, according to DARPA officials, provide the Director with information on the transition status of the program. According to DARPA officials, the scope of these reviews is reflective of and consistent with the agency’s top priority of creating innovative technologies. However, these reviews do not assess a program’s strategy for achieving technology transition. DARPA leadership delegates oversight and review of technology transition strategies to the agency’s Adaptive Execution Office, which coordinates with program managers to review and provide input on technology transition strategies, particularly in the latter stages of programs.

DOD policy, however, assigns to the Director, DARPA, the responsibility to pursue “strategies” that “increase the impact of DARPA’s research and development programs” and “speed the transition of successful research and development programs to the military departments and defense agencies,” among other scientific and technical functions. Consequently, by not assessing technology transition strategies at the program milestone reviews it chairs, the Director, DARPA, is forgoing key opportunities to perform this function. This approach undermines transition planning and introduces risk that DARPA programs will not achieve their full transition potential.

Apart from the policy cited above, the Office of the Secretary of Defense does not maintain other instructions or directives related to technology transition at DARPA. In previous years, different components within the Office of the Secretary of Defense have issued nonmandatory guidance on technology transition, which has, at times, applied to DARPA programs. However, the guidance is now outmoded in that it does not address changes in key science, technology, and acquisition processes that have occurred during the last 10 years. The Office of the Assistant Secretary of Defense for Research and Engineering, which has primary oversight of DARPA and other DOD research agencies, provides a great deal of latitude to these agencies to define their own technology transition policies and procedures. Most notably, officials from this office stated to us that technology transition is no longer an explicit function of the office and that the DOD division formerly responsible for technology transition no longer exists. Instead, the office now limits its technology policy responsibilities to minimizing unnecessary duplication of research efforts within DOD, disseminating research knowledge throughout DOD, and sharing that information with the general public.

Current Training for Managers Does Not Consistently Position Programs for Transition Success

DARPA’s program managers receive limited training on how to effect technology transition in their programs. This training consists primarily of overviews on DARPA’s technology transition paths and considerations to make at program milestones with respect to technology transition. DARPA program managers are not subject to the formal training and certification requirements applicable to permanently hired science and technology managers at military service laboratories. DOD requires managers in these laboratories to complete Defense Acquisition University (DAU) training courses in science and technology, including how they apply to technology transition. These courses lead to progressively higher knowledge and certifications over their careers. DARPA officials countered that the training necessary to complete these courses and achieve science and technology manager certifications would require DARPA program managers to devote an inordinate amount of time to training, particularly if DAU requires DARPA staff to complete all the typical prerequisite courses that other managers are required to complete. Further, given the agency’s unique mission, DARPA officials stated that they do not consider the DAU training courses to be as relevant to their program managers given the agency’s broad discretion to pursue breakthrough technologies versus specific management of acquisition programs.

In lieu of more robust training, DARPA officials stated that the agency supports its program managers’ transition efforts by providing them with access to various transition planning and outreach resources. For example, DARPA program managers are also supported by the agency’s Adaptive Execution Office, which provides them with assistance in developing their transition plans and in communicating with the military’s transition stakeholders at DOD’s combatant commands. Program managers are further supported by DARPA’s military liaison officers from the Air Force, Army, Navy, and Marine Corps, who help program managers identify and reach out to potential transition partners or other stakeholders in the military services. These liaisons also arrange for DARPA leadership to meet with the military’s senior leaders, which is done in part to advocate for the transition of DARPA programs to the military services. As we found in September 2006, these liaisons can also provide operational advice for planning and strategy development and provide an understanding of service perspectives, issues, and needs so that potential customers can be identified and effective agreements can be written. Program managers also are authorized to use program funds to hire experienced contractors and government staff from other agencies to aid technology transition activities in their programs.

Previous guidance and studies, including one commissioned by DARPA in 2001, have recommended that DARPA improve its technology transition training for program managers through additional training and mentoring programs related to technology transition. Further, in 2005, DOD issued guidance on technology transition stating that developing and executing a training plan for the members of the team supporting technology transition is essential to their success. Similarly, Standards for Internal Control in the Federal Government indicates that effective management of an organization’s workforce, which includes providing necessary training to the organization’s staff, is essential to achieving results. DARPA’s limited training for program managers on technology transition is inadequate to consistently position programs for transition success. Without sufficient training, program managers may not develop the skills and knowledge that they need to identify and engage potential transition partners and facilitate transition successes.

While DARPA does not currently rely on other DOD entities for technology transition training, individual DARPA program managers may voluntarily elect to take training related to technology transition in DOD or other federal organizations. For example, the Federal Laboratory Consortium for Technology Transfer was established by law in the Federal Technology Transfer Act of 1986 to, among other things, (1) develop training for federal lab employees engaged in technology transfer and (2) facilitate communication and cooperation between federal laboratories. The Consortium offers both in-person and online training regarding commercialization of technologies, as well as guidance regarding best practices. In response to our inquiries on this subject, DARPA officials indicated they were in discussions with the DAU staff regarding potential future training options.

Shortfalls in Disseminating Program Information May Lead to Missed Transition Opportunities

Disseminating information regarding developed technologies is a way for agencies to promote technology transition after the conclusion of a program, particularly once program managers and staff are no longer actively advocating for the transition of their program’s technologies. For many years, DOD has maintained website-accessible databases that disseminate information within the department, and to a lesser extent, to the public and to private companies. These websites allow their users to search for related technologies while considering new programs or products that could possibly use them.

While DARPA disseminates information on past programs through the use of public government websites, its selective approach to posting this information does not maximize the chances of DARPA technologies being identified and selected by potential transition partners. Currently, DARPA disseminates information on past programs through both internal and external means, but does not share information with key data repositories that the federal government sponsors, which may obscure visibility into its programs and lead to missed transition opportunities. Since the 1960s, DARPA has provided substantial amounts of information regarding its technologies to the official DOD dissemination website managed by the Defense Technical Information Center (DTIC). Although the majority of DARPA-related information in this database is restricted to DOD staff, it is by far the largest repository that private companies and the general public can access for information on DARPA technologies. For instance, we found that while non-DOD users can access approximately 3,600 DARPA technical records through DTIC’s public website, DOD users can access over 30,000 of these records. DARPA also maintains an “Open Catalog” public website for disseminating information on its programs, although it currently only has technical information on about a few dozen active and completed programs. In comparison, DARPA’s public website also provides brief, non-technical descriptions of 194 active DARPA programs. Two other government-sponsored websites, operated by the DOD TechLink public-private partnership and the Federal Laboratory Consortium for Technology Transfer, also exist to help science and technology agencies disseminate technology information. DARPA officials indicated they do not share information with either of these entities and instead exclusively rely on DTIC, which DARPA officials stated represented DOD’s official repository.

In recent years, the White House has provided direction to broaden access to non-sensitive information on government-developed technologies, in recognition of government research’s potential for catalyzing innovative breakthroughs that drive the U.S. economy, and helping to drive progress in areas such as health, energy, the environment, agriculture, and national security. In February 2013, the White House’s Office of Science and Technology Policy instructed the federal government’s science and technology community to begin planning how to disseminate information on technologies they have developed. According to DARPA officials, the lead DOD agency for implementing this system is DTIC, and they do not expect DOD to have a dissemination system in place that fully addresses the requirements of the memorandum until 2017.

DARPA Participates in Mandated Small Business Programs, but Refrains from Other DOD-Wide Programs Intended to Facilitate Technology Transition

The Office of the Secretary of Defense manages several DOD programs intended to accelerate development, testing, and delivery of mature technologies that provide new solutions for military needs. The general purpose of these programs is to facilitate the transition of technologies, but vary in terms of what types of technology developers and operational needs they target. For example, in partnership with the military services, the Joint Capability Technology Demonstration program addresses joint warfighting needs of the combatant commands by demonstrating mature technology prototypes that may transition to acquisition programs or directly to the warfighter in the field. Other programs such as the Small Business Innovation Research (SBIR) program fund small business research and development with the goal that innovations produced will be commercialized and eventually sold back to DOD. Table 5 lists these programs.

According to DARPA officials, the only programs that DARPA participates in are the SBIR and Small Business Technology Transfer (STTR) programs because it is legally required to do so. However, the agency’s knowledge of transition outcomes associated with SBIR and STTR expenditures is limited. DARPA officials said they do not maintain a comprehensive list of agency programs using SBIR or STTR funds—or the transition paths of technologies developed with these funds—because these data are not always reported or accessible. According to DOD small business program management officials, who oversee the use of these funds throughout the department, transition outcome data is not required from any military service or agency, including DARPA. These officials further stated that once a small business contract ends, DOD’s means for compelling contractors to identify and report on successful transitions expires. In December 2013, we recommended that DOD improve its tracking of technology transition outcomes in SBIR-funded programs by establishing a common definition of technology transition for all SBIR projects and improving the completeness, quality, and reliability of SBIR transition data that it reports. These tracking shortfalls precluded us from assessing the extent to which DARPA’s SBIR and STTR funds contribute to successful technology transitions. In lieu of comprehensive transition data, DARPA officials have worked with some of their prior program contractors—who successfully developed and transitioned technologies—to identify small business program success stories. In addition, DARPA officials stated that they are developing contract language for future SBIR awards that would require firms to identify their transition and commercialization outcomes as an addendum to their final report.

Apart from the legally required small business programs, DARPA officials said that the processes and reporting requirements associated with participating in DOD’s other technology transition programs are generally cumbersome and do not align with DARPA’s time frames for executing programs or mission of creating disruptive technologies over relatively long periods of time. Conversely, DOD transition programs are mainly intended for mature technologies, or short-term efforts that can be fielded quickly. DARPA officials explained that technologies their programs develop usually require additional maturation in subsequent technology development efforts, either within DARPA or at military service laboratories, before transitioning to acquisition programs or warfighters. DARPA officials also said that agency leadership generally views the use of these funds as unnecessary given that DARPA’s budget currently provides adequate funding to support its research endeavors.

DARPA officials also indicated that they are exploring stronger relationships with the Joint Capability Technology Demonstration (JCTD) program, particularly in the area of prototyping. In previous decades, DARPA used funds from the predecessor to the JCTD program—then known as the Advanced Concept Technology Demonstration program—to develop and demonstrate technologies. These efforts include currently fielded systems such as the Air Force’s Global Hawk and Predator unmanned aircraft and Miniature Air-Launched Decoy systems. DARPA officials also stated that Manufacturing Technology program funds have been applied after DARPA program completions to improve the affordability of and manufacturing base for semiconductors developed by DARPA.

Conclusions

Technology transition does not have to occur at the expense of innovation, but should instead be viewed as a natural extension of innovation. When DARPA places technology in the hands of a user, operational knowledge is gained that can be used to improve the technology and further scientific innovation. However, DARPA leadership does not fully subscribe to this viewpoint; instead, it is satisfied with maturing technology to the point where feasibility, but not functionality, is proven. Today, programs progress through DARPA without the agency head fully assessing whether transition strategies make sense. Such assessments, if measured against key transition factors, could improve a program’s potential for transition success. Transition responsibilities then fall almost exclusively on individual program managers, who are often not sufficiently trained to achieve the favorable transition outcomes they seek. Further, when the program manager’s tenure expires, the primary advocate for transitioning the program’s technology is also lost. This turnover increases the need for technical gains to be appropriately documented and disseminated so that user communities have visibility into potential solutions available to meet their emerging needs. An important part of this process is the tracking of transition outcomes, as we recommended DOD undertake for its technology transition programs in March 2013, and which we have also found lacking at DARPA.

Recommendations for Executive Action

To improve technology transition planning and outcomes at DARPA, we recommend the Secretary of Defense direct the Director, DARPA, to take the following three actions:

Oversee assessments of technology transition strategies for new and existing DARPA programs as part of existing milestone reviews used to assess scientific and technical progress to inform transition planning and program changes, as necessary. Our analysis identified four factors that could underpin these assessments, but the uniqueness of individual DARPA programs suggests that other considerations may also be warranted.

Increase technology transition training requirements and offerings for DARPA program managers, leveraging existing DOD science and technology training curricula, as appropriate.

Increase the dissemination of technical data on completed DARPA programs through Open Catalog and other government-sponsored information repositories aimed at facilitating commercialization of technologies.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD for review and comment. In its written comments, which are included in appendix II, DOD partially agreed with our recommendations to oversee assessments of technology transition strategies for DARPA programs and to increase technology transition training requirements and offerings for DARPA program managers. In doing so, DOD agreed with most of the principles contained in our recommendations, but disagreed with the actions we recommended. DOD did not agree with our recommendation to increase the dissemination of technical data on completed DARPA programs. DOD also separately provided technical comments, which we incorporated, as appropriate.

DOD agreed that assessments of technology transition strategies, which consider the four factors we identified for transition success, would help inform program decisions by DARPA leadership. However, DOD did not agree that such assessments be required at milestone reviews for DARPA programs, citing active participation by the Director, DARPA, in technology transition discussions throughout the life of a program. We agree that leadership is focused on technology transition and holds discussions often; however, we found it difficult to be able to identify transitions—or changes to transition strategies—that arise from these discussions. We believe that these discussions are an inadequate substitute for assessing technology transition strategies as part of the comprehensive program reviews that DARPA already undertakes. Assessing transition strategies at these reviews, as we recommended, would provide the opportunity to coordinate and prioritize transition goals, objectives, and planned actions in the context of scientific and technical developments in the program. By overseeing technology transition strategies separate from these reviews, the Director, DARPA, risks making decisions related to a program’s transition that are not appropriately informed by other important program considerations. Although DARPA asserted that our recommendation runs counter to its current efforts to improve processes and procedures, we found no evidence that processes and procedures were improving.

DOD also agreed that technology transition training improves transition planning and outcomes, citing DOD science and technology training curricula as a “rich repository of transition insight.” Yet, despite the value it sees in its own training resources, DOD stated that DARPA program managers’ relatively short tenure leaves few opportunities to expose them to such “generic” training opportunities. Consequently, DOD did not agree that technology transition training requirements should be increased for DARPA program managers and stated that DARPA’s current approach of “tailored curricula focused on a program’s unique transition needs” remained appropriate. However, we did not find evidence of such tailored curricula in our review. Instead, we found that DARPA program managers all received the same limited training upon hiring, which was inadequate to consistently position programs for transition success. DOD also stated that DARPA continues to explore opportunities to offer tailored, concise, and streamlined training to its program managers. Therefore, we stand by our recommendation and continue to believe that expanded training opportunities are necessary for achieving better transition outcomes in DARPA programs, and we encourage DOD to capitalize on its existing investments in this area, to the extent possible.

Further, DOD did not agree that increased dissemination of technical data on completed DARPA programs was warranted. DOD stated that using multiple information repositories “thins the DOD technology market by spreading it across several venues,” in turn reducing the likelihood that technology providers and potential transition partners will find a match. DOD also stated that it intends to make DTIC the central data storage for all DOD technical activities, including DARPA technologies, and views the use of multiple information repositories as unconducive to improving technology transition outcomes. In our review, we found DARPA’s existing reliance on DTIC limited the chances of the agency’s technologies being identified and selected by potential transition partners, particularly those outside of DOD. We fail to see how increased dissemination of technical data would actually “thin” the DOD technology market. To the contrary, it would allow more portals with which to gain access. Similarly, in 2013, the White House identified a government-wide need to broaden access to non-sensitive information on government- developed technologies, but improvements remain incomplete. Consequently, we continue to believe that DARPA should pursue dissemination of non-sensitive technical data through as many existing government-sponsored outlets as possible, including its own Open Catalog website and DOD TechLink, to improve the likelihood of transition successes in the agency’s programs.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and the Director, DARPA. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

This report covers the Defense Advanced Research Projects Agency’s (DARPA) (1) effectiveness at transitioning technologies since fiscal year 2010, including identifying the factors that contributed to successful technology transitions, and (2) implementation of Department of Defense (DOD) policies and programs intended to facilitate the transition of technologies.

To assess DARPA’s effectiveness at transitioning technologies since fiscal year 2010, including identifying factors that contributed to successful transitions, we requested and reviewed portfolio-level data identifying the names, funding amounts, and technology transition of those DARPA programs successfully completing technology development during fiscal years 2010 through 2014. We confined our analysis to this time frame owing to availability of data from DARPA. These data included 150 programs funded under DOD’s budget activities for (1) applied research and (2) advanced technology development that DARPA identified as having completed as planned and producing a substantive technological gain or innovation, regardless of whether that technology or innovation transitioned to an end user. We used these data as the basis for selecting a simple random sample of 10 case study programs—5 that transitioned and 5 that did not transition. In conducting our case study analyses, we reviewed relevant program documentation to identify factors that facilitate transition success. While reviewing these case study programs, we identified inconsistencies between agency portfolio-level transition outcome data and program-level information. DARPA officials stated to us that this was due to the transition status of these programs changing after they had collected the portfolio-level data. As a result, we concluded that DARPA’s portfolio-level data were not sufficiently reliable for the purposes of assessing agency-wide transition rates and outcomes since fiscal year 2010. However, these inconsistencies did not significantly affect our program selections; therefore, these data were sufficiently reliable for the purposes of selecting the 10 case study programs.

To identify factors that facilitated technology transition within the 10 selected programs, we analyzed DARPA provided documentation— including program briefings, memorandums of agreement, broad area announcements, budget documents, and program completion reports— for selected programs to identify factors that facilitated or precluded their individual transitions. We then conducted a content analysis of these individual factors to identify common themes among the programs, which led to us determining that four significant factors underpinned technology transition outcomes in the programs we reviewed. Once we identified these four factors, we developed a rating system to assess the extent to which each factor was present in each of our 10 programs, as supported through our analysis of program documentation. Our measures for each of the four factors were as follows: Military or commercial demand for the planned technology

Fully present: Demand for the technology from a potential transition partner existed throughout the program, which would include (1) agreement between DARPA and a military service, DOD laboratory or other warfighter representatives that a related military capability gap or requirement exists; or (2) a private company identified a commercial demand for the technology or showed an interest in commercializing it.

Partially present: Potential transition partners indicated to DARPA that they believed a demand existed for the technology, as is described above, although their interest was not consistent through the end of the program.

Not present: The factor did not exist at all, and DARPA appears to have initiated the program without a potential transition partner agreeing that a capability gap or potential commercial use existed at any point during the program.

Linkage to a research area where DARPA has sustained interest

Fully present: In the years preceding the program’s initiation, at least two related DARPA or other DOD science and technology program had been completed.

Partially present: In the years preceding the program’s initiation, at least one related DARPA or other science and technology program had been completed (this was the second DOD science and technology program of its kind).

Not present: The factor did not exist at all, and this program appears to not have any roots in previous similar DARPA or other DOD science and technology programs.

Active collaboration with potential transition partners

Fully present: Potential transition partners consistently participated in, advised, or otherwise supported the program.

Partially present: Potential transition partners participated in, advised, or otherwise supported the program, although their involvement was not consistent through the end of the program or was not present until after the prototype demonstration (relatively late in the program).

Not present: The factor did not exist at all. The program appears to have lacked assistance from any potential transition partners, or their assistance was very infrequent or insignificant.

Achievement of clearly defined technical goals

Fully present: Measurable technical goals were set in the program and fully achieved to the satisfaction of DARPA and any transition partner involved in the program, to the extent that one or more had been identified for the technology.

Partially present: Measurable technical goals were set in the program, but met with varying levels of success. The technical successes achieved, however, were sufficient to produce a technology responsive to the interests of a transition partner, to the extent that one or more had been identified for the technology.

Not present: Measurable technical goals were either not set or sufficiently met in the program. The level of technical success was not sufficient to produce a technology responsive to the interests of a transition partner, to the extent that one or more had been identified for the technology.

Using this rating system, two GAO analysts analyzed and coded whether each of the four factors was fully present, partially present, or not present in each of the 10 programs we reviewed. Each GAO analyst coded all the constituent items independently, and the two analysts then met to discuss and reconcile the differences between their codings. Following this initial round of coding, another GAO analyst independently verified the accuracy of the coding by reviewing the supporting program documentation. The final assessment reflected the analysts’ consensus based on the individual assessments.

To assess DARPA’s implementation of DOD policies and programs intended to facilitate the transition of technologies, we identified and analyzed information sources including policy instructions, guidance, training materials, and technical data repositories intended to promote technology transition within DARPA, DOD and the federal government. We also reviewed previous federal directives issued by the Executive Office of the President that were related to technology transition at DARPA. We reviewed previous DARPA-sponsored reports on technology transition produced in previous years. We reviewed our prior related reports and program information regarding DOD’s technology transition programs and relevant DOD funding information. We reviewed available training, resources, and tools used by DARPA officials to help bring about technology transition. We reviewed the contents of DOD computer systems used to disseminate information on DARPA programs to potential transition partners. We reviewed our prior reports and DOD documentation on DOD transition programs, including the Small Business Innovation Research, Small Business Technology Transfer, and Joint Capability Technology Demonstration programs, among others, to understand the extent to which DARPA participates in these programs. We reviewed the extent to which DARPA uses DOD transition funds, and requested data regarding DARPA’s use of small business funds and its technology transition outcomes, although these data were unavailable for our analysis, as is discussed elsewhere in this report. We also reviewed historical information on DARPA’s use of DOD transition funds available from public sources, including DOD budget documentation.

To gather additional information in support of our review for both objectives, we conducted interviews with current and former officials responsible for executing, managing, and overseeing transition of DARPA-developed technologies, including representatives of DARPA’s senior leadership and Adaptive Execution Office, program management offices and selected program managers, military services liaisons, and small business program officials. We also interviewed officials from the Office of the Assistant Secretary of Defense for Research and Engineering and DOD’s Office of Small Business Programs. Further, we interviewed officials from selected military requirements and acquisition offices, including the Joint Staff’s Force Structure, Resource, and Assessment directorate; Office of the Deputy Chief of Staff of the Army for Operation, Plans and Training; Office of the Deputy Chief of Staff of the Army for Logistics; Office of the Assistant Secretary of the Army for Acquisition, Logistics, and Technology; Army Program Executive Offices for Intelligence Electronic Warfare and Sensors and Command, Control, Communications—Tactical; and Marine Corps Systems Command. We also met with staff from selected DOD research centers, including the Air Force Research Laboratory and the Office of Naval Research, and with the Director of Science and Technology curriculum at the Defense Acquisition University.

We conducted this performance audit from January 2015 to November 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Diana Moldafsky, Assistant Director; Christopher R. Durbin, Analyst in Charge; Emily Bond; Nathan Foster; Aaron M. Greenberg; John Krump; Jean L. McSween; Sean Seales; and Roxanna T. Sun made key contributions to this report.