Defense Advanced Research Projects Agency: Key Factors Drive Transition of Technologies, but Better Training and Data Dissemination Can Increase Success
Published: 2015-11-18T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: After the Soviet Union launched the first satellite into orbit in 1957, the U.S. government made a commitment to initiate, rather than react to, strategic technological surprises. DOD relies on DARPA's disruptive innovations to maintain this promise, backed by congressional appropriations of over $2.9 billion in fiscal year 2015 alone. In April 2015, DOD reported that U.S. technological superiority is again being challenged by potential adversaries and renewed efforts to improve its products. Meanwhile, GAO found deficiencies in DOD's technology transition processes that may hinder these efforts and DARPA's goals.
Senate Report 113-176 included a provision for GAO to review DOD's technology transition processes, practices, and results. This report focuses on DARPA and assesses its (1) effectiveness at transitioning technologies since fiscal year 2010, including identifying factors that contribute to successful transitions, and (2) implementation of DOD policies and programs intended to facilitate technology transition. GAO reviewed DARPA programs completed since 2010; identified transition factors by analyzing program documentation for a random sample of 10 cases; reviewed DOD policies; and interviewed DOD officials.
section_title: What GAO Found
paragraphs: Since 2010, the Defense Advanced Research Projects Agency (DARPA) has had success in technology transition—the process of migrating new technologies from the research environment to military users, including Department of Defense (DOD) acquisition programs and warfighters. However, inconsistencies in how the agency defines and assesses its transition outcomes preclude GAO from reliably reporting on transition performance across DARPA's portfolio of 150 programs that were successfully completed between fiscal years 2010 and 2014. These inconsistencies are due in part to shortfalls in agency processes for tracking technology transition. Nevertheless, GAO's analysis of 10 selected programs identified four factors that contributed to transition success, the most important being military or commercial demand for the planned technology and linkage to a research area where DARPA has sustained interest. Both of these factors were generally evident at the time a program started, while the other two factors were observed later, once the program was underway. The figure below highlights the four factors.
DARPA's implementation of DOD programs intended to foster technology transition has been limited and neither DOD nor DARPA have defined policies for managing transition activities. DARPA has also largely elected not to participate in DOD technology transition programs, with the exception of federally mandated small business programs, citing challenges in meeting program requirements within DARPA's typical three- to five-year timeframe for executing its research initiatives. Instead, DARPA primarily focuses its time and resources on creating radically innovative technologies that support DOD's warfighting mission and relegates technology transition to a secondary priority. DARPA leadership defers to its program managers to foster technology transition, but provides limited related training. Moreover, while its leadership conducts oversight of program managers' activities through periodic program reviews, these reviews do not regularly assess technology transition strategies. GAO has found that this approach does not consistently position programs for transition success. Further, while DARPA disseminates information on its past programs within DOD, to the public, and among private companies, it does not take full advantage of government-sponsored resources for sharing technical data, which may obscure visibility into its programs and lead to missed transition opportunities.
section_title: What GAO Recommends
paragraphs: DARPA should regularly assess technology transition strategies, refine training requirements, and increase dissemination of technical data for completed programs. DOD did not agree to take GAO's recommended actions, which remain warranted, as discussed in the report.

section_title: Letter
paragraphs: The Defense Advanced Research Projects Agency (DARPA) is a  Department of Defense (DOD) agency whose mission is to create new,  disruptive technologies in the interest of national security. Its genesis  dates to the Soviet Union’s launch of the first satellite, Sputnik 1, into orbit  in 1957, which spurred a commitment by the U.S. government that, from  that time forward, it would initiate, rather than react to, strategic  technological surprises. DOD relies on technological innovation produced  by DARPA to maintain this promise—and ensure superiority of the  nation’s weapon systems and armed forces—to protect U.S. interests  both at home and abroad. In fiscal year 2015, Congress appropriated  over $2.9 billion to DARPA to support its mission and objectives, which  include basic and applied research activities, as well as advanced  technology development.
In April 2015, the Under Secretary of Defense for Acquisition, Technology  and Logistics reported concern that U.S. technological superiority is being  challenged by potential adversaries in ways not seen since the Cold  War. As a result, the Under Secretary is leading new efforts aimed at  strengthening DOD innovation and technical excellence and the ability of  DOD products to continue providing military technological superiority.  However, we have previously found deficiencies in DOD’s processes for  migrating new technologies from the laboratory or research environment  to the acquisition and warfighter communities—a pursuit known as  technology transition—that may undermine the Under Secretary’s efforts  and DARPA’s goals. Congress also recently expressed concern  regarding barriers to technology transition. Senate Report 113-176 to the  National Defense Authorization Act for Fiscal Year 2015 included a  provision that GAO review DARPA and other DOD technology transition  processes, practices, and results. Our work focused exclusively on  DARPA and assessed the agency’s (1) effectiveness at transitioning  technologies since fiscal year 2010, including identifying the factors that  contributed to successful technology transitions, and (2) implementation  of DOD policies and programs intended to facilitate the transition of  technologies.
To assess DARPA’s effectiveness at transitioning technologies, including  identifying the factors that contribute to transitions, we reviewed data on  150 programs that DARPA identified as having completed as planned and  producing a substantive technological gain or innovation for fiscal years  2010 through 2014, regardless of whether that technology or innovation  transitioned to an end user. We confined our analysis to this time frame  owing to availability of data from DARPA. Using this portfolio-level data  set, we randomly selected 10 programs—5 that transitioned and 5 that  did not—for case study analyses. We then analyzed relevant program- level documentation for our selected cases to identify factors that facilitate  transition success. While reviewing these case study programs, we  identified inconsistencies between agency portfolio-level transition  outcome data and program-level information. As a result, we concluded  that DARPA’s portfolio-level data were not sufficiently reliable for the  purposes of assessing agency-wide transition rates and outcomes since  fiscal year 2010. However, these inconsistencies did not significantly  affect our program selections; therefore, these data were sufficiently  reliable for case study selection purposes. We then analyzed DARPA  provided documentation—including program briefings, memorandums of  agreement, and program completion reports—for our selected programs  to identify factors that facilitated or precluded their individual transitions.  We then conducted a content analysis of these individual factors to  identify common themes among the programs, which led us to identifying  the four significant factors that underpinned transition outcomes in the  programs we reviewed.
To assess DARPA’s implementation of DOD policies and programs  intended to facilitate the transition of technologies, we analyzed policy  instructions, guidance, training materials, and technical data repositories  intended to promote technology transition. We also reviewed our prior  reports and DOD documentation on the Small Business Innovation  Research (SBIR) and Small Business Technology Transfer (STTR)  programs and other DOD-wide initiatives aimed at facilitating technology  transition to understand the extent to which DARPA participates in these  programs. We supplemented the above analyses through interviews with  current and former officials from DARPA, the Office of the Secretary of  Defense, selected military service acquisition and requirements offices,  and selected DOD research centers.
We conducted this performance audit from January 2015 to November  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: DOD Technology Management Process
paragraphs: DOD invests about $12 billion in funding to support its science and  technology community, which it relies upon to identify, pursue, and  develop new technologies to improve and enhance military capabilities.  This community is comprised of DOD-wide research agencies, including  DARPA, as well as military service research agencies and laboratories,  test facilities, private industry, and academic institutions, and is overseen  by the Office of the Assistant Secretary of Defense for Research and  Engineering. The research and development activities these different  components engage in are intended to produce mature technologies that  DOD can integrate and deliver in systems that support its warfighters.  This integration process, known as product development, represents the  handover of breakthrough technologies from DOD’s science and  technology community to its acquisition community. Although not  precisely defined, technology transition generally occurs at the point when  advanced technology development ends and this new product  development begins. Figure 1 illustrates DOD’s technology management  process.
DOD has long noted the existence of a chasm between its science and  technology community and its acquisition community that impedes  technology transition from consistently occurring. This chasm, often  referred to by department insiders as “the valley of death,” exists because  the acquisition community often requires a higher level of technology  maturity than the science and technology community is willing to fund and  develop. In 2007, DOD reported that this gap can only be bridged through  cooperative efforts and investments from both communities, such as early  and frequent collaboration among the developer, acquirer, and user.
We have also reported extensively on shortfalls across DOD’s technology  management enterprise in transitioning technologies from development to  acquisition and fielding. In June 2005, we found that DOD technology  transition programs faced challenges selecting, managing, and  overseeing projects, and assessing outcomes. In September 2006, we  found that DOD lacked the key planning, processes, and metrics used by  leading commercial companies to successfully develop and transition  technologies. More recently, in March 2013, we found that the vast  majority of DOD technology transition programs provide technologies to  military users, but tracking of project outcomes and other benefits derived  after transition remained limited.
section_title: DARPA Processes and Programs
paragraphs: DARPA’s scientific investigations run the gamut from laboratory efforts to  the creation of full-scale technology demonstrations in the fields of  biology, medicine, computer science, chemistry, physics, engineering,  mathematics, material sciences, social sciences, neurosciences, and  more. The agency solicits proposals for research work in support of its  scientific endeavors through broad agency announcements. These  solicitations seek thought leaders and technological pioneers that can  leverage new ideas in science to advance the state of the art beyond the  practical application of knowledge. Non-DARPA entities respond to broad  agency announcements by submitting proposals for executing work to  meet the agency’s stated needs. DARPA reviews those proposals based  on technical merit, and entities receiving awards are thereafter referred to  as performers.
To execute solicitations, awards, and program oversight, DARPA relies  on approximately 220 government employees, including nearly 100  program managers. Program managers report to DARPA’s office  directors and their deputies, who are responsible for charting the strategic  directions of six technical offices. The technical staff is supported by  experts in security, legal and contracting issues, finance, human  resources, and communications. DARPA’s Director and Deputy Director  approve new programs and lead scientific and technical reviews of  ongoing programs, while setting agency-wide priorities and ensuring a  balanced investment portfolio. Currently, DARPA has about 250 ongoing  research and development programs in its portfolio.
The 10 recently completed programs that we reviewed for this report  together spanned a broad range of research areas, including  communications, navigation, and health and marine sciences. Table 1  highlights the research focuses of these 10 programs in more detail.
section_title: Overall Transition Rates Not Determinable, but Selected Programs Illustrate Factors for Transition Success
paragraphs: Since 2010, DARPA has had success in transitioning new technologies  from the research environment to military users, including DOD  acquisition programs and warfighters. DARPA maintains a portfolio-level  database that identifies these outcomes by program. However, the  agency’s process for tracking technology transition outcomes is not  designed to capture transitions that occur after a program completes and  does not provide DARPA with an effective means for updating its  database. We used outputs from this database to select 10 case study  programs, but later identified inconsistencies affecting three programs in  how transition outcomes were reported in the portfolio-level database  versus how they were reported in other program documentation. We then  concluded that DARPA’s portfolio-level database was unreliable for  assessing transition rates and outcomes since fiscal year 2010. Our  analysis of the 10 selected programs did, however, identify four factors  that contributed to transition successes, the most important of which were  military or commercial demand for the planned technology and linkage to  a research area where DARPA has sustained interest.
subsections: section_title: DARPA’s Approach to Developing Technology Contributes to a Broad Definition of Transition Success
paragraphs: DARPA’s technological approach focuses on radical innovation that  addresses future warfighting needs, rather than developing technologies  that address current warfighting needs. This approach shapes how the  agency defines, pursues, and tracks technology transition. DARPA  considers a successful transition to be one where its program, or a  portion of its program, influences or introduces new knowledge. This  knowledge is often passed through program performers, which DARPA  relies on to execute technology development in its programs. Typical  performers include commercial enterprises; other DOD entities, such as  military service laboratories and research agencies; and academic  institutions. Further, DARPA generally does not develop technologies to  full maturity. Instead, the agency focuses on demonstrating the feasibility  of new technologies, which includes verifying that the concepts behind  the technologies have potential for real life applications. As a result, most  DARPA technologies require additional development before they are  ready for operational or commercial use. Therefore, follow-on  development is the predominant path of technology transition at DARPA.  Table 2 highlights the different technology transition paths that DARPA  technologies can take.
DARPA’s definition of what constitutes technology transition reflects one  of many in use within DOD. In June 2005, the Office of the Deputy Under  Secretary of Defense for Advanced Systems and Concepts in  collaboration with the Defense Acquisition University (DAU) published  guidance defining technology transition as “the use of technology in  military systems to create effective weapons and support systems—in the  quantity and quality needed by the warfighter to carry out assigned  missions at the ‘best value’ as measured by the warfighter.” However,  DOD officials told us the 2005 guidance is outdated, does not constitute  department policy, and should only be considered as a useful reference  source. In the absence of current DOD policy, in a March 2013 report we  identified three communities that DOD technologies typically transitioned  to: acquisition programs; directly to the field for use by the warfighter; and  to other users such as science and technology organizations, test and  evaluation centers, or industry. The communities we identified in 2013  are similar to the transition outcomes listed in the 2005 guidance, which  broadly lists commercialization, acquisition program, and follow-on  development by the prime contractor as primary pathways of technology  transition. In a subsequent report in December 2013, we found further  differences among what the military services define as technology  transition and additional confirmation that DOD itself lacks a formal  definition for technology transition across the department. These  variations, in tandem with the absence of a standard DOD-wide definition  of technology transition, prevents the military services, DOD research  agencies, and other DOD entities from consistently defining and tracking  technology transition. This lack of a formal definition of technology  transition means that DOD entities, such as DARPA, are free to define  and categorize technology transition for themselves.
section_title: DARPA’s Tracking of Technology Transition Concludes at Program Completion, Which Limits Data Reliability and Precludes Evaluation of Transition Rates
paragraphs: Following a program’s completion, DARPA officials identify and record  transition outcomes in accordance with the technology transition paths  identified in table 2. DARPA collects this information within a portfolio- level database that spans all of its recently completed programs. The  agency uses this database primarily to provide incoming program  managers with training on potential transition opportunities. Figure 2  illustrates in more detail DARPA’s process for assessing technology  transition outcomes in its programs.
DARPA’s process for tracking technology transition outcomes is not  designed to capture transitions that occur after a program completes and  the agency’s agreements with program performers have ended. After this  point, however, program performers often continue to develop their  technologies using non-DARPA sources of funding. According to DARPA  officials, these efforts can result in later transitions of technologies to  commercial products—including ones that are sold back to DOD for  military use—without the agency’s knowledge.
This process for tracking technology transition outcomes also does not  provide DARPA with an effective means for updating its portfolio-level  database. We used outputs from this database to select 10 case study  programs (5 that transitioned and 5 that did not transition), but later  identified inconsistencies affecting three programs in how transition  outcomes were reported in the portfolio-level database versus how they  were reported in other program documentation that we reviewed. This  confusion about ultimate transition outcomes persisted during our  interviews with DARPA officials. As a result, we concluded that DARPA’s  portfolio-level database was unreliable for assessing transition rates and  outcomes since fiscal year 2010. Table 3 highlights the inconsistencies  we found in our reviews.
The inconsistencies we identified suggest that DARPA’s current approach  to tracking technology transitions can limit its understanding of transition  outcomes. This may undermine its ability to craft transition plans for new  programs based on the lessons learned from previous programs. We  have previously identified technology transition tracking as a longstanding  issue at DOD. For example, in September 2006, we found that tracking  technology transitions and the effect of transitions, such as cost savings  or deployment of the technology in a product, provided key feedback that  can inform the future management of programs. However, in March 2013,  we found that DOD stopped tracking transition outcomes in many  programs once a program stopped receiving funding, which consequently  limited visibility into the extent of successful transitions within the DOD  portfolio.
section_title: Selected Programs Indicate That Several Factors Contribute to Successful Transitions
paragraphs: DARPA has undertaken efforts to understand the elements that contribute  to or impede successful technology transitions. According to DARPA  officials, a technology’s maturity level, availability of military service  funding, alignment with military service requirements, and transition  planning by the program manager influence whether or not a DARPA- developed technology successfully transitions. These characteristics align  with the findings of a 2001 DARPA-funded study, which reported that  mission, program manager turnover, timing, funding, and regulations,  among other elements, affect transition success.
In our review of 10 case study programs, we found different, but related,  factors for transition success as compared to the ones put forward by  DARPA: (1) military or commercial demand for the technology, (2) linkage  to a research area where DARPA has sustained interest, (3) active  collaboration with potential transition partners, and (4) achievement of  clearly defined technical goals. Based on our analyses, we identified two  factors—military or commercial demand for the planned technology and  linkage to a research area where DARPA has sustained interest—as  factors that were generally evident at program initiation and were most  important to transition. The remaining two factors—active collaboration  with potential transition partners and achievement of clearly defined  technical goals—sequentially follow the first two factors and become  observable once a program is underway. Figure 3 highlights these four  factors.
In reviewing the 10 programs, we found that the existence of the factors  identified varied from program to program. We assessed the extent to  which the four factors were present within the 10 programs we reviewed,  and table 4 highlights these results.
We found that successful transitions were often underpinned by existing  military or commercial demand for the technology. DARPA officials told us  that all of the agency’s programs are linked to military and joint service  needs at a high level, but through our analyses, we found that this  commitment was exemplified when any of the following components were  present in the program files:
Agreement between DARPA and (1) a military service, (2) a DOD  research agency or laboratory, or (3) other warfighter representative  that a related military capability gap or requirement exists; or
A private company identified a commercial demand for the technology  or showed an interest in commercializing it.
For example, the Spoken Language Communication and Translation  System for Tactical Use (TRANSTAC) program addressed a known  capability gap for speech translation technology within the Army. As a  result, the Army developed the appropriate requirements documents that  allowed the technology to successfully transition to an Army acquisition  program of record. These documents identified desired performance  attributes and system parameters, which served to better define and  communicate the Army’s need for TRANSTAC. The Army’s decision to  validate specific performance requirements provided TRANSTAC an  opportunity to transition into an Army program of record.
Within our 10 case studies, we found that a military or commercial  demand was fully present within four of the five programs that  successfully transitioned. In two cases—TRANSTAC and Quint  Networking Technology (QNT)—near-term military demand was a result  of DOD’s ongoing involvement in warfighting operations. However, in the  other two cases, an immediate military need for the technology was not  as prevalent. A fifth program that transitioned, Advanced Wireless  Networks for Soldier (AWNS), initially was in demand by the Army, but  interest waned over time as other options for radio networking platforms  emerged. In addition, several programs developed technologies that  demonstrated military applicability but lacked a military or commercial  demand, which precluded successful transition. For example, the  Predicting Health and Disease (PHD) and Nastic Materials programs  successfully demonstrated innovative research concepts that had  potential military applications, but an immediate military/commercial  demand simply did not exist without further maturation of technologies  past the point of program completion.
subsections: section_title: Linkage to a Research Area Where DARPA Has Sustained Interest
paragraphs: We also found that a program’s linkage to a research area in which  DARPA has sustained interest often facilitated successful transition. This  interest was demonstrated by evidence that in the years preceding the  program’s initiation, at least two related DARPA or other related DOD  science and technology programs had been completed. Sustained  interest is also exemplified by a program’s reuse of existing research  facilities and data from related programs, among other things. Of our 10  case studies, all 5 programs that successfully transitioned were fully  linked to sustained research interests, whereas 4 of the 5 non- transitioning programs did not have any such linkage.
DARPA’s program portfolio is currently organized around 10 research  focus areas under four key research themes. DARPA officials report  that the Hypersonics Capability focus area, for example, reflects an  ongoing interest for the agency that dates back to the mid-1980s. The  Falcon Combined-cycle Engine Technology (FaCET) is one of several  recent DARPA programs within the Hypersonics Capability focus area. In  addition, FaCET’s research was done in concert with other hypersonic  programs within DOD. As a result of this sustained interest, FaCET  technologies transitioned to other hypersonics programs, including  DARPA’s Mode Transition program, the joint DARPA/Air Force  Hypersonic Air-breathing Weapon Concept program, and the Air Force  Research Laboratory’s Robust Scramjet and Enhanced Operability  Scramjet Technology. Moreover, due to the National Aeronautics and  Space Administration’s (NASA) sustained involvement in FaCET,  technologies were also transitioned to NASA’s Glenn Research Center’s  Combined-Cycle Engine Large Scale Inlet Mode Transition Experiment  program.
section_title: Active Collaboration with Potential Transition Partners
paragraphs: We found that in all five cases where transition occurred, active  collaboration with potential transition partners was fully present. This  collaboration generally consisted of early program involvement by  stakeholders within the government and commercial sectors, service  requirements officials, and military liaison officers, among others. DARPA  program managers were responsible for facilitating this early stakeholder  involvement, including identifying the potential transition partners needed  to assist with their programs. According to DARPA officials, achieving  active collaboration with potential transition partners is highly dependent  on the nature of the program and background of the program manager,  which might be in academia, private industry, or military services. For  example, a program manager with a military background might be familiar  with DOD’s acquisition process and have connections with service  officials who can facilitate transition. On the other hand, a program  manager with an academic background might lack DOD service  connections, in which case DARPA’s military liaison officers can be used  to facilitate collaboration.
DARPA’s Architecture for Diode High Energy Laser Systems (ADHELS)  program exemplifies how active collaboration with potential transition  partners can facilitate successful technology transition. ADHELS  development included several technological components, including  volume bragg grating (VBG) technology. VBG is a transparent device  made of refractive glass that when combined with a diode laser can  control the laser output—such as by magnifying laser power, narrowing a  laser beam, or controlling the beam quality of the laser diode. According  to DARPA officials, the agency contracted with the foremost experts on  VBG technology to develop ADHELS components, recognizing that  adaptations of the VBG technology had potential applications within the  commercial marketplace. As ADHELS development progressed, DARPA  continued to engage its performers, who then licensed the VBG  technology to an ADHELS subcontractor. This subcontractor formed the  commercial entity Optigrate to further develop the VBG technology for  commercial sale.
Conversely, the programs that lacked active collaboration with potential  transition partners encountered challenges such as funding shortfalls,  requirements uncertainties, and underperforming technologies. For  example, early technical challenges prompted DARPA to restructure the  Self-Regenerative Systems (SRS) program to focus exclusively on  technology maturation, canceling initial plans to demonstrate and  evaluate SRS technologies on a transition partner’s system. This decision  constrained opportunities to identify potential transition partners and  actively collaborate with them during the program.
section_title: Achievement of Clearly Defined Technical Goals
paragraphs: We found that defining and, ultimately, achieving clear technical goals  helped facilitate technology transition. Of the five programs that  successfully transitioned, this factor was fully present in three programs  and partially present in the remaining two. Clearly defined technical goals  often existed in the form of documented agreements among stakeholders  that outlined technical specifications and desired capabilities, funding  requirements, development schedule, and organizational responsibilities  for technology development. These agreements allowed DARPA to share  development, management, and funding responsibilities with its service  partners, which facilitated shared understanding of technical goals and  mutual commitments to the program’s success and transition. Equally  important to this factor though was the degree to which a program  achieved its stated technical goals. Most of the programs we reviewed  identified clear technical goals, but fewer than half actually achieved the  technical goals that were originally set.
DARPA’s QNT program represents one example where clearly defined  technical goals were set and achieved. QNT was initiated with support  from the Air Force and Navy, which helped DARPA craft clear technical  goals including size, weight, robustness, transmission rates, and other  performance attributes of the technology. Defining technical goals during  the early stages of the program also secured each organization’s  commitment to playing a role in managing, developing, funding,  demonstrating, and testing QNT. As a result, stakeholders then worked  together to test QNT technical performance at several military exercises  and in theater, where the system performed to expectations and gained  added exposure within DOD. Ultimately, QNT transitioned to the Army’s  Intelligence, Surveillance, and Reconnaissance Network program, which  fielded the system in Afghanistan in September 2011. QNT also  transitioned to two Navy weapons programs and was also selected by the  Air Force for use in its Battlefield Airborne Communications Node  program, which hosts a data link communications system between aircraft  and ground units.
In other cases, such as AWNS, ADHELS, and Nastic Materials, technical  goals were clearly defined, but only partially met. These partial  successes, nonetheless, produced substantive technological gains. In the  cases of AWNS and ADHELS, these gains—coupled with the presence of  other key factors—proved sufficient to promote technology transition. On  the other hand, three programs lacked clearly defined goals—or did not  substantively achieve those goals—which led to significant restructuring  or development of technologies that did not align with the needs of a  planned transition partner. For example, Marine Corps’ officials stated  that the Tactical Underwater Navigation System relied on divers  swimming at unsustainable speeds to calibrate its positioning, which was  not responsive to their interests.
section_title: DARPA Prioritizes Innovation and Deemphasizes Technology Transition in Key Processes
paragraphs: DARPA’s investment of program funds and staff are primarily focused on  the highest priority of its agency mission, which is creating radically  innovative technologies that support DOD’s warfighting mission.  Technology transition is a secondary priority at the agency. DARPA  leadership conducts periodic reviews of agency programs, but these  reviews are focused on scientific and technical aspects of the programs  and do not assess technology transition strategies. Instead, the Director,  DARPA, delegates responsibility for oversight and assessment of  technology transition strategies to a subordinate office. DARPA also  provides limited training to program managers related to technology  transition, instead relying on others within the agency to assist program  managers with this activity, as needed. In addition, although DARPA  disseminates information on its past programs, it does not take full  advantage of available, government-sponsored resources for sharing  technical data. DARPA has also elected not to participate in most DOD  programs intended to facilitate technology transition, with the exception of  mandated small business programs, citing the challenges it perceives in  meeting the process and reporting requirements of these DOD programs  within DARPA’s typical timeframes for executing its research initiatives.
subsections: section_title: DARPA Investments in People and Programs Drive a Culture of Innovation
paragraphs: At DARPA, the desire for innovation drives investment, both in terms of  recruitment and programs. DARPA hires world-class scientists and  engineers from private industry, universities, government laboratories,  and research centers to serve as program managers. According to  DARPA officials, program managers are given great flexibility in leading  their programs, building their teams, and allocating funds to achieve their  programs’ objectives, including technology transition. DARPA officials  stated that these expectations are outlined to program managers during  new hire orientations, but are not codified in any agency-wide policy or  guidance. To ensure that new ideas for advanced technologies are  continuously coming into DARPA, the agency usually limits the tenure of  its program managers, as well as the duration of its programs, to 3 to 5  years. In this environment, program managers prioritize achieving  programs’ technical objectives, which can require the overwhelming  majority of their available time.
This focus on innovation, which corresponds with undertaking bold,  ambitious programs, makes the pursuit of technology transition a  secondary priority for the agency. Consequently, programs generally seek  to prove the art of “what is possible” rather than refining, producing, and  delivering tactical equipment to warfighters. According to DARPA officials,  the agency views these latter processes as the responsibility of military  service research agencies, laboratories, and acquisition programs of  record. However, DARPA officials report that potential transition partners  in the acquisition community are often unwilling to commit to incorporating  new technology into their programs of record without additional  maturation, and service research agencies and laboratories both have  their own programs and priorities to pursue. Consequently, the additional  maturation work needed to position DARPA programs for effective  transitions can go unfunded. According to DARPA officials, this dynamic  has proven to be a major impediment for the agency in transitioning  technology.
In addition, the introduction of DARPA’s radically innovative technologies  can disrupt the status quo for military programs, budgets, and warfighting  doctrine, which can drive cultural opposition within the military services.  DARPA officials stated that the agency’s research sometimes leads to the  identification of technologies and capabilities that military service officials  do not initially want or think their services will need, although these  technologies can eventually provide important military capabilities. For  example, DARPA officials said that the Air Force was initially highly  resistant to investments in stealth technologies for aircraft. Despite this  resistance, DARPA proceeded with the development of stealth  technologies, and today they are in use on multiple DOD weapon  systems, including the F-22 Raptor and F-35 Lightning II fighter aircraft.
DARPA’s secondary emphasis on transition is a long-standing  characteristic of the agency’s culture, as evidenced in studies  commissioned by DARPA in 1985 and 2001, which found that the agency  does not place enough emphasis on technology transition. The 1985  report recommended that DARPA designate full-time technology  transition facilitators, due to problems that were identified in the transition  of technologies to the military services. The 2001 report recommended  matching program manager tenure to the expected length of the  programs to which they are assigned—rather than setting arbitrary dates  of departure—and defining additional training and incentives for  technology transition.
According to DARPA officials, the Director, DARPA, has undertaken  several initiatives to improve the agency’s emphasis on technology  transition, including transition-focused quarterly meetings with each of the  military service chiefs or their deputies and establishment of the Adaptive  Execution Office in 2013, which was chartered to accelerate the transition  of game-changing DARPA technologies into DOD capabilities. In addition,  DARPA officials stated that the Director has shifted the role of the  agency’s military service liaisons to focus exclusively on assisting  program managers with military service engagement and transition of  DARPA technologies. DARPA officials report that these actions have  elevated the priority of and resources devoted to technology transition  within the agency.
section_title: DARPA Leadership Forgoes Opportunities to Assess, and Thus Potentially Improve, Technology Transition Strategies
paragraphs: The Director, DARPA, conducts oversight of programs through periodic  milestone reviews. These reviews assess a program’s scientific and  technical merit, and, according to DARPA officials, provide the Director  with information on the transition status of the program. According to  DARPA officials, the scope of these reviews is reflective of and consistent  with the agency’s top priority of creating innovative technologies.  However, these reviews do not assess a program’s strategy for achieving  technology transition. DARPA leadership delegates oversight and review  of technology transition strategies to the agency’s Adaptive Execution  Office, which coordinates with program managers to review and provide  input on technology transition strategies, particularly in the latter stages of  programs.
DOD policy, however, assigns to the Director, DARPA, the responsibility  to pursue “strategies” that “increase the impact of DARPA’s research and  development programs” and “speed the transition of successful research  and development programs to the military departments and defense  agencies,” among other scientific and technical functions.  Consequently, by not assessing technology transition strategies at the  program milestone reviews it chairs, the Director, DARPA, is forgoing key  opportunities to perform this function. This approach undermines  transition planning and introduces risk that DARPA programs will not  achieve their full transition potential.
Apart from the policy cited above, the Office of the Secretary of Defense  does not maintain other instructions or directives related to technology  transition at DARPA. In previous years, different components within the  Office of the Secretary of Defense have issued nonmandatory guidance  on technology transition, which has, at times, applied to DARPA  programs. However, the guidance is now outmoded in that it does not  address changes in key science, technology, and acquisition processes  that have occurred during the last 10 years. The Office of the Assistant  Secretary of Defense for Research and Engineering, which has primary  oversight of DARPA and other DOD research agencies, provides a great  deal of latitude to these agencies to define their own technology transition  policies and procedures. Most notably, officials from this office stated to  us that technology transition is no longer an explicit function of the office  and that the DOD division formerly responsible for technology transition  no longer exists. Instead, the office now limits its technology policy  responsibilities to minimizing unnecessary duplication of research efforts  within DOD, disseminating research knowledge throughout DOD, and  sharing that information with the general public.
section_title: Current Training for Managers Does Not Consistently Position Programs for Transition Success
paragraphs: DARPA’s program managers receive limited training on how to effect  technology transition in their programs. This training consists primarily of  overviews on DARPA’s technology transition paths and considerations to  make at program milestones with respect to technology transition.  DARPA program managers are not subject to the formal training and  certification requirements applicable to permanently hired science and  technology managers at military service laboratories. DOD requires  managers in these laboratories to complete Defense Acquisition  University (DAU) training courses in science and technology, including  how they apply to technology transition. These courses lead to  progressively higher knowledge and certifications over their careers.  DARPA officials countered that the training necessary to complete these  courses and achieve science and technology manager certifications  would require DARPA program managers to devote an inordinate amount  of time to training, particularly if DAU requires DARPA staff to complete  all the typical prerequisite courses that other managers are required to  complete. Further, given the agency’s unique mission, DARPA officials  stated that they do not consider the DAU training courses to be as  relevant to their program managers given the agency’s broad discretion to  pursue breakthrough technologies versus specific management of  acquisition programs.
In lieu of more robust training, DARPA officials stated that the agency  supports its program managers’ transition efforts by providing them with  access to various transition planning and outreach resources. For  example, DARPA program managers are also supported by the agency’s  Adaptive Execution Office, which provides them with assistance in  developing their transition plans and in communicating with the military’s  transition stakeholders at DOD’s combatant commands. Program  managers are further supported by DARPA’s military liaison officers from  the Air Force, Army, Navy, and Marine Corps, who help program  managers identify and reach out to potential transition partners or other  stakeholders in the military services. These liaisons also arrange for  DARPA leadership to meet with the military’s senior leaders, which is  done in part to advocate for the transition of DARPA programs to the  military services. As we found in September 2006, these liaisons can also  provide operational advice for planning and strategy development and  provide an understanding of service perspectives, issues, and needs so  that potential customers can be identified and effective agreements can  be written. Program managers also are authorized to use program funds  to hire experienced contractors and government staff from other agencies  to aid technology transition activities in their programs.
Previous guidance and studies, including one commissioned by DARPA  in 2001, have recommended that DARPA improve its technology  transition training for program managers through additional training and  mentoring programs related to technology transition. Further, in 2005,  DOD issued guidance on technology transition stating that developing  and executing a training plan for the members of the team supporting  technology transition is essential to their success. Similarly, Standards  for Internal Control in the Federal Government indicates that effective  management of an organization’s workforce, which includes providing  necessary training to the organization’s staff, is essential to achieving  results. DARPA’s limited training for program managers on technology  transition is inadequate to consistently position programs for transition  success. Without sufficient training, program managers may not develop  the skills and knowledge that they need to identify and engage potential  transition partners and facilitate transition successes.
While DARPA does not currently rely on other DOD entities for  technology transition training, individual DARPA program managers may  voluntarily elect to take training related to technology transition in DOD or  other federal organizations. For example, the Federal Laboratory  Consortium for Technology Transfer was established by law in the  Federal Technology Transfer Act of 1986 to, among other things, (1)  develop training for federal lab employees engaged in technology transfer  and (2) facilitate communication and cooperation between federal  laboratories. The Consortium offers both in-person and online training  regarding commercialization of technologies, as well as guidance  regarding best practices. In response to our inquiries on this subject,  DARPA officials indicated they were in discussions with the DAU staff  regarding potential future training options.
section_title: Shortfalls in Disseminating Program Information May Lead to Missed Transition Opportunities
paragraphs: Disseminating information regarding developed technologies is a way for  agencies to promote technology transition after the conclusion of a  program, particularly once program managers and staff are no longer  actively advocating for the transition of their program’s technologies. For  many years, DOD has maintained website-accessible databases that  disseminate information within the department, and to a lesser extent, to  the public and to private companies. These websites allow their users to  search for related technologies while considering new programs or  products that could possibly use them.
While DARPA disseminates information on past programs through the  use of public government websites, its selective approach to posting this  information does not maximize the chances of DARPA technologies being  identified and selected by potential transition partners. Currently, DARPA  disseminates information on past programs through both internal and  external means, but does not share information with key data repositories  that the federal government sponsors, which may obscure visibility into its  programs and lead to missed transition opportunities. Since the 1960s,  DARPA has provided substantial amounts of information regarding its  technologies to the official DOD dissemination website managed by the  Defense Technical Information Center (DTIC). Although the majority of  DARPA-related information in this database is restricted to DOD staff, it is  by far the largest repository that private companies and the general public  can access for information on DARPA technologies. For instance, we  found that while non-DOD users can access approximately 3,600 DARPA  technical records through DTIC’s public website, DOD users can access  over 30,000 of these records. DARPA also maintains an “Open Catalog”  public website for disseminating information on its programs, although it  currently only has technical information on about a few dozen active and  completed programs. In comparison, DARPA’s public website also  provides brief, non-technical descriptions of 194 active DARPA programs.  Two other government-sponsored websites, operated by the DOD  TechLink public-private partnership and the Federal Laboratory  Consortium for Technology Transfer, also exist to help science and  technology agencies disseminate technology information. DARPA officials  indicated they do not share information with either of these entities and  instead exclusively rely on DTIC, which DARPA officials stated  represented DOD’s official repository.
In recent years, the White House has provided direction to broaden  access to non-sensitive information on government-developed  technologies, in recognition of government research’s potential for  catalyzing innovative breakthroughs that drive the U.S. economy, and  helping to drive progress in areas such as health, energy, the  environment, agriculture, and national security. In February 2013, the  White House’s Office of Science and Technology Policy instructed the  federal government’s science and technology community to begin  planning how to disseminate information on technologies they have  developed. According to DARPA officials, the lead DOD agency for  implementing this system is DTIC, and they do not expect DOD to have a  dissemination system in place that fully addresses the requirements of  the memorandum until 2017.
section_title: DARPA Participates in Mandated Small Business Programs, but Refrains from Other DOD-Wide Programs Intended to Facilitate Technology Transition
paragraphs: The Office of the Secretary of Defense manages several DOD programs  intended to accelerate development, testing, and delivery of mature  technologies that provide new solutions for military needs. The general  purpose of these programs is to facilitate the transition of technologies,  but vary in terms of what types of technology developers and operational  needs they target. For example, in partnership with the military services,  the Joint Capability Technology Demonstration program addresses joint  warfighting needs of the combatant commands by demonstrating mature  technology prototypes that may transition to acquisition programs or  directly to the warfighter in the field. Other programs such as the Small  Business Innovation Research (SBIR) program fund small business  research and development with the goal that innovations produced will be  commercialized and eventually sold back to DOD. Table 5 lists these  programs.
According to DARPA officials, the only programs that DARPA participates  in are the SBIR and Small Business Technology Transfer (STTR)  programs because it is legally required to do so. However, the agency’s  knowledge of transition outcomes associated with SBIR and STTR  expenditures is limited. DARPA officials said they do not maintain a  comprehensive list of agency programs using SBIR or STTR funds—or  the transition paths of technologies developed with these funds—because  these data are not always reported or accessible. According to DOD  small business program management officials, who oversee the use of  these funds throughout the department, transition outcome data is not  required from any military service or agency, including DARPA. These  officials further stated that once a small business contract ends, DOD’s  means for compelling contractors to identify and report on successful  transitions expires. In December 2013, we recommended that DOD  improve its tracking of technology transition outcomes in SBIR-funded  programs by establishing a common definition of technology transition for  all SBIR projects and improving the completeness, quality, and reliability  of SBIR transition data that it reports. These tracking shortfalls  precluded us from assessing the extent to which DARPA’s SBIR and  STTR funds contribute to successful technology transitions. In lieu of  comprehensive transition data, DARPA officials have worked with some  of their prior program contractors—who successfully developed and  transitioned technologies—to identify small business program success  stories. In addition, DARPA officials stated that they are developing  contract language for future SBIR awards that would require firms to  identify their transition and commercialization outcomes as an addendum  to their final report.
Apart from the legally required small business programs, DARPA officials  said that the processes and reporting requirements associated with  participating in DOD’s other technology transition programs are generally  cumbersome and do not align with DARPA’s time frames for executing  programs or mission of creating disruptive technologies over relatively  long periods of time. Conversely, DOD transition programs are mainly  intended for mature technologies, or short-term efforts that can be fielded  quickly. DARPA officials explained that technologies their programs  develop usually require additional maturation in subsequent technology  development efforts, either within DARPA or at military service  laboratories, before transitioning to acquisition programs or warfighters.  DARPA officials also said that agency leadership generally views the use  of these funds as unnecessary given that DARPA’s budget currently  provides adequate funding to support its research endeavors.
DARPA officials also indicated that they are exploring stronger  relationships with the Joint Capability Technology Demonstration (JCTD)  program, particularly in the area of prototyping. In previous decades,  DARPA used funds from the predecessor to the JCTD program—then  known as the Advanced Concept Technology Demonstration program—to  develop and demonstrate technologies. These efforts include currently  fielded systems such as the Air Force’s Global Hawk and Predator  unmanned aircraft and Miniature Air-Launched Decoy systems. DARPA  officials also stated that Manufacturing Technology program funds have  been applied after DARPA program completions to improve the  affordability of and manufacturing base for semiconductors developed by  DARPA.
section_title: Conclusions
paragraphs: Technology transition does not have to occur at the expense of  innovation, but should instead be viewed as a natural extension of  innovation. When DARPA places technology in the hands of a user,  operational knowledge is gained that can be used to improve the  technology and further scientific innovation. However, DARPA leadership  does not fully subscribe to this viewpoint; instead, it is satisfied with  maturing technology to the point where feasibility, but not functionality, is  proven. Today, programs progress through DARPA without the agency  head fully assessing whether transition strategies make sense. Such  assessments, if measured against key transition factors, could improve a  program’s potential for transition success. Transition responsibilities then  fall almost exclusively on individual program managers, who are often not  sufficiently trained to achieve the favorable transition outcomes they seek.  Further, when the program manager’s tenure expires, the primary  advocate for transitioning the program’s technology is also lost. This  turnover increases the need for technical gains to be appropriately  documented and disseminated so that user communities have visibility  into potential solutions available to meet their emerging needs. An  important part of this process is the tracking of transition outcomes, as we  recommended DOD undertake for its technology transition programs in  March 2013, and which we have also found lacking at DARPA.
section_title: Recommendations for Executive Action
paragraphs: To improve technology transition planning and outcomes at DARPA, we  recommend the Secretary of Defense direct the Director, DARPA, to take  the following three actions:
Oversee assessments of technology transition strategies for new and  existing DARPA programs as part of existing milestone reviews used  to assess scientific and technical progress to inform transition  planning and program changes, as necessary. Our analysis identified  four factors that could underpin these assessments, but the  uniqueness of individual DARPA programs suggests that other  considerations may also be warranted.
Increase technology transition training requirements and offerings for  DARPA program managers, leveraging existing DOD science and  technology training curricula, as appropriate.
Increase the dissemination of technical data on completed DARPA  programs through Open Catalog and other government-sponsored  information repositories aimed at facilitating commercialization of  technologies.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOD for review and comment. In its  written comments, which are included in appendix II, DOD partially  agreed with our recommendations to oversee assessments of technology  transition strategies for DARPA programs and to increase technology  transition training requirements and offerings for DARPA program  managers. In doing so, DOD agreed with most of the principles contained  in our recommendations, but disagreed with the actions we  recommended. DOD did not agree with our recommendation to increase  the dissemination of technical data on completed DARPA programs. DOD  also separately provided technical comments, which we incorporated, as  appropriate.
DOD agreed that assessments of technology transition strategies, which  consider the four factors we identified for transition success, would help  inform program decisions by DARPA leadership. However, DOD did not  agree that such assessments be required at milestone reviews for  DARPA programs, citing active participation by the Director, DARPA, in  technology transition discussions throughout the life of a program. We  agree that leadership is focused on technology transition and holds  discussions often; however, we found it difficult to be able to identify  transitions—or changes to transition strategies—that arise from these  discussions. We believe that these discussions are an inadequate  substitute for assessing technology transition strategies as part of the  comprehensive program reviews that DARPA already undertakes.  Assessing transition strategies at these reviews, as we recommended,  would provide the opportunity to coordinate and prioritize transition goals,  objectives, and planned actions in the context of scientific and technical  developments in the program. By overseeing technology transition  strategies separate from these reviews, the Director, DARPA, risks  making decisions related to a program’s transition that are not  appropriately informed by other important program considerations.   Although DARPA asserted that our recommendation runs counter to its  current efforts to improve processes and procedures, we found no  evidence that processes and procedures were improving.
DOD also agreed that technology transition training improves transition  planning and outcomes, citing DOD science and technology training  curricula as a “rich repository of transition insight.” Yet, despite the value  it sees in its own training resources, DOD stated that DARPA program  managers’ relatively short tenure leaves few opportunities to expose them  to such “generic” training opportunities. Consequently, DOD did not agree  that technology transition training requirements should be increased for  DARPA program managers and stated that DARPA’s current approach of  “tailored curricula focused on a program’s unique transition needs”  remained appropriate. However, we did not find evidence of such tailored  curricula in our review. Instead, we found that DARPA program managers  all received the same limited training upon hiring, which was inadequate  to consistently position programs for transition success. DOD also stated  that DARPA continues to explore opportunities to offer tailored, concise,  and streamlined training to its program managers. Therefore, we stand by  our recommendation and continue to believe that expanded training  opportunities are necessary for achieving better transition outcomes in  DARPA programs, and we encourage DOD to capitalize on its existing  investments in this area, to the extent possible.
Further, DOD did not agree that increased dissemination of technical data  on completed DARPA programs was warranted. DOD stated that using  multiple information repositories “thins the DOD technology market by  spreading it across several venues,” in turn reducing the likelihood that  technology providers and potential transition partners will find a match.  DOD also stated that it intends to make DTIC the central data storage for  all DOD technical activities, including DARPA technologies, and views the  use of multiple information repositories as unconducive to improving  technology transition outcomes. In our review, we found DARPA’s  existing reliance on DTIC limited the chances of the agency’s  technologies being identified and selected by potential transition partners,  particularly those outside of DOD. We fail to see how increased  dissemination of technical data would actually “thin” the DOD technology  market.  To the contrary, it would allow more portals with which to gain  access.  Similarly, in 2013, the White House identified a government-wide  need to broaden access to non-sensitive information on government- developed technologies, but improvements remain incomplete.  Consequently, we continue to believe that DARPA should pursue  dissemination of non-sensitive technical data through as many existing  government-sponsored outlets as possible, including its own Open  Catalog website and DOD TechLink, to improve the likelihood of transition  successes in the agency’s programs.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense, and the Director, DARPA. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report covers the Defense Advanced Research Projects Agency’s  (DARPA) (1) effectiveness at transitioning technologies since fiscal year  2010, including identifying the factors that contributed to successful  technology transitions, and (2) implementation of Department of Defense  (DOD) policies and programs intended to facilitate the transition of  technologies.
To assess DARPA’s effectiveness at transitioning technologies since  fiscal year 2010, including identifying factors that contributed to  successful transitions, we requested and reviewed portfolio-level data  identifying the names, funding amounts, and technology transition of  those DARPA programs successfully completing technology development  during fiscal years 2010 through 2014. We confined our analysis to this  time frame owing to availability of data from DARPA. These data included  150 programs funded under DOD’s budget activities for (1) applied  research and (2) advanced technology development that DARPA  identified as having completed as planned and producing a substantive  technological gain or innovation, regardless of whether that technology or  innovation transitioned to an end user. We used these data as the basis  for selecting a simple random sample of 10 case study programs—5 that  transitioned and 5 that did not transition. In conducting our case study  analyses, we reviewed relevant program documentation to identify factors  that facilitate transition success. While reviewing these case study  programs, we identified inconsistencies between agency portfolio-level  transition outcome data and program-level information. DARPA officials  stated to us that this was due to the transition status of these programs  changing after they had collected the portfolio-level data. As a result, we  concluded that DARPA’s portfolio-level data were not sufficiently reliable  for the purposes of assessing agency-wide transition rates and outcomes  since fiscal year 2010. However, these inconsistencies did not  significantly affect our program selections; therefore, these data were  sufficiently reliable for the purposes of selecting the 10 case study  programs.
To identify factors that facilitated technology transition within the 10  selected programs, we analyzed DARPA provided documentation— including program briefings, memorandums of agreement, broad area  announcements, budget documents, and program completion reports— for selected programs to identify factors that facilitated or precluded their  individual transitions. We then conducted a content analysis of these  individual factors to identify common themes among the programs, which  led to us determining that four significant factors underpinned technology  transition outcomes in the programs we reviewed. Once we identified  these four factors, we developed a rating system to assess the extent to  which each factor was present in each of our 10 programs, as supported  through our analysis of program documentation. Our measures for each  of the four factors were as follows:  Military or commercial demand for the planned technology
Fully present: Demand for the technology from a potential transition  partner existed throughout the program, which would include (1)  agreement between DARPA and a military service, DOD laboratory or  other warfighter representatives that a related military capability gap  or requirement exists; or (2) a private company identified a  commercial demand for the technology or showed an interest in  commercializing it.
Partially present: Potential transition partners indicated to DARPA that  they believed a demand existed for the technology, as is described  above, although their interest was not consistent through the end of  the program.
Not present: The factor did not exist at all, and DARPA appears to  have initiated the program without a potential transition partner  agreeing that a capability gap or potential commercial use existed at  any point during the program.
Linkage to a research area where DARPA has sustained interest
Fully present: In the years preceding the program’s initiation, at least  two related DARPA or other DOD science and technology program  had been completed.
Partially present: In the years preceding the program’s initiation, at  least one related DARPA or other science and technology program  had been completed (this was the second DOD science and  technology program of its kind).
Not present: The factor did not exist at all, and this program appears  to not have any roots in previous similar DARPA or other DOD  science and technology programs.
Active collaboration with potential transition partners
Fully present: Potential transition partners consistently participated in,  advised, or otherwise supported the program.
Partially present: Potential transition partners participated in, advised,  or otherwise supported the program, although their involvement was  not consistent through the end of the program or was not present until  after the prototype demonstration (relatively late in the program).
Not present: The factor did not exist at all. The program appears to  have lacked assistance from any potential transition partners, or their  assistance was very infrequent or insignificant.
Achievement of clearly defined technical goals
Fully present: Measurable technical goals were set in the program  and fully achieved to the satisfaction of DARPA and any transition  partner involved in the program, to the extent that one or more had  been identified for the technology.
Partially present: Measurable technical goals were set in the program,  but met with varying levels of success. The technical successes  achieved, however, were sufficient to produce a technology  responsive to the interests of a transition partner, to the extent that  one or more had been identified for the technology.
Not present: Measurable technical goals were either not set or  sufficiently met in the program. The level of technical success was not  sufficient to produce a technology responsive to the interests of a  transition partner, to the extent that one or more had been identified  for the technology.
Using this rating system, two GAO analysts analyzed and coded whether  each of the four factors was fully present, partially present, or not present  in each of the 10 programs we reviewed. Each GAO analyst coded all the  constituent items independently, and the two analysts then met to discuss  and reconcile the differences between their codings. Following this initial  round of coding, another GAO analyst independently verified the  accuracy of the coding by reviewing the supporting program  documentation. The final assessment reflected the analysts’ consensus  based on the individual assessments.
To assess DARPA’s implementation of DOD policies and programs  intended to facilitate the transition of technologies, we identified and  analyzed information sources including policy instructions, guidance,  training materials, and technical data repositories intended to promote  technology transition within DARPA, DOD and the federal government.  We also reviewed previous federal directives issued by the Executive  Office of the President that were related to technology transition at  DARPA. We reviewed previous DARPA-sponsored reports on technology  transition produced in previous years. We reviewed our prior related  reports and program information regarding DOD’s technology transition  programs and relevant DOD funding information. We reviewed available  training, resources, and tools used by DARPA officials to help bring about  technology transition. We reviewed the contents of DOD computer  systems used to disseminate information on DARPA programs to  potential transition partners. We reviewed our prior reports and DOD  documentation on DOD transition programs, including the Small Business  Innovation Research, Small Business Technology Transfer, and Joint  Capability Technology Demonstration programs, among others, to  understand the extent to which DARPA participates in these programs.  We reviewed the extent to which DARPA uses DOD transition funds, and  requested data regarding DARPA’s use of small business funds and its  technology transition outcomes, although these data were unavailable for  our analysis, as is discussed elsewhere in this report. We also reviewed  historical information on DARPA’s use of DOD transition funds available  from public sources, including DOD budget documentation.
To gather additional information in support of our review for both  objectives, we conducted interviews with current and former officials  responsible for executing, managing, and overseeing transition of  DARPA-developed technologies, including representatives of DARPA’s  senior leadership and Adaptive Execution Office, program management  offices and selected program managers, military services liaisons, and  small business program officials. We also interviewed officials from the  Office of the Assistant Secretary of Defense for Research and  Engineering and DOD’s Office of Small Business Programs. Further, we  interviewed officials from selected military requirements and acquisition  offices, including the Joint Staff’s Force Structure, Resource, and  Assessment directorate; Office of the Deputy Chief of Staff of the Army  for Operation, Plans and Training; Office of the Deputy Chief of Staff of  the Army for Logistics; Office of the Assistant Secretary of the Army for  Acquisition, Logistics, and Technology; Army Program Executive Offices  for Intelligence Electronic Warfare and Sensors and Command, Control,  Communications—Tactical; and Marine Corps Systems Command. We  also met with staff from selected DOD research centers, including the Air  Force Research Laboratory and the Office of Naval Research, and with  the Director of Science and Technology curriculum at the Defense  Acquisition University.
We conducted this performance audit from January 2015 to November  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Diana Moldafsky, Assistant  Director; Christopher R. Durbin, Analyst in Charge; Emily Bond; Nathan  Foster; Aaron M. Greenberg; John Krump; Jean L. McSween; Sean  Seales; and Roxanna T. Sun made key contributions to this report.