Defense Technology Development: Technology Transition Programs Support Military Users, but Opportunities Exist to Improve Measurement of Outcomes
Published: 2013-03-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: DOD and Congress recognize that technology innovation sometimes moves too slowly from the lab to the field. Programs have been created in DOD to help facilitate the transition of new technologies. The conference report accompanying the fiscal year 2012 National Defense Authorization Act directed GAO to undertake a body of work that will provide a holistic assessment of DOD's S&T enterprise. This report reflects the results from GAO's first review, which focuses on technology transition. Generally, when technologies have been sufficiently matured in the S&T environment, the technologies are available to transition to a military user. GAO's specific objectives were to (1) determine what DOD programs are dedicated to facilitating technology transition, (2) assess the outcomes of these transition programs, and (3) identify practices among the programs that may facilitate technology transition. GAO conducted interviews with and collected information from each technology transition program to identify their selection, management, and assessment practices, as well as project outcomes.
section_title: What GAO Found
paragraphs: GAO identified 20 technology transition programs--managed by the Office of the Secretary of Defense (OSD) and the military departments--that provide structured mechanisms and funding to facilitate technology transition. All of the programs GAO reviewed are consistent in providing opportunities to transition technologies from the science and technology (S&T) environment to a user, such as a weapon system acquisition program or the warfighter in the field. To help speed the delivery of technologies to users, most transition programs target fairly mature technologies, which are suitable for final stages of development and demonstration. Collectively, the programs GAO reviewed obligated about $7.9 billion in Department of Defense (DOD) research, development, test, and evaluation funding for fiscal years 2010 through 2012 to support technology transition.
Most programs that GAO assessed track whether their projects were completed and successfully transitioned to intended users. On average, programs reported a historical transition rate of over 70 percent for projects. The vast majority of these projects resulted in technologies transitioning to acquisition programs or directly to the warfighter. However, about one-quarter of the projects transitioned to other organizations, such as test and evaluation centers, for further development. Prior GAO work found that tracking technology transitions and the impact of those transitions, such as cost savings or deployment of the technology in a product, provides key feedback that can inform the management of programs. For the most part, transition programs that GAO reviewed do not track projects beyond transition, which limits their ability to know and report final outcomes for transitioned technologies and the associated benefits realized from those technologies.
As GAO has reported in the past, effective selection and management processes as well as tools are needed to ensure that new technologies can be successfully transitioned to military users. GAO found that OSD's and the Military Departments' technology transition programs make use of these practices to varying degrees. Most programs have formal review processes to determine whether candidate projects have sufficiently mature technologies, are in demand by users, and have schedules and costs that fit within the programs' criteria. Once selected, projects require effective management to ensure risks are minimized and transition commitments are confirmed. Many program officials indicated that regular stakeholder communication during project execution is important to ensure projects stay on track and transition commitments are sustained. Moreover, many program officials identified the use of formal management tools, such as technology transition agreements, as key mechanisms to help hold stakeholders accountable and facilitate technology transition.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD require programs to track and measure project outcomes to document transition results and benefits from transition, as well as assess programs to identify opportunities for more widespread use of existing transition management tools. DOD generally concurred with these recommendations and stated that it will initiate actions to address potential opportunities for improvement identified in the report.

section_title: Letter
paragraphs: The Department of Defense (DOD) plans to spend about $70 billion in  fiscal year 2013 to develop and enhance weapon systems. Despite the  high caliber of systems delivered to the warfighter, weapon system  programs too often experience cost, schedule, and performance  problems, in part because they attempt to incorporate advanced  technologies that have not been proven. DOD’s science and technology  (S&T) community, which receives about $12 billion annually, is tasked  with identifying, developing, and ensuring that high value technologies are  mature and available for use by the department’s acquisition and military  user communities. Generally speaking, technology transition means  identifying technologies that have been sufficiently matured in the S&T  environment and are ready to transition to a user such as a weapon  acquisition program or the warfighter in the field. Transitioning  technologies from defense S&T labs and research organizations to  military users, however, has been a long-standing challenge for DOD.  Sometimes technologies are not ready to transition when needed  because they may still be too risky or too costly to adopt. At other times  promising technologies are not taken advantage of because of insufficient  processes and mechanisms to expedite their transition to users.
Congress and DOD recognize the difficulties involved in transitioning  technologies and have established many programs over time aimed at  removing barriers to technology transition and accelerating the flow of  technologies to military users. GAO was directed by the Congress, in the  conference report accompanying the fiscal year 2012 National Defense  Authorization Act, to undertake a body of work that will provide a holistic  review of DOD’s S&T enterprise. This report reflects the findings from our  first review, which focuses on technology transition. Specifically, as  discussed with your offices, the objectives of this review were to (1)  determine what DOD programs exist that are dedicated to facilitating  technology transition from the S&T base to military users, (2) assess the  outcomes for these transition programs, and (3) identify practices among  the programs that may facilitate technology transition.
To conduct this work, we evaluated documentation from, as well as  interviewed, Office of the Secretary of Defense (OSD) and military  department officials as appropriate. We reviewed 20 technology transition  programs and assessed information on program structure, processes,  transition tools, and metrics kept to track transition outcomes. This  included data on each transition program and its technology transition  projects for fiscal years 2010 through 2012.
We conducted this performance audit from April 2012 to March 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix I provides further  details on the scope and methodology used for this work.
subsections: section_title: Background
paragraphs: DOD relies on its S&T community—DOD research laboratories, test  facilities, industry, and academia—to identify, pursue, and develop new  technologies that improve and enhance military operations and ensure  technological superiority over adversaries. These activities by DOD’s S&T  community support the development of technologies for new or existing  weapon systems, as well as development of capabilities that are  transitioned directly to warfighters in the field. The S&T community  engages in activities ranging from basic research through advanced  technology development that are conducted by the government or  externally by universities and commercial industry. Once the S&T  community has completed its technology development, additional product  development activities, such as technology demonstration and testing,  are often needed before incorporating the technologies into military  weapon systems. Under the management of the acquisition community,  product development further advances technology received from S&T  developers and integrates it into systems that are ultimately delivered to  support the warfighter.
Supporting these activities is DOD’s research, development, test, and  evaluation budget. As shown in figure 1 below, this budget is divided into  seven categories that follow a generally sequential path for developing  technologies from basic research to operational system development.  The first three categories represent DOD’s S&T activities to advance  research and technology development, while the remaining categories  support product development for DOD acquisition programs. Appendix II  provides further details on research and development activities in these  budget categories.
The large size and scope of DOD’s national security mission imposes  broad and competing demands on the S&T community, and determining  the right mix of technology investments to pursue is challenging. One key  expectation is that S&T funding will result in relevant and feasible  technologies that can transition into weapon system programs or go  directly to the warfighter in the field. DOD has also tasked its S&T  community with anticipating its technological needs for an uncertain  future. Further, the S&T community serves as a technology innovation  mechanism to expand scientific knowledge and investigate technologies  that may provide breakthrough warfighting capabilities. As a result, some  investments focus on conducting research to generate scientific  knowledge, exploring new technologies, demonstrating the feasibility of a  technology concept, and other science and technology endeavors. The  challenge is finding the right balance between taking risk to develop  breakthrough technologies versus investing in moderate technology  enhancements that are likely to transition to a military user. Although not  precisely defined, technology transition generally occurs when advanced  technology development ends and product development begins. Figure 2  below provides a notional picture of how DOD’s S&T community  manages technology investment, development, and transition to a user.
DOD’s research laboratories and test facilities, as well as industry and  academia, provide many examples of developing new technologies that  improve warfighter capabilities and enhance military operations. However,  as we have reported in the past, for a variety of reasons DOD historically  has experienced problems in transitioning technologies out of its S&T  environment and into military systems. Technologies may not leave the  lab because their potential has not been adequately demonstrated or  recognized, acquisition programs may be unwilling to fund final stages of  development, or private industry chooses to develop the technologies  itself.
Despite these challenges, technology transition can be facilitated in many  ways, and is affected by choices made throughout the S&T management  cycle. Before technology development is begun, critical S&T investment  decisions must be made that balance needs, resources, and technical  feasibility in a way that ensures the S&T community is responsive to  warfighter priorities. Once S&T projects are underway, technology  development provides opportunities to demonstrate feasibility and  management decisions must be made along the way by S&T  developers—with input from the acquisition community—on which  technologies should be developed to a final state that would facilitate  technology transition.
section_title: DOD Has a Variety of Technology Transition Programs That Support Military Users
paragraphs: We identified 20 technology transition programs, managed by OSD and  the military departments, that provide structured mechanisms and funding  to facilitate technology transition. These programs vary in size, mission,  approach, funding, and technology maturity expectations. However, all  are consistent in providing opportunities to transition technologies from  the S&T community to a military user, such as an acquisition program or  the warfighter in the field. Many of the transition programs are focused on  rapid response to warfighter needs, which has received greater emphasis  in the recent past because of the United States’ operations in Iraq and  Afghanistan. In addition, most programs target fairly mature technologies,  which are suitable for final stages of development and demonstration.  Collectively, the programs we reviewed use a mix of S&T and other  research, development, test, and evaluation funding—about $7.9 billion  obligated from fiscal years 2010 through 2012—to facilitate technology  transition.
subsections: section_title: OSD Has Seven Technology Transition Programs That Vary in Their Broader Missions, Approaches, and Funding
paragraphs: OSD manages seven transition programs that are intended to accelerate  development, testing, and delivery of mature technologies that provide  new solutions for military needs. Six programs are managed by the  Deputy Assistant Secretary of Defense, Rapid Fielding, who reports to the  Under Secretary of Defense for Acquisition, Technology, and Logistics.  They include three programs authorized by Congress—the Defense  Acquisition Challenge (DAC), Foreign Comparative Testing (FCT), and  Rapid Innovation Fund (RIF)—and three programs established by  DOD—Joint Capability Technology Demonstration (JCTD), Quick  Reaction Fund (QRF), and Rapid Reaction Fund (RRF). In addition, OSD  manages a Small Business Innovation Research (SBIR) program, which  was established by Congress and has technology transition—described  by the program as “commercialization”—as a tenet.
The OSD programs share the general purpose to transition technologies,  but vary in what type of technology developers and operational needs are  targeted. For example, the Joint Capability Technology Demonstration  program addresses joint warfighting needs of the combatant commands,  in partnership with the military services, by demonstrating mature  technology prototypes that may transition to acquisition programs or  directly to the warfighter in the field. In contrast, the Foreign Comparative  Testing program mission is to identify and test technologies that have  already been developed by other countries and may have utility in  addressing U.S. military needs. Other programs like the Rapid Innovation  Fund and Small Business Innovation Research seek to leverage  technology solutions from small businesses. And still others, like the  Quick Reaction Fund and Rapid Reaction Fund programs, seek to target  specific urgent conventional and irregular warfighting problems,  respectively, and rapidly deliver technologies to operational users.
Collectively, research, development, test, and evaluation funding for the  OSD technology transition programs we reviewed included about $1.75  billion for fiscal years 2010 through 2012. The Joint Capability  Technology Demonstration and Rapid Innovation Fund programs were  the most substantial part of that funding—over $1.2 billion for fiscal years  2010 through 2012. The majority of OSD technology transition programs  constrain project time frames to 2 years or less, since their focus is rooted  predominately on providing funds to accelerate transition for technologies  that rapidly respond to current or near-term military needs. Individual  projects also represent relatively small funding commitments for the  department, with $3 million on the high end of costs, although Joint  Capability Technology Demonstration projects can be higher. It also is  important to note that several of the programs utilize co-funding provided  by prospective technology users to further advance projects. Table 1  provides further information on funding as well as project-related details  for each OSD program.
The Departments of the Navy, Army, and Air Force have created distinct  programs that rapidly respond to pressing warfighter needs, while also  initiating programs that work to address needs that are more mid-term in  nature. In addition, as directed by Congress, DOD has established a  Manufacturing Technology (ManTech) program and Small Business  Innovation Research program that provide technology transition  opportunities. All three of DOD’s military departments manage ManTech  and Small Business Innovation Research programs.
In the Navy, the Office of Naval Research (ONR) has a well-established  technology transition focus. ONR’s Office of Transition manages the  Future Naval Capabilities (FNC) portfolio, which is the Navy’s largest  transition program—for which nearly $450 million was budgeted in fiscal  year 2013. The program, which was initiated in 1999, seeks to provide the  best technology solutions to address operational requirements, delivering  technology products to acquisition programs that enhance capabilities  within a 5-year time frame. ONR’s Offices of Transition and Innovation  also support rapid technology transition to the fleet, force, and acquisition  communities via the Rapid Technology Transition (RTT), Technology  Insertion Program for Savings (TIPS), TechSolutions (TS), and  SwampWorks and Experimentation (SW/Exp) programs.
The Army’s efforts to facilitate transition are evolving. In 2011, the Army  began a new Technology Enabled Capability Demonstration (TECD)  program, which is replacing Army Technology Objectives-Demonstrations  (ATO-D). According to Army officials, Technology Enabled Capability  Demonstration characteristics were influenced by the Navy’s Future  Naval Capabilities program, with the Army identifying a need for a more  integrated S&T approach. Key characteristics of Technology Enabled  Capability Demonstrations include senior leadership input at the outset for  S&T decisions, greater focus on providing capabilities, and better  planning on how a technology will transition to provide an operational  capability. When fully implemented, the Army expects Technology  Enabled Capability Demonstrations to garner about 50 percent of its  advanced technology development budget. The Army also established  another technology transition effort in 2012, the Technology Maturation  Initiative (TMI), which is intended to encourage a stronger partnership  between the S&T and acquisition communities. The Army’s Rapid  Equipping Force (REF) program straddles the boundary between being a  technology transition program and rapid acquisition program, scanning  the S&T base for technologies that require limited development, testing,  or both, as well as readily available technologies that can be delivered  immediately to the warfighter to fill capability gaps.
Similar to the Army, the Air Force is modifying its approach to technology  transition. The Advanced Technology Demonstration (ATD) program is an  established Air Force Research Laboratory program that provides  technology transition opportunities, though the size of the program has  declined in recent years. According to Air Force officials, a challenge for  the Advanced Technology Demonstration program is that priorities of the  major commands, such as Air Combat Command, can change due to  environmental factors, internal analysis, and funding constraints. This can  result in ongoing Advanced Technology Demonstration projects not being  completed because of new, higher priorities overtaking existing projects.  A new program—Flagship Capability Concepts (FCC)—is expected to  provide a mechanism through which the Air Force can ensure  commitments from senior leadership as well as from the major commands  to transition technologies to the acquisition community. Specifically,  Flagship Capability Concepts are vetted through Air Force senior  leadership to ensure they align with strategic priorities and have clear  intent for transition. Finally, the Core Process 3 (CP3) program provides  the Air Force with its own program focused on rapidly responding to  urgent needs.
In total, the military departments’ technology transition programs received  over $6 billion for fiscal years 2010 through 2012. Over 40 percent of this  funding—about $2.6 billion—supported DOD’s Small Business Innovation  Research program, which in addition to pursuing technology development  that leads to technology transition, also supports early-stage research  and development to explore technology feasibility. In addition to the Small  Business Innovation Research funding, the military departments made  substantial investments in their other technology transition programs  during the fiscal year 2010-2012 timeframe—about $1.7 billion by the  Navy, over $1.5 billion by the Army, and nearly $370 million by the Air  Force. Table 2 provides a distribution of funding and project-related  details for each of the military department’s transition programs.
Beyond funding availability and technology turnaround times, technology  maturation plays a key role in technology transition. DOD uses  Technology Readiness Levels (TRL) to measure technology maturity  using a scale of one to nine. Level one starts at the lowest level of  technology readiness, where scientific research begins to be translated  into applied research and development. TRL 1, for example, may begin  with a paper study of a technology’s basic properties. It proceeds through  laboratory demonstrations, and ends with an application of the technology  in its final form and under mission conditions. DOD generally expects a  TRL 6 or higher for technologies to be included in a new acquisition  program. Figure 3 portrays each transition program’s TRLs expectations  for a technology to be selected for inclusion in the program and for a  technology at project completion. For example, OSD’s Defense  Acquisition Challenge program typically accepts projects that are already  a TRL 7 or higher, though on occasion will accept projects at a TRL 6,  and will advance the technologies to as high as a TRL 9 before project  completion and transition. Further details on the demonstrated maturity  associated with each TRL are provided in appendix III. In addition to  technology maturity, the figure distinguishes programs that target  conventional transition with longer project timelines and with acquisition  programs often the intended recipients, and programs focused on rapid  response to warfighter needs.
While the figure shows the full range of TRLs that may be accepted by  programs, programs generally prefer to limit the amount of less mature  technologies they will accept, particularly because immature technologies  pose greater risk to successfully completing projects in the short time  frames that many programs demand. Higher TRLs tend to be desired by  most OSD transition programs as well as several military department  programs, such as the Navy’s Technology Insertion Program for Savings,  Army’s Rapid Equipping Force, and the Air Force’s Core Process 3  program. Because these programs are focused on rapidly delivering  technology solutions for the warfighter, many require technologies of TRL  6 or higher for each project from the outset.
Figure 3 also provides context on the outlook for technology transition  programs that are focused more on addressing anticipated technology  needs for future warfighters. Military department programs like the Navy’s  Future Naval Capabilities, Army’s Technology Enabled Capability  Demonstrations, and Air Force’s Flagship Capability Concepts,  encompass a broader focus that primarily targets technology  development opportunities that are expected to meet capability needs  further into the future. Typical transitions for technologies in these  programs are expected to occur in 3 to 6 years, and may be to acquisition  programs that can take several more years to actually deploy the  technologies for warfighter use. Because there is more time afforded by  these programs to develop technologies, they are able to accept less  mature technologies (which is demonstrated in figure 3), as these types of  programs are shown to select technologies that are TRLs 3 and 4 and  advance them to TRL 6 or beyond.
section_title: Technology Transition Programs Provide Technologies to Military Users, But Tracking of Project Outcomes and Benefits after Transition Is Limited
paragraphs: Most programs that we assessed track whether their projects were  completed and successfully transitioned to intended users. On average,  programs reported a historical transition rate of over 70 percent for their  technology transition projects. The vast majority of these projects resulted  in technologies transitioning to acquisition programs or directly to the  warfighter. However, about one-quarter of the projects transitioned to  other organizations, such as test and evaluation centers and industry, for  further development. For the most part, the programs do not track their  projects beyond transition, which limits their ability to know and report  final outcomes for transitioned technologies and any associated benefits  DOD achieved from those technologies.
subsections: section_title: Most Programs Reported the Majority of Their Projects Successfully Transitioned
paragraphs: Technology transition programs reported rates of technology transition  ranging from 56 to 86 percent. Newer programs—such as Army’s  Technology Enabled Capability Demonstrations and Technology  Maturation Initiative, Air Force’s Flagship Capability Concepts, and OSD’s  Rapid Innovation Fund—are still in the process of determining what steps  they will take to measure transition outcomes. The DOD Small Business  Innovation Research program relies heavily on self-reporting by small  businesses for transition data, and because there is no requirement that  all small businesses participating in the program report transitions,  comprehensive program data on transitions is not available. The Air  Force Advanced Technology Demonstration program does not track  transition outcomes for completed projects, which precludes the program  from readily identifying a transition rate. Table 3 provides details on  reported transition rates by program.
While most programs have demonstrated success in transitioning  technologies to military users, differences in expectations across  programs are key to understanding what programs provide to the  warfighter. For example, programs have different definitions for what  project outcomes are considered as “transitions,” particularly as they  relate to transition recipients. To gain a better understanding of transition  outcomes for each program, we evaluated who the recipients of  transitioned technologies were in recent years. Figure 4 shows the  distribution of transitions reported by the transition programs we assessed  for fiscal years 2010-2012 that fall into one of three categories—transition  to an acquisition program, transition directly to the field for use by the  warfighter, and transitions to “other” users such as S&T organizations,  test and evaluation centers, or industry.
Collectively, these programs reported that 545 projects transitioned in  fiscal years 2010 through 2012. The vast majority of the projects resulted  in technologies transitioning to acquisition programs or directly to the  warfighter in the field. However, some programs take a broader view and  consider the transfer of technologies to other organizations for further  development and demonstration, such as test and evaluation centers, or  to industry as technology transition. In total, nearly one-fourth of reported  transitions constituted this type of transition. For example, Navy’s  SwampWorks and Experimentation, which has a mission to focus on  high-risk technologies and concepts, stated that successful transition can  be showing the feasibility of a high risk technology to allow further  development of it at another level in the Navy’s S&T community.
The following are a few examples of transition program projects that  provide context for the array of technologies and the users supported by  these transitions.
Persistent Ground Surveillance System: A Joint Capability  Technology Demonstration project that transitioned a family of wide  area surveillance systems consisting of intelligence, surveillance, and  reconnaissance technologies, as well as communications  technologies to forward operating bases in Afghanistan. The systems  were integrated into tailored tower and aerostat platforms that are in  use and sustained by the Army.
Mosquito prototype: An Air Force Core Process 3 project that  provided a field-tested prototype of a backpack-carried system for  rapidly determining ground bearing strength for landing aircraft in  remote sites. Following evaluations by users, the Air Force funded  efforts to ruggedize and deliver additional units to the field for use.
Tactical Control System Digital Video Enhancements: A Navy  Rapid Technology Transition project to upgrade the computer  interface and video processing capability for the Vertical Takeoff and  Landing Tactical Unmanned Aerial Vehicle (VTUAV) system’s ground  control station. The VTUAV acquisition program has already  purchased and fielded the complete system upgrade, and plans to  acquire more of the systems.
section_title: Few Programs Track Project Outcomes and Benefits after Transition
paragraphs: As GAO has reported in the past, tracking technology transitions and the  impact of those transitions, such as cost savings or deployment of the  technology in a product, provides key feedback that can inform the  management of programs. In particular, GAO found that leading  commercial companies tracked technology transition not only to enable  them to measure success, but also to assess their processes and  determine what changes are necessary to improve transition rates. In  addition, tracking outcomes provides the laboratories and management  with feedback on the impact technology investments have on their  companies. The transition programs we reviewed predominately stop  tracking a project once the program is no longer funding it, or shortly  thereafter, and generally have limited insight into benefits realized by  users for their transition project portfolios. For example, most program  officials we spoke to indicated they lack the ability to obtain feedback that  would indicate a technology that transitioned to a weapon system  acquisition program was ultimately integrated into the weapon system  and deployed for use by the warfighter, or was not integrated into the  system, but did support competition that led to enhanced capabilities or  reduced cost for the weapon system.
A few programs did identify processes for tracking transitioned projects  through the acquisition process and to deployment within a fielded  weapon system. The Navy uses a Transition Review Board to monitor  completed projects from the Future Naval Capabilities, Rapid Technology  Transition, and Technology Insertion Program for Savings programs. The  board determines and reports on whether transitioned projects are utilized  in systems that support Navy warfighters. The Navy determined, for  example, that of the 155 technology products the Future Naval  Capabilities program delivered to acquisition programs between fiscal  years 2006-2011, 21 percent were subsequently deployed to fleet forces,  35 percent were still with the acquisition programs, and 44 percent failed  to deploy. For projects that do not successfully deploy, the board  assesses whether there are other benefits achieved, such as  technologies leveraged for follow-on S&T work. The board also identifies  obstacles to transition, such as loss of interest by the user or inadequacy  of funding. These findings, along with a detailed one-page summary for  each project, are then used to inform the Navy’s annual review process.  We found that by maintaining this level of tracking, the Navy is better  aware of the benefits and obstacles associated with a substantial portion  of their S&T portfolio, which may better inform decisions made by Navy  leadership.
In addition to tracking from transition to deployment, another notable  project outcome measure we observed among the transition programs— specifically within the ManTech program—is the amount of cost savings  or cost avoidance for projects. ManTech is somewhat unique when  compared to other transition programs because its mission is focused on  reducing acquisition and supportability costs of DOD weapon systems  and improving manufacturing and repair efficiency for weapon systems.  The ability of the Navy, Army, and Air Force components of ManTech to  provide actual as well as projected cost-reduction benefits coming from  transition and implementation of their projects presents an opportunity to  clearly demonstrate value to the warfighter and value of the program. A  couple of examples for ManTech cost savings and cost avoidance  measures are illustrated by the projects below.
Energy Storage Manufacturing Very High Power Batteries: Army  project that reportedly has produced $20 million in cost avoidance  benefit by developing an automated lithium-ion battery manufacturing  line to include improvements that yield affordable, high-power battery  packs.
Virginia Class Submarine Affordability Initiative: The Navy  estimates that this effort has already yielded over $21 million in cost  savings per hull through a series of projects transitioned to the  acquisition program that provides manufacturing efficiencies.
section_title: Programs Facilitate Transition Through Sound Project Selection and Management Practices
paragraphs: As GAO has reported in the past, technology transition programs need to  establish disciplined selection and management processes as well as  tools to ensure that new technologies can be effectively transitioned to a  military user. Programs must have processes in place that support the  selection of projects that have realistic schedule and cost expectations,  as well as sufficiently mature technologies. In addition, early project  endorsement from intended users and other key stakeholders, including  senior leadership and S&T developers, is critical. Once selected, projects  require effective management to ensure technology risks are minimized,  costs and schedules are maintained, and transition commitments are  confirmed. To achieve this, stakeholders must engage in regular  communication during project execution. Formal agreements and other  project assessment measures also are important to ensure projects stay  on track and stakeholders sustain their commitments to transition. In  conducting this review, we found that OSD and military department  technology transition programs make use of these practices to varying  degrees. All of the programs we reviewed have established project  selection processes that strive to select relevant and feasible  technologies that have strong stakeholder support. However,  management practices varied across programs, with some programs  using more robust, formal tools, such as technology transition  agreements, to guide project management and communicate project  progress and expectations.
subsections: section_title: Clear Expectations for Cost, Schedule, and Technology Maturity Objectives Are Central to Project Selection and Management
paragraphs: Establishing a path to facilitate technology transition begins with each  transition program’s ability to select relevant and feasible technologies to  meet military needs. Programs must identify technologies that have the  potential to provide new or enhanced capabilities, and can generally be  matured or demonstrated in a timely manner with modest program  investments. Identifying viable candidate technologies requires that  programs clearly communicate what capabilities are needed and solicit  proposals from the right audience.
Most of the transition programs we reviewed identify and set forth high- priority capability needs or topics for project solicitations based on input  from some combination of the S&T and acquisition communities,  operational users, and senior military leadership. Having individuals with  different focuses involved in generating topics helps ensure that the right  technology projects will be proposed and that projects will have interested  users. A limited number of programs we reviewed do not set topics, but  do require submitted project proposals to demonstrate a capability need.  For example, Navy TechSolutions, which has a relatively small budget— about $9 million per year—offers an open invitation for warfighters in the  field to propose projects based on identified capability needs. Regardless  of solicitation approach, all transition programs we reviewed have  established criteria for evaluating whether to pursue proposed projects.  As we previously identified, each program has its own project cost and  timeline constraints, as well as technology maturity criteria in place to  support projection selection. Clear expectations for these factors can  reduce the risk of a technology failing to transition because of cost,  schedule, or technology maturity concerns.
section_title: Early Project Endorsements by Stakeholders Can Influence Transition
paragraphs: Securing comprehensive stakeholder endorsements for projects from  their outset is another important influence on whether projects transition.  Most of the transition programs we reviewed have processes or  mechanisms they use to foster some type of commitment from  prospective users to transition projects once completed. However,  officials in some programs also reported that garnering senior leadership  support for proposed projects is critical, as it ensures that projects are  aligned with the priorities of department leadership as well as user needs.  For example, the Air Force’s Flagship Capability Concept program  emphasizes senior leadership involvement by requiring all projects to be  approved by the Air Force Requirements Oversight Council. Approval is  supported by recommendations from the Air Force S&T governance  structure, which is composed of senior-level leadership from Air Force  headquarters and major commands. According to an Air Force official,  this governance structure serves to ensure S&T efforts are properly  aligned with Air Force needs, thereby facilitating transition and  deployment to the operational community.
The commitment of sponsor funding, or co-funding is another means  used to solidify stakeholder endorsement for transition projects. The  investment by prospective users serves as an endorsement of sorts, as it  can create greater buy-in for transition since the intended users have a  monetary stake in the project. Specifically, we found that several of the  transition programs we reviewed require or encourage prospective users  to help fund projects. For example, programs like OSD’s Foreign  Comparative Testing and Rapid Reaction Fund and the Navy’s Rapid  Technology Transition program do not require co-funding but do use it in  some cases. The Joint Capability Technology Demonstration program  requires that at least 50 percent of a project is co-funded, and program  officials noted that their projects normally have multiple sponsors that  provide funding.
section_title: Stakeholder Commitment and Communication Are Key to Effective Management of Transition Programs and their Projects
paragraphs: Establishing clear and consistent commitments and communication  channels among stakeholders is fundamental to managing transition  projects and achieving transition. Senior leadership plays a key role in  initiating and reinforcing effective program management practices. At the  organizational level, we found several examples, such as with the Army,  where officials indicated that their top S&T leadership championed  institutional changes to emphasize transition, including the introduction of  the Technology Enabled Capability Demonstration and Technology  Maturation Initiative programs. These programs are intended to produce  an integrated Army S&T strategy that uses dedicated transition programs  to more effectively respond to warfighter needs. At the program level,  many program officials indicated that senior leadership engagement,  particularly in providing oversight for projects through to transition, is  essential to having an effective program. We found the Future Naval  Capabilities program provides a good example of senior leadership  positively affecting project management activities. Specifically, due to  funding constraints in its fiscal year 2013 S&T budget, Navy senior  leadership supported the termination of ongoing Future Naval Capabilities  projects that were determined to be lower priorities so that new, higher  priority projects could be pursued. Navy officials stated that this type of  awareness and understanding at senior levels enables the Future Naval  Capabilities program to make efficient decisions that are less likely to  meet resistance and that support the highest priority projects being  developed for transition opportunities.
Several transition programs also emphasized the relationship between  “working-level” stakeholders—S&T developers and acquisition programs  or warfighters in the field—when discussing the keys to technology  transition. These stakeholders manage expectations throughout a project  and ensure it will meet user needs. This reduces the risk of completed  projects languishing because funding is not available or because user  requirements have changed, or both. Some programs that we reviewed  use integrated product teams, which may be composed of individuals  representing the requirements, acquisition, operational, and S&T  communities, among others, to facilitate continuous communication with  stakeholders and ensure that transition planning is on track. In the case of  the Navy, integrated product teams identify capability gaps, provide input  on which S&T projects may address those gaps, assess project progress,  make sure transition strategies remain valid, and confirm funding is  aligned to support transition. According to Navy officials, the results of  integrated product team efforts also support information sharing across  senior- and working-level stakeholders to validate development status  and transition planning activities.
section_title: Transition Agreements and Other Project Assessment Tools Are Used to Formalize Project Expectations and Help Sustain Commitments to Transition Technologies
paragraphs: We found a range of management tools being used by some transition  programs that underlie support and communication among stakeholders.  Technology transition agreements are used by 9 of the 20 transition  programs we reviewed to manage projects. These are “good-faith”  agreements between stakeholders that document the expectations for  developing, demonstrating, delivering, and integrating technologies into  systems, or using them as standalone products. Agreement specifics vary  by program and can be tailored for each project, but typically outline  technology and readiness metrics, such as cost, schedule, and  performance parameters that labs must meet for transition to occur. As  GAO has reported in prior work, these metrics help sustain a strong  transition path by providing a formal way to track progress against  requirements.agreements are used by stakeholders to make informed decisions on  projects that are not meeting expectations. For example, if the labs  cannot develop a technology within cost and schedule expectations, or  cannot meet specified performance characteristics, the prospective user  can terminate the agreement and pursue alternative technology options.
Also as GAO has reported, technology transition  The Joint Capability Technology Demonstration program within OSD and  the Navy’s Future Naval Capabilities program both require technology  transition agreements for all projects and provide insights into the content  and value these agreements can offer to stakeholders. The Joint  Capability Technology Demonstration program agreements, at a  minimum, must outline:
Operational need
Proposed technical solution
Transition target information
Transition requirements
Integration strategy
Business case
Risks
Costs and schedule
Project points of contact  We found the Future Naval Capabilities program uses technology  transition agreements as management tools to increase the level of  documented commitment as a project progresses over time. To  accomplish this, the program has three levels for agreements that reflect  the requisite knowledge available at different phases of a project. Key  elements of an initial agreement include a basic project description,  identification of initial exit criteria, a high-level integration strategy, and a  likely transition funding source. As a project progresses, the other two  levels of agreement require increasing commitment and specificity of  requirements from stakeholders to develop, deliver, and integrate a  Future Naval Capabilities project into an acquisition program or other  form of deployment. Key elements of the second and third tier  agreements involve refining and finalizing project descriptions, detailing  exit criteria, providing greater specificity about the integration strategy,  and providing estimates for transition costs and eventually executing  transition funding. Stakeholders review the agreements annually to  revalidate the commitments laid out within the document.
We also found Transition Commitment Level (TCL) assessment tools— used by the Joint Capability Technology Demonstration and Future Naval  Capabilities programs—offer another means of validating that transition  programs are investing in projects that have a firm transition commitment  from prospective users. These tools provide scorecards that chart how  well-defined the fundamental characteristics that support a strong  commitment to transition projects are at a given point in time. The Future  Naval Capabilities program uses a single TCL tool that documents level  of transition commitment from project start to completion. The Joint  Capability Technology Demonstration program uses two TCL assessment  tools—one to identify transition commitment to support project selection  and one to track commitment through project execution and to  completion. Figure 5 shows the TCL assessment tool used for Joint  Capability Technology Demonstration project execution and completion.
As shown by the figure, a TCL tool provides a formal way for determining  which proposed and active projects have a high, medium, and low  probability to transition. For example, in order for a Joint Capability  Technology Demonstration project to be designated as “green,” project  managers must show the project has a clear path to one of the three  transition recipients as defined by the program. The TCL tools used by  both the Joint Capability Technology Demonstration and Future Naval  Capabilities programs provide insights into the progress of projects based  on established benchmarks. In addition, Navy officials stated that the TCL  tool used by Future Naval Capabilities provides information that can be  used by program managers to inform investment decisions. In particular,  the TCL tool can help identify active projects that may require greater  attention to solidify transition commitments from stakeholders, and can  also be used to support decisions to end projects that no longer have  stakeholder support.
Eleven of the programs we reviewed do not use formal technology  transition agreements, but incorporate some of the elements of  agreements in managing projects. For example, OSD’s Foreign  Comparative Testing program requires documentation of a valid need for  a technology when a project starts and a letter of endorsement from a  prospective military user that supports procuring the technology at  completion. The Rapid Reaction Fund program does not require  commitments from users, but does require them to identify a potential  transition path at project start and then the program will follow-up with the  users to assess their interest in the technology if it is successfully  developed. Although prior GAO work found that technology transition  agreements do not guarantee transition success because requirements  and funding changes can occur, the agreements are useful tools to  solidify expectations and secure user commitment. At the time, GAO  recommended that DOD expand the use of technology transition  agreements. DOD concurred with the recommendation, but did not  identify specific actions it planned to take to do so.
section_title: Conclusions
paragraphs: For the most part, technology transition programs we reviewed have been  reasonably successful in delivering technologies to military users.  However, a limited number of programs do not have metrics established  to track completed projects. In addition, few programs measure whether  transitioned technologies actually result in a benefit to users. For  example, many programs track whether a technology is delivered to an  acquisition program of record, but often have limited or no insight into  whether the technology resulted in enhanced performance or new  capabilities, cost savings, or reduced times for testing and evaluation and  system integration. Without some measure of technology implementation  or other benefit after transition, questions remain as to whether programs  are providing the right technologies at the right time to users, are using  effective approaches to select, develop, and transition technologies, and  are providing tangible benefits.
The transition programs we reviewed generally have disciplined project  selection and management processes in place to facilitate the transition  of technologies to intended users. To varying degrees, programs  emphasize the need for early and sustained commitments from senior  military leadership, S&T developers, and military users in order to ensure  projects are needed, have a sound basis, and risks are reduced. A key  mechanism some programs use for obtaining these commitments are  formal tools, such as technology transition agreements and technology  commitment level assessments. These tools help to clarify expectations,  hold stakeholders accountable for what they must deliver, and gauge  progress towards achieving project objectives. There may be  opportunities for more widespread use of these tools among the  programs we reviewed, which could help strengthen technology transition  success.
section_title: Recommendations for Executive Action
paragraphs: To improve visibility and management of the department’s efforts to  transition technologies to support the needs of the warfighter, we  recommend that the Secretary of Defense take the following two actions:
Require that all technology transition programs track and measure  project outcomes, to include not only whether technologies  transitioned to an intended user but also the longer-term impact of  whether the technologies benefitted acquisition programs or military  users in the field.
Assess transition programs to identify opportunities for more  widespread use of existing transition management tools, such as  technology transition agreements and technology commitment level  evaluation mechanisms.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a copy of a draft of this report to DOD for review and  comment. In written comments on the draft, DOD partially concurred with  our first recommendation and fully concurred with the second  recommendation. DOD’s comments appear in appendix IV.
In responding to the first recommendation, DOD agreed that all transition  programs should track and measure project outcomes, including how the  technology or capability is being employed by the intended user.  However, the department raised concerns that tracking and measuring  technology project outcomes would be a labor-intensive and time- consuming process requiring significant investment. DOD also suggested  that the recommendation be revised to have the department assess  technology transition programs for opportunities to implement processes  to track and measure project outcomes. We continue to believe that it is  important that DOD have a formal means to track and document  technology transition in order to demonstrate not only whether technology  investments under these programs are reaching their intended users, but  also that they are providing the desired benefits. We believe that DOD  should require all programs to track and measure technology transition,  and that the department’s suggested revision to our recommendation  could potentially remove any responsibility for programs to do so.  Moreover, while we acknowledge that tracking and measuring project  benefits following transition would require additional time and effort, we  believe our recommendation provides DOD with significant flexibility to  measure outcomes in different ways that would be less resource- intensive than DOD envisions. For example, rather than tracking all  transitioned projects, the department could assess a subset of projects or  establish mechanisms for military users to report back on the results. In  addition, as we reported, the Navy has recently used an approach that  could be adopted by other programs in DOD to measure potential  benefits from transitioned projects. Specifically, the Office of Naval  Research has employed a Technology Review Board to complete  independent reviews of Navy transition programs and report whether  transitioned projects were being used by the warfighter. Consequently,  we have not revised our recommendation as suggested by DOD. In  concurring with our second recommendation, the department agreed that  all technology transition programs should identify opportunities to use  existing transition management tools. DOD stated that it will ensure  guidelines are published and widely disseminated to improve the  understanding as to when, how, and under what circumstances transition  management tools can be employed.
We are sending copies of this report to appropriate congressional  committees, the Secretary of Defense, and the Secretaries of the Air  Force, Army, and Navy. In addition, this report will be available at no  charge on the GAO website at http://www.gao.gov. If you or your staff  have any questions concerning this report, please contact me at (202)  512-4841 or by email at sullivanm@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on   the last page of this report. Key contributors to this report are listed in  appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: To identify what Department of Defense (DOD) programs exist that are  dedicated to facilitating technology transition from the science and  technology (S&T) community to a user, we reviewed DOD reports and  documents. As a starting point, we reviewed a March 2012 report to  Congress by the Under Secretary of Defense for Acquisition, Technology  and Logistics that assessed technology transition programs within DOD.  In conducting this assessment, DOD reviewed budget program elements  to identify potential technology transition programs and then surveyed  these programs to validate those whose primary function is technology  transition. We discussed the programs identified in the DOD report with  officials from the Office of the Secretary of Defense (OSD) and the  military departments and made adjustments to our scope based on their  inputs and our own analysis. The resulting list of programs, which is the  focus of our review, includes 20 transition programs across OSD and the  military departments—Air Force, Army, and Navy. While there are other  technology transition activities that occur within the military departments  and other DOD organizations, such as those within the Joint Improvised  Explosive Device Defeat Organization and Defense Advanced Research  Projects Agency, we limited our scope to programs within DOD dedicated  to technology transition.
We then interviewed officials from the programs, as well as other officials  within OSD and the military departments affiliated with technology  transition activities, to become familiar with their respective technology  transition programs. We collected and analyzed information from the  programs on their structures, processes, tools, and technology transition  metrics used. To assess what technology transition programs are doing to  meet their objectives, we reviewed program policies and procedures for  selecting technology transition projects, monitoring and managing their  progress, and determining transition outcomes. For the purpose of  assessing the programs’ activities—including projects started, completed,  and transitioned to users—we gathered data from the programs for fiscal  years 2010 through 2012. For the purpose of reviewing and reporting  historical transition rates for the programs, we collected self-reported  transition rates from program officials for those programs that could  provide such information. We did not validate the data provided by the  technology transition program officials, but reviewed and discussed the  data with them and determined it was sufficiently reliable for the purposes  of this report. To identify what practices may facilitate technology  transition, we reviewed prior GAO studies on DOD technology transition  and best practices for transition, interviewed officials from OSD and the  military departments, and reviewed programmatic information and other  related DOD documentation. Using this information, we were able to  isolate a number of practices that influence a program’s ability to  transition technology from the S&T community to a prospective user.  Additionally, we reviewed what practices were present in each program.  In performing our work, we obtained information and interviewed OSD  and military department officials from:  Assistant Secretary of Defense, Research and  Engineering, Arlington, Virginia; Defense Advanced  Research Projects Agency, Arlington, Virginia  Office of Science and Technology, Arlington, Virginia;  Rapid Equipping Force, Ft. Belvoir, Virginia; Small  Business Innovation Research and Manufacturing  Technology programs, Aberdeen Proving Ground,  Maryland  Air Force Research Laboratory, Wright-Patterson Air Force  Base, Ohio; Air Force Acquisition and Research, Arlington,  Virginia;  Office of Naval Research, Arlington, Virginia; Navy  Research and Development, U.S. Naval Yard,  Washington, D.C.; Deputy Assistant Secretary of the Navy  for Research, Development, Test, and Evaluation,  Arlington, Virginia  We conducted this performance audit from April 2012 to March 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: DOD Research, Development, Test, and Evaluation Budget Activities
subsections: section_title: Science and technology funding
paragraphs: DOD RDT&E Budget Activity  Description  Basic Research (6.1)
Includes all scientific study and experimentation directed toward increasing  fundamental knowledge and understanding in physical, engineering, environmental,  and life sciences related to long-term national security needs. It is farsighted high  payoff research that provides the basis for technological progress. Program elements  in this category involve pre-Milestone A efforts, which is the point where entry into the  Technology Development phase is approved.
Varies from systematic mission-directed research beyond that done through basic  research, to sophisticated breadboard hardware, study, programming and planning  efforts that establish the initial feasibility and practicality of proposed solutions to  technological challenges. The dominant characteristic is it is directed toward general  military needs with a view toward developing and evaluating the feasibility and  practicality of proposed solutions and determining their parameters. Program elements  in this category involve pre-Milestone B efforts, also known as Technology  Development phase tasks.
Includes concept and technology demonstrations of components and subsystems or  system models. The results of this type of effort are proof of technological feasibility  and assessment of subsystem and component operability and producibility rather than  the development of hardware for service use. Advanced Technology Development  demonstrates the general military utility or cost reduction potential of technology when  applied to different types of military equipment or techniques. Program elements in this  category involve pre-Milestone B efforts, such as system concept demonstration, and  generally have technology readiness levels of 4, 5, or 6. Projects in this category  should have the goal of moving out of S&T and into the acquisition process within the  future years defense program. Upon successful completion of projects that have  military utility, the technology should be available for transition.
section_title: Acquisition- based Funding
paragraphs: Advanced Component  Development & Prototypes  (6.4)
Efforts necessary to evaluate integrated technologies, representative modes or  prototype systems in a high fidelity and realistic operating environment. Includes  system specific efforts that help expedite technology transition from the laboratory to  operational use. Emphasis is on proving component and subsystem maturity prior to  integration in major and complex systems and may involve risk reduction initiatives.  Program elements in this category involve efforts prior to Milestone B acquisition  program start. Completion of technology readiness levels 6 and 7 should be achieved  for major programs. Program control is exercised at the program and project level.
Post-Milestone B, conducting engineering and manufacturing development tasks  aimed at meeting validated requirements prior to full-rate production. Prototype  performance is near or at planned operational system levels. Characteristics involve  mature system development, integration and demonstration to support Milestone C  production decisions, and conducting live fire test and evaluation and initial operational  test and evaluation of production representative articles.
RDT&E Management Support  (6.6)
Efforts to sustain and/or modernize the installations or operations required for general  RDT&E. Includes test ranges, military construction, maintenance support of  laboratories, operation and maintenance of test aircraft and ships, and studies and  analyses in support of the RDT&E program. Military construction costs directly related  to major development programs are included.
Efforts to upgrade systems that have been fielded or have received approval for full- rate production and anticipate production funding in the current or subsequent fiscal  year. Programs in this category involve systems that have received Milestone C  production decision approval.
section_title: Appendix III: DOD Technology Readiness Levels
paragraphs: Appendix III: DOD Technology Readiness  Levels Technology readiness  level (TRL)  1. Basic principles  observed and reported.
Description  Lowest level of technology readiness. Scientific research  begins to be translated into applied research and development.  Examples might include paper studies of a technology’s basic  properties.
2. Technology concept  and/or application  formulated.
Invention begins. Once basic principles are observed, practical  applications can be invented. Applications are speculative, and  there may be no proof or detailed analysis to support the  assumptions. Examples are limited to analytic studies.
3. Analytical and  experimental critical  function and/or  characteristic proof of  concept.
Active research and development is initiated. This includes  analytical studies and laboratory studies to physically validate  the analytical predictions of separate elements of the  technology. Examples include components that are not yet  integrated or representative.
4. Component and/or  breadboard validation in  laboratory environment.
Basic technological components are integrated to establish  that they will work together. This is relatively low fidelity  compared with the eventual system. Examples include  integration of ad hoc hardware in the laboratory.
5. Component and/or  breadboard validation in  relevant environment.
Fidelity of breadboard technology increases significantly. The  basic technological components are integrated with reasonably  realistic supporting elements so they can be tested in a  simulated environment. Examples include high-fidelity  laboratory integration of components.
6. System/subsystem  model or prototype  demonstration in a  relevant environment.
Representative model or prototype system, which is well  beyond that of TRL 5, is tested in a relevant environment.  Represents a major step up in a technology’s demonstrated  readiness. Examples include testing a prototype in a high- fidelity laboratory environment or in a simulated operational  environment.
7. System prototype  demonstration in an  operational  environment.
Prototype near or at planned operational system. Represents a  major step up from TRL 6 by requiring demonstration of an  actual system prototype in an operational environment (e.g., in  an aircraft, in a vehicle, or in space).
8. Actual system  completed and qualified  through test and  demonstration.
Technology has been proven to work in its final form and under  expected conditions. In almost all cases, this TRL represents  the end of true system development. Examples include  developmental test and evaluation of the system in its intended  weapon system to determine if it meets design specifications.
9. Actual system proven  through successful  mission operations.
Actual application of the technology in its final form and under  mission conditions, such as those encountered in operational  test and evaluation. Examples include using the system under  operational mission conditions.
section_title: Appendix IV: Comments from the Department of Defense
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, John Oppenheim, Assistant  Director; Danielle Greene; John Krump; Sean Merrill; Robert Miller; Carol  Petersen; and Scott Purdy also made key contributions to the report.