Defense Technology Development: Technology Transition Programs Support Military Users, but Opportunities Exist to Improve Measurement of Outcomes
Released: 2013-03-07T00:00:00

Letter

Background

DOD relies on its S&T community—DOD research laboratories, test facilities, industry, and academia—to identify, pursue, and develop new technologies that improve and enhance military operations and ensure technological superiority over adversaries. These activities by DOD’s S&T community support the development of technologies for new or existing weapon systems, as well as development of capabilities that are transitioned directly to warfighters in the field. The S&T community engages in activities ranging from basic research through advanced technology development that are conducted by the government or externally by universities and commercial industry. Once the S&T community has completed its technology development, additional product development activities, such as technology demonstration and testing, are often needed before incorporating the technologies into military weapon systems. Under the management of the acquisition community, product development further advances technology received from S&T developers and integrates it into systems that are ultimately delivered to support the warfighter.

Supporting these activities is DOD’s research, development, test, and evaluation budget. As shown in figure 1 below, this budget is divided into seven categories that follow a generally sequential path for developing technologies from basic research to operational system development. The first three categories represent DOD’s S&T activities to advance research and technology development, while the remaining categories support product development for DOD acquisition programs. Appendix II provides further details on research and development activities in these budget categories.

The large size and scope of DOD’s national security mission imposes broad and competing demands on the S&T community, and determining the right mix of technology investments to pursue is challenging. One key expectation is that S&T funding will result in relevant and feasible technologies that can transition into weapon system programs or go directly to the warfighter in the field. DOD has also tasked its S&T community with anticipating its technological needs for an uncertain future. Further, the S&T community serves as a technology innovation mechanism to expand scientific knowledge and investigate technologies that may provide breakthrough warfighting capabilities. As a result, some investments focus on conducting research to generate scientific knowledge, exploring new technologies, demonstrating the feasibility of a technology concept, and other science and technology endeavors. The challenge is finding the right balance between taking risk to develop breakthrough technologies versus investing in moderate technology enhancements that are likely to transition to a military user. Although not precisely defined, technology transition generally occurs when advanced technology development ends and product development begins. Figure 2 below provides a notional picture of how DOD’s S&T community manages technology investment, development, and transition to a user.

DOD’s research laboratories and test facilities, as well as industry and academia, provide many examples of developing new technologies that improve warfighter capabilities and enhance military operations. However, as we have reported in the past, for a variety of reasons DOD historically has experienced problems in transitioning technologies out of its S&T environment and into military systems. Technologies may not leave the lab because their potential has not been adequately demonstrated or recognized, acquisition programs may be unwilling to fund final stages of development, or private industry chooses to develop the technologies itself.

Despite these challenges, technology transition can be facilitated in many ways, and is affected by choices made throughout the S&T management cycle. Before technology development is begun, critical S&T investment decisions must be made that balance needs, resources, and technical feasibility in a way that ensures the S&T community is responsive to warfighter priorities. Once S&T projects are underway, technology development provides opportunities to demonstrate feasibility and management decisions must be made along the way by S&T developers—with input from the acquisition community—on which technologies should be developed to a final state that would facilitate technology transition.

DOD Has a Variety of Technology Transition Programs That Support Military Users

We identified 20 technology transition programs, managed by OSD and the military departments, that provide structured mechanisms and funding to facilitate technology transition. These programs vary in size, mission, approach, funding, and technology maturity expectations. However, all are consistent in providing opportunities to transition technologies from the S&T community to a military user, such as an acquisition program or the warfighter in the field. Many of the transition programs are focused on rapid response to warfighter needs, which has received greater emphasis in the recent past because of the United States’ operations in Iraq and Afghanistan. In addition, most programs target fairly mature technologies, which are suitable for final stages of development and demonstration. Collectively, the programs we reviewed use a mix of S&T and other research, development, test, and evaluation funding—about $7.9 billion obligated from fiscal years 2010 through 2012—to facilitate technology transition.

OSD Has Seven Technology Transition Programs That Vary in Their Broader Missions, Approaches, and Funding

OSD manages seven transition programs that are intended to accelerate development, testing, and delivery of mature technologies that provide new solutions for military needs. Six programs are managed by the Deputy Assistant Secretary of Defense, Rapid Fielding, who reports to the Under Secretary of Defense for Acquisition, Technology, and Logistics. They include three programs authorized by Congress—the Defense Acquisition Challenge (DAC), Foreign Comparative Testing (FCT), and Rapid Innovation Fund (RIF)—and three programs established by DOD—Joint Capability Technology Demonstration (JCTD), Quick Reaction Fund (QRF), and Rapid Reaction Fund (RRF). In addition, OSD manages a Small Business Innovation Research (SBIR) program, which was established by Congress and has technology transition—described by the program as “commercialization”—as a tenet.

The OSD programs share the general purpose to transition technologies, but vary in what type of technology developers and operational needs are targeted. For example, the Joint Capability Technology Demonstration program addresses joint warfighting needs of the combatant commands, in partnership with the military services, by demonstrating mature technology prototypes that may transition to acquisition programs or directly to the warfighter in the field. In contrast, the Foreign Comparative Testing program mission is to identify and test technologies that have already been developed by other countries and may have utility in addressing U.S. military needs. Other programs like the Rapid Innovation Fund and Small Business Innovation Research seek to leverage technology solutions from small businesses. And still others, like the Quick Reaction Fund and Rapid Reaction Fund programs, seek to target specific urgent conventional and irregular warfighting problems, respectively, and rapidly deliver technologies to operational users.

Collectively, research, development, test, and evaluation funding for the OSD technology transition programs we reviewed included about $1.75 billion for fiscal years 2010 through 2012. The Joint Capability Technology Demonstration and Rapid Innovation Fund programs were the most substantial part of that funding—over $1.2 billion for fiscal years 2010 through 2012. The majority of OSD technology transition programs constrain project time frames to 2 years or less, since their focus is rooted predominately on providing funds to accelerate transition for technologies that rapidly respond to current or near-term military needs. Individual projects also represent relatively small funding commitments for the department, with $3 million on the high end of costs, although Joint Capability Technology Demonstration projects can be higher. It also is important to note that several of the programs utilize co-funding provided by prospective technology users to further advance projects. Table 1 provides further information on funding as well as project-related details for each OSD program.

The Departments of the Navy, Army, and Air Force have created distinct programs that rapidly respond to pressing warfighter needs, while also initiating programs that work to address needs that are more mid-term in nature. In addition, as directed by Congress, DOD has established a Manufacturing Technology (ManTech) program and Small Business Innovation Research program that provide technology transition opportunities. All three of DOD’s military departments manage ManTech and Small Business Innovation Research programs.

In the Navy, the Office of Naval Research (ONR) has a well-established technology transition focus. ONR’s Office of Transition manages the Future Naval Capabilities (FNC) portfolio, which is the Navy’s largest transition program—for which nearly $450 million was budgeted in fiscal year 2013. The program, which was initiated in 1999, seeks to provide the best technology solutions to address operational requirements, delivering technology products to acquisition programs that enhance capabilities within a 5-year time frame. ONR’s Offices of Transition and Innovation also support rapid technology transition to the fleet, force, and acquisition communities via the Rapid Technology Transition (RTT), Technology Insertion Program for Savings (TIPS), TechSolutions (TS), and SwampWorks and Experimentation (SW/Exp) programs.

The Army’s efforts to facilitate transition are evolving. In 2011, the Army began a new Technology Enabled Capability Demonstration (TECD) program, which is replacing Army Technology Objectives-Demonstrations (ATO-D). According to Army officials, Technology Enabled Capability Demonstration characteristics were influenced by the Navy’s Future Naval Capabilities program, with the Army identifying a need for a more integrated S&T approach. Key characteristics of Technology Enabled Capability Demonstrations include senior leadership input at the outset for S&T decisions, greater focus on providing capabilities, and better planning on how a technology will transition to provide an operational capability. When fully implemented, the Army expects Technology Enabled Capability Demonstrations to garner about 50 percent of its advanced technology development budget. The Army also established another technology transition effort in 2012, the Technology Maturation Initiative (TMI), which is intended to encourage a stronger partnership between the S&T and acquisition communities. The Army’s Rapid Equipping Force (REF) program straddles the boundary between being a technology transition program and rapid acquisition program, scanning the S&T base for technologies that require limited development, testing, or both, as well as readily available technologies that can be delivered immediately to the warfighter to fill capability gaps.

Similar to the Army, the Air Force is modifying its approach to technology transition. The Advanced Technology Demonstration (ATD) program is an established Air Force Research Laboratory program that provides technology transition opportunities, though the size of the program has declined in recent years. According to Air Force officials, a challenge for the Advanced Technology Demonstration program is that priorities of the major commands, such as Air Combat Command, can change due to environmental factors, internal analysis, and funding constraints. This can result in ongoing Advanced Technology Demonstration projects not being completed because of new, higher priorities overtaking existing projects. A new program—Flagship Capability Concepts (FCC)—is expected to provide a mechanism through which the Air Force can ensure commitments from senior leadership as well as from the major commands to transition technologies to the acquisition community. Specifically, Flagship Capability Concepts are vetted through Air Force senior leadership to ensure they align with strategic priorities and have clear intent for transition. Finally, the Core Process 3 (CP3) program provides the Air Force with its own program focused on rapidly responding to urgent needs.

In total, the military departments’ technology transition programs received over $6 billion for fiscal years 2010 through 2012. Over 40 percent of this funding—about $2.6 billion—supported DOD’s Small Business Innovation Research program, which in addition to pursuing technology development that leads to technology transition, also supports early-stage research and development to explore technology feasibility. In addition to the Small Business Innovation Research funding, the military departments made substantial investments in their other technology transition programs during the fiscal year 2010-2012 timeframe—about $1.7 billion by the Navy, over $1.5 billion by the Army, and nearly $370 million by the Air Force. Table 2 provides a distribution of funding and project-related details for each of the military department’s transition programs.

Beyond funding availability and technology turnaround times, technology maturation plays a key role in technology transition. DOD uses Technology Readiness Levels (TRL) to measure technology maturity using a scale of one to nine. Level one starts at the lowest level of technology readiness, where scientific research begins to be translated into applied research and development. TRL 1, for example, may begin with a paper study of a technology’s basic properties. It proceeds through laboratory demonstrations, and ends with an application of the technology in its final form and under mission conditions. DOD generally expects a TRL 6 or higher for technologies to be included in a new acquisition program. Figure 3 portrays each transition program’s TRLs expectations for a technology to be selected for inclusion in the program and for a technology at project completion. For example, OSD’s Defense Acquisition Challenge program typically accepts projects that are already a TRL 7 or higher, though on occasion will accept projects at a TRL 6, and will advance the technologies to as high as a TRL 9 before project completion and transition. Further details on the demonstrated maturity associated with each TRL are provided in appendix III. In addition to technology maturity, the figure distinguishes programs that target conventional transition with longer project timelines and with acquisition programs often the intended recipients, and programs focused on rapid response to warfighter needs.

While the figure shows the full range of TRLs that may be accepted by programs, programs generally prefer to limit the amount of less mature technologies they will accept, particularly because immature technologies pose greater risk to successfully completing projects in the short time frames that many programs demand. Higher TRLs tend to be desired by most OSD transition programs as well as several military department programs, such as the Navy’s Technology Insertion Program for Savings, Army’s Rapid Equipping Force, and the Air Force’s Core Process 3 program. Because these programs are focused on rapidly delivering technology solutions for the warfighter, many require technologies of TRL 6 or higher for each project from the outset.

Figure 3 also provides context on the outlook for technology transition programs that are focused more on addressing anticipated technology needs for future warfighters. Military department programs like the Navy’s Future Naval Capabilities, Army’s Technology Enabled Capability Demonstrations, and Air Force’s Flagship Capability Concepts, encompass a broader focus that primarily targets technology development opportunities that are expected to meet capability needs further into the future. Typical transitions for technologies in these programs are expected to occur in 3 to 6 years, and may be to acquisition programs that can take several more years to actually deploy the technologies for warfighter use. Because there is more time afforded by these programs to develop technologies, they are able to accept less mature technologies (which is demonstrated in figure 3), as these types of programs are shown to select technologies that are TRLs 3 and 4 and advance them to TRL 6 or beyond.

Technology Transition Programs Provide Technologies to Military Users, But Tracking of Project Outcomes and Benefits after Transition Is Limited

Most programs that we assessed track whether their projects were completed and successfully transitioned to intended users. On average, programs reported a historical transition rate of over 70 percent for their technology transition projects. The vast majority of these projects resulted in technologies transitioning to acquisition programs or directly to the warfighter. However, about one-quarter of the projects transitioned to other organizations, such as test and evaluation centers and industry, for further development. For the most part, the programs do not track their projects beyond transition, which limits their ability to know and report final outcomes for transitioned technologies and any associated benefits DOD achieved from those technologies.

Most Programs Reported the Majority of Their Projects Successfully Transitioned

Technology transition programs reported rates of technology transition ranging from 56 to 86 percent. Newer programs—such as Army’s Technology Enabled Capability Demonstrations and Technology Maturation Initiative, Air Force’s Flagship Capability Concepts, and OSD’s Rapid Innovation Fund—are still in the process of determining what steps they will take to measure transition outcomes. The DOD Small Business Innovation Research program relies heavily on self-reporting by small businesses for transition data, and because there is no requirement that all small businesses participating in the program report transitions, comprehensive program data on transitions is not available. The Air Force Advanced Technology Demonstration program does not track transition outcomes for completed projects, which precludes the program from readily identifying a transition rate. Table 3 provides details on reported transition rates by program.

While most programs have demonstrated success in transitioning technologies to military users, differences in expectations across programs are key to understanding what programs provide to the warfighter. For example, programs have different definitions for what project outcomes are considered as “transitions,” particularly as they relate to transition recipients. To gain a better understanding of transition outcomes for each program, we evaluated who the recipients of transitioned technologies were in recent years. Figure 4 shows the distribution of transitions reported by the transition programs we assessed for fiscal years 2010-2012 that fall into one of three categories—transition to an acquisition program, transition directly to the field for use by the warfighter, and transitions to “other” users such as S&T organizations, test and evaluation centers, or industry.

Collectively, these programs reported that 545 projects transitioned in fiscal years 2010 through 2012. The vast majority of the projects resulted in technologies transitioning to acquisition programs or directly to the warfighter in the field. However, some programs take a broader view and consider the transfer of technologies to other organizations for further development and demonstration, such as test and evaluation centers, or to industry as technology transition. In total, nearly one-fourth of reported transitions constituted this type of transition. For example, Navy’s SwampWorks and Experimentation, which has a mission to focus on high-risk technologies and concepts, stated that successful transition can be showing the feasibility of a high risk technology to allow further development of it at another level in the Navy’s S&T community.

The following are a few examples of transition program projects that provide context for the array of technologies and the users supported by these transitions.

Persistent Ground Surveillance System: A Joint Capability Technology Demonstration project that transitioned a family of wide area surveillance systems consisting of intelligence, surveillance, and reconnaissance technologies, as well as communications technologies to forward operating bases in Afghanistan. The systems were integrated into tailored tower and aerostat platforms that are in use and sustained by the Army.

Mosquito prototype: An Air Force Core Process 3 project that provided a field-tested prototype of a backpack-carried system for rapidly determining ground bearing strength for landing aircraft in remote sites. Following evaluations by users, the Air Force funded efforts to ruggedize and deliver additional units to the field for use.

Tactical Control System Digital Video Enhancements: A Navy Rapid Technology Transition project to upgrade the computer interface and video processing capability for the Vertical Takeoff and Landing Tactical Unmanned Aerial Vehicle (VTUAV) system’s ground control station. The VTUAV acquisition program has already purchased and fielded the complete system upgrade, and plans to acquire more of the systems.

Few Programs Track Project Outcomes and Benefits after Transition

As GAO has reported in the past, tracking technology transitions and the impact of those transitions, such as cost savings or deployment of the technology in a product, provides key feedback that can inform the management of programs. In particular, GAO found that leading commercial companies tracked technology transition not only to enable them to measure success, but also to assess their processes and determine what changes are necessary to improve transition rates. In addition, tracking outcomes provides the laboratories and management with feedback on the impact technology investments have on their companies. The transition programs we reviewed predominately stop tracking a project once the program is no longer funding it, or shortly thereafter, and generally have limited insight into benefits realized by users for their transition project portfolios. For example, most program officials we spoke to indicated they lack the ability to obtain feedback that would indicate a technology that transitioned to a weapon system acquisition program was ultimately integrated into the weapon system and deployed for use by the warfighter, or was not integrated into the system, but did support competition that led to enhanced capabilities or reduced cost for the weapon system.

A few programs did identify processes for tracking transitioned projects through the acquisition process and to deployment within a fielded weapon system. The Navy uses a Transition Review Board to monitor completed projects from the Future Naval Capabilities, Rapid Technology Transition, and Technology Insertion Program for Savings programs. The board determines and reports on whether transitioned projects are utilized in systems that support Navy warfighters. The Navy determined, for example, that of the 155 technology products the Future Naval Capabilities program delivered to acquisition programs between fiscal years 2006-2011, 21 percent were subsequently deployed to fleet forces, 35 percent were still with the acquisition programs, and 44 percent failed to deploy. For projects that do not successfully deploy, the board assesses whether there are other benefits achieved, such as technologies leveraged for follow-on S&T work. The board also identifies obstacles to transition, such as loss of interest by the user or inadequacy of funding. These findings, along with a detailed one-page summary for each project, are then used to inform the Navy’s annual review process. We found that by maintaining this level of tracking, the Navy is better aware of the benefits and obstacles associated with a substantial portion of their S&T portfolio, which may better inform decisions made by Navy leadership.

In addition to tracking from transition to deployment, another notable project outcome measure we observed among the transition programs— specifically within the ManTech program—is the amount of cost savings or cost avoidance for projects. ManTech is somewhat unique when compared to other transition programs because its mission is focused on reducing acquisition and supportability costs of DOD weapon systems and improving manufacturing and repair efficiency for weapon systems. The ability of the Navy, Army, and Air Force components of ManTech to provide actual as well as projected cost-reduction benefits coming from transition and implementation of their projects presents an opportunity to clearly demonstrate value to the warfighter and value of the program. A couple of examples for ManTech cost savings and cost avoidance measures are illustrated by the projects below.

Energy Storage Manufacturing Very High Power Batteries: Army project that reportedly has produced $20 million in cost avoidance benefit by developing an automated lithium-ion battery manufacturing line to include improvements that yield affordable, high-power battery packs.

Virginia Class Submarine Affordability Initiative: The Navy estimates that this effort has already yielded over $21 million in cost savings per hull through a series of projects transitioned to the acquisition program that provides manufacturing efficiencies.

Programs Facilitate Transition Through Sound Project Selection and Management Practices

As GAO has reported in the past, technology transition programs need to establish disciplined selection and management processes as well as tools to ensure that new technologies can be effectively transitioned to a military user. Programs must have processes in place that support the selection of projects that have realistic schedule and cost expectations, as well as sufficiently mature technologies. In addition, early project endorsement from intended users and other key stakeholders, including senior leadership and S&T developers, is critical. Once selected, projects require effective management to ensure technology risks are minimized, costs and schedules are maintained, and transition commitments are confirmed. To achieve this, stakeholders must engage in regular communication during project execution. Formal agreements and other project assessment measures also are important to ensure projects stay on track and stakeholders sustain their commitments to transition. In conducting this review, we found that OSD and military department technology transition programs make use of these practices to varying degrees. All of the programs we reviewed have established project selection processes that strive to select relevant and feasible technologies that have strong stakeholder support. However, management practices varied across programs, with some programs using more robust, formal tools, such as technology transition agreements, to guide project management and communicate project progress and expectations.

Clear Expectations for Cost, Schedule, and Technology Maturity Objectives Are Central to Project Selection and Management

Establishing a path to facilitate technology transition begins with each transition program’s ability to select relevant and feasible technologies to meet military needs. Programs must identify technologies that have the potential to provide new or enhanced capabilities, and can generally be matured or demonstrated in a timely manner with modest program investments. Identifying viable candidate technologies requires that programs clearly communicate what capabilities are needed and solicit proposals from the right audience.

Most of the transition programs we reviewed identify and set forth high- priority capability needs or topics for project solicitations based on input from some combination of the S&T and acquisition communities, operational users, and senior military leadership. Having individuals with different focuses involved in generating topics helps ensure that the right technology projects will be proposed and that projects will have interested users. A limited number of programs we reviewed do not set topics, but do require submitted project proposals to demonstrate a capability need. For example, Navy TechSolutions, which has a relatively small budget— about $9 million per year—offers an open invitation for warfighters in the field to propose projects based on identified capability needs. Regardless of solicitation approach, all transition programs we reviewed have established criteria for evaluating whether to pursue proposed projects. As we previously identified, each program has its own project cost and timeline constraints, as well as technology maturity criteria in place to support projection selection. Clear expectations for these factors can reduce the risk of a technology failing to transition because of cost, schedule, or technology maturity concerns.

Early Project Endorsements by Stakeholders Can Influence Transition

Securing comprehensive stakeholder endorsements for projects from their outset is another important influence on whether projects transition. Most of the transition programs we reviewed have processes or mechanisms they use to foster some type of commitment from prospective users to transition projects once completed. However, officials in some programs also reported that garnering senior leadership support for proposed projects is critical, as it ensures that projects are aligned with the priorities of department leadership as well as user needs. For example, the Air Force’s Flagship Capability Concept program emphasizes senior leadership involvement by requiring all projects to be approved by the Air Force Requirements Oversight Council. Approval is supported by recommendations from the Air Force S&T governance structure, which is composed of senior-level leadership from Air Force headquarters and major commands. According to an Air Force official, this governance structure serves to ensure S&T efforts are properly aligned with Air Force needs, thereby facilitating transition and deployment to the operational community.

The commitment of sponsor funding, or co-funding is another means used to solidify stakeholder endorsement for transition projects. The investment by prospective users serves as an endorsement of sorts, as it can create greater buy-in for transition since the intended users have a monetary stake in the project. Specifically, we found that several of the transition programs we reviewed require or encourage prospective users to help fund projects. For example, programs like OSD’s Foreign Comparative Testing and Rapid Reaction Fund and the Navy’s Rapid Technology Transition program do not require co-funding but do use it in some cases. The Joint Capability Technology Demonstration program requires that at least 50 percent of a project is co-funded, and program officials noted that their projects normally have multiple sponsors that provide funding.

Stakeholder Commitment and Communication Are Key to Effective Management of Transition Programs and their Projects

Establishing clear and consistent commitments and communication channels among stakeholders is fundamental to managing transition projects and achieving transition. Senior leadership plays a key role in initiating and reinforcing effective program management practices. At the organizational level, we found several examples, such as with the Army, where officials indicated that their top S&T leadership championed institutional changes to emphasize transition, including the introduction of the Technology Enabled Capability Demonstration and Technology Maturation Initiative programs. These programs are intended to produce an integrated Army S&T strategy that uses dedicated transition programs to more effectively respond to warfighter needs. At the program level, many program officials indicated that senior leadership engagement, particularly in providing oversight for projects through to transition, is essential to having an effective program. We found the Future Naval Capabilities program provides a good example of senior leadership positively affecting project management activities. Specifically, due to funding constraints in its fiscal year 2013 S&T budget, Navy senior leadership supported the termination of ongoing Future Naval Capabilities projects that were determined to be lower priorities so that new, higher priority projects could be pursued. Navy officials stated that this type of awareness and understanding at senior levels enables the Future Naval Capabilities program to make efficient decisions that are less likely to meet resistance and that support the highest priority projects being developed for transition opportunities.

Several transition programs also emphasized the relationship between “working-level” stakeholders—S&T developers and acquisition programs or warfighters in the field—when discussing the keys to technology transition. These stakeholders manage expectations throughout a project and ensure it will meet user needs. This reduces the risk of completed projects languishing because funding is not available or because user requirements have changed, or both. Some programs that we reviewed use integrated product teams, which may be composed of individuals representing the requirements, acquisition, operational, and S&T communities, among others, to facilitate continuous communication with stakeholders and ensure that transition planning is on track. In the case of the Navy, integrated product teams identify capability gaps, provide input on which S&T projects may address those gaps, assess project progress, make sure transition strategies remain valid, and confirm funding is aligned to support transition. According to Navy officials, the results of integrated product team efforts also support information sharing across senior- and working-level stakeholders to validate development status and transition planning activities.

Transition Agreements and Other Project Assessment Tools Are Used to Formalize Project Expectations and Help Sustain Commitments to Transition Technologies

We found a range of management tools being used by some transition programs that underlie support and communication among stakeholders. Technology transition agreements are used by 9 of the 20 transition programs we reviewed to manage projects. These are “good-faith” agreements between stakeholders that document the expectations for developing, demonstrating, delivering, and integrating technologies into systems, or using them as standalone products. Agreement specifics vary by program and can be tailored for each project, but typically outline technology and readiness metrics, such as cost, schedule, and performance parameters that labs must meet for transition to occur. As GAO has reported in prior work, these metrics help sustain a strong transition path by providing a formal way to track progress against requirements.agreements are used by stakeholders to make informed decisions on projects that are not meeting expectations. For example, if the labs cannot develop a technology within cost and schedule expectations, or cannot meet specified performance characteristics, the prospective user can terminate the agreement and pursue alternative technology options.

Also as GAO has reported, technology transition The Joint Capability Technology Demonstration program within OSD and the Navy’s Future Naval Capabilities program both require technology transition agreements for all projects and provide insights into the content and value these agreements can offer to stakeholders. The Joint Capability Technology Demonstration program agreements, at a minimum, must outline:

Operational need

Proposed technical solution

Transition target information

Transition requirements

Integration strategy

Business case

Risks

Costs and schedule

Project points of contact We found the Future Naval Capabilities program uses technology transition agreements as management tools to increase the level of documented commitment as a project progresses over time. To accomplish this, the program has three levels for agreements that reflect the requisite knowledge available at different phases of a project. Key elements of an initial agreement include a basic project description, identification of initial exit criteria, a high-level integration strategy, and a likely transition funding source. As a project progresses, the other two levels of agreement require increasing commitment and specificity of requirements from stakeholders to develop, deliver, and integrate a Future Naval Capabilities project into an acquisition program or other form of deployment. Key elements of the second and third tier agreements involve refining and finalizing project descriptions, detailing exit criteria, providing greater specificity about the integration strategy, and providing estimates for transition costs and eventually executing transition funding. Stakeholders review the agreements annually to revalidate the commitments laid out within the document.

We also found Transition Commitment Level (TCL) assessment tools— used by the Joint Capability Technology Demonstration and Future Naval Capabilities programs—offer another means of validating that transition programs are investing in projects that have a firm transition commitment from prospective users. These tools provide scorecards that chart how well-defined the fundamental characteristics that support a strong commitment to transition projects are at a given point in time. The Future Naval Capabilities program uses a single TCL tool that documents level of transition commitment from project start to completion. The Joint Capability Technology Demonstration program uses two TCL assessment tools—one to identify transition commitment to support project selection and one to track commitment through project execution and to completion. Figure 5 shows the TCL assessment tool used for Joint Capability Technology Demonstration project execution and completion.

As shown by the figure, a TCL tool provides a formal way for determining which proposed and active projects have a high, medium, and low probability to transition. For example, in order for a Joint Capability Technology Demonstration project to be designated as “green,” project managers must show the project has a clear path to one of the three transition recipients as defined by the program. The TCL tools used by both the Joint Capability Technology Demonstration and Future Naval Capabilities programs provide insights into the progress of projects based on established benchmarks. In addition, Navy officials stated that the TCL tool used by Future Naval Capabilities provides information that can be used by program managers to inform investment decisions. In particular, the TCL tool can help identify active projects that may require greater attention to solidify transition commitments from stakeholders, and can also be used to support decisions to end projects that no longer have stakeholder support.

Eleven of the programs we reviewed do not use formal technology transition agreements, but incorporate some of the elements of agreements in managing projects. For example, OSD’s Foreign Comparative Testing program requires documentation of a valid need for a technology when a project starts and a letter of endorsement from a prospective military user that supports procuring the technology at completion. The Rapid Reaction Fund program does not require commitments from users, but does require them to identify a potential transition path at project start and then the program will follow-up with the users to assess their interest in the technology if it is successfully developed. Although prior GAO work found that technology transition agreements do not guarantee transition success because requirements and funding changes can occur, the agreements are useful tools to solidify expectations and secure user commitment. At the time, GAO recommended that DOD expand the use of technology transition agreements. DOD concurred with the recommendation, but did not identify specific actions it planned to take to do so.

Conclusions

For the most part, technology transition programs we reviewed have been reasonably successful in delivering technologies to military users. However, a limited number of programs do not have metrics established to track completed projects. In addition, few programs measure whether transitioned technologies actually result in a benefit to users. For example, many programs track whether a technology is delivered to an acquisition program of record, but often have limited or no insight into whether the technology resulted in enhanced performance or new capabilities, cost savings, or reduced times for testing and evaluation and system integration. Without some measure of technology implementation or other benefit after transition, questions remain as to whether programs are providing the right technologies at the right time to users, are using effective approaches to select, develop, and transition technologies, and are providing tangible benefits.

The transition programs we reviewed generally have disciplined project selection and management processes in place to facilitate the transition of technologies to intended users. To varying degrees, programs emphasize the need for early and sustained commitments from senior military leadership, S&T developers, and military users in order to ensure projects are needed, have a sound basis, and risks are reduced. A key mechanism some programs use for obtaining these commitments are formal tools, such as technology transition agreements and technology commitment level assessments. These tools help to clarify expectations, hold stakeholders accountable for what they must deliver, and gauge progress towards achieving project objectives. There may be opportunities for more widespread use of these tools among the programs we reviewed, which could help strengthen technology transition success.

Recommendations for Executive Action

To improve visibility and management of the department’s efforts to transition technologies to support the needs of the warfighter, we recommend that the Secretary of Defense take the following two actions:

Require that all technology transition programs track and measure project outcomes, to include not only whether technologies transitioned to an intended user but also the longer-term impact of whether the technologies benefitted acquisition programs or military users in the field.

Assess transition programs to identify opportunities for more widespread use of existing transition management tools, such as technology transition agreements and technology commitment level evaluation mechanisms.

Agency Comments and Our Evaluation

We provided a copy of a draft of this report to DOD for review and comment. In written comments on the draft, DOD partially concurred with our first recommendation and fully concurred with the second recommendation. DOD’s comments appear in appendix IV.

In responding to the first recommendation, DOD agreed that all transition programs should track and measure project outcomes, including how the technology or capability is being employed by the intended user. However, the department raised concerns that tracking and measuring technology project outcomes would be a labor-intensive and time- consuming process requiring significant investment. DOD also suggested that the recommendation be revised to have the department assess technology transition programs for opportunities to implement processes to track and measure project outcomes. We continue to believe that it is important that DOD have a formal means to track and document technology transition in order to demonstrate not only whether technology investments under these programs are reaching their intended users, but also that they are providing the desired benefits. We believe that DOD should require all programs to track and measure technology transition, and that the department’s suggested revision to our recommendation could potentially remove any responsibility for programs to do so. Moreover, while we acknowledge that tracking and measuring project benefits following transition would require additional time and effort, we believe our recommendation provides DOD with significant flexibility to measure outcomes in different ways that would be less resource- intensive than DOD envisions. For example, rather than tracking all transitioned projects, the department could assess a subset of projects or establish mechanisms for military users to report back on the results. In addition, as we reported, the Navy has recently used an approach that could be adopted by other programs in DOD to measure potential benefits from transitioned projects. Specifically, the Office of Naval Research has employed a Technology Review Board to complete independent reviews of Navy transition programs and report whether transitioned projects were being used by the warfighter. Consequently, we have not revised our recommendation as suggested by DOD. In concurring with our second recommendation, the department agreed that all technology transition programs should identify opportunities to use existing transition management tools. DOD stated that it will ensure guidelines are published and widely disseminated to improve the understanding as to when, how, and under what circumstances transition management tools can be employed.

We are sending copies of this report to appropriate congressional committees, the Secretary of Defense, and the Secretaries of the Air Force, Army, and Navy. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or by email at sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V.

Appendix I: Scope and Methodology

To identify what Department of Defense (DOD) programs exist that are dedicated to facilitating technology transition from the science and technology (S&T) community to a user, we reviewed DOD reports and documents. As a starting point, we reviewed a March 2012 report to Congress by the Under Secretary of Defense for Acquisition, Technology and Logistics that assessed technology transition programs within DOD. In conducting this assessment, DOD reviewed budget program elements to identify potential technology transition programs and then surveyed these programs to validate those whose primary function is technology transition. We discussed the programs identified in the DOD report with officials from the Office of the Secretary of Defense (OSD) and the military departments and made adjustments to our scope based on their inputs and our own analysis. The resulting list of programs, which is the focus of our review, includes 20 transition programs across OSD and the military departments—Air Force, Army, and Navy. While there are other technology transition activities that occur within the military departments and other DOD organizations, such as those within the Joint Improvised Explosive Device Defeat Organization and Defense Advanced Research Projects Agency, we limited our scope to programs within DOD dedicated to technology transition.

We then interviewed officials from the programs, as well as other officials within OSD and the military departments affiliated with technology transition activities, to become familiar with their respective technology transition programs. We collected and analyzed information from the programs on their structures, processes, tools, and technology transition metrics used. To assess what technology transition programs are doing to meet their objectives, we reviewed program policies and procedures for selecting technology transition projects, monitoring and managing their progress, and determining transition outcomes. For the purpose of assessing the programs’ activities—including projects started, completed, and transitioned to users—we gathered data from the programs for fiscal years 2010 through 2012. For the purpose of reviewing and reporting historical transition rates for the programs, we collected self-reported transition rates from program officials for those programs that could provide such information. We did not validate the data provided by the technology transition program officials, but reviewed and discussed the data with them and determined it was sufficiently reliable for the purposes of this report. To identify what practices may facilitate technology transition, we reviewed prior GAO studies on DOD technology transition and best practices for transition, interviewed officials from OSD and the military departments, and reviewed programmatic information and other related DOD documentation. Using this information, we were able to isolate a number of practices that influence a program’s ability to transition technology from the S&T community to a prospective user. Additionally, we reviewed what practices were present in each program. In performing our work, we obtained information and interviewed OSD and military department officials from: Assistant Secretary of Defense, Research and Engineering, Arlington, Virginia; Defense Advanced Research Projects Agency, Arlington, Virginia Office of Science and Technology, Arlington, Virginia; Rapid Equipping Force, Ft. Belvoir, Virginia; Small Business Innovation Research and Manufacturing Technology programs, Aberdeen Proving Ground, Maryland Air Force Research Laboratory, Wright-Patterson Air Force Base, Ohio; Air Force Acquisition and Research, Arlington, Virginia; Office of Naval Research, Arlington, Virginia; Navy Research and Development, U.S. Naval Yard, Washington, D.C.; Deputy Assistant Secretary of the Navy for Research, Development, Test, and Evaluation, Arlington, Virginia We conducted this performance audit from April 2012 to March 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: DOD Research, Development, Test, and Evaluation Budget Activities

Science and technology funding

DOD RDT&E Budget Activity Description Basic Research (6.1)

Includes all scientific study and experimentation directed toward increasing fundamental knowledge and understanding in physical, engineering, environmental, and life sciences related to long-term national security needs. It is farsighted high payoff research that provides the basis for technological progress. Program elements in this category involve pre-Milestone A efforts, which is the point where entry into the Technology Development phase is approved.

Varies from systematic mission-directed research beyond that done through basic research, to sophisticated breadboard hardware, study, programming and planning efforts that establish the initial feasibility and practicality of proposed solutions to technological challenges. The dominant characteristic is it is directed toward general military needs with a view toward developing and evaluating the feasibility and practicality of proposed solutions and determining their parameters. Program elements in this category involve pre-Milestone B efforts, also known as Technology Development phase tasks.

Includes concept and technology demonstrations of components and subsystems or system models. The results of this type of effort are proof of technological feasibility and assessment of subsystem and component operability and producibility rather than the development of hardware for service use. Advanced Technology Development demonstrates the general military utility or cost reduction potential of technology when applied to different types of military equipment or techniques. Program elements in this category involve pre-Milestone B efforts, such as system concept demonstration, and generally have technology readiness levels of 4, 5, or 6. Projects in this category should have the goal of moving out of S&T and into the acquisition process within the future years defense program. Upon successful completion of projects that have military utility, the technology should be available for transition.

Acquisition- based Funding

Advanced Component Development & Prototypes (6.4)

Efforts necessary to evaluate integrated technologies, representative modes or prototype systems in a high fidelity and realistic operating environment. Includes system specific efforts that help expedite technology transition from the laboratory to operational use. Emphasis is on proving component and subsystem maturity prior to integration in major and complex systems and may involve risk reduction initiatives. Program elements in this category involve efforts prior to Milestone B acquisition program start. Completion of technology readiness levels 6 and 7 should be achieved for major programs. Program control is exercised at the program and project level.

Post-Milestone B, conducting engineering and manufacturing development tasks aimed at meeting validated requirements prior to full-rate production. Prototype performance is near or at planned operational system levels. Characteristics involve mature system development, integration and demonstration to support Milestone C production decisions, and conducting live fire test and evaluation and initial operational test and evaluation of production representative articles.

RDT&E Management Support (6.6)

Efforts to sustain and/or modernize the installations or operations required for general RDT&E. Includes test ranges, military construction, maintenance support of laboratories, operation and maintenance of test aircraft and ships, and studies and analyses in support of the RDT&E program. Military construction costs directly related to major development programs are included.

Efforts to upgrade systems that have been fielded or have received approval for full- rate production and anticipate production funding in the current or subsequent fiscal year. Programs in this category involve systems that have received Milestone C production decision approval.

Appendix III: DOD Technology Readiness Levels

Appendix III: DOD Technology Readiness Levels Technology readiness level (TRL) 1. Basic principles observed and reported.

Description Lowest level of technology readiness. Scientific research begins to be translated into applied research and development. Examples might include paper studies of a technology’s basic properties.

2. Technology concept and/or application formulated.

Invention begins. Once basic principles are observed, practical applications can be invented. Applications are speculative, and there may be no proof or detailed analysis to support the assumptions. Examples are limited to analytic studies.

3. Analytical and experimental critical function and/or characteristic proof of concept.

Active research and development is initiated. This includes analytical studies and laboratory studies to physically validate the analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative.

4. Component and/or breadboard validation in laboratory environment.

Basic technological components are integrated to establish that they will work together. This is relatively low fidelity compared with the eventual system. Examples include integration of ad hoc hardware in the laboratory.

5. Component and/or breadboard validation in relevant environment.

Fidelity of breadboard technology increases significantly. The basic technological components are integrated with reasonably realistic supporting elements so they can be tested in a simulated environment. Examples include high-fidelity laboratory integration of components.

6. System/subsystem model or prototype demonstration in a relevant environment.

Representative model or prototype system, which is well beyond that of TRL 5, is tested in a relevant environment. Represents a major step up in a technology’s demonstrated readiness. Examples include testing a prototype in a high- fidelity laboratory environment or in a simulated operational environment.

7. System prototype demonstration in an operational environment.

Prototype near or at planned operational system. Represents a major step up from TRL 6 by requiring demonstration of an actual system prototype in an operational environment (e.g., in an aircraft, in a vehicle, or in space).

8. Actual system completed and qualified through test and demonstration.

Technology has been proven to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications.

9. Actual system proven through successful mission operations.

Actual application of the technology in its final form and under mission conditions, such as those encountered in operational test and evaluation. Examples include using the system under operational mission conditions.

Appendix IV: Comments from the Department of Defense

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, John Oppenheim, Assistant Director; Danielle Greene; John Krump; Sean Merrill; Robert Miller; Carol Petersen; and Scott Purdy also made key contributions to the report.