Defense Technology Development: Management Process Can Be Strengthened for New Technology Transition Programs
Published: 2005-06-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) and Congress both recognize that Defense technology innovations sometimes move too slowly from the lab to the field. Three new programs have been recently created in DOD to help speed and enhance the transition of new technologies. A report accompanying the fiscal year 2003 National Defense Authorization Act required GAO to review two of these programs--the Technology Transition Initiative (TTI) and Defense Acquisition Challenge Program (DACP). The first is designed to speed transition of technologies from DOD labs to acquisition programs and the second is designed to introduce cost-saving technologies from inside and outside DOD. We were also asked to review the Quick Reaction Fund, which is focused on rapidly field testing promising new technology prototypes. We assessed the impact the programs had on technology transition and the programs' selection, management and oversight, and assessment practices.
section_title: What GAO Found
paragraphs: The ability to spur and leverage technological advances is vital to sustaining DOD's ability to maintain its superiority over others and to improve and even transform how military operations are conducted. The three new transition programs we reviewed are all appropriately targeted on what has been a critical problem in this regard--quickly moving promising technologies from the laboratory and commercial environment into actual use. Moreover, by tailoring processes and criteria to focus on different objectives, whether that may be saving time or money or broadening the industrial base, DOD has had an opportunity to experiment with a variety of management approaches and criteria that can be used to help solve transition problems affecting the approximately $69 billion spent over the past 3 years on later stages of technology development. However, it is too soon for us to determine the impact the three new DOD technology transition programs are having. At the time of our review, the programs--the TTI, DACP, and Quick Reaction Fund--had completed only 11 of 68 projects funded in fiscal years 2003 and 2004; of those, only 4 were providing full capability to users. Additionally, the programs have limited measures to gauge success of individual projects and return on investment. Nonetheless, reports from the programs have pointed to an array of benefits, including quicker fielding of technological improvements, cost savings, and the opportunity for DOD to tap into innovative technologies from firms that are new to defense work. Some sponsored technologies are bringing benefits to warfighters, such as a small, unmanned aircraft that can detect chemical and biological agents, and a device the size of an ink pen that can be used to purify water on the battlefield or in disaster areas. Furthermore, DOD officials credit the programs with giving senior leaders the flexibility to rapidly address current warfighter needs and for highlighting smaller technology projects that might otherwise be ignored. Long-term success for the programs likely will depend on how well the programs are managed and overseen. The programs must have effective processes for selecting the best projects, and management and oversight processes that will catch potential problems early. Thus far, of the three programs, the DACP has adopted the most disciplined and structured process for selecting and managing projects, and has encountered few problems managing projects. However, the program has had some difficulties processing the large number of proposals it receives. The TTI has also established selection criteria and processes, but it is unclear the extent to which it is reaching its intended audience and has had less success in tracking its projects. The Quick Reaction Fund has the least structured processes of the three programs--a deliberate approach seen as providing the flexibility needed to field innovations rapidly. It has had some difficulty selecting, managing and tracking projects.

section_title: Letter
paragraphs: Over the past 3 years, the Department of Defense (DOD) has invested  almost $69 billion on a wide range of advanced technology development  projects—from detecting and neutralizing deeply buried facilities that  protect weapons of mass destruction, to miniaturizing power and energy  technologies to reduce the weight of gear soldiers carry into battle, to  improving access to space. As we have previously reported, the majority of  these dollars are spent within large weapons programs that have taken too  long to get to the warfighter, in large part because these programs often  attempt to incorporate technology advances that have not been proven.   Invariably, this practice has resulted not only in large cost and schedule  increases for large programs, but less available funding and commitment  for small-scale development.
Both DOD and Congress recognize these as well as other problems in  transitioning technology and have initiated a number of programs over the  past decade aimed at spurring and fielding smaller-scale technology  advances. Recently, the fiscal year 2003 National Defense Authorization  Act required DOD to establish two new programs, the Technology  Transition Initiative (TTI) and the Defense Acquisition Challenge Program  (DACP)—the first designed to speed transition of technologies from  DOD’s science and technology (S&T) programs to acquisition programs  and the second designed to introduce innovative and cost-saving  technologies from inside and outside DOD. In addition, DOD established  the Quick Reaction Fund to rapidly field-test promising new technology  prototypes. All three programs target relatively small-scale projects and  their total combined annual budget is about $64 million in fiscal year  2005—a very small portion of the overall dollars dedicated to technology  research and development. We were required by the conference report  accompanying the act to assess the implementation of the first two  programs and were later asked by your offices to also review the Quick  Reaction Fund. As discussed with your offices, our specific objectives  were to assess (1) the impact the programs have had on technology  transition and (2) practices that have helped or hindered the selection,  management and oversight, and assessment of the programs.
In conducting our work, we interviewed the managers of the three  programs we reviewed and analyzed documents that detailed how the  programs have been implemented, expected project benefits, and the  types of performance metrics they used to gauge individual project and  overall program success. We met with several high-level DOD officials  who were appointed to a council to oversee the Technology Transition  Initiative, and we observed meetings of a working group in charge of  supporting the same initiative. In addition, we judgmentally selected 24 of  68 projects funded by the three transition programs for more in-depth  review, targeting those that had already been completed or were in the  completion stages as well as assuring we had projects representing all of  the military services and key Defense agencies. We conducted interviews  with the managers of these 24 projects to discuss implementation,  management, and oversight activities. We analyzed pertinent project  documents, including original proposals and quarterly status reports. We  also analyzed documents relating to the transition programs as well as  broader transition issues and conducted interviews with officials in DOD,  the Army, the Navy, the Air Force, the Marine Corps, the Defense  Advanced Research Projects Agency, the Missile Defense Agency, the  Defense Threat Reduction Agency, Defense Information Systems Agency,  Special Operations Command, Joint Chiefs of Staff, and military  department research laboratories. We relied on our previous best  practices work, other transition program guidance, and general  management practices as criteria when assessing the program offices’  efforts to select, manage and oversee, and assess projects. We also  analyzed DOD budget documents to identify program funding, as well as  applied research and advanced technology development funding for fiscal  years 2003 to 2009 and confirmed with DOD officials responsible for  maintaining this information that our analysis was correct. Our review was  conducted from June 2004 to April 2005 in accordance with generally  accepted government auditing standards.
subsections: section_title: Background
paragraphs: DOD relies on its research laboratories and test facilities as well as  industry and academia to develop new technologies and systems that  improve and enhance military operations and ensure technological  superiority over adversaries. Yet, historically, DOD has experienced  problems in bringing technologies out of the lab environment and into real  use. At times, technologies do not leave the lab because their potential has  not been adequately demonstrated or recognized. In other cases,  acquisition programs—which receive the bulk of DOD’s funding in  research, development, testing and evaluation of technology—are simply  unwilling to fund final stages of development of a promising technology,  preferring to invest in other aspects of the program that are viewed as  more vital to success. Other times, they choose to develop the  technologies themselves, rather than rely on DOD labs to do so—a  practice that brings cost and schedule risk since programs may well find  themselves addressing problems related to technology immaturity that  hamper other aspects of the acquisition process. And often, DOD’s  budgeting process, which requires investments to be targeted at least 2  years in advance of their activation, makes it difficult for DOD to seize  opportunities to introduce technological advances into acquisition  programs. In addition, it is challenging just to identify and pursue  technologies that could be used to enhance military operations given the  very wide range of organizations inside and outside of DOD that are  focused on technology development and the wide range of capabilities  that DOD is interested in advancing.
In recognizing this array of challenges, DOD and Congress have  established a number of “technology transition” programs, each with a  particular focus. (See table 1.) The Advanced Concept Technology  Demonstration (ACTD) program, for example, was initiated by DOD in  1994 as a way to get technologies that meet critical military needs into the  hands of users faster and at less cost than the traditional acquisition  process. Under this program, military operators test prototypes that have  already been developed and matured in realistic settings. If they find the  items to have military utility, DOD may choose to buy additional quantities  or just use the items remaining after the demonstration. In 1980, DOD  established the Foreign Comparative Testing (FCT) Program to identify,  evaluate, and procure technologies that have already been developed and  tested in other countries—saving DOD the costly burden of maturing the  technology itself. Other programs include those that seek to quickly  identify and solve production problems associated with technology  transition (the Manufacturing Technology Program—MANTECH) and to  partner with the commercial sector in completing projects that are useful  to both military and industry (the Dual Use Science and Technology  program). Even taken together, however, these programs represent a very  small portion of DOD dollars spent on applied research and advanced  technology development—about $9 billion annually—and considerably  less of total money spent on the later stages of technology development,  which includes an additional $60 billion spent on advanced component  development and prototypes, largely within weapons acquisition  programs. As such, they cannot single-handedly overcome transition  problems, but rather demonstrate various ways to ease transition and  broaden participation from the industrial base.
Three of the more recent initiatives include the TTI and DACP, both  established by Congress in fiscal year 2003, and the Quick Reaction Fund,  established by DOD the same year. TTI is focused on speeding the  transition of technologies developed by DOD’s S&T programs into  acquisition programs, while DACP is focused on introducing innovative  and cost-saving technologies developed inside and outside DOD. The  Quick Reaction Fund is focused on field testing technology prototypes.
All three programs are managed by DOD’s Office of Defense Research and  Engineering, which reports to the Under Secretary of Defense for  Acquisition, Technology and Logistics.
Together, these three programs received about $64 million in fiscal year  2005–a fraction of the $9.2 billion DOD invested in applied research and  advanced technology development the same year and a relatively small  budget compared to some of the other transition programs. Nevertheless,  DOD has been increasing its investment in these programs and plans to  further increase it over the next few years. (See figure 1.)
Table 2 highlights similarities and differences between DACP, TTI, and  Quick Reaction Fund. Table 3 provides examples of projects that have  already been funded.
section_title: Technology Transition Programs Offering Benefits, but It Is too Early to Determine Their Impact
paragraphs: The three transition programs, which are being implemented consistent  with congressional intent, reported that benefits can already be seen in  many projects, including improvements to performance, affordability,  manufacturability, and operational capability for the warfighter. While  such benefits may have eventually been achieved through normal  processes, program officials believe the three transition programs enabled  DOD to realize them sooner due to the immediate funding that was  provided to complete testing and evaluation as well as attention received  from senior managers. DOD officials also emphasized that these programs  are calling attention to emerging technologies that have the potential to  offer important performance gains and cost savings but, due to their size  and relative obscurity, may otherwise be overlooked when competing  against other, larger-scaled technologies and/or technologies already  deemed as vital to a particular acquisition program’s success. Another  benefit cited with the DACP is an expansion of the Defense industrial  base, because the program invites participation from companies and  individuals that have not been traditional business partners with DOD.  Nevertheless, it is too early for us to determine the impact that these  programs have had on technology transition. At the time we selected  projects to review, few projects had been completed. In addition, the  programs had limited performance measures to gauge success of  individual projects or track return on investment over time.
The following examples highlight some of the reported benefits of  individual projects.
Host Weapons Shock Profile Database—DOD spends a significant  amount of time and resources to test new accessories (e.g., night vision  scopes) for Special Operations Forces weapons. Currently, when new  accessories are added, they must go through live fire testing to  determine if they work properly and will meet reliability standards.  This process could take several months to complete as the acquisition  office must schedule time at a test range to complete the testing.  Program officials must also identify and pay for an expert to conduct  the testing and pay for ammunition that will be used in the test. The  DACP is funding the test and evaluation of a database that will simulate  the vibration or shock of various machine guns in order to test new  accessories for that gun. This will eliminate almost all of the testing  costs mentioned above and greatly reduce the amount of time needed  for testing. The project office estimates that it will save almost $780,000  per year in ammunition costs alone.
Enhanced Optics for the Rolling Airframe Missile—The Rolling  Airframe Missile is part of the Navy’s ship self-defense system to  counter attacks from missiles and aircraft.  However, the missile  experiences operational deficiencies in certain weather conditions, and  the program has had problems producing components for the optics.   The DACP is providing funding to a small business to test and evaluate  a new sapphire dome and optics for the missile to resolve these  problems.  Program officials estimate that program funding will  accelerate the development of a solution 1 to 2 years earlier than  anticipated.  If the DACP project is successful, an added benefit will be  that the dome material will be readily available from manufacturers in  the United States instead of a single overseas supplier, as is currently  the case.
Water Purification System— For tactical situations in which deployed  troops do not have quick and easy access to potable water, the pen will  allow soldiers to treat up to 300 liters of any available, non-brackish  water source on one set of lithium camera batteries and common table  salt. The pen eliminates the risk of the soldiers’ exposure to diseases  and bio-chemical pollutants. TTI funding was used to purchase  approximately 6,600 water pens for distribution to the military services.   In addition, TTI funding enabled this item to be placed on a General  Services Administration schedule, where approximately 8,600  additional water pens have been purchased by DOD customers.  DOD  and the company that produces the pen donated hundreds of these  systems to the tsunami relief effort in Southeast Asia.
Dragon Eye—The Dragon Eye is a small, unmanned aerial vehicle with  video surveillance capabilities used by the marines. To address the  concerns over a chemical and biological threat to troops in Iraq, the  Quick Reaction Fund funded the integration of a small chemical  detection and biological collection device on the Dragon Eye. The low- flying Dragon Eye can tell troops in real time where and when it is  collecting samples, and in cases where a plume is detected, it can  determine the direction the plume is moving. According to program  officials, Quick Reaction funding allowed the chemical and biological  detection capability to be developed 2 years ahead of schedule. The  technology was available to a limited number of Special Operations  Forces at the beginning of the Iraqi conflict.
Despite the evident benefits of certain projects, it is too early to determine  the programs’ impact on technology transition.  At the time we selected  projects for review, only 11 of 68 projects started in fiscal years 2003 and  2004 had been completed, and, of those, only 4 were currently available to  warfighters.  These include one TTI project—a miniaturized water  purification system that is now being offered through a General Services  Administration schedule to the warfighter—and three projects under the  Quick Reaction Fund, including the Dragon Eye chemical and biological  sensor, planning software used by Combatant Commanders dealing with  weapons of mass destruction targets, and special materials that strengthen  unmanned aerial vehicles.
Since the time we selected projects, 20 have been reported as completed  and 13 have been reported as available to warfighters.  The latest project  completion information by program is shown in Table 4.
It is important to note that, even though 20 TTI and Quick Reaction Fund  projects are considered to be complete, not all of the capabilities have  reached the warfighter.  For example:    The T58 Titanium Nitride Erosion Protection is a TTI project that has  transitioned to an acquisition program but has not yet reached the  warfighter.  The project is being developed to improve the reliability of  T-58-16A helicopter engines used in Iraq. While the compressor blades  are designed for 3000 operating hours, the Marine Corps has had to  remove engines with fewer than 150 operational hours due to sand  ingestion. The project received funding from the TTI in fiscal years  2003 and 2004 to develop a titanium nitride coating for engine blades  that would significantly mitigate erosion problems in a desert  environment. According to program documents, blades with the new  coating will be included in future production lots beginning in July  2005. Modification kits will also be developed for retrofitting engines  already produced. Program officials expect the project will double the  compressor life of the engine in a sand environment and save about $12  million in life-cycle costs through fiscal year 2012.
The Ping project, funded by the Quick Reaction Fund, is an example of  a project that is considered complete, but a prototype was never field  tested by the warfighter.  The Air Force had hoped to broaden the  capability of the microwave technology it used to identify large objects  such as tanks or cars to also detect concealed weapons or explosives— such as suicide vests. However, the project was cancelled after some  initial testing revealed that the technology was not accurate enough to  determine the microwave signatures of small arms or suicide vests that  could have numerous configurations and materials. DOD officials  stated that, even though the project was unsuccessful, they gained a  better understanding of microwave technologies and are continuing to  develop these technologies for other applications.
The long-term impact of the programs will also be difficult to determine  because the technology transition programs have a limited set of metrics  to gauge project success or the impact of program funding over time.  While each funded project had to identify potential impact in terms of  dollar savings, performance improvements, or acceleration to the field as  part of the proposal process, actual impact of specific projects as well as  the transition programs as a whole is not being tracked consistently. The  value of having performance measures as well as DOD’s progress in  adopting them for these transition programs is discussed in the next  section of this report.
section_title: Selection, Management and Oversight, and Assessment Processes Could Be Improved by Adopting Additional Practices
paragraphs: To ensure that new technologies can be effectively transitioned and  integrated into acquisitions, transition programs need to establish effective  selection, management and oversight, and assessment processes. For  example, programs must assure that proposals being accepted have  established a sound business case, that is, technologies being transitioned  are fairly mature and in demand and schedules and cost for transition fit  within the program’s criteria. Once projects are selected, there needs to be  continual and effective communication between labs and acquisition  programs so that commitment can be sustained even when problems arise.  To assure that the return on investment is being maximized, the impact of  programs must be tracked, including cost and time savings as well as  performance enhancements. Our work over the past 7 years has found that  high-performing organizations adopt these basic practices as a means for  successfully transitioning technologies into acquisitions. Moreover, several  larger DOD technology transition programs, such as the ACTD program  and some Defense Advanced Research Projects Agency (DARPA) projects,  embrace similar practices and have already developed tools to help  sustain commitment, such as memorandums of agreement between  technology developers and acquirers.  Both DARPA and ACTD manage  budgets that are considerably larger than the programs included in this  review.  As such, the level of detail and rigor associated with their  management processes may not be appropriate for TTI, DACP, or Quick  Reaction Fund.  However, the concepts and basic ingredients of their  criteria and guidance could serve as a useful starting point for the smaller  programs to strengthen their own processes.
The three programs we reviewed adopted these practices to varying  degrees. Overall, the DACP had disciplined and well-defined processes for  selecting and managing, and overseeing projects. The TTI had disciplined  and well-defined processes for selecting projects, but less formal  processes for management and oversight. The Quick Reaction Fund was  the least formal and disciplined of all three, believing that success was  being achieved through flexibility and a high degree of senior management  attention. All three programs had limited performance measures to gauge  progress and return on investment.  Generally, we found that the more the  programs adopted structured and disciplined management processes, the  fewer problems they encountered with individual efforts.
subsections: section_title: Selection
paragraphs: Success in transitioning technologies from a lab to the field or an  acquisition program hinges on a transition program’s ability to choose the  most promising technology projects. This includes technologies that can  substantially enhance an existing or new system either through better  performance or cost savings and those with technologies at a fairly mature  stage, in other words, suitable for final stages of testing and evaluation. A  program can only do this, however, if it is able to clearly communicate its  purpose and reach the right audience to submit proposals in the first  place. It is also essential that a program have a systematic process for  determining the relative technical maturity of the project as well as for  evaluating other aspects of the project, such as its potential to benefit  specific acquisition programs. Involving individuals in the selection  process from various functions within an organization—e.g., technical,  business, and acquisition—further helps to assure that the right projects  are being chosen and that they will have interested customers. An  analytical tool that can be particularly useful in selecting projects is a  technology readiness level (TRL) assessment, which assesses the maturity  level of a technology ranging from paper studies (level 1), to prototypes  that can be tested in a realistic environment (level 7), to an actual system  that has proven itself in mission operations (level 9). Our prior work has  found TRLs to be a valuable decision-making tool because it can presage  the likely consequences of incorporating a technology at a given level of  maturity into a product development.
As further detailed in table 5, the DACP program has a fairly robust  selection process. The program relies on internet-based tools to  communicate its goals and announce its selection process and ensure a  broad audience is targeted. As a result, it receives a wide array of  proposals from which the program office assesses their potential for  generating improvements to existing programs as well as actual interest  from the acquisition community. The DACP also solicits technical experts  from inside and outside DOD to assess potential benefits and risks. Once  the number of projects is whittled down, the program takes extra steps to  secure commitments from acquisition program managers as well as  program executive officers. The program’s popularity, however, has had  some drawbacks. For example, the sheer number of proposals have  tended to overwhelm DACP staff and slowed down the selection process,  particularly in the first year. In addition, while technology benefits and  risks are assessed in making selection decisions, DACP does not formally  confirm the technology readiness levels being reported. The TTI program  also has a fairly rigorous selection process, with specific criteria for  selection, including technology readiness, and a team of representatives of  higher-level DOD S&T officials in charge of disseminating information  about the program in their organization, assessing their organization’s  proposals based on TTI criteria as well as other criteria they developed,  and ranking their top proposals. The program, which is focused on  reaching DOD’s S&T community rather than outside industry, had been  communicating in a relatively informal manner and it was unclear during  our review the extent to which the TTI was reaching its intended audience.   The program, however, has been taking steps to strengthen its ability to  reach out to the S&T community.  In addition, TTI does not confirm TRLs.  At the time of our review, the Quick Reaction Program selection process  was much less structured and disciplined than DACP and TTI. This was by  design, because the program wants to select projects quickly and get them  out to the field where they can be of use in military operations in Iraq,  Afghanistan, and elsewhere. However, the program experienced problems  related to selection and as a result—for example, significant gaps in  knowledge about technology readiness led to the cancellation of one  project. To program officials, the risk associated with less formal selection  is worth the benefit of being able to move rapidly evolving technologies  into an environment where they can begin to immediately enhance  military operations and potentially save lives.  Nevertheless, the program  is now taking steps to strengthen selection processes.
Selecting promising projects for funding is not enough to ensure  successful transition. Program managers must also actively oversee  implementation to make sure that project goals are being met and the  program is working as intended and to identify potential barriers to  transition. They must also sustain commitment from acquirers. Moreover,  the transition program as a whole must have good visibility over progress  and be positioned to shift attention and resources to problems as they  arise.
A tool that has proven particularly useful for other established DOD  technology transition programs is designating individuals, preferably with  experience in acquisitions or operations and/or the S&T world, as “deal  brokers” or agents to facilitate communication between the lab and the  acquisition program and to resolve problems as they arise. DARPA  employs such individuals, for example, as well as some Navy-specific  transition programs. Both have found that these agents have been integral  to transition success. Another tool that is useful for sustaining  commitment from the acquirers is a formal agreement. Our previous work  found that best practice companies develop agreements with cost and  schedule targets to achieve and sustain buy-in and that the agreements are  modified as a project progresses to reflect more specific terms for  accepting or rejecting a technology. DARPA develops similar agreements  that describe how projects will be executed and funded as well as how  projects will be terminated if the need arises. The agreements are signed  by high-level officials, including the director of DARPA and senior-level  representatives of the organizations DARPA is working with. The ACTD  program develops “implementation directives” that clarify roles and  responsibilities of parties executing an ACTD, time frames, funding, and  the operational parameters by which military effectiveness is to be  evaluated. The agreements are also signed by high-level officials.
DACP has fairly robust management and oversight mechanisms. Status is  monitored via formal quarterly reporting as well as interim meetings  which, at a minimum, involve the customer, the developer, and the DACP  project manager. The meetings provide an opportunity to ensure the  acquisition program is still committed to the project and to resolve  problems. Though formal memoranda of agreements are not usually  employed, the program establishes test and evaluation plans that detail  pass/fail criteria so that funding does not continue on projects that  experience insurmountable problems. TTI also employs periodic status  reports and meetings; however, communication has not been as open. In  two cases, projects ran into significant problems, such as loss of  acquisition program office support in one case and logistics issues that  had not been addressed to transition a technology smoothly in the other,  which had not come to the attention of the TTI program office. As a result,  the TTI office thought the projects had transitioned when in actuality,  significant problems still needed to be addressed. Per legislation, TTI had  also established a formal council comprised of high-level DOD officials to  help oversee the program; however, the Council has only met once in          2 years, while the act requires that it meet at least semiannually. In  addition, there is some confusion among Council members and others we  spoke with as to what the purpose of the Council should be—that is,  focused on TTI only or broader transition issues. Congressional officials  expressed that they intended for the Council to focus on broader  transition issues and how best to solve them.  Although the Quick  Reaction Fund does not require status reports to assess progress, project  managers are required to submit after-action reports. However, these were  not regularly reviewed by the office. We identified several problems that  arose during transition that were not known to the Quick Reaction Fund  program manager.  The program manager is currently taking steps to  improve the management and oversight of projects.  For example, a  website has been developed to help monitor and execute the program.   Among other things, the website will allow for the automatic collection of  monthly status reports.
Though the transition programs we reviewed are relatively small in scale  compared to other transition programs in DOD, the government’s  investment is still considerable and it will continue to grow if DOD’s  funding plans for the programs are approved. As a result, it is important  that these programs demonstrate that they are generating a worthwhile  return on investment—whether through cost savings to acquisition  programs, reduced times for completing testing and evaluation and  integrating technologies into programs, and/or enhanced performance or  new capabilities. Developing such information can enable transition  program managers to identify what is or is not working well within a  program; how well the program is measuring up to its goals, as well as to  make trade-off decisions between individual projects. On a broader level,  it can enable senior managers and oversight officials to compare and  contrast the performance of transition programs across DOD.
Finding the right measures to use for this purpose is challenging, however,  given the wide range of projects being pursued, the different environments  to which they are being applied, and difficulties associated with measuring  certain aspects of return on investment. For example, measuring long-term  cost savings could be problematical because some projects could have  impacts on platforms and systems that were not part of the immediate  transition effort. As a result, the best place to start may be with high-level  or broad metrics or narratives that focus on the spectrum of benefits and  cost savings being achieved through the program, complemented by more  specific quantifiable metrics that do not require enormous efforts to  develop and support, such as time saved in transition or short-term cost  savings. At this time, however, the transition programs have limited  measures to gauge individual project success and program impact or  return on investment in the long term.  At best, they are collecting after  action reports that describe the results of transition projects, and  occasionally identify some cost savings, but not in a consistent manner. In  addition, there are inconsistencies in how the reports are being prepared,  reviewed, and used. The Quick Reaction Fund program manager, in fact,  had trouble just getting projects to submit after action reports.
Officials from all three transition programs we reviewed as well as higher  level officials agreed that they should be doing more to capture  information regarding return on investments for the programs. They also  agreed that there may already be readily available starting points within  DOD. For example, the Foreign Comparative Testing Program has  established metrics to measure the health, success, and cost-effectiveness  of the program and has developed a database to facilitate return on  investment analyses. The program also captures general performance  enhancements in written narratives. The program has refined and  improved its metrics over time and used them to develop annual reports.  The specific metrics established by the FCT program may not be readily  transferable to DACP, TTI, or the Quick Reaction Fund because the nature  of FCT projects is quite different—technologies themselves are more  mature and costs savings are achieved by virtue of the fact that DOD is  essentially avoiding the cost of developing the technologies rather than  applying the technologies to improve larger development efforts.  However, the process by which the program came to identify useful  metrics as well as the automated tools it uses could be valuable to the  other transition programs. In addition, DOD has asked the Naval Post  Graduate School to study metrics that would be useful for assessing the  ACTD program. The results of this study may also serve as a starting point  for the transition programs in developing their own ways to assess return  on investment.
section_title: Conclusions
paragraphs: The ability to spur and leverage technological advances is vital to  sustaining DOD’s ability to maintain its superiority over others and to  improve and even transform how military operations are conducted. The  three new transition programs are all appropriately targeted on what has  been a critical problem in this regard—quickly moving promising  technologies from the laboratory and commercial environment into actual  use. Moreover, by tailoring processes and criteria to focus on different  objectives, whether that may be saving time or money or broadening the  industrial base, DOD has had an opportunity to experiment with a variety  of management approaches and criteria that can be used to help solve  transition problems affecting the approximately $69 billion spent annually  on advanced stages of technology development. Already, it is evident that  an element missing from all three programs is good performance  measurement. Without having this capability, DOD will not be able to  effectively assess which approaches are working best and whether the  programs individually or as a whole are truly worthwhile. In addition, it is  evident that having well-established tools for selecting and managing  projects as well as communicating with technology developers and  acquisitions helps programs to reduce risk and achieve success, and that  there are opportunities for all three programs for strengthening their  capabilities in this regard. In light of its plans to increase funding for the  three programs, DOD should consider actions to strengthen selection and  management capabilities, while taking into account resources needed for  implementing them as well as their impact on the ability of the programs  to maintain flexibility.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense take the following five  actions:  To optimize DOD’s growing investment in the Technology Transition  Initiative, the Defense Acquisition Challenge Program, and the Quick  Reaction Fund, we recommend that the Secretary of Defense direct the  Under Secretary of Defense (Acquisition, Technology, and Logistics) to  develop data and measures that can be used to support assessments of the  performance of the three transition programs as well as broader  assessments of  the return on investment that would track the long-term  impact of the programs. DOD could use measures already developed by  other transition programs, such as FCT, as a starting point as well as the  results of its study on performance measurement being conducted by the  Naval Post Graduate School. To complement this effort, we recommend  that DOD develop formal feedback mechanisms, consisting of interim and  after action reporting, as well as project reviews if major deviations occur  in a project. Deviations include, but are not limited to, changes in the  technology developer, acquirer, or user, or an inability for the technology  developer to meet cost, schedule, or performance parameters at key points  in time.
We also recommend that the Secretary of Defense direct the Under  Secretary of Defense (Acquisition, Technology, and Logistics) to  implement the following, as appropriate, for each of the transition  programs: (1) formal agreements to solidify up-front technology  development agreements related to cost, schedule, and performance  parameters that must be met at key points in time and (2) confirmation of  technology readiness levels as part of the proposal acceptance process.
In addition, we recommend that DOD identify and implement mechanisms  to ensure that transition program managers, developers, and acquirers are  able to better communicate to collectively identify and resolve problems  that could hinder technology transition. There may be opportunities to  strengthen communication by improving the structure and content of  interim progress meetings and possibly even designating individuals to act  as deal brokers.
Lastly, as DOD considers solutions to broader technology transition  problems, we recommend that Secretary of Defense direct the Under  Secretary of Defense (Acquisition, Technology, and Logistics) to assess  how the Technology Transition Council can be better used.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us with written comments on a draft of this report.  DOD  partially concurred with four of the five recommendations and concurred  with one recommendation.  The reason DOD only partially concurred with  four of the recommendations is because it does not believe the Quick  Reaction Fund fits the definition of a transition program.  However, we  continue to believe it is important for DOD to institute better management  controls and have better visibility of the Quick Reaction Fund as it  increases its investment in this program over the next several years.  DOD  comments appear in appendix I.
DOD partially concurred with our recommendation that the programs  develop data and measures that can be used to support assessments of the  performance of the three transition programs as well as broader  assessments of return on investment that would track the long term  impact of the programs.  DOD agreed that performance measures for the  DACP and TTI programs could be improved but does not believe that  measuring the impact of the Quick Reaction Fund is necessary because it  does not technically fit the definition of a transition program.  We  disagree.  DOD should track the progress of its various programs to  determine if the programs are worthwhile and should be continued, if the  program should receive additional funding, or if changes should be made  in the selection or implementation process that could result in better  outcomes.  Further, failure to track even the most basic information, such  as the number of projects completed, could result in a lack of ability to  manage the program properly and poor stewardship of taxpayer money.
DOD partially concurred with our recommendation that the three  programs develop formal feedback mechanisms consisting of interim and  after action reporting, as well as project reviews if major deviations occur  in a project.  DOD agrees that the TTI and DACP can be improved and has  recently taken steps in this regard.  However, DOD believes that due to the  limited scope and duration of Quick Reaction Fund projects, formal  feedback mechanisms may not be necessary for this program.  We believe  that regular feedback on the progress of each program is important to help  program managers mitigate risk.  As stated in the report, the Quick  Reaction Fund program manager was unaware that one project ran out of  funding prior to field testing the technology.  Had the program manager  been aware of the problem, money that had not yet been allocated could  have been used to finish the project.  In addition, based upon our  discussions with the current program manager, DOD is planning to require  monthly status reports for funded projects.
DOD partially concurred with our recommendation that the programs  implement, as appropriate: (1) formal agreements to solidify up-front  technology development agreements related to cost, schedule, and  performance parameters that must be met at key points in time and (2)  confirmation of technology readiness levels as part of the proposal  acceptance process.  DOD indicated that it recently implemented  Technology Transition Agreements for the TTI, and the DACP program  also uses formal agreements.  However, DOD does not believe formal  agreements are necessary for the Quick Reaction Fund because it is not  intended to be a transition program.  Also, it does not believe TRLs should  be a factor in the proposal acceptance process.  As stated in the report, we  agree that formal agreements may not be appropriate for Quick Reaction  Fund projects.  However, TRLs should be considered during the selection  process.  Since the goal of this particular program is to prototype a new  technology in 12 months or less, it is important that DOD has some  assurance that the technology is ready to be field tested.  As discussed in  the report, the Quick Reaction Fund had to cancel one project, after $1.5  million had already been spent, because it had only achieved a TRL 3.  Had  the selecting official known the TRLs of each proposed project during the  selection phase, he may have decided to fund another, more mature  project instead.
DOD also partially concurred with our recommendation that the programs  identify and implement mechanisms to ensure that transition program  managers, developers, and acquirers better communicate and collectively  identify and resolve problems that could hinder technology transition.   DOD established a Transition Overarching Integrated Product Team  earlier this year to provide the necessary oversight structure to address  this issue, but does not believe this recommendation applies to the Quick  Reaction Fund program.  We believe that if DOD receives monthly status  reports on the Quick Reaction Fund, as planned by the program manager,  it should be in a good position to identify and resolve problems that could  hinder the testing of new technology prototypes.
DOD concurred with our recommendation that the Under Secretary of  Defense (Acquisition, Technology and Logistics) assess how the  Technology Transition Council can be better used as DOD considers  solutions to broader technology transition problems.  Although DOD did  not indicate how it plans to do this, the Deputy Under Secretary of  Defense (Advanced Systems and Concepts) has a goal that the Council not  limit itself to just the TTI program, but look at broader technology  transition issues across DOD.
We are sending copies of this report to the Secretary of Defense, the  Director of the Office of Management and Budget, and interested  congressional committees. We will also make copies available at no charge  on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (937) 258-7915. Key contributors to this report were Cristina  Chaplain, Cheryl Andrew, Art Cobb, Gary Middleton, and Sean D. Merrill.