DOD Rapid Innovation Program: Some Technologies Have Transitioned to Military Users, but Steps Can Be Taken to Improve Program Metrics and Outcomes
Published: 2015-05-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: DOD relies on technology innovation to maintain superior weapon systems. However, a long-standing challenge has been ensuring that high value technologies are mature and available for military users. The Ike Skelton National Defense Authorization Act for Fiscal Year 2011 required DOD to establish RIP to facilitate the transition of innovative technologies into acquisition programs, and over $1.3 billion has been appropriated to the program since its inception.
Senate Report No. 113-44 included a provision that GAO review the execution of RIP. This report examines the extent to which (1) DOD has established a competitive, merit-based process to award RIP contracts; (2) DOD has established practices to manage the execution of RIP projects; and (3) RIP is meeting its objective of rapidly inserting innovative technologies in defense acquisition programs. GAO reviewed RIP documentation, interviewed DOD and military department officials, and reviewed a nongeneralizable sample of 40 projects awarded with fiscal year 2011 and 2012 funding to assess DOD management practices as well as 52 fiscal year 2011 funded projects scheduled for completion through July 2014 to assess transition outcomes.
section_title: What GAO Found
paragraphs: The Department of Defense (DOD) has established a competitive, merit-based process to solicit proposals from interested contractors, review and select projects based on military needs and standardized evaluation criteria, and award contracts to execute Rapid Innovation Program (RIP) projects. To date, the process has been lengthy, taking about 18 months to implement, but interest from contractors has been high. Between fiscal years 2011 and 2015, the military services and defense agency components received more than 11,000 white papers on proposed technologies from contractors and will have awarded contracts for about 435 projects—primarily to small businesses—when the fiscal year 2014 solicitation is completed.
The military services and defense components have practices and tools in place to manage and monitor the execution of projects, which are similar to those they use for other science and technology projects. For example, project officials review contractor reports, conduct system reviews, and engage in regular communications with contractors to determine the progress of projects. Also, DOD's 2014 annual review found that 85 percent of the fiscal year 2011 funded projects and 78 percent of the fiscal year 2012 funded projects were likely to meet 80 percent or more of their technical performance goals.
Some completed projects have successfully transitioned technology to acquisition programs and other military users. DOD officials estimate that 50 percent of all fiscal year 2011 funded projects (88 of 175) have out-year funding commitments from military users, indicating the likelihood they will transition technologies. GAO assessed 44 projects completed in July 2014, and found that 50 percent successfully transitioned to acquisition programs or other users. However, it is too soon to accurately assess the overall success of RIP due to the limited number of completed projects as well as the lack of an overall program transition goal and effective metrics to track the degree to which projects have actually transitioned. GAO found that several factors can contribute to transition success of RIP projects, such as having military user commitment and mature technology when projects are started. However, DOD has not made an effort to understand how these factors may be contributing to differences in transition success from defense components with a higher rate of transition.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD establish a program transition goal, and identify and apply factors that lead to transition success. DOD disagreed on the need for a goal, stating it would impede RIP flexibility, but agreed to identify transition success factors. GAO believes having a goal would improve DOD's ability to transition technologies.

section_title: Letter
paragraphs: The Department of Defense (DOD) relies on technological innovation to  maintain the superiority of its weapon systems and armed forces to  protect U.S. interests at home and abroad. However, a long-standing  challenge for DOD has been ensuring that high value technologies from  defense research organizations and businesses are mature and available  for use by the department’s acquisition community. Sometimes  technologies are not ready to transition to a defense acquisition program  when needed because they may still be too risky or too costly to adopt. At  other times, promising technologies are not taken advantage of due to  insufficient processes and mechanisms to expedite their transition to  military users. DOD and Congress recognize these as well as other  problems in transitioning technology and have initiated a number of  programs over the past decade aimed at removing transition barriers and  accelerating the flow of innovative technologies to users. The Ike Skelton  National Defense Authorization Act for Fiscal Year 2011 required DOD to  establish the Rapid Innovation Program (RIP) to facilitate the insertion of  innovative technologies, into defense acquisition programs. Between  fiscal years 2011 and 2015, Congress appropriated about $1.3 billion to  fund technology projects through the program.
S. Rep. No. 113-44 at 54-56 (2013). of RIP projects, and (3) RIP is meeting its objective of rapidly inserting  innovative technologies into defense acquisition programs.
To conduct this work, we evaluated documentation from the Under  Secretary of Defense for Acquisition, Technology, and Logistics; the Air  Force; the Army; Navy; Special Operations Command; and other defense  agencies involved in RIP; and interviewed key program and project  officials. To assess how DOD solicited and awarded RIP projects, we  reviewed DOD and military department polices and guidance on the  program as well as solicitation and source selection documents. Also, we  interviewed officials responsible for developing the guidance and those  involved in the source selection process. To assess how DOD manages  and oversees RIP projects, we reviewed a non-generalizable random  sample of 40 RIP projects across DOD that were awarded with fiscal year  2011 or 2012 funding. We selected this time frame to capture projects  that would be at or near completion. In addition, we reviewed project  contracts, contractor reports, agency project assessments, and  interviewed officials responsible for project oversight and management.  To determine whether the RIP is meeting its goal of rapidly inserting  innovative technologies into acquisition programs, we reviewed available  program monitoring information and assessed the transition status of all  projects that were scheduled for completion by the end of July 2014. This   For these  totaled 52 projects awarded with fiscal year 2011 funding.projects, we interviewed DOD RIP project officials and collected and  analyzed documents outlining processes and procedures used to manage  the projects and promote transition opportunities. We discussed factors  that may have helped or hindered project execution and/or successful  transition to defense acquisition programs or other military users. We also  used data from DOD’s annual in-process review on the status and  progress of RIP projects, and determined through discussions with DOD  subject matter experts that the data were sufficiently reliable for the  purposes of this report. See appendix I for additional details on the scope  and methodology used for this work.
We conducted this performance audit from April 2014 to May 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: To identify, pursue, and develop new technologies to improve and  enhance military capabilities, DOD relies on its science and technology  (S&T) community which is comprised of DOD research laboratories, test  facilities, industry, and academia. The S&T community receives about  $12 billion in funding each year to support activities ranging from basic  research through advanced technology development that are conducted  by the government or externally by universities and commercial industry.  Once the S&T community has completed its technology development,  additional product development activities, such as technology  demonstration and testing, are often needed before incorporating the  technologies into military weapon systems. Under the management of the  acquisition community, product development further advances technology  received from S&T developers and integrates it into systems that are  ultimately delivered to support the warfighter (see figure 1).
However, as we have reported in the past, for a variety of reasons DOD  historically has experienced problems in transitioning technologies out of  the S&T environment and into military systems. For example,  technologies may not leave the laboratory because their potential has not  been adequately demonstrated or recognized, acquisition programs may  be unwilling to fund final stages of development, or private industry  chooses to develop the technologies itself.
DOD has a variety of technology transition programs managed by the  Office of the Secretary of Defense (OSD) and the military departments  that provide mechanisms and funding to facilitate technology transitions.  The programs vary in size, approach, and funding, but most of them are  intended to target fairly mature technologies that are suitable for the final  stages of development and demonstration. Some, such as the Small  Business Innovation Research (SBIR) program, have a stated purpose of  using small businesses to meet federal research and development  needs.feasibility and development projects, small businesses are expected to  obtain funding from the private sector or government sources outside the  SBIR program to commercialize or transition technologies for sale to the  military or elsewhere.
However, while the program provides funding for technology  Congress established RIP as another mechanism, to include  consideration of innovation research projects from small businesses, as  they are often viewed as a key source for developing innovative  technologies in areas of military need. Specifically, Congress directed  DOD to establish RIP to accelerate the transition of technologies  developed by  small businesses participating in SBIR phase II projects; the defense laboratories; and other small or large businesses.
The projects funded by RIP are to primarily support major defense  acquisition programs, but also other defense acquisition programs that  meet a critical national security need. In addition, the projects selected  are to accelerate the fielding of technologies with the purpose of reducing  acquisition or lifecycle costs; addressing technical risks; and improving  test and evaluation outcomes. Furthermore, DOD was directed to develop  a competitive, merit-based program that at a minimum provides for  the use of a broad agency announcement or the use of any other  competitive or merit-based processes for the solicitation of proposals; merit-based selection of the most promising cost-effective proposals  for funding through contracts, cooperative agreements, and other  transactions; a limit on funding for each RIP project not exceeding $3 million; and no project is funded under the program for more than 2 years.
Within DOD, the Under Secretary of Defense for Acquisition, Technology,  and Logistics issues overall program guidelines, and representatives  appointed by the military service acquisition executives, Assistant  Secretary of Defense for Research and Engineering (ASD/R&E), and the  Director, Office of Small Business Programs have the responsibility for  establishing the RIP processes that support DOD’s goal and meet the  guidelines. The ASD/R&E is also responsible for coordinating RIP  activities among the military departments and other defense components.  In addition, ASD/R&E is responsible for preparing and submitting a report,  at the end of the fiscal year, to the congressional defense committees that  includes a list and description of each project funded, the amount of  funding provided, and anticipated timeline for transition. The program is  implemented and managed by each of the military departments and by  ASD/R&E, which represents the other defense components that  participate in the program. To date, DOD has not included RIP in its  annual budget requests because there is no formal requirement within the  department for the program, but Congress has appropriated funding each  year. Congress has authorized RIP until September 30, 2015, when it will  expire unless further legislative action is taken.
section_title: DOD Uses a Competitive and Merit-Based Process to Select RIP Projects, with Most Awards Going to Small Businesses
paragraphs: DOD has established a competitive, merit-based solicitation process to  select and award RIP contracts that address military needs. Each year,  the Office of the Under Secretary of Defense for Acquisition, Technology,  and Logistics begins by issuing broad guidance to the military  departments and defense components for implementing a multi-phase  process to identify military needs, solicit proposals from interested  contractors, review and select projects, and award contracts. The process  is somewhat lengthy, taking about 18 months to implement and award  contracts, but interest from contractors has been high. Between fiscal  year 2011 and 2015, the military departments and defense components  received more than 11,000 white papers from interested contractors and  will have awarded contracts for about 435 projects when the fiscal year  2014 solicitation is completed, with the vast majority going to small  business. This high level of interest in the program has presented some  administrative challenges to reducing the time from identification of need  to contract award.
subsections: section_title: DOD Has Implemented a Multi-Step Process to Select Promising RIP Technology Projects
paragraphs: DOD has established a multi-step process to solicit, evaluate, and select  RIP projects to fund (see figure 2).
Before the yearly RIP acquisition cycle begins, the Office of the Under  Secretary of Defense for Acquisition, Technology, and Logistics issues  implementation guidelines to the military departments and defense  components which outline program goals and specific guidance on the  program’s implementation and reporting requirements. For example, the  guidelines include the funding available to the military departments and  defense components; DOD-wide research areas, if applicable; guidance  on how the solicitation process will be structured, such as whether a  single solicitation will be used and the length of time it will be open; and  the process for obtaining a waiver for projects expected to exceed the 24  month or $3 million limits.
subsections: section_title: Identification of Technology Needs
paragraphs: The RIP process begins with the military departments and defense  components identifying the technology topics that will be addressed in the  program. In addition to the 3 military departments, participation by  defense components has increased, from 4 in fiscal year 2011 to 16 in  fiscal year 2014. The topics are intended to reflect technology capability  needs and requirements for weapon systems and acquisition programs,  and are one of the criteria used in the selection of projects. The military  departments and defense components have used different approaches to  identify the technology research topics, but over time have moved to a  more common approach. For example, while the Air Force and Navy  have largely relied on their acquisition communities to identify technology  needs, the Army relied primarily on its science and technology community  to identify needs for the first 3 years of the program. For the Air Force and  Navy, this entailed going directly to systems commands, Program  Executive Offices, program managers, and others directly involved in the  development and production of weapon systems to solicit needs. In  relying on its science and technology community, the Army focused on  S&T research areas that included more than the specific technology  needs of acquisition programs. In fiscal year 2014, the Army expanded its  approach and solicited topics from Program Executive Offices as well as  from the science and technology community. The defense components  used a variety of methods for identifying technology requirements. For  example, the Missile Defense Agency develops research topics by  reviewing technology road maps and prior SBIR research topics, and  obtaining input from Program Executive Officers and program managers.  The combatant command participants, such as the United States Special  Operations Command (SOCOM), include operational capabilities linked to  internal requirements or urgent needs.
section_title: Competitive, Merit-Based Process Used to Solicit and Award Projects
paragraphs: The military departments and defense components have used broad  agency announcements (BAA) to solicit technical solutions that address  technology topics and meet other submission requirements. The Federal  Acquisition Regulation (FAR) provides that the selection of basic and  applied research is a competitive procedure if an award results from (1) a  broad agency announcement that is general in nature, identifying areas of  research interest, including criteria for selecting proposals, and soliciting  the participation of all offerors capable of satisfying the government’s  needs; and (2) a peer or scientific review. The FAR also requires that  BAAs specify the period of time for parties to respond, and contain  instructions for preparing and submitting proposals. Also, we have found  in a prior review that the use of competitive contracting procedures  encourages firms to offer their best proposals when competing for work.  The procedures used by the military departments and defense  components for selecting RIP projects generally follow FAR competition  guidelines because the BAAs  identified general areas of research interest and criteria for selecting  proposals; solicited the participation of all offerors capable of satisfying the  government’s needs; and that subject matter experts will review project proposals.
For fiscal years 2011 to 2013, the military departments and the ASD/R&E  published four separate BAAs, although they used similar selection and  evaluation criteria. In fiscal year 2014, a single BAA was used to solicit  proposals because officials wanted to make it easier for businesses to  participate in the program and reduce some of the internal overhead  involved in having to maintain four separate BAAs within DOD.
DOD has established a merit-based review process to select projects  through a two-step approach—submission and review of brief white  papers submitted by contractors, then comprehensive technical proposals  from a subset of these contractors whose white papers are deemed most  promising based on four criteria:
Contribution to Requirement—the degree to which the technical  approach is relevant to the proposed requirement;
Technical Approach and Qualifications—the degree to which the  technical approach is innovative, feasible, achievable, complete and  supported by a technical team that has the expertise and experience  to accomplish the proposed tasks;
Schedule—the degree to which the proposed schedule is achievable  within 24 months from contract award; and
Cost—the degree to which the proposed cost or price is realistic for  the proposed technical approach and does not exceed $3 million.
According to RIP officials, this approach is viewed positively by small  businesses because they do not have to invest in developing full  proposals at the outset.
Evaluation teams of subject matter experts, nominated by the military  departments and defense activities, rate the white papers “go” or “no go”  based on the criteria above. Greater consideration is given to the criteria  pertaining to contribution to requirement and technical approach; if papers  do not meet either of these two criteria they are not considered for further  review. The contractors submitting white papers that are approved by the  source selection authority for further consideration are then invited to  submit a full proposal. According to a DOD official, more white papers  are approved than can be funded because some technologies will not  remain competitive when further details and support are provided in a full  proposal.
Once businesses are notified that their white papers have been selected,  they have 30 days to submit full proposals that amplify the information in  the white paper. The military departments and defense components  assign subject matter evaluators to review the proposals using the same  criteria as for the white paper evaluations. DOD uses a 5-level descriptive  rating scale, ranging from outstanding to unacceptable, when evaluating  the proposals against the criteria. As with the white paper evaluations,  RIP gives greater importance to obtaining superior technical capabilities  that will transition than in making awards at lower cost to the government.  Selection preference is also given to small business proposals. Selection  of large businesses is allowed, but only if their offers for the same  requirement are superior to those of the small businesses. Once all  proposals are evaluated and ranked, the source selection authority  makes the final decisions on contract awards. The number of proposals  selected for contract award will depend on the level of RIP funding  appropriated and the proposed project costs. The businesses are notified  of the results and the military departments and defense components then  negotiate and complete all contracting and award procedures with the  businesses. Contracts establish the cost, schedule, and key performance  parameters, and deliverables required for projects.
section_title: RIP Has Awarded a Large Number of Contracts, but Faced Administrative Challenges in Reducing Process Time
paragraphs: From fiscal years 2011 through 2013, the military departments and  defense components awarded $760 million for 365 projects, with 89  percent of the awards going to small businesses. Program officials said  that RIP attracted some new small businesses that had not worked with  DOD previously. However, most of the projects selected (60-74 percent)  had also received DOD funding through the SBIR program for earlier  technology development activities. Table 1 presents an overview of the  numbers of proposals, contract awards and dollar values, and small  business participation rates for the fiscal year 2011 through 2014 project  selection cycles.
Projects funded by RIP cover a broad range of technologies, including  software tools and applications, measurement and testing devices, the  reconfiguration of existing technologies, and the development,  demonstration, or prototyping of new technologies. Examples of projects  include  software to improve the rate and accuracy of the transmission of data, an effort to improve the manufacturing of a component used in  thermal battery insulation technology to increase the reliability and life  span of missile power sources, and a compact water hand pump and filter for purifying water on the  battlefield.
According to RIP officials, although the program is intended to target  innovative technologies, the department’s process enables the defense  components to identify requirements based on evolving operational needs  and determine the kinds of projects to fund.
Since its inception, RIP has received a high level of interest from  businesses─more than 11,000 white papers submitted in 4 years─which  has presented some program administrative challenges. In the first year,  several of the military departments and defense components had to  “scramble” to complete reviews and award the contracts before program  funds expired, in part because they did not have sufficient infrastructure in  place to administer the program. According to a RIP official, SOCOM  received 800 of the 3,600 papers submitted that year and was  overwhelmed due to the high volume of white papers. Since the first year,  the components have made adjustments to try and administer the  process more efficiently, such as the move towards issuing a single  consolidated BAA to solicit proposals in fiscal year 2014. However, the  time needed to prepare the RIP implementation guidelines; identify  technology topics; prepare and execute the BAAs; and review and  evaluate white papers as well as complete the other steps in the  solicitation and project selection process has been lengthy in each  acquisition cycle, taking about 18 months to complete (see figure 3).
In 2014, RIP program officials set a qualitative goal to reduce cycle time  for the fiscal year 2014 solicitation process, but said it is challenging due  to the desire to employ the two-step solicitation process. For the fiscal  2015 cycle, they plan to release the BAA earlier than in previous cycles  and have moved the milestone for completing this acquisition cycle  forward about 30 days.
section_title: DOD Uses a Variety of Practices and Tools to Manage and Oversee Projects
paragraphs: The military departments and defense components used management  practices and tools to manage RIP projects similar to those they use for  other science and technology projects. Project managers and contracting  officials review progress reports submitted by contractors and maintain  regular communications with contractors to monitor whether projects are  meeting cost, schedule, and performance requirements specified in the  contracts. ASD/R&E established an annual DOD-wide reporting  mechanism in 2013 to assess the technical performance of the projects,  and most of them are meeting their technical performance metrics.
The FAR requires that contract quality assurance be performed at such  times and places as may be necessary to determine that the goods or  services conform to the contract requirements. The RIP project contracts  describe the deliverables, reporting schedules, and financial and technical  reports that the project contractor must submit. For the 40 projects we  reviewed, RIP project officials and contracting officer’s representatives— the technical subject matter expert—conducted a variety of activities to  manage and monitor the status of RIP projects, including continued  involvement throughout contract implementation to support contractors  and help ensure that the contracted services would be delivered  according to the schedule, cost, quality, and quantity specified in the  contract.
RIP project officials said that they manage and oversee projects in a  similar fashion as other DOD technology development projects. In  managing projects, officials said they review progress and financial  reports submitted by the contractor, conduct system review meetings  tailored to the size and complexity of the project, and engage in regular  communications with contractors through e-mails, phone discussions, and  occasional visits to a contractor’s facility. For example, on one project, the  government team met weekly with a contractor for the first few months of  the contract to review the contractor’s deliverables. In addition to the  written reports, project officials also conducted quarterly program  management reviews which involved an in-person meeting. A project  official explained that working with small businesses presented unique  challenges for oversight, because it requires the contractor to be  educated on DOD requirements such as preliminary design reviews,  critical design reviews, and acceptance of testing plans. Despite requiring  these extra education efforts, officials said that small companies were  very flexible and responsive to oversight activities. In addition to the  project officials that help manage the project, the contracting officer also  appointed a contracting officer’s representative to assess the contractor’s  performance against contract performance standards and to record and  report this information.
subsections: section_title: Oversight of RIP Projects
paragraphs: ASD/R&E established an annual reporting mechanism in 2013 to assess  the status and progress of RIP projects. Federal government internal  control standards provide that effective and efficient control activities be  established to enforce management directives; these include  performance measures and indicators to compare against program goals   We found that RIP officials within DOD are taking these  and objectives.steps. ASD/R&E collects information from project officials on the  contractor’s ability to meet key technical performance parameters, using  the following three-level scale to assess the performance indicators for  each project:
Green—80 percent or more of the key performance parameters,  goals, or thresholds will be met;
Yellow—less than 80 percent of the key performance parameters,  goals, or thresholds were being met; and
Red—none of the key performance parameters, goals, or thresholds  In the first in-process review, conducted in September 2013, ASD/R&E  assessed the performance of the 175 ongoing projects that were funded  in the fiscal year 2011 acquisition cycle. The review found that 86 percent  of the projects were likely to meet 80 percent or more of their key  performance parameters or goals. ASD/R&E conducted another in- process review in October 2014 and found that 85 percent of the fiscal  year 2011 projects and 78 percent of the fiscal year 2012 projects were  likely to meet 80 percent or more of their key performance parameters or  goals (see figure 4).
In addition to the ASD/R&E in-process review, some military departments  and defense components conduct their own performance reviews. For  example, the Air Force obtains semi-annual status reports from project  managers. These status reports include information associated with  technical risks that could negatively impact a project or its technologies  and additional information such as accomplishments, planned actions,  and information on projects’ transition strategies. The Navy requires its  system commands’ chief technology officer or designee to semi-annually  update the project execution plan for their projects. Project execution  plans include a description of what constitutes a project’s transition, the  criteria for test events, and funding. This information enables the Navy to  maintain awareness of a project’s execution and to provide assistance  with transition and eventual deployment to end users if needed. Similarly,  the Missile Defense Agency and SOCOM also perform periodic reviews  of their RIP projects.
section_title: RIP Has Transitioned Some Projects, but It Is Too Early to Determine Overall Results
paragraphs: Some completed projects successfully transitioned to acquisition  programs and other users, but opportunities may exist to improve overall  RIP transition outcomes. DOD reported that half of all fiscal year 2011  projects (88 of 175) had funding commitments from military users, which  indicates a likelihood they will transition. We also found that half of the  fiscal year 2011 completed projects we reviewed (22 of 44) had  technology that actually transitioned to an acquisition program, another  military user, a prime contractor, or was commercialized. The majority of  these projects had previously participated in the SBIR program. These  transition rates are lower than what we found in our prior review of other  DOD technology transition programs that reported transition rates ranging  from about 55 to 85 percent.assess the overall success of RIP due to the limited number of completed  projects and lack of established metrics to track whether projects have  successfully transitioned to users. We also found several factors that can  contribute to the transition success of RIP projects, such as user  commitment and mature technology at the beginning of projects, which  were not consistently emphasized by all the defense components in the  program.
subsections: section_title: DOD Estimates Half of 2011 Projects Will Secure Transition Funding
paragraphs: As part of its annual in process review, DOD collects data from RIP  project managers on the transition status of their projects. Based on the  October 2014 review, DOD reported that half of fiscal year 2011 projects  and 41 percent of fiscal year 2012 projects had out-year funding  committed by a partner or user, which DOD uses as an indicator of the  likelihood transition will occur after project completion (see figure 5).
Funding is a marker that illustrates the transition partner’s level of  commitment to transition the technology. A transition partner can be a  program of record, a prime contractor, or a user that is willing to dedicate  funding to mature a technology past the development stage. The  transition status of fiscal year 2011 and 2012 projects varied across the  military departments and defense components. At the time of DOD’s  review, many of the fiscal year 2012 projects were ongoing and may be  up to a year away from completion. As a result, transition indicators for  fiscal year 2012 projects may not reflect final outcomes. As depicted in  the figure, the Air Force and the Navy reported that most of their fiscal  year 2011 projects have out-year funding committed, but the Army had  significantly fewer projects with a transition funding commitment.  According to an Army RIP official, this may be due in part to the Army’s  selection of projects that addressed a broader set of science and  technology needs rather than needs defined by the acquisition  community, where partner funding commitments would be more likely.  For projects that did not have a commitment from a transition partner, the  military departments and defense components reported that in several  cases, the expected follow-on acquisition, procurement or support funds  were redirected to higher priorities or requirements for the technologies  were cancelled while the projects were underway.
section_title: Half of Completed Projects We Reviewed Successfully Transitioned
paragraphs: To gain additional insights into the range of transition outcomes for RIP  projects, we assessed all 52 fiscal year 2011 projects scheduled for  completion through July 2014, to determine their transition results. We  found that 44 of these projects had been completed, meaning the contract  period of performance ended. Although projects are typically required to  be completed within 2 years, several of them had encountered delays and  obtained no-cost schedule extensions of 6 months or more from RIP  officials to complete the remaining work. Officials told us extensions were  granted for various reasons: completing additional testing; better aligning  projects with acquisition program schedules; and further demonstrating  project technology. Of the 44 projects that were completed, half had  technology that successfully transitioned to an acquisition program, a  military user, a prime contractor, or became a commercially available  product. Table 2 depicts these outcomes, including the range of scenarios  for projects that did not transition.
The following are a few specific examples of RIP projects that  successfully transitioned, demonstrating the array of technologies and the  users supported by these technology transitions:
Wireless Vibration Recorder: This Navy project provides a compact  and lightweight measurement device that can acquire data quickly  and easily on aircraft, reducing flight test costs and system  development time. The device includes specialized software and  sensors to acquire vibration and acceleration data for aircraft internal  components to determine why and when components fail. The Navy  currently utilizes a test instrumented aircraft to measure vibrations,  which is time consuming and costly. RIP officials reported this  technology is expected to save the Naval Air Systems Command $3  million to $5 million in the next 4 years in flight test costs. This  technology is also available commercially.
Enhanced Ground Moving Target Indicator (GMTI)-based  Intelligence, Surveillance, and Reconnaissance: This Air Force  project provides software applications to be used by joint surveillance  target attack radar system operators. The applications are intended to  interact with GMTI data to facilitate pattern recognition, automatically  identify targets and perform analyses over any set period of time. For  example, instead of a dot on the water, a target could be automatically  identified as a carrier or a person in a rowboat so the analyst can  determine if they need to send a camera to the area to investigate  further. RIP officials reported the Air Force Research Laboratory has  installed the software suite on a data repository that can be used by a  wide variety of intelligence analysts.
Multi-Missile Common Launch Tube: This United States Special  Operations Command project increases the number of munitions that  can be carried and launched from a single common launch tube. This  technology doubles the number of targets that can be attacked and  allows the use of smaller warheads with precision delivery, which can  minimize collateral damage. RIP officials reported that ground and  flight tests were successful, and the technology has been transitioned  into the Command’s common launch tube program.
Automated Intelligent Training with a Tactical Decision-Making  Serious Game: This Army-funded project enhances a software tool  for training Army officers in the classroom or field. The enhanced tool  improves upon a capability to practice key leader cognitive skills, such  as the ability to rapidly assess a dynamic situation, make sound  decisions, and effectively direct subordinate units through scenario- based exercises. According to RIP officials, the software was  delivered to end users at West Point and future Army leaders are  using this software to improve their tactical command skills.
Mine Roller Wheel Assembly Improvement: This project improves  effectiveness of the Marine Corps mine roller wheel assembly over  the legacy system which makes it more effective at neutralizing  threats. A mine roller is used to detonate and clear certain classes of  buried, pressure-activated explosive threats. This upgrade is expected  to be less expensive than the cost of the original roller wheel  assembly it is modifying while increasing its effectiveness at a greater  range of speeds. In February 2015 the Marine Corps established an  acquisition program known as the Wheelbank Suspension Upgrade  Program to purchase the upgraded mine roller wheel assembly and is  now working toward a production decision expected in the third  quarter of fiscal year 2015.
In contrast, we found several different reasons projects did not transition.  DOD officials told us it may take a year or more to transition a project  because project completion may not easily line up with user timelines or  the DOD budget cycle. A little more than half had successfully  demonstrated the technology as planned, but had no user commitment.  According to RIP project managers, some of the projects provided value  to DOD and may yet transition in the future. One of the Navy projects, for  example, which provides a technology to filter out interference in certain  military radios, has an interested transition partner, but has not yet  transitioned due to funding constraints. Officials reported they were  unable to secure funding in the fiscal year 2015 and 2016 budgets, but  have submitted another request for fiscal year 2017. DOD officials also  said that several projects failed to transition due to changing user  priorities and requirements, such as the phase-out of ongoing missions in  Iraq and Afghanistan. Further, according to DOD officials, other projects  demonstrated a proof of concept or provided data to develop  requirements, but were not planned to transition into a program of record.  For example, one Army project delivered a prototype field kitchen that  demonstrated potential capabilities, and the results will be used to inform  future requirements for a program of record planned in 2019.
Other RIP projects that did not transition needed further technology  development or additional testing—such as software that needed security  accreditation—before they could be integrated into a DOD program. For  example, a Navy-funded project for a torpedo array nose assembly,  designed to improve the array’s performance in shallow water, needs  additional testing to complete the array qualification effort. If successful,  this array nose assembly is expected to provide a second source at a  lower cost than the current array.
On the other hand, officials told us some of the projects did not meet user  requirements or did not demonstrate that their technology was better than  existing technology. For example, a SOCOM-funded project to develop  an upgraded antenna that could operate in the presence of jamming did  not meet a performance standard. Also, an Air Force project to test  capabilities of a spectral flare showed the flare did not meet all of the  requirements and did not offer significant improvement in performance  compared to the current flare. Officials said that although projects may  appear unsuccessful, in some cases they provided valuable information  that could improve future decisions.
section_title: Mature Technology, User Commitment, and Program Support Contribute to Successful Transition; However, DOD Has Not Taken Steps to Evaluate This
paragraphs: In our past work on DOD technology transition activities, we found that  several factors are important to successful transition, including the  selection of projects that have sufficiently mature technologies and early  endorsement from intended military users. In addition, once projects are  selected, they require effective management processes to support  technology transition. RIP officials and project managers that we  interviewed also noted that these factors can contribute to successful  transition. In general, however, at the DOD level there has not been an  effort to understand the extent to which these factors may be contributing  to differences in transition and to communicate lessons learned from  military departments or defense components with a higher percentage of  transition success. Without such an understanding, DOD may be missing  opportunities to leverage practices from certain program components that  could help improve overall transition success rates.
subsections: section_title: Technology Maturity
paragraphs: GAO-13-286. to review the support for the technical approach presented in proposal  submissions, including information on TRLs.
section_title: User Commitment
paragraphs: We previously reported that early project endorsement from intended  users and other key stakeholders in the acquisition community is  important for project transition. RIP officials also agreed that having a  transition partner, such as a specific program of record, identified prior to  the beginning of the project contributes to transition success. Other RIP  officials said user participation in ongoing project reviews further ensures  the final product is aligned with user requirements. In the required  reporting to Congress, all of the projects we reviewed identified an  acquisition program or potential user as the project started; however, we  found the level of user interest and commitment varied and in some cases  changed during project execution.
section_title: Program Support
paragraphs: The military departments and defense components varied in the approach  and level of support they used to manage RIP technology transition.  According to an OSD-level RIP official, the Navy and the Air Force have  more robust processes to support transition than the other components.  For example, the Navy and the Air Force provide additional internal  guidance for RIP participants that describe the roles and responsibilities  of participants. These departments also require project reviews every 6  months until project completion and stress the importance of lessons  learned for RIP project managers. For example, the Air Force Life Cycle  Management Center’s guidance states that analysis of RIP outcomes at  the conclusion of each project cycle can identify lessons learned from  processes, successes, challenges, and recommendations that can help  build the program knowledge base and establish best practices in  program implementation and stakeholder relations.
The Navy’s guidance describes the importance of quality control,  consistent management, and continuous improvement to help avoid  demonstrating a technology that has nowhere to transition. The Navy also  uses project execution plans that describe transition plans, codify  transition partner agreements, and describe criteria to evaluate seminal  transition events. In addition, the Office of Naval Research’s risk  management team can provide support for small businesses to stay on  track in fulfilling RIP contracts, including making sure companies can  ramp up production if their projects are transitioned. We have previously  reported that this office has a well-established technology transition focus  which may contribute to project success. Because of this, the Navy may  be better aware of the benefits and obstacles associated with a  substantial portion of their S&T portfolio. This knowledge can better  inform investment decisions made by Navy leadership.
section_title: RIP Does Not Track Final Transition Outcomes
paragraphs: While the program has established metrics to determine whether projects  have a funding commitment from users and are therefore likely to  transition, it does not track the degree to which completed projects have  actually transitioned. RIP officials view transition success broadly, as a  technology which is inserted into an acquisition program of record,  incorporated into a weapon system manufacturing process, adopted for  use by a depot or logistics center, or available for purchase on the  General Services Administration federal supply schedule or in the  commercial market. However, RIP does not formally track projects  beyond completion and whether they are inserted into acquisition  programs or used for other purposes, which limits DOD’s ability to know  the final transition outcomes and whether any benefits were achieved. As  we have found in the past, tracking and measuring technology transition  and the impact of those transitions, such as cost savings or deployment  of a technology into a weapon system, provides key feedback that can  inform the management of programs such as RIP. We previously  recommended the Secretary of Defense require all technology transition  programs in the department to track and measure project outcomes—  DOD  including long term benefits for acquisition programs or users.generally agreed with our recommendations but we found no evidence  that DOD has taken any action yet; this recommendation would also  apply to RIP, as it is a technology transition program. In addition, DOD  has not established an overall transition goal for RIP, so it is not clear  what is expected in terms of success. Federal government internal control  standards require that effective and efficient control activities be  established which include performance measures and indicators to  compare against program goals and objectives.DOD annually provides a report to Congress on the number and  description of projects that are funded through RIP, the department is not  required to provide information on transition results. Without this  information, Congress lacks insight about the program’s performance  which is important for conducting program oversight.
In our discussions with DOD RIP officials, they estimated that about 50  percent or more of the RIP projects will transition when completed,  although they said it is too early to accurately assess the overall  effectiveness of the program. Some officials indicated that there are not  enough projects completed and it could take two or more years after a  project is completed before it is successfully transitioned and used. In  addition, projects awarded in RIP’s first acquisition cycle may not  accurately reflect program performance for the following years, in part  because there were implementation challenges associated with starting a  new program. Therefore, more data that captures final project outcomes  and long-term program experience is required to accurately assess  performance. In our prior review of DOD technology transition programs  that provide structured mechanisms and funding to facilitate technology  transition, we found that programs reported rates of technology transition  ranging from about 55 to 85 percent. percent is at the lower end of this range, as is experience to date with the  transition of completed projects.
section_title: Conclusions
paragraphs: GAO-13-286. report program results to Congress, as this is not currently required.  Consequently, without an overall transition performance target and better  measure of outcomes, it is unclear whether RIP has been successful at  transitioning innovative technologies and is worth continuing. We continue  to believe that the recommendation we made in 2013—for DOD to track  and measure the outcomes of its numerous technology transition  programs in order to improve the visibility and management of these  efforts—has merit and is applicable to RIP.
In addition, although the military departments and defense components  have implemented a structured process for soliciting projects, they do not  always select projects that have a high likelihood of successfully  transitioning, such as those with mature technologies or commitments  from transition partners. DOD components are capitalizing on these  success factors to varying degrees. The Air Force and Navy, in particular,  incorporate them in RIP projects and, as a result, appear to be realizing  significantly higher transition rates for their projects, even in the program’s  early years. Without a more consistent focus on these factors during  project selection, opportunities to achieve higher levels of transition  success for RIP may remain limited.
section_title: Matter for Congressional Consideration
paragraphs: Congress should consider, if it decides to re-authorize RIP, requiring  DOD to submit annual reports to Congress on the transition results of the  program to improve accountability and transparency of the program.
section_title: Recommendations for Executive Action
paragraphs: If Congress re-authorizes RIP then, to improve visibility and management  of DOD’s ability to transition technologies through the program, we  recommend that the Secretary of Defense direct the Under Secretary of  Defense for Acquisition, Technology, and Logistics to take the following  two actions:
Establish an overall technology transition goal for RIP; and  Identify and apply factors that contribute to the likelihood of  technology transition success more consistently across the program.
section_title: Agency Comments and our Evaluation
paragraphs: We provided a draft of this report to DOD for comment. In its written  comments, DOD disagreed with our first recommendation and concurred  with the second recommendation. DOD’s comments are reproduced in  appendix II.
DOD disagreed with our first recommendation, to establish an overall  technology transition goal for RIP. The department stated that  establishing a transition goal will impede the program’s objective of  encouraging innovative technologies in defense programs. In addition, the  department raised the concern that, since only a limited number of  projects are complete, it is too soon to accurately assess the overall  success of RIP and therefore, establishing a goal is not in the best  interest of the program. Further, DOD said that in line with the Secretary’s  release of the Defense Innovation Initiative in November 2014—a  department initiative to pursue innovative ways to sustain and advance  military superiority—it needs to maintain flexibility in RIP to address risky  technical requirements that may not be mature enough to transition to  acquisition programs, but may present opportunities for prototyping,  experimentation, or innovative test and evaluation. However, the  department said it would continue to measure and assess program  transition results annually.
We continue to believe that it is important for DOD to establish specific  transition performance targets for RIP and assess the extent to which the  program is successfully transitioning technologies to support acquisition  programs. While we agree that flexibility may be necessary at times in  RIP to address risky technologies that may not be mature enough for  acquisition programs, the purpose of the program is to target innovative  technologies that have the potential to support acquisition programs in the  near term. We believe that there are many other technology development  programs and activities within the department’s science and technology  enterprise with broader objectives than RIP and more closely align with  the goal of the Defense Innovation Initiative. Furthermore, as we pointed  out in this report, DOD has historically experienced problems in  transitioning technologies out of its science and technology enterprise  and into acquisition programs, and RIP was established as one  mechanism aimed at facilitating transition. Establishing a goal will not  impede, but instead help focus efforts on meeting this objective. We  recognize it is early in the program and that transition goals may need to  be adjusted as the program matures.
The department agreed on the need to identify and apply factors that  contribute to the likelihood of technology transition success more  consistently across the program. Their response identified several actions  already taken, and indicated that if Congress reauthorizes RIP the  department will continue to identify additional best practices that  contribute to transition success.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense, the Under Secretary of Defense  for Acquisition, Technology, and Logistics, and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-4841 or mackinm@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this review were to assess the extent to which (1) the  Department of Defense (DOD) has established a competitive and merit  based process to solicit and award Rapid Innovation Program (RIP)  contracts, (2) DOD has established practices to manage and oversee the  execution of RIP projects, and (3) RIP is meeting its objective of rapidly  inserting innovative technologies in defense acquisition programs.
To assess how DOD solicited and awarded RIP projects, we examined  DOD and military department policies and guidance on the program such  as DOD’s annual implementation guidelines, any internal guidance  developed by the military departments and defense agencies participating  in the program, and other policy documents. For purposes of this report,  we defined competitive awards as those using the competitive procedures  listed in Federal Acquisition Regulation (FAR) Subpart 6.1 (Full and Open  Competition). Also, we used FAR Part 35 (Research and Development  Contracting) to identify the policies and procedures for using broad  agency announcements. We analyzed the broad agency announcements  issued between 2011 and 2014 by the military departments and for the  defense components. For a sample of 40 projects awarded using fiscal  year 2011 and 2012 funding, we examined source selection documents  such as project white papers, proposals, and project scoring sheets used  in the source selection process. We interviewed officials responsible for  developing the guidance, RIP leads for each of the services and  components, and those involved in the source selection process. In  addition, we reviewed the Federal Acquisition Regulation and DOD  Source Selection Procedures. The results from the selected 40 projects  cannot be generalized to all RIP projects, but provide valuable insight.
To assess whether DOD has established practices to manage and  oversee the execution of RIP projects, we reviewed a nongeneralizable  random sample of 40 RIP projects across DOD that were awarded with  fiscal year 2011 and 2012 funding. The sample included 20 projects from  each fiscal year, with 5 projects from each of the military departments and  the defense agency component. For these projects, we reviewed project  contracts, contractor reports, and agency project assessments and  technical performance data. We also interviewed officials responsible for  high-level project oversight and management such as program leads for  the services and defense agency components in addition to contracting  officers and contracting officer’s representatives responsible for day-to- day management activities. Further, we reviewed data from DOD’s in- process review on the technical performance of ongoing RIP projects. We  did not independently assess the accuracy of technical performance data,  but reviewed and discussed the data with DOD subject matter experts  and determined it was sufficiently reliable for the purposes of this report.
To determine whether the RIP is meeting its goal of rapidly inserting  innovative technologies into acquisition programs, we reviewed available  program monitoring information and assessed the transition status of all  projects that were scheduled for completion by the end of July 2014,  which included 52 projects that were awarded with fiscal year 2011  funding. For these projects, we interviewed RIP project officials and  collected and analyzed documents outlining processes and procedures  used to manage the projects and promote transition opportunities. In  these interviews, we discussed whether projects successfully transitioned  and factors that may have helped or hindered project execution and/or  successful transition to defense acquisition programs or other military  users. We also reviewed prior GAO studies on DOD technology transition  and best practices for transition to identify what practices may facilitate  technology transition. Further, we reviewed data from DOD in-process  reviews on the status of RIP projects which provided information on the  likelihood of fiscal year 2011 and 2012 projects transitioning. We  assessed the accuracy of the transition performance data from DOD’s in- process review by comparing its results to the 52 projects that we  examined. We determined the data from the in-process review was  sufficiently reliable for the purposes of this report.
We conducted this performance audit from April 2014 to May 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, John Oppenheim (Assistant  Director), Marie P. Ahearn, LeAnna Parkey, Kenneth E. Patton, Mark F.  Ramage, Jose Ramos, Robert Swierczek, and Oziel A. Trevino made key  contributions to this report.