DOD Rapid Innovation Program: Some Technologies Have Transitioned to Military Users, but Steps Can Be Taken to Improve Program Metrics and Outcomes
Released: 2015-05-07T00:00:00

Letter

Background

To identify, pursue, and develop new technologies to improve and enhance military capabilities, DOD relies on its science and technology (S&T) community which is comprised of DOD research laboratories, test facilities, industry, and academia. The S&T community receives about $12 billion in funding each year to support activities ranging from basic research through advanced technology development that are conducted by the government or externally by universities and commercial industry. Once the S&T community has completed its technology development, additional product development activities, such as technology demonstration and testing, are often needed before incorporating the technologies into military weapon systems. Under the management of the acquisition community, product development further advances technology received from S&T developers and integrates it into systems that are ultimately delivered to support the warfighter (see figure 1).

However, as we have reported in the past, for a variety of reasons DOD historically has experienced problems in transitioning technologies out of the S&T environment and into military systems. For example, technologies may not leave the laboratory because their potential has not been adequately demonstrated or recognized, acquisition programs may be unwilling to fund final stages of development, or private industry chooses to develop the technologies itself.

DOD has a variety of technology transition programs managed by the Office of the Secretary of Defense (OSD) and the military departments that provide mechanisms and funding to facilitate technology transitions. The programs vary in size, approach, and funding, but most of them are intended to target fairly mature technologies that are suitable for the final stages of development and demonstration. Some, such as the Small Business Innovation Research (SBIR) program, have a stated purpose of using small businesses to meet federal research and development needs.feasibility and development projects, small businesses are expected to obtain funding from the private sector or government sources outside the SBIR program to commercialize or transition technologies for sale to the military or elsewhere.

However, while the program provides funding for technology Congress established RIP as another mechanism, to include consideration of innovation research projects from small businesses, as they are often viewed as a key source for developing innovative technologies in areas of military need. Specifically, Congress directed DOD to establish RIP to accelerate the transition of technologies developed by small businesses participating in SBIR phase II projects; the defense laboratories; and other small or large businesses.

The projects funded by RIP are to primarily support major defense acquisition programs, but also other defense acquisition programs that meet a critical national security need. In addition, the projects selected are to accelerate the fielding of technologies with the purpose of reducing acquisition or lifecycle costs; addressing technical risks; and improving test and evaluation outcomes. Furthermore, DOD was directed to develop a competitive, merit-based program that at a minimum provides for the use of a broad agency announcement or the use of any other competitive or merit-based processes for the solicitation of proposals; merit-based selection of the most promising cost-effective proposals for funding through contracts, cooperative agreements, and other transactions; a limit on funding for each RIP project not exceeding $3 million; and no project is funded under the program for more than 2 years.

Within DOD, the Under Secretary of Defense for Acquisition, Technology, and Logistics issues overall program guidelines, and representatives appointed by the military service acquisition executives, Assistant Secretary of Defense for Research and Engineering (ASD/R&E), and the Director, Office of Small Business Programs have the responsibility for establishing the RIP processes that support DOD’s goal and meet the guidelines. The ASD/R&E is also responsible for coordinating RIP activities among the military departments and other defense components. In addition, ASD/R&E is responsible for preparing and submitting a report, at the end of the fiscal year, to the congressional defense committees that includes a list and description of each project funded, the amount of funding provided, and anticipated timeline for transition. The program is implemented and managed by each of the military departments and by ASD/R&E, which represents the other defense components that participate in the program. To date, DOD has not included RIP in its annual budget requests because there is no formal requirement within the department for the program, but Congress has appropriated funding each year. Congress has authorized RIP until September 30, 2015, when it will expire unless further legislative action is taken.

DOD Uses a Competitive and Merit-Based Process to Select RIP Projects, with Most Awards Going to Small Businesses

DOD has established a competitive, merit-based solicitation process to select and award RIP contracts that address military needs. Each year, the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics begins by issuing broad guidance to the military departments and defense components for implementing a multi-phase process to identify military needs, solicit proposals from interested contractors, review and select projects, and award contracts. The process is somewhat lengthy, taking about 18 months to implement and award contracts, but interest from contractors has been high. Between fiscal year 2011 and 2015, the military departments and defense components received more than 11,000 white papers from interested contractors and will have awarded contracts for about 435 projects when the fiscal year 2014 solicitation is completed, with the vast majority going to small business. This high level of interest in the program has presented some administrative challenges to reducing the time from identification of need to contract award.

DOD Has Implemented a Multi-Step Process to Select Promising RIP Technology Projects

DOD has established a multi-step process to solicit, evaluate, and select RIP projects to fund (see figure 2).

Before the yearly RIP acquisition cycle begins, the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics issues implementation guidelines to the military departments and defense components which outline program goals and specific guidance on the program’s implementation and reporting requirements. For example, the guidelines include the funding available to the military departments and defense components; DOD-wide research areas, if applicable; guidance on how the solicitation process will be structured, such as whether a single solicitation will be used and the length of time it will be open; and the process for obtaining a waiver for projects expected to exceed the 24 month or $3 million limits.

Identification of Technology Needs

The RIP process begins with the military departments and defense components identifying the technology topics that will be addressed in the program. In addition to the 3 military departments, participation by defense components has increased, from 4 in fiscal year 2011 to 16 in fiscal year 2014. The topics are intended to reflect technology capability needs and requirements for weapon systems and acquisition programs, and are one of the criteria used in the selection of projects. The military departments and defense components have used different approaches to identify the technology research topics, but over time have moved to a more common approach. For example, while the Air Force and Navy have largely relied on their acquisition communities to identify technology needs, the Army relied primarily on its science and technology community to identify needs for the first 3 years of the program. For the Air Force and Navy, this entailed going directly to systems commands, Program Executive Offices, program managers, and others directly involved in the development and production of weapon systems to solicit needs. In relying on its science and technology community, the Army focused on S&T research areas that included more than the specific technology needs of acquisition programs. In fiscal year 2014, the Army expanded its approach and solicited topics from Program Executive Offices as well as from the science and technology community. The defense components used a variety of methods for identifying technology requirements. For example, the Missile Defense Agency develops research topics by reviewing technology road maps and prior SBIR research topics, and obtaining input from Program Executive Officers and program managers. The combatant command participants, such as the United States Special Operations Command (SOCOM), include operational capabilities linked to internal requirements or urgent needs.

Competitive, Merit-Based Process Used to Solicit and Award Projects

The military departments and defense components have used broad agency announcements (BAA) to solicit technical solutions that address technology topics and meet other submission requirements. The Federal Acquisition Regulation (FAR) provides that the selection of basic and applied research is a competitive procedure if an award results from (1) a broad agency announcement that is general in nature, identifying areas of research interest, including criteria for selecting proposals, and soliciting the participation of all offerors capable of satisfying the government’s needs; and (2) a peer or scientific review. The FAR also requires that BAAs specify the period of time for parties to respond, and contain instructions for preparing and submitting proposals. Also, we have found in a prior review that the use of competitive contracting procedures encourages firms to offer their best proposals when competing for work. The procedures used by the military departments and defense components for selecting RIP projects generally follow FAR competition guidelines because the BAAs identified general areas of research interest and criteria for selecting proposals; solicited the participation of all offerors capable of satisfying the government’s needs; and that subject matter experts will review project proposals.

For fiscal years 2011 to 2013, the military departments and the ASD/R&E published four separate BAAs, although they used similar selection and evaluation criteria. In fiscal year 2014, a single BAA was used to solicit proposals because officials wanted to make it easier for businesses to participate in the program and reduce some of the internal overhead involved in having to maintain four separate BAAs within DOD.

DOD has established a merit-based review process to select projects through a two-step approach—submission and review of brief white papers submitted by contractors, then comprehensive technical proposals from a subset of these contractors whose white papers are deemed most promising based on four criteria:

Contribution to Requirement—the degree to which the technical approach is relevant to the proposed requirement;

Technical Approach and Qualifications—the degree to which the technical approach is innovative, feasible, achievable, complete and supported by a technical team that has the expertise and experience to accomplish the proposed tasks;

Schedule—the degree to which the proposed schedule is achievable within 24 months from contract award; and

Cost—the degree to which the proposed cost or price is realistic for the proposed technical approach and does not exceed $3 million.

According to RIP officials, this approach is viewed positively by small businesses because they do not have to invest in developing full proposals at the outset.

Evaluation teams of subject matter experts, nominated by the military departments and defense activities, rate the white papers “go” or “no go” based on the criteria above. Greater consideration is given to the criteria pertaining to contribution to requirement and technical approach; if papers do not meet either of these two criteria they are not considered for further review. The contractors submitting white papers that are approved by the source selection authority for further consideration are then invited to submit a full proposal. According to a DOD official, more white papers are approved than can be funded because some technologies will not remain competitive when further details and support are provided in a full proposal.

Once businesses are notified that their white papers have been selected, they have 30 days to submit full proposals that amplify the information in the white paper. The military departments and defense components assign subject matter evaluators to review the proposals using the same criteria as for the white paper evaluations. DOD uses a 5-level descriptive rating scale, ranging from outstanding to unacceptable, when evaluating the proposals against the criteria. As with the white paper evaluations, RIP gives greater importance to obtaining superior technical capabilities that will transition than in making awards at lower cost to the government. Selection preference is also given to small business proposals. Selection of large businesses is allowed, but only if their offers for the same requirement are superior to those of the small businesses. Once all proposals are evaluated and ranked, the source selection authority makes the final decisions on contract awards. The number of proposals selected for contract award will depend on the level of RIP funding appropriated and the proposed project costs. The businesses are notified of the results and the military departments and defense components then negotiate and complete all contracting and award procedures with the businesses. Contracts establish the cost, schedule, and key performance parameters, and deliverables required for projects.

RIP Has Awarded a Large Number of Contracts, but Faced Administrative Challenges in Reducing Process Time

From fiscal years 2011 through 2013, the military departments and defense components awarded $760 million for 365 projects, with 89 percent of the awards going to small businesses. Program officials said that RIP attracted some new small businesses that had not worked with DOD previously. However, most of the projects selected (60-74 percent) had also received DOD funding through the SBIR program for earlier technology development activities. Table 1 presents an overview of the numbers of proposals, contract awards and dollar values, and small business participation rates for the fiscal year 2011 through 2014 project selection cycles.

Projects funded by RIP cover a broad range of technologies, including software tools and applications, measurement and testing devices, the reconfiguration of existing technologies, and the development, demonstration, or prototyping of new technologies. Examples of projects include software to improve the rate and accuracy of the transmission of data, an effort to improve the manufacturing of a component used in thermal battery insulation technology to increase the reliability and life span of missile power sources, and a compact water hand pump and filter for purifying water on the battlefield.

According to RIP officials, although the program is intended to target innovative technologies, the department’s process enables the defense components to identify requirements based on evolving operational needs and determine the kinds of projects to fund.

Since its inception, RIP has received a high level of interest from businesses─more than 11,000 white papers submitted in 4 years─which has presented some program administrative challenges. In the first year, several of the military departments and defense components had to “scramble” to complete reviews and award the contracts before program funds expired, in part because they did not have sufficient infrastructure in place to administer the program. According to a RIP official, SOCOM received 800 of the 3,600 papers submitted that year and was overwhelmed due to the high volume of white papers. Since the first year, the components have made adjustments to try and administer the process more efficiently, such as the move towards issuing a single consolidated BAA to solicit proposals in fiscal year 2014. However, the time needed to prepare the RIP implementation guidelines; identify technology topics; prepare and execute the BAAs; and review and evaluate white papers as well as complete the other steps in the solicitation and project selection process has been lengthy in each acquisition cycle, taking about 18 months to complete (see figure 3).

In 2014, RIP program officials set a qualitative goal to reduce cycle time for the fiscal year 2014 solicitation process, but said it is challenging due to the desire to employ the two-step solicitation process. For the fiscal 2015 cycle, they plan to release the BAA earlier than in previous cycles and have moved the milestone for completing this acquisition cycle forward about 30 days.

DOD Uses a Variety of Practices and Tools to Manage and Oversee Projects

The military departments and defense components used management practices and tools to manage RIP projects similar to those they use for other science and technology projects. Project managers and contracting officials review progress reports submitted by contractors and maintain regular communications with contractors to monitor whether projects are meeting cost, schedule, and performance requirements specified in the contracts. ASD/R&E established an annual DOD-wide reporting mechanism in 2013 to assess the technical performance of the projects, and most of them are meeting their technical performance metrics.

The FAR requires that contract quality assurance be performed at such times and places as may be necessary to determine that the goods or services conform to the contract requirements. The RIP project contracts describe the deliverables, reporting schedules, and financial and technical reports that the project contractor must submit. For the 40 projects we reviewed, RIP project officials and contracting officer’s representatives— the technical subject matter expert—conducted a variety of activities to manage and monitor the status of RIP projects, including continued involvement throughout contract implementation to support contractors and help ensure that the contracted services would be delivered according to the schedule, cost, quality, and quantity specified in the contract.

RIP project officials said that they manage and oversee projects in a similar fashion as other DOD technology development projects. In managing projects, officials said they review progress and financial reports submitted by the contractor, conduct system review meetings tailored to the size and complexity of the project, and engage in regular communications with contractors through e-mails, phone discussions, and occasional visits to a contractor’s facility. For example, on one project, the government team met weekly with a contractor for the first few months of the contract to review the contractor’s deliverables. In addition to the written reports, project officials also conducted quarterly program management reviews which involved an in-person meeting. A project official explained that working with small businesses presented unique challenges for oversight, because it requires the contractor to be educated on DOD requirements such as preliminary design reviews, critical design reviews, and acceptance of testing plans. Despite requiring these extra education efforts, officials said that small companies were very flexible and responsive to oversight activities. In addition to the project officials that help manage the project, the contracting officer also appointed a contracting officer’s representative to assess the contractor’s performance against contract performance standards and to record and report this information.

Oversight of RIP Projects

ASD/R&E established an annual reporting mechanism in 2013 to assess the status and progress of RIP projects. Federal government internal control standards provide that effective and efficient control activities be established to enforce management directives; these include performance measures and indicators to compare against program goals We found that RIP officials within DOD are taking these and objectives.steps. ASD/R&E collects information from project officials on the contractor’s ability to meet key technical performance parameters, using the following three-level scale to assess the performance indicators for each project:

Green—80 percent or more of the key performance parameters, goals, or thresholds will be met;

Yellow—less than 80 percent of the key performance parameters, goals, or thresholds were being met; and

Red—none of the key performance parameters, goals, or thresholds In the first in-process review, conducted in September 2013, ASD/R&E assessed the performance of the 175 ongoing projects that were funded in the fiscal year 2011 acquisition cycle. The review found that 86 percent of the projects were likely to meet 80 percent or more of their key performance parameters or goals. ASD/R&E conducted another in- process review in October 2014 and found that 85 percent of the fiscal year 2011 projects and 78 percent of the fiscal year 2012 projects were likely to meet 80 percent or more of their key performance parameters or goals (see figure 4).

In addition to the ASD/R&E in-process review, some military departments and defense components conduct their own performance reviews. For example, the Air Force obtains semi-annual status reports from project managers. These status reports include information associated with technical risks that could negatively impact a project or its technologies and additional information such as accomplishments, planned actions, and information on projects’ transition strategies. The Navy requires its system commands’ chief technology officer or designee to semi-annually update the project execution plan for their projects. Project execution plans include a description of what constitutes a project’s transition, the criteria for test events, and funding. This information enables the Navy to maintain awareness of a project’s execution and to provide assistance with transition and eventual deployment to end users if needed. Similarly, the Missile Defense Agency and SOCOM also perform periodic reviews of their RIP projects.

RIP Has Transitioned Some Projects, but It Is Too Early to Determine Overall Results

Some completed projects successfully transitioned to acquisition programs and other users, but opportunities may exist to improve overall RIP transition outcomes. DOD reported that half of all fiscal year 2011 projects (88 of 175) had funding commitments from military users, which indicates a likelihood they will transition. We also found that half of the fiscal year 2011 completed projects we reviewed (22 of 44) had technology that actually transitioned to an acquisition program, another military user, a prime contractor, or was commercialized. The majority of these projects had previously participated in the SBIR program. These transition rates are lower than what we found in our prior review of other DOD technology transition programs that reported transition rates ranging from about 55 to 85 percent.assess the overall success of RIP due to the limited number of completed projects and lack of established metrics to track whether projects have successfully transitioned to users. We also found several factors that can contribute to the transition success of RIP projects, such as user commitment and mature technology at the beginning of projects, which were not consistently emphasized by all the defense components in the program.

DOD Estimates Half of 2011 Projects Will Secure Transition Funding

As part of its annual in process review, DOD collects data from RIP project managers on the transition status of their projects. Based on the October 2014 review, DOD reported that half of fiscal year 2011 projects and 41 percent of fiscal year 2012 projects had out-year funding committed by a partner or user, which DOD uses as an indicator of the likelihood transition will occur after project completion (see figure 5).

Funding is a marker that illustrates the transition partner’s level of commitment to transition the technology. A transition partner can be a program of record, a prime contractor, or a user that is willing to dedicate funding to mature a technology past the development stage. The transition status of fiscal year 2011 and 2012 projects varied across the military departments and defense components. At the time of DOD’s review, many of the fiscal year 2012 projects were ongoing and may be up to a year away from completion. As a result, transition indicators for fiscal year 2012 projects may not reflect final outcomes. As depicted in the figure, the Air Force and the Navy reported that most of their fiscal year 2011 projects have out-year funding committed, but the Army had significantly fewer projects with a transition funding commitment. According to an Army RIP official, this may be due in part to the Army’s selection of projects that addressed a broader set of science and technology needs rather than needs defined by the acquisition community, where partner funding commitments would be more likely. For projects that did not have a commitment from a transition partner, the military departments and defense components reported that in several cases, the expected follow-on acquisition, procurement or support funds were redirected to higher priorities or requirements for the technologies were cancelled while the projects were underway.

Half of Completed Projects We Reviewed Successfully Transitioned

To gain additional insights into the range of transition outcomes for RIP projects, we assessed all 52 fiscal year 2011 projects scheduled for completion through July 2014, to determine their transition results. We found that 44 of these projects had been completed, meaning the contract period of performance ended. Although projects are typically required to be completed within 2 years, several of them had encountered delays and obtained no-cost schedule extensions of 6 months or more from RIP officials to complete the remaining work. Officials told us extensions were granted for various reasons: completing additional testing; better aligning projects with acquisition program schedules; and further demonstrating project technology. Of the 44 projects that were completed, half had technology that successfully transitioned to an acquisition program, a military user, a prime contractor, or became a commercially available product. Table 2 depicts these outcomes, including the range of scenarios for projects that did not transition.

The following are a few specific examples of RIP projects that successfully transitioned, demonstrating the array of technologies and the users supported by these technology transitions:

Wireless Vibration Recorder: This Navy project provides a compact and lightweight measurement device that can acquire data quickly and easily on aircraft, reducing flight test costs and system development time. The device includes specialized software and sensors to acquire vibration and acceleration data for aircraft internal components to determine why and when components fail. The Navy currently utilizes a test instrumented aircraft to measure vibrations, which is time consuming and costly. RIP officials reported this technology is expected to save the Naval Air Systems Command $3 million to $5 million in the next 4 years in flight test costs. This technology is also available commercially.

Enhanced Ground Moving Target Indicator (GMTI)-based Intelligence, Surveillance, and Reconnaissance: This Air Force project provides software applications to be used by joint surveillance target attack radar system operators. The applications are intended to interact with GMTI data to facilitate pattern recognition, automatically identify targets and perform analyses over any set period of time. For example, instead of a dot on the water, a target could be automatically identified as a carrier or a person in a rowboat so the analyst can determine if they need to send a camera to the area to investigate further. RIP officials reported the Air Force Research Laboratory has installed the software suite on a data repository that can be used by a wide variety of intelligence analysts.

Multi-Missile Common Launch Tube: This United States Special Operations Command project increases the number of munitions that can be carried and launched from a single common launch tube. This technology doubles the number of targets that can be attacked and allows the use of smaller warheads with precision delivery, which can minimize collateral damage. RIP officials reported that ground and flight tests were successful, and the technology has been transitioned into the Command’s common launch tube program.

Automated Intelligent Training with a Tactical Decision-Making Serious Game: This Army-funded project enhances a software tool for training Army officers in the classroom or field. The enhanced tool improves upon a capability to practice key leader cognitive skills, such as the ability to rapidly assess a dynamic situation, make sound decisions, and effectively direct subordinate units through scenario- based exercises. According to RIP officials, the software was delivered to end users at West Point and future Army leaders are using this software to improve their tactical command skills.

Mine Roller Wheel Assembly Improvement: This project improves effectiveness of the Marine Corps mine roller wheel assembly over the legacy system which makes it more effective at neutralizing threats. A mine roller is used to detonate and clear certain classes of buried, pressure-activated explosive threats. This upgrade is expected to be less expensive than the cost of the original roller wheel assembly it is modifying while increasing its effectiveness at a greater range of speeds. In February 2015 the Marine Corps established an acquisition program known as the Wheelbank Suspension Upgrade Program to purchase the upgraded mine roller wheel assembly and is now working toward a production decision expected in the third quarter of fiscal year 2015.

In contrast, we found several different reasons projects did not transition. DOD officials told us it may take a year or more to transition a project because project completion may not easily line up with user timelines or the DOD budget cycle. A little more than half had successfully demonstrated the technology as planned, but had no user commitment. According to RIP project managers, some of the projects provided value to DOD and may yet transition in the future. One of the Navy projects, for example, which provides a technology to filter out interference in certain military radios, has an interested transition partner, but has not yet transitioned due to funding constraints. Officials reported they were unable to secure funding in the fiscal year 2015 and 2016 budgets, but have submitted another request for fiscal year 2017. DOD officials also said that several projects failed to transition due to changing user priorities and requirements, such as the phase-out of ongoing missions in Iraq and Afghanistan. Further, according to DOD officials, other projects demonstrated a proof of concept or provided data to develop requirements, but were not planned to transition into a program of record. For example, one Army project delivered a prototype field kitchen that demonstrated potential capabilities, and the results will be used to inform future requirements for a program of record planned in 2019.

Other RIP projects that did not transition needed further technology development or additional testing—such as software that needed security accreditation—before they could be integrated into a DOD program. For example, a Navy-funded project for a torpedo array nose assembly, designed to improve the array’s performance in shallow water, needs additional testing to complete the array qualification effort. If successful, this array nose assembly is expected to provide a second source at a lower cost than the current array.

On the other hand, officials told us some of the projects did not meet user requirements or did not demonstrate that their technology was better than existing technology. For example, a SOCOM-funded project to develop an upgraded antenna that could operate in the presence of jamming did not meet a performance standard. Also, an Air Force project to test capabilities of a spectral flare showed the flare did not meet all of the requirements and did not offer significant improvement in performance compared to the current flare. Officials said that although projects may appear unsuccessful, in some cases they provided valuable information that could improve future decisions.

Mature Technology, User Commitment, and Program Support Contribute to Successful Transition; However, DOD Has Not Taken Steps to Evaluate This

In our past work on DOD technology transition activities, we found that several factors are important to successful transition, including the selection of projects that have sufficiently mature technologies and early endorsement from intended military users. In addition, once projects are selected, they require effective management processes to support technology transition. RIP officials and project managers that we interviewed also noted that these factors can contribute to successful transition. In general, however, at the DOD level there has not been an effort to understand the extent to which these factors may be contributing to differences in transition and to communicate lessons learned from military departments or defense components with a higher percentage of transition success. Without such an understanding, DOD may be missing opportunities to leverage practices from certain program components that could help improve overall transition success rates.

Technology Maturity

GAO-13-286. to review the support for the technical approach presented in proposal submissions, including information on TRLs.

User Commitment

We previously reported that early project endorsement from intended users and other key stakeholders in the acquisition community is important for project transition. RIP officials also agreed that having a transition partner, such as a specific program of record, identified prior to the beginning of the project contributes to transition success. Other RIP officials said user participation in ongoing project reviews further ensures the final product is aligned with user requirements. In the required reporting to Congress, all of the projects we reviewed identified an acquisition program or potential user as the project started; however, we found the level of user interest and commitment varied and in some cases changed during project execution.

Program Support

The military departments and defense components varied in the approach and level of support they used to manage RIP technology transition. According to an OSD-level RIP official, the Navy and the Air Force have more robust processes to support transition than the other components. For example, the Navy and the Air Force provide additional internal guidance for RIP participants that describe the roles and responsibilities of participants. These departments also require project reviews every 6 months until project completion and stress the importance of lessons learned for RIP project managers. For example, the Air Force Life Cycle Management Center’s guidance states that analysis of RIP outcomes at the conclusion of each project cycle can identify lessons learned from processes, successes, challenges, and recommendations that can help build the program knowledge base and establish best practices in program implementation and stakeholder relations.

The Navy’s guidance describes the importance of quality control, consistent management, and continuous improvement to help avoid demonstrating a technology that has nowhere to transition. The Navy also uses project execution plans that describe transition plans, codify transition partner agreements, and describe criteria to evaluate seminal transition events. In addition, the Office of Naval Research’s risk management team can provide support for small businesses to stay on track in fulfilling RIP contracts, including making sure companies can ramp up production if their projects are transitioned. We have previously reported that this office has a well-established technology transition focus which may contribute to project success. Because of this, the Navy may be better aware of the benefits and obstacles associated with a substantial portion of their S&T portfolio. This knowledge can better inform investment decisions made by Navy leadership.

RIP Does Not Track Final Transition Outcomes

While the program has established metrics to determine whether projects have a funding commitment from users and are therefore likely to transition, it does not track the degree to which completed projects have actually transitioned. RIP officials view transition success broadly, as a technology which is inserted into an acquisition program of record, incorporated into a weapon system manufacturing process, adopted for use by a depot or logistics center, or available for purchase on the General Services Administration federal supply schedule or in the commercial market. However, RIP does not formally track projects beyond completion and whether they are inserted into acquisition programs or used for other purposes, which limits DOD’s ability to know the final transition outcomes and whether any benefits were achieved. As we have found in the past, tracking and measuring technology transition and the impact of those transitions, such as cost savings or deployment of a technology into a weapon system, provides key feedback that can inform the management of programs such as RIP. We previously recommended the Secretary of Defense require all technology transition programs in the department to track and measure project outcomes— DOD including long term benefits for acquisition programs or users.generally agreed with our recommendations but we found no evidence that DOD has taken any action yet; this recommendation would also apply to RIP, as it is a technology transition program. In addition, DOD has not established an overall transition goal for RIP, so it is not clear what is expected in terms of success. Federal government internal control standards require that effective and efficient control activities be established which include performance measures and indicators to compare against program goals and objectives.DOD annually provides a report to Congress on the number and description of projects that are funded through RIP, the department is not required to provide information on transition results. Without this information, Congress lacks insight about the program’s performance which is important for conducting program oversight.

In our discussions with DOD RIP officials, they estimated that about 50 percent or more of the RIP projects will transition when completed, although they said it is too early to accurately assess the overall effectiveness of the program. Some officials indicated that there are not enough projects completed and it could take two or more years after a project is completed before it is successfully transitioned and used. In addition, projects awarded in RIP’s first acquisition cycle may not accurately reflect program performance for the following years, in part because there were implementation challenges associated with starting a new program. Therefore, more data that captures final project outcomes and long-term program experience is required to accurately assess performance. In our prior review of DOD technology transition programs that provide structured mechanisms and funding to facilitate technology transition, we found that programs reported rates of technology transition ranging from about 55 to 85 percent. percent is at the lower end of this range, as is experience to date with the transition of completed projects.

Conclusions

GAO-13-286. report program results to Congress, as this is not currently required. Consequently, without an overall transition performance target and better measure of outcomes, it is unclear whether RIP has been successful at transitioning innovative technologies and is worth continuing. We continue to believe that the recommendation we made in 2013—for DOD to track and measure the outcomes of its numerous technology transition programs in order to improve the visibility and management of these efforts—has merit and is applicable to RIP.

In addition, although the military departments and defense components have implemented a structured process for soliciting projects, they do not always select projects that have a high likelihood of successfully transitioning, such as those with mature technologies or commitments from transition partners. DOD components are capitalizing on these success factors to varying degrees. The Air Force and Navy, in particular, incorporate them in RIP projects and, as a result, appear to be realizing significantly higher transition rates for their projects, even in the program’s early years. Without a more consistent focus on these factors during project selection, opportunities to achieve higher levels of transition success for RIP may remain limited.

Matter for Congressional Consideration

Congress should consider, if it decides to re-authorize RIP, requiring DOD to submit annual reports to Congress on the transition results of the program to improve accountability and transparency of the program.

Recommendations for Executive Action

If Congress re-authorizes RIP then, to improve visibility and management of DOD’s ability to transition technologies through the program, we recommend that the Secretary of Defense direct the Under Secretary of Defense for Acquisition, Technology, and Logistics to take the following two actions:

Establish an overall technology transition goal for RIP; and Identify and apply factors that contribute to the likelihood of technology transition success more consistently across the program.

Agency Comments and our Evaluation

We provided a draft of this report to DOD for comment. In its written comments, DOD disagreed with our first recommendation and concurred with the second recommendation. DOD’s comments are reproduced in appendix II.

DOD disagreed with our first recommendation, to establish an overall technology transition goal for RIP. The department stated that establishing a transition goal will impede the program’s objective of encouraging innovative technologies in defense programs. In addition, the department raised the concern that, since only a limited number of projects are complete, it is too soon to accurately assess the overall success of RIP and therefore, establishing a goal is not in the best interest of the program. Further, DOD said that in line with the Secretary’s release of the Defense Innovation Initiative in November 2014—a department initiative to pursue innovative ways to sustain and advance military superiority—it needs to maintain flexibility in RIP to address risky technical requirements that may not be mature enough to transition to acquisition programs, but may present opportunities for prototyping, experimentation, or innovative test and evaluation. However, the department said it would continue to measure and assess program transition results annually.

We continue to believe that it is important for DOD to establish specific transition performance targets for RIP and assess the extent to which the program is successfully transitioning technologies to support acquisition programs. While we agree that flexibility may be necessary at times in RIP to address risky technologies that may not be mature enough for acquisition programs, the purpose of the program is to target innovative technologies that have the potential to support acquisition programs in the near term. We believe that there are many other technology development programs and activities within the department’s science and technology enterprise with broader objectives than RIP and more closely align with the goal of the Defense Innovation Initiative. Furthermore, as we pointed out in this report, DOD has historically experienced problems in transitioning technologies out of its science and technology enterprise and into acquisition programs, and RIP was established as one mechanism aimed at facilitating transition. Establishing a goal will not impede, but instead help focus efforts on meeting this objective. We recognize it is early in the program and that transition goals may need to be adjusted as the program matures.

The department agreed on the need to identify and apply factors that contribute to the likelihood of technology transition success more consistently across the program. Their response identified several actions already taken, and indicated that if Congress reauthorizes RIP the department will continue to identify additional best practices that contribute to transition success.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Under Secretary of Defense for Acquisition, Technology, and Logistics, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4841 or mackinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The objectives of this review were to assess the extent to which (1) the Department of Defense (DOD) has established a competitive and merit based process to solicit and award Rapid Innovation Program (RIP) contracts, (2) DOD has established practices to manage and oversee the execution of RIP projects, and (3) RIP is meeting its objective of rapidly inserting innovative technologies in defense acquisition programs.

To assess how DOD solicited and awarded RIP projects, we examined DOD and military department policies and guidance on the program such as DOD’s annual implementation guidelines, any internal guidance developed by the military departments and defense agencies participating in the program, and other policy documents. For purposes of this report, we defined competitive awards as those using the competitive procedures listed in Federal Acquisition Regulation (FAR) Subpart 6.1 (Full and Open Competition). Also, we used FAR Part 35 (Research and Development Contracting) to identify the policies and procedures for using broad agency announcements. We analyzed the broad agency announcements issued between 2011 and 2014 by the military departments and for the defense components. For a sample of 40 projects awarded using fiscal year 2011 and 2012 funding, we examined source selection documents such as project white papers, proposals, and project scoring sheets used in the source selection process. We interviewed officials responsible for developing the guidance, RIP leads for each of the services and components, and those involved in the source selection process. In addition, we reviewed the Federal Acquisition Regulation and DOD Source Selection Procedures. The results from the selected 40 projects cannot be generalized to all RIP projects, but provide valuable insight.

To assess whether DOD has established practices to manage and oversee the execution of RIP projects, we reviewed a nongeneralizable random sample of 40 RIP projects across DOD that were awarded with fiscal year 2011 and 2012 funding. The sample included 20 projects from each fiscal year, with 5 projects from each of the military departments and the defense agency component. For these projects, we reviewed project contracts, contractor reports, and agency project assessments and technical performance data. We also interviewed officials responsible for high-level project oversight and management such as program leads for the services and defense agency components in addition to contracting officers and contracting officer’s representatives responsible for day-to- day management activities. Further, we reviewed data from DOD’s in- process review on the technical performance of ongoing RIP projects. We did not independently assess the accuracy of technical performance data, but reviewed and discussed the data with DOD subject matter experts and determined it was sufficiently reliable for the purposes of this report.

To determine whether the RIP is meeting its goal of rapidly inserting innovative technologies into acquisition programs, we reviewed available program monitoring information and assessed the transition status of all projects that were scheduled for completion by the end of July 2014, which included 52 projects that were awarded with fiscal year 2011 funding. For these projects, we interviewed RIP project officials and collected and analyzed documents outlining processes and procedures used to manage the projects and promote transition opportunities. In these interviews, we discussed whether projects successfully transitioned and factors that may have helped or hindered project execution and/or successful transition to defense acquisition programs or other military users. We also reviewed prior GAO studies on DOD technology transition and best practices for transition to identify what practices may facilitate technology transition. Further, we reviewed data from DOD in-process reviews on the status of RIP projects which provided information on the likelihood of fiscal year 2011 and 2012 projects transitioning. We assessed the accuracy of the transition performance data from DOD’s in- process review by comparing its results to the 52 projects that we examined. We determined the data from the in-process review was sufficiently reliable for the purposes of this report.

We conducted this performance audit from April 2014 to May 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, John Oppenheim (Assistant Director), Marie P. Ahearn, LeAnna Parkey, Kenneth E. Patton, Mark F. Ramage, Jose Ramos, Robert Swierczek, and Oziel A. Trevino made key contributions to this report.