Small Business Innovation Research: DOD's Program Supports Weapon Systems, but Lacks Comprehensive Data on Technology Transition Outcomes
Released: 2013-12-20T00:00:00

Letter

Background

DOD relies on its science and technology community—DOD research laboratories, test facilities, industry, and academia—to identify, pursue, and develop new technologies that address military needs. The DOD SBIR program is one mechanism through which DOD attempts to accomplish its science and technology goals and develop technologies that contribute to weapon systems or transition directly to warfighters for use in the field. Within DOD, the Office of Small Business Programs oversees the department’s SBIR program activities, develops policy, and manages program reporting. This office generally relies on the agencies, such as the Army, Air Force, and Navy, to oversee and execute their own SBIR program activities. Each agency has flexibility to tailor its SBIR program to meet its needs, including determining what type of research to pursue, which projects to fund, and how to monitor ongoing projects.

To initiate the project award process, SBIR programs work with the science and technology and acquisition communities to generate and prioritize research and development topics. These topics describe technical areas of interest and capability needs, which the programs use in their solicitations for proposals from small businesses. DOD conducts three solicitations each year where small businesses compete for Phase I contract awards that are expected to respond to the needs identified in each topic. Once awarded, SBIR projects are managed through a three- phase program structure, which is outlined in table 1.

The number of Phase I and Phase II projects varies from year to year based on technology needs and funding availability. Table 2 shows the budgets and project awards reported for the military department SBIR programs in fiscal year 2012.

We and others have previously found that DOD and its technology development programs, such as the SBIR program, have encountered challenges to transitioning their technologies to acquisition programs or directly to the warfighter for use in the field. For instance, in our past work we found several reasons why technologies may not transition, including insufficient maturity, inadequate demonstration or recognition by users of a technology’s potential, and unwillingness or inability of acquisition programs to fund final stages of development.

To address SBIR technology transition challenges, DOD, the Small Business Administration, and Congress have established additional program provisions, incentives, and reporting requirements. For example, the Commercialization Readiness Program was initiated to accelerate the transition of SBIR funded technologies to Phase III, especially those that lead to acquisition programs and high priority military requirements, such as fielded systems. military departments to use up to 1 percent of SBIR funding for administrative activities that facilitate transition. This funding is used to support program staff and contractors who provide assistance to SBIR awardees, including efforts to enhance networking and build relationships among small businesses, prime contractors, and DOD science and technology and acquisition communities.

The National Defense Authorization Act for Fiscal Year 2006 authorized the Commercialization Pilot Program under the Secretary of Defense and the Secretary of each military department. Pub. L. No. 109-163, § 252. The National Defense Authorization Act for Fiscal Year 2012 continued the program and renamed it the Commercialization Readiness Program. Although the program may support any Phase III awards, such as technology transition to commercial products, DOD is required to provide goals to increase the number of Phase II SBIR contracts that lead to technology transition into programs of record or fielded systems and to use incentives to meet those goals. Pub. L. No. 112-81, § 5122(a).

Requires DOD to set a goal to increase the number of Phase II contracts awarded that lead to technology transition into acquisition programs or fielded systems, and use incentives to encourage program managers and prime contractors to meet the goal.

Requires that DOD report specific transition-related information to the Administrator of the Small Business Administration for inclusion in an annual report to designated congressional committees. This includes reporting the number and percentage of Phase II contracts that led to technology transition into acquisition programs or fielded systems, information on the status of each project that received funding through the Commercialization Readiness Program and efforts to transition those projects, and a description of each incentive used to meet the department’s transition goal.

Authorizes DOD to establish goals for the transition of Phase III technologies in subcontracting plans for contracts of $100 million or more, and to require prime contractors on such contracts to report the number and dollar amount of contracts entered into by prime contractors for Phase III projects.

Sets the ceiling for discretionary technical assistance that can be provided annually for all Phase I and Phase II projects at $5,000 per project. Programs can use this funding to assist awardees in making technical decisions on projects, solving technical problems, minimizing technical risks, and commercializing projects.

Establishes a pilot effort to allow DOD SBIR programs to use not more than 3 percent of their SBIR budgets for, among other things, program administration, technical assistance, and the implementation of commercialization and outreach initiatives.

Military Department SBIR Programs Use a Variety of Practices, Tools, and Administrative Funds to Support Technology Transition Efforts

The military department SBIR programs use several management practices and tools to support technology transition efforts. We identified some common transition elements across the programs, but also found some differences in how each program approaches its technology transition efforts. The programs’ technology transition efforts are supported through use of administrative funds coming from their SBIR budgets and other funds provided by their respective military department. The transition facilitation practices, tools, and funds used to promote the transition of SBIR technologies include the following: Early focus on transition through topic generation and project selection: Technology transition efforts begin with topic generation and project selection processes that emphasize the pursuit of projects for which there is a demonstrated military need and potential transition opportunities. To do this, the military department programs formally engage stakeholders from the science and technology and acquisition communities in generating and endorsing topics for SBIR solicitations. In proposing topics and selecting projects, programs have to balance their desire for technological innovation with meeting pressing warfighter needs. SBIR officials stated that projects that pursue incremental improvements generally are more likely to deliver the technical capability expected for technology transition to occur. In contrast, they noted that projects that focus more on “leap-forward” technology innovations that can support future warfighting needs tend to require more long-term development and have greater technical and transition risks. DOD policy requires that at least 50 percent of military department topics are endorsed by the acquisition community, such as program executive offices. This helps ensure that the acquisition community is engaged with the SBIR programs and that a significant portion of projects are dedicated to addressing specific needs identified by military users.

Phase II transition initiatives: Transition-focused activities increase as Phase II projects progress, commensurate with an increasing technology maturity and understanding of a project’s potential opportunities for use. In particular, the military department SBIR programs target transition opportunities through their Commercialization Readiness Programs and other initiatives that provide additional support to select Phase II projects. In some cases, the SBIR programs require formal technology transition agreements or matching funding as a condition to receiving additional Phase II funding. Technology transition agreements, which Air Force and Navy officials reported using, help manage project expectations and formalize stakeholder commitments by outlining cost, schedule, and performance expectations for transition to occur. Matching funds from intended users, which are required by the Navy for some projects, can help create greater buy-in for transition because the intended users have a monetary stake in the project.

Transition facilitators: Each military department SBIR program has a network of transition facilitators who manage the Commercialization Readiness Program and other enhancement efforts, as well as broader SBIR activities that support technology transition. The facilitators are located at military labs, acquisition centers, and program executive offices to work directly with government stakeholders and help ensure projects are responsive to warfighter needs. They also help small businesses identify and position themselves for opportunities to transition their SBIR technologies. Although the roles and responsibilities vary somewhat across the programs, in general, transition facilitators assist with topic generation and prioritization; foster communication among small businesses, research laboratories, and the acquisition community in support of transition opportunities; and monitor project progress, including outcomes.

Navy Transition Assistance Program: The Navy established an additional program over a decade ago to prepare its SBIR participants for technology transition opportunities. The Transition Assistance Program is a voluntary 11-month program with, on average, about two-thirds of Phase II recipients participating each year. It provides consulting services focused on improving the small businesses’ abilities to transition their SBIR products, including assistance in transition planning and developing marketing tools. Under the program, profiles are used to describe the expected capability, level of technology maturity, and potential technology transition opportunities for each project. These profiles are available in electronic form through a web-based portal called the Navy Virtual Acquisition Showcase, and support the annual Navy Opportunity Forum conference. The conference provides Transition Assistance Program participants with direct exposure and one-on-one opportunities to interact with prospective transition partners in the government and industry.

Other transition facilitation tools: SBIR programs also use technology roadmaps and formal relationship-building activities, such as conferences and workshops, to support transition efforts. Technology roadmaps are schedule-based planning documents used to identify opportunities for SBIR technology insertion into acquisition programs or direct use by the warfighter. Conferences and workshops, such as the annual Beyond Phase II conference hosted by the Office of Small Business Programs, are used by the programs to provide opportunities for SBIR Phase II companies to interact directly with prospective government and industry users and showcase their projects.

Administrative funds: The technology transition practices and tools used by the programs are supported by administrative funds provided through their SBIR budgets as well as non-SBIR sources from their respective agencies. The Commercialization Readiness Program and discretionary technical assistance provisions enable programs to use portions of their SBIR budgets to fund administrative activities, including transition support. For fiscal year 2012, this funding totaled about $12 million across the three military departments. The fiscal year 2012 SBIR reauthorization included a new provision—which DOD officials advocated—that allows the programs to use up to 3 percent of their funds to support administrative activities, which is in addition to funds available through the Commercialization Readiness Program and discretionary technical assistance. SBIR officials stated that although this additional funding allowance is in the initial stages of being used, they believe these funds will help enhance transition facilitation measures for their programs going forward. Additional agency funding outside of the SBIR budget is also used to manage programs and support transition activities, but the amount of such funding is not readily identifiable because the military departments do not all require that the amount of funding used to support associated administrative efforts be documented.

DOD Lacks Comprehensive Data on SBIR Technology Transition Outcomes, but Some Transition Successes Have Been Identified

We were unable to assess the extent of technology transition associated with the military department SBIR programs because comprehensive and reliable technology transition data are not collected. Tracking mechanisms used by DOD—Company Commercialization Reports (CCR) and the Federal Procurement Data System-Next Generation (FPDS- NG)—provide some information on SBIR Phase III activities, but these mechanisms have significant gaps in coverage and data reliability concerns that limit their transition tracking capabilities. The military departments have additional measures through which they have identified a number of successful SBIR transitions to DOD acquisition programs and directly to fielded systems, but these efforts capture a limited amount of transition information. DOD is assessing how to comply with the new transition reporting requirements directed by Congress, but has yet to develop a plan that will support identification and annual reporting of the extent to which SBIR technologies transition to DOD acquisition programs or to fielded systems.

SBIR Programs Use Three Resources to Identify Transition Results, but Each Has Inherent Data Limitations

The military department SBIR programs rely, to varying degrees, on two data systems—CCR and FPDS-NG—as well as their own agency-specific data collection activities to identify transition results. Table 3 more fully describes the data sources used and their limitations.

Although the CCR and FPDS-NG data systems do not capture complete data on the transition of SBIR technologies, they do provide high-level commercialization information that the SBIR programs use to track progress in achieving program goals. Because the data help support program management efforts, the Office of Small Business Programs and the military departments, to varying degrees, take steps to verify the quality of CCR and FPDS-NG data. For example, the Army assesses and validates CCR data for its projects on an ongoing basis. This process involves comparing recent updates to the database with FPDS-NG contract data and internal Army tracking data to confirm the accuracy of commercialization funding reported by the small businesses. The Navy SBIR program uses FPDS-NG as its primary source of commercialization data and employs similar validation techniques to improve the accuracy of commercialization data tracked through this system. By comparing contracts in FPDS-NG flagged as SBIR-related to DOD contract management systems, the Navy is able to verify the accuracy of Phase III awards data tied to government contracts. Both the Army and Navy officials acknowledged, however, that even with their data validation efforts, problems persist because of the limitations of the Company Commercialization Reports and FPDS-NG.

The military department programs have developed some internal capabilities to track certain projects and provide insight into the types of capabilities enabled by them. Like Company Commercialization Reports and FPDS-NG, these capabilities do not provide comprehensive transition information, but may help the departments to gain more insight into transition outcomes for some technologies developed within SBIR programs and to respond to DOD and congressional inquiries about program results. In particular, the programs identify transition success stories for a limited number of projects, ranging from Phase III awards for additional research and development to transition to major acquisition programs or fielded systems. Information on these success stories can come from SBIR program officials, acquisition program officials, prime contractors, or directly from the small businesses. The Air Force’s database of identified transition successes includes 95 transition stories dating back to 2004. The Army’s program produces an annual report describing transition outcomes for 20-30 successful projects. The Navy’s program maintains a searchable database of SBIR projects that includes profiles on select transitioned projects as well. Table 4 provides examples of transition outcomes for projects identified through our review of these reporting mechanisms.

SBIR program officials within the military departments emphasized that, in addition to their broader program efforts to identify transition outcomes, some acquisition organizations have implemented their own practices to track transition. For example, the Navy Program Executive Office for Submarines tracks the transition of SBIR technologies to its acquisition programs by managing a list of companies, the value of contract awards, the specific program office associated with each contract award, and the SBIR technology associated with the award. The office indicated that 20 active Phase III awards associated with its acquisition program efforts are being tracked.

DOD Is in the Early Stages of Developing a Plan to Improve Technology Transition Tracking and Reporting

The National Defense Authorization Act for Fiscal Year 2012 mandated that DOD report new transition-related information to the Administrator of the Small Business Administration who will report this information annually to designated congressional committees. This reporting will include information on the number and percentage of Phase II projects that transition into acquisition programs or to fielded systems, the efficacy of steps taken by DOD to increase the number of transitioned projects, and additional information specific to the transition of projects funded through Commercialization Readiness Programs. In order to provide more complete and accurate transition data to support the new reporting requirements, DOD recognizes it may need to modify its existing data systems or develop new tools to better capture the transition results for SBIR projects.

According to the Office of Small Business Programs, DOD’s response to the new reporting requirements is still being evaluated, in part because there are several challenges to compiling complete and accurate technology transition data. One such challenge we found was variation across the military departments in their definitions of technology transition. Specifically, transition definitions ranged from any commercialization dollars applied to a project, to only when a technology is actually incorporated into a weapon system or in direct use by the warfighter. The Office of Small Business Programs acknowledged that a standard DOD SBIR definition of technology transition must be ensured before the congressionally-required reporting begins. Standards for internal control state that management should establish procedures to ensure that it is able to achieve its objectives, such as being able to compile and report consistent, complete, and accurate data.according to SBIR officials, tracking transition outcomes can be Additionally, challenging because the sometimes lengthy period between SBIR project completion and transition to a DOD user can obscure a project’s SBIR linkages. Time lags can occur because of delays in transition funding availability, additional development or testing needs before transition, or schedule delays encountered by intended users. During the time between project completion and transition, personnel associated with projects may change and technologies may evolve. This increases the likelihood that transitions associated with SBIR technologies go unacknowledged. SBIR officials within the military departments also stated that limited resources for administrative activities constrain their ability to effectively follow up on the transition outcomes for completed projects.

Although the Office of Small Business Programs acknowledges the limitations of CCR data, the initial plan is to use this data source—viewed by DOD as the best available—as the primary means for beginning to address the new transition reporting requirements. Additionally, in an effort to improve DOD’s future technology transition reporting and its understanding of transition results in general, the Office of Small Business Programs has initiated an assessment of different options for enhancing transition data. For example, as part of this assessment, DOD is examining whether CCR could be modified to improve reporting. Additionally, existing DOD reporting mechanisms, such as Selected Acquisition Reports—annually required for major defense acquisition programs—are being considered as potential vehicles for supporting SBIR technology transition reporting. Opportunities to build more SBIR awareness directly into acquisition activities are being considered as well, such as including provisions in acquisition strategy documents or formal program reviews. According to the Office of Small Business Programs, DOD intends to issue a policy directive in fiscal year 2014 that will provide guidance for implementing overall SBIR program requirements. However, SBIR officials indicated that addressing technology transition reporting requirements is viewed as a longer-term effort because of the challenges we have discussed, and no specific plan including a time line has been established for when DOD will be able to support those requirements. Without a plan that establishes a time line, it is unclear how and when DOD will begin to provide the technology transition information expected by Congress. Although Congress did not specify when reporting was to begin, it expects DOD to report new transition-related information to the Administrator of the Small Business Administration to meet the National Defense Authorization Act for Fiscal Year 2012 requirement. However, as stated above, DOD expects this to be a longer-term effort and designated congressional committees may not be aware of when DOD will likely have developed the capability to provide comprehensive and accurate data.

Further, unless DOD communicates its plan and accompanying time line, these committees may be unaware that the transition-related information DOD plans to provide in the near-term to address the National Defense Authorization Act for Fiscal Year 2012 requirements has data quality issues. Standards for internal control emphasize the need for federal agencies to establish plans to help ensure goals and objectives can be met, including compliance with applicable laws and regulations. Further, communicating internal control efforts on a timely basis to external stakeholders, such as congressional committees, helps ensure that effective oversight can take place.

Conclusions

The SBIR program efforts within DOD provide opportunities for small businesses to develop new technologies that may improve current U.S. military capabilities and provide innovative solutions to address future needs of the warfighter. However, information on technology transition outcomes for SBIR projects is limited. Consequently, DOD cannot identify the extent to which the program is supporting military users. The Office of Small Business Programs is taking steps to respond to new technology transition reporting requirements, but has not yet determined how and when it will more completely and reliably track and report on the extent of transition for SBIR technologies. While initial reporting efforts are expected to use existing data systems, such as CCR, DOD will need to overcome the inherent limitations of data collected through those systems if it expects to provide a comprehensive picture of transition outcomes.

Recommendations for Executive Action

To improve tracking and reporting of technology transition outcomes for SBIR projects, we recommend that the Secretary of Defense direct the Office of Small Business Programs to take the following three actions: 1. Establish a common definition of technology transition for all SBIR projects to support annual reporting requirements; 2. Develop a plan to meet new technology transition reporting requirements that will improve the completeness, quality, and reliability of SBIR transition data; and 3. Report to Congress on the department’s plan for meeting the new SBIR reporting requirements set forth in the program’s fiscal year 2012 reauthorization, including the specific steps for improving the technology transition data.

Agency Comments and Our Evaluation

We provided a copy of a draft of this report to DOD for review and comment. Written comments from the department are included in appendix II of this report.

DOD partially concurred with our recommendations. In its response, DOD stated that it has established a working group that is currently working with all stakeholders to develop a common definition of technology transition for all SBIR projects. DOD also agreed that it is important to improve the completeness, quality, and reliability of SBIR transition data, but noted that it has significant concerns related to the difficulty in actually capturing the data. The department indicated that the full scope of data collection challenges and associated resource needs is unknown at this time.

While we recognize there are challenges to improving transition data, we believe there are avenues already available that DOD could pursue to improve transition data that may not require extensive resource commitments. For example, DOD’s SBIR program could work more closely with its acquisition community to track transition outcomes. As outlined in this report, some acquisition organizations have developed their own practices to track transition outcomes, which the program may be able to leverage for use on a broader scale. In addition, DOD could consider greater use of contracting provisions to require contractors to report on SBIR project activities, or use existing program reporting mechanisms, such as Selected Acquisition Reports, to capture additional transition information.

We believe that collection of better data is not only needed to support the congressional reporting requirements, but also to help DOD assess the efficacy of existing transition efforts and the benefits the program yields for the warfighter. DOD stated it will continue with initiatives that seek to improve the collection of SBIR technology transition data. However, it did not specify if or when it intends to develop a plan for meeting the transition reporting requirements. We continue to believe a plan that includes a time line for when DOD will begin to support reporting requirements should be provided to the designated congressional committees in the near term to make clear the limitations of reported transition data and the department’s approach to improving the data over time.

We are sending copies of this report to appropriate congressional committees and the Secretary of Defense. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or by email at sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Scope and Methodology

To identify what processes are used by the Department of Defense (DOD) to facilitate transition for Small Business Innovation Research (SBIR) technologies, we reviewed prior reports by GAO, DOD and other organizations, such as the National Research Council and the Rand Corporation, as well as reviewed DOD policies, procedures, and funding information. Using this information, we scoped our work to focus on the SBIR activities conducted by the Air Force, Army, and Navy. These three organizations typically receive about three-fourths of the annual SBIR funding that supports the 13 participating DOD organizations. With the military department SBIR programs as our focus, we interviewed DOD officials from the Office of Small Business Programs and the SBIR program offices at the Air Force, Army, and Navy on practices and tools used to facilitate technology transition. In addition, we interviewed and collected documentation from DOD officials within the acquisition community concerning their use of and interactions with the SBIR program. Specifically, we interacted with officials at the Air Force Life Cycle Management Center and Air Force Research Laboratory; the Army Aviation and Missile, Research, Development, and Engineering Center; the Naval Sea Systems Command; and the F-35 Joint Program Office. This included interviewing SBIR program management and transition facilitation personnel at each location, as applicable.

Similarly, to assess the extent to which SBIR technologies are transitioning to DOD users, we met with officials in the Office of Small Business Programs, military department SBIR program offices, and the aforementioned military acquisition organizations to discuss what data are available to measure transition of SBIR technologies to acquisition programs, or directly to warfighters in the field. We determined that DOD uses two primary data systems—Company Commercialization Reports and the Federal Procurement Data System-Next Generation. We discussed with DOD officials what data are collected by these systems, how the data are validated and used, and whether there are limitations to the data collected. We also reviewed available documentation on the systems. In assessing data limitations, we discussed with SBIR officials whether the systems provide accurate, reliable, and comprehensive data on SBIR projects that transition to military users. In addition, we interviewed military department officials about other data collection practices they may have implemented to track SBIR projects and results. Any limitations that were identified for the data collection practices and data systems used to identify technology transition outcomes for SBIR projects are discussed in this report.

We conducted this performance audit from April 2013 to December 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, John Oppenheim, Assistant Director; Danielle Greene; Victoria Klepacz; Sean Merrill; Scott Purdy; and Sylvia Schatz also made key contributions to the report.