Small Business Research Programs: Agencies Need to Take Steps to Assess Progress Toward Commercializing Technologies
Released: 2018-01-31T00:00:00

Letter

Background

The SBIR program was initiated in 1982 and has four main purposes: (1) use small businesses to meet federal R&D needs, (2) stimulate technological innovation, (3) increase commercialization of innovations derived from federal R&D efforts, and (4) encourage participation in technological innovation by small businesses owned by women and disadvantaged individuals. The STTR program was initiated a decade later, in 1992, and has three main purposes: (1) stimulate technological innovation, (2) foster technology transfer through cooperative R&D between small businesses and research institutions, and (3) increase private-sector commercialization of innovations derived from federal R&D.

The SBIR and STTR programs are similar in that participating agencies identify topics for R&D projects and support small businesses, but the STTR program requires the small business to partner with a nonprofit research institution, such as a college or university or a federally funded research and development center.

Each participating agency must manage its SBIR and STTR programs in accordance with program laws and regulations and the policy directives issued by SBA. In general, the programs are similar across participating agencies. All of the participating agencies follow the same general process to obtain proposals from and make awards to small businesses for both the SBIR and STTR programs. However, each participating agency has considerable flexibility in designing and managing specific aspects of these programs, such as determining research topics, selecting award recipients, and administering funding agreements. At least once a year, each participating agency issues a solicitation requesting proposals for projects in topic areas determined by the agency. Each participating agency uses its own process to review proposals and determine which proposals should receive awards. The agencies that participate in both SBIR and STTR programs usually use the same process for both programs. Also, each participating agency determines whether to provide the funding for awards as grants or contracts.

According to the policy directives, SBA maintains a system that records SBIR and STTR award information—using data submitted by the agencies—as well as commercialization information, such as information about patents, sales, and investments reported by small businesses that received these awards. SBA is to use these data to assess small businesses that received awards against the benchmarks and identify any small businesses that did not meet the benchmarks. SBA is to initially assess the small businesses against the benchmarks and then in April of each year notify those that do not meet the benchmarks so that the businesses can review their award data and work with participating agencies to correct the database if necessary. SBA then is to analyze the award data again to identify, on June 1, those small businesses that still do not meet the benchmarks. These small businesses are then ineligible for certain awards from that date through May 31 of the following year.

Data Challenges Have Limited the Implementation of the Benchmarks, and SBA and Participating Agencies Have Provided Inconsistent Information about the Consequence SBA and Participating Agencies Assessed Small Businesses against the Transition Rate Benchmark, but the Assessments Have Been Based on Inaccurate or Incomplete Data

Data challenges have limited SBA’s and the 11 participating agencies’ efforts to fully implement the benchmarks. Since 2014, SBA and the participating agencies have regularly assessed small businesses against the Transition Rate Benchmark, but the assessments have been based on inaccurate or incomplete data. SBA and the participating agencies have assessed small businesses against the Commercialization Benchmark only once, in 2014, because of challenges in collecting and verifying the accuracy of data. In addition, SBA and the participating agencies have provided inconsistent information to small businesses about the consequence of not meeting the benchmarks.

Since 2014, SBA and the participating agencies have regularly assessed small businesses against the Transition Rate Benchmark, which, in general, measures the rate at which businesses move projects from phase I to phase II. From 2014 through 2017, SBA determined that 4 to 7 small businesses did not meet the benchmark each year and placed those businesses on a list of those ineligible to receive certain additional awards.

However, we found instances in which the data used to generate the list were inaccurate or incomplete. For example, we identified an instance in which the data in the awards database changed considerably after SBA’s initial assessment, indicating that the data used for that assessment were inaccurate. SBA’s list of small businesses subject to the benchmark in 2015 showed that a small business received 297 phase I awards during the assessment period. However, data received from SBA officials in August 2017 showed that this small business received only 1 phase I award. Agencies can update their data in the awards database at any time to, for example, submit additional award data or correct previously submitted award data, which is what an SBA official stated may have caused this change. Because the small business received only 1 award, it would not have been subject to the Transition Rate Benchmark. In this case, the change meant that SBA did not miss identifying a small business that should have been ineligible for an award; however, in other instances, changes to the data may lead SBA to miss identifying a small business that should have been ineligible for awards.

In addition, we identified instances in which the publicly available data on awards were incomplete, including data that were missing or otherwise unusable. For example, based on our review of the award data from 2007 through 2016, we identified more than 2,700 small businesses that had multiple records with different spellings of the same business’s name. Furthermore, we identified more than 1,400 instances in which a unique identification number had errors, such as having an incorrect number of digits, all zeros, or hyphens. SBA officials told us that the quality of the award information in the database has been an issue, and that accurate information is important because small businesses may avoid being identified as subject to the benchmark if their business names and identification numbers are different across multiple records. For example, if the database contains 18 phase I awards made within the assessment period to a small business with a certain unique identification number but also contains 3 other phase I awards within that period with a different or missing unique identification number, the small business may avoid being identified as subject to the benchmark because the data would suggest it did not meet the threshold of receiving more than 20 phase I awards, even if it did. As a result, it could be difficult to determine which small businesses actually received more than 20 awards and should be subject to the benchmark.

Standards for Internal Control in the Federal Government state that management should use quality information to achieve the entity’s objectives, and SBA’s Information Quality Guidelines state that SBA seeks to ensure the quality, utility, and integrity of the information it shares with the public, among other things. SBA’s policy directives for the SBIR and STTR programs state that SBA maintains a system that records SBIR and STTR award information, which is publicly available, and uses this information to calculate small businesses’ performance against the benchmark. SBA officials told us they depend on the accuracy of the data received from the participating agencies to perform SBA’s assessment. These officials also acknowledged that confirming the accuracy of SBA’s annual assessments against the benchmarks has been challenging because agencies can update their data over time. SBA officials stated that they have sought to improve the quality of the data after the data are entered into the database, such as fixing instances in which small businesses’ names were spelled differently across multiple records; however, the officials said that correcting the data already entered in the awards database is an ongoing and time-consuming process. SBA officials told us that there are errors in the database, in part because SBA has not worked with participating agencies to ensure that agencies enter high-quality, accurate data into the database. SBA officials provided us guidance on how to enter data that they said is available to agencies, but the errors we found suggest that agencies are not fully utilizing this guidance. As a result, SBA cannot reasonably ensure the quality and reliability of its award data and therefore cannot reasonably ensure that it has correctly assessed small businesses against the Transition Rate Benchmark.

SBA and the Participating Agencies Assessed Small Businesses against the Commercialization Benchmark Only in 2014

The Small Business Act requires agencies to evaluate whether small businesses have met a minimum performance standard for commercializing their technology. SBA and participating agencies do not know the extent to which small businesses are meeting the Commercialization Benchmark because SBA and the agencies have assessed businesses against the benchmark only once, in 2014, when SBA determined that 12 businesses did not meet the benchmark. This is in part because, according to officials from SBA and several agencies, they cannot collect and verify the accuracy of the data needed to implement the benchmark as written.

For SBA and participating agencies to assess whether small businesses meet the Commercialization Benchmark, these small businesses must provide data on sales, investments, or patents resulting from the awards. However, agency officials told us about challenges related to obtaining the data they need to implement this benchmark. For example, agency officials told us that the needed data are not consistently applicable across agencies or projects. Specifically, these officials said that an agency may purchase the technology developed as a result of the SBIR or STTR award, while another agency may focus on funding technologies that will be sold on the commercial market, leading to different kinds of data on “sales.” Additionally, officials from SBA and several of the participating agencies told us they have been unable to collect and verify the accuracy of the information from small businesses to assess them against the Commercialization Benchmark. In addition, officials from 2 agencies told us that small businesses can easily circumvent the benchmark by submitting incorrect data.

The Small Business Act and the policy directives provide agencies flexibility in how they can implement the Commercialization Benchmark. Officials from participating agencies said that they thought the Commercialization Benchmark should be revised, but they provided differing views on how to do it. Officials from SBA and 2 agencies told us that they would consider having individual agencies develop a benchmark or metric tailored to their agency, in part because the definition of successful commercialization could vary across the agencies. However, officials acknowledged that collecting and verifying the accuracy of the data would still be a concern with this approach. Officials from 2 participating agencies told us that collecting and verifying the accuracy of the data is a significant amount of work, and officials from a third agency added that implementing the benchmark independently is impractical because they do not have the capability to track small businesses’ commercialization efforts. Officials from 1 agency said they preferred to keep a uniform benchmark across the agencies, in part because having varying benchmarks could lead to a small business being eligible to participate in the programs with one agency but not with another. Although views differed across agencies, working together to find a way to implement the benchmark as designed or revising it so that it can be implemented could allow the agencies to fulfill the requirement in the Small Business Act.

Officials from 3 agencies told us they would prefer to consider businesses’ prior commercialization experience as part of their overall evaluation of businesses’ proposals, rather than implement the current Commercialization Benchmark. The SBIR and STTR policy directives currently allow agencies to define the benchmark in terms other than revenue or investment, such as using a commercialization scoring system that rates awardees on their past commercialization success. Defining the benchmark in these terms could help agencies to implement the statutory requirement. Officials from SBA said they see the value of allowing reviewers to use professional judgment in determining the commercialization success of applicants, rather than assessing small businesses against standard criteria. Officials from 1 agency said that such a change could help achieve the goal of the benchmark without the challenges of collecting data from all small businesses participating in the programs.

Nine of the 11 participating agencies currently consider prior commercialization experience as part of their evaluation when making award selections (see table 2), which shows that evaluating commercialization experience at individual agencies can be feasible. For example, project solicitations from the Department of Agriculture, the Department of Defense, and the National Science Foundation state that these agencies require applicants to provide sales or revenue information for products resulting from SBIR or STTR awards, and the Department of Homeland Security’s solicitation requires applicants to provide a history of previous federal and nonfederal funding and subsequent commercialization of their products. All agencies consider commercialization potential when selecting these awards.

SBA and Participating Agencies Have Provided Inconsistent Information to Small Businesses on the Consequence of Not Meeting the Benchmarks

The consequence for small businesses not meeting the benchmarks is ineligibility to participate in phase I of the SBIR or STTR program for a year, according to the Small Business Act. SBA officials stated that they and the agencies initially interpreted this to mean that small businesses could not receive awards during the ineligibility period of June 1 through May 31 of the following year, and this is how the consequence is described in the SBIR and STTR policy directives. SBA officials told us that they and the participating agencies sought to change how to implement the consequence of businesses not meeting the benchmarks because of SBA’s and agencies’ difficulties in implementing the benchmarks. Officials from 4 agencies said that they generally evaluate and select awards shortly before SBA releases the list of ineligible companies, leading them to potentially select projects from small businesses that will be on the ineligible list by the time the award period begins. Based on our review of award data from October 2014 to May 2017, we identified 13 phase I awards across 5 small businesses with award start dates during the period that the business was ineligible to receive such awards. According to agency officials, each of these awards was selected before the small business became ineligible to receive the award. SBA and the participating agencies agreed to change how the consequence would be implemented, starting in 2017, so that small businesses that do not meet the benchmarks are ineligible to submit proposals, according to SBA officials.

As of November 2017, however, the information available about this new way to implement the consequence was inconsistent because some of the agencies had not updated their project solicitations. Specifically, information in the most recent project solicitations available at that time for 2 agencies and one subunit of an agency stated that businesses that do not meet the benchmarks are ineligible to submit certain proposals, consistent with the revised approach for how to implement the consequence. However, the most recent project solicitations available at that time for 7 other agencies and the other subunit of the agency mentioned above instead stated that those businesses that do not meet the benchmarks are ineligible to receive certain awards, consistent with the prior approach for how to implement the consequence. One other agency directed users to SBA’s website in its solicitation. Table 3 shows the information about the consequence of not meeting the benchmarks that each agency included in its most recent project solicitations as of November 2017.

As of November 2017, the SBIR and STTR policy directives stated that the consequence for not meeting these benchmarks is ineligibility to receive certain awards. SBA officials told us they are in the process of updating the policy directives to reflect this change in how the consequence is implemented, but these officials said that it is a long process and they could not provide a timeframe for when the update would be complete. As mentioned earlier in this report, SBA’s Information Quality Guidelines state that SBA seeks to ensure the quality, utility, and integrity of the information it shares with the public, among other things. Until participating agencies update their project solicitations and SBA updates its policy directives to accurately reflect agreed-upon practices about the consequence for small businesses that do not meet the benchmarks, small businesses may be confused about their eligibility to submit proposals and could invest time developing and submitting proposals when they are not eligible to do so.

Conclusions

Under the SBIR and STTR programs, federal agencies have awarded billions of dollars to small businesses to help these businesses develop and commercialize innovative technologies. SBA and the participating agencies have assessed these small businesses against the Transition Rate Benchmark, but those assessments have been based on inaccurate or incomplete data. Without ensuring the reliability of its data, SBA cannot reasonably ensure that it has correctly assessed small businesses against the Transition Rate Benchmark.

SBA and the participating agencies developed a Commercialization Benchmark across all the participating agencies but have not fully implemented it, in part because they have been unable to collect information from the small businesses and verify the accuracy of that information. Working together to implement the benchmark as written or revise it so that it can be implemented could allow the agencies to fulfill the requirement in the Small Business Act to evaluate whether small businesses have met a minimum performance standard for commercializing their technology.

Lastly, SBA and the participating agencies have provided inconsistent information to small businesses about the consequence of not meeting the benchmarks. Officials from SBA and the participating agencies had agreed to change how the consequence would be implemented, starting in 2017, because of difficulties implementing the benchmarks. However, as of November 2017, seven agencies, and a subunit of one agency, had not updated their project solicitations and SBA had not updated its policy directives. Without consistent information on the benchmarks, small businesses may be confused about their eligibility to submit proposals and could invest time developing proposals that they are not eligible to submit.

Recommendations for Executive Action

We are making a total of 11 recommendations, including 3 to SBA and 1 each to the Department of Commerce’s National Oceanic and Atmospheric Administration; the Departments of Defense, Education, Energy, Health and Human Services, and Homeland Security; the Environmental Protection Agency; and the National Science Foundation. Specifically: The Director of the Office of Investment and Innovation within SBA should work with participating agencies to improve the reliability of its SBIR and STTR award data (Recommendation 1).

The Director of the Office of Investment and Innovation within SBA should work with participating agencies to implement the Commercialization Benchmark or, if that is not feasible, revise the benchmark so that it can be implemented (Recommendation 2).

The Director of the Office of Investment and Innovation within SBA should update the SBIR and STTR policy directives to accurately reflect how the consequence of the benchmarks is to be implemented (Recommendation 3).

The SBIR Program Manager of the Department of Commerce’s National Oceanic and Atmospheric Administration should update the agency’s SBIR project solicitation to accurately reflect how the consequence of not meeting the benchmarks is to be implemented (Recommendation 4).

The SBIR Program Administrator within the Department of Defense should update the agency’s SBIR and STTR project solicitations to accurately reflect how the consequence of not meeting the benchmarks is to be implemented (Recommendation 5).

The SBIR Program Manager within the Department of Education should update the agency’s SBIR project solicitation to accurately reflect how the consequence of not meeting the benchmarks is to be implemented (Recommendation 6).

The SBIR Program Manager within the Department of Energy should update the agency’s combined SBIR and STTR project solicitation to accurately reflect how the consequence of not meeting the benchmarks is to be implemented (Recommendation 7).

The SBIR/STTR Program Coordinator within the Department of Health and Human Services should update the agency’s SBIR and STTR project solicitations to accurately reflect how the consequence of not meeting the benchmarks is to be implemented (Recommendation 8).

The SBIR Program Director within the Department of Homeland Security should update the agency’s SBIR project solicitation to accurately reflect how the consequence of not meeting the benchmarks is to be implemented (Recommendation 9).

The SBIR Program Manager within the Environmental Protection Agency should update the agency’s SBIR project solicitation to accurately reflect how the consequence of not meeting the benchmarks is to be implemented (Recommendation 10).

The SBIR and STTR Program Manager within the National Science Foundation should update the agency’s SBIR and STTR project solicitations to accurately reflect how the consequence of not meeting the benchmarks is to be implemented (Recommendation 11).

Agency Comments and Our Evaluation

We provided a draft of this report to SBA and the 11 participating agencies for review and comment. In written comments, the Department of Commerce’s National Oceanic and Atmospheric Administration; the Departments of Defense, Education, Energy, Health and Human Services, and Homeland Security; the Environmental Protection Agency; and SBA agreed with the respective recommendations directed to their agencies. Agencies’ written comments are reproduced in appendixes I through VIII. An official from one agency—the National Science Foundation—stated in an email that the agency concurred with the recommendation and did not have any further comments. Two agencies—the Department of Homeland Security and SBA—also provided technical comments, which we incorporated as appropriate. Three agencies—the Departments of Agriculture and Transportation, and the National Aeronautics and Space Administration—as well as the Department of Commerce’s National Institute of Standards and Technology stated via email that they had no technical or written comments.

In its comments, SBA stated that it disagreed with a statement in our draft report that SBA had not worked with agencies to enter high-quality and accurate data into the database and provided us documentation of an instruction guide on entering data that SBA officials said was available to agencies. Based on our review of this information, we clarified the text of the report and modified the draft report’s recommendation by removing the suggested example that SBA provide guidance to the agencies to improve SBIR and STTR award data reliability. SBA agreed with the revised recommendation.

After we provided a draft of the report to the agencies for comment, the Departments of Education and Homeland Security took action on their respective recommendations. Specifically, in December 2017, the agencies issued new project solicitations that reflected the updated consequence of not meeting the benchmarks. We agree that these agencies fully implemented the recommendations we made to them in this report.

We are sending copies of this report to the appropriate congressional committees; the Secretaries of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, and Transportation; the Administrators of the Small Business Administration, the Environmental Protection Agency, and the National Aeronautics and Space Administration; the Director of the National Science Foundation; and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or neumannj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IX.

Appendix I: Comments from the Small Business Administration

Appendix II: Comments from the Department of Commerce

Appendix III: Comments from the Department of Defense

Appendix IV: Comments from the Department of Education

Error! No text of specified style in document.

Appendix V: Comments from the Department of Energy

Appendix VI: Comments from the Department of Health and Human Services

Appendix VII: Comments from the Department of Homeland Security

Appendix VII: Comments from the Department of Homeland Security Error! No text of specified style in document.

Appendix VIII: Comments from the Environmental Protection Agency

Appendix IX: GAO Contact and Staff Acknowledgments

Appendix IX: GAO Contact and Staff Acknowledgments Error! No text of specified style in document.

GAO Contacts

Staff Acknowledgments

In addition to the contact named above, Hilary Benedict (Assistant Director), John Barrett, Natalie Block, Antoinette Capaccio, Tanya Doriss, Justin Fisher, Ellen Fried, Juan Garay, Cindy Gilbert, Perry Lusk, William Shear, and Elaine Vaurio made key contributions to this report.