Federal Research: Observations on the Small Business Innovation Research Program
Released: 2005-06-28T00:00:00

Letter

SBIR Program Has Generally Met Its Goals

Between July 1985 and June 1999, we reviewed, reported, and testified on the SBIR program many times at the request of the Congress. While our work focused on many different aspects of the program, we generally found that SBIR is achieving its goals to enhance the role of small businesses in federal R&D, stimulate commercialization of research results, and support the participation of small businesses owned by women and/or disadvantaged persons. Participating agencies and companies that we surveyed during the course of our reviews generally rated the program highly. Specific examples of program success that we identified include the following: High-quality research. Throughout the life of the program, awards have been based on technical merit and are generally of good quality. For example, in 1989 we reported that according to agency officials, more than three-quarters of the research conducted with SBIR funding was as good as or better than other agency-funded research. Agency officials also rated the research as more likely than other research they oversaw to result in the invention and commercialization of new products. When we again looked at the quality of research proposals in 1995, we found that while it was too early to make a conclusive judgment about the long-term quality of the research, the quality of proposals remained good, according to agency officials.

Widespread competition. The SBIR program successfully attracts many qualified companies, has had a high level of competition, and consistently has had a high number of first-time participants. Specifically, we reported that the number of proposals that agencies received each year had been increasing. In addition, as we reported in 1998, agencies rarely received only a single proposal in response to a solicitation, indicating a sustained level of competition for the awards. We also found that the agencies deemed many more proposals worthy of awards than they were able to fund. For example, the Air Force deemed 1,174 proposals worthy of awards in fiscal year 1993 but funded only 470. Moreover, from fiscal years 1993 through 1997, one third of the companies that received awards were first-time participants. This suggests that the program attracts hundreds of new companies annually.

Effective outreach. SBIR agencies consistently reach out to foster participation by women-owned or socially and economically disadvantaged small businesses. For example, we found that DOD’s SBIR managers participated in a number of regional small business conferences and workshops that are specifically designed to foster increased participation by women-owned and socially and economically disadvantaged small businesses.

Successful commercialization. SBIR successfully fosters commercialization of research results. At various points in the life of the program we have reported that SBIR has been successful in increasing private sector commercialization of innovations. For example, past GAO and DOD surveys of companies that received SBIR Phase II funding have determined that approximately 35 percent of the projects resulted in the sales of products or services, and approximately 45 percent of the projects received additional developmental funding. We have also reported that agencies were using various techniques to foster commercialization. For example, in an attempt to get those companies with the greatest potential for commercial success to the marketplace sooner, DOD instituted a Fast Track Program, whereby companies that are able to attract outside commitments/capital for their research during phase I are given higher priority in receiving a phase II award.

Helping to serve mission needs. SBIR has helped serve agencies’ missions and R&D needs. Agencies differ in the emphasis they place on funding research to support their mission and to support more generalized research. Specifically, we found that DOD links its projects more closely to its mission. In comparison, other agencies emphasize research that will be commercialized by the private sector. Many of the projects DOD funded have specialized military applications while NIH projects have access to the biomedical market in the private sector. Moreover, we found that SBIR promotes research on the critical technologies identified in lists developed by DOD and/or the National Critical Technologies Panel. Generally agencies reviewed these listings of critical technologies to develop research topics or conducted research that fell within one of the two lists.

Improvements Made to the SBIR Program Over Time

We have also identified areas of weaknesses and made recommendations that, if addressed, could strengthen the program further. Many of our recommendations for program improvement have been either fully or partially addressed by the Congress in various reauthorizations of the program or by the agencies themselves. For example, Duplicate funding. In 1995, we identified duplicate funding for similar, or even identical, research projects by more than one agency. A few companies received funding for the same proposals two, three, and even five times before agencies became aware of the duplication. Contributing factors included the fraudulent evasion of disclosure by companies applying for awards, the lack of a consistent definition for key terms such as “similar research,” and the lack of interagency sharing of data on awards. In response to our recommendations, SBA strengthened the language agencies use in their application packages to clearly warn applicants about the illegality of entering into multiple agreements for essentially the same effort and developed Internet capabilities to access SBIR data for all of the agencies. In SBA’s view, the stronger language regarding the illegality of seeking funding for similar or identical projects addresses the need to develop consistent definitions to help agencies determine when projects are “similar.”

Inconsistent interpretations of extramural research budgets. In 1998, we found that while agency officials adhered to SBIR’s program and statutory funding requirements, they used differing interpretations of how to calculate their “extramural research budgets.” As a result some agencies were inappropriately including or excluding some types of expenses. To address our recommendation that SBA provide additional guidance on how participating agencies were to calculate their extramural research budgets, the Congress in 2000 required that the agencies report annually to SBA on the methods used to calculate their extramural research budgets.

Geographical concentration of awards. In 1999, in response to congressional concerns about the geographical concentration of SBIR awards, we reported that companies in a small number of states, especially California and Massachusetts, have submitted the most proposals and won the majority of awards. The distribution of awards generally followed the pattern of distribution of non-SBIR expenditures for R&D, venture capital investments, and academic research funds. We reported that some agencies had undertaken efforts to broaden the geographic distribution of awards and that the program implemented by the National Science Foundation had been particularly effective. Although we did not make any recommendations on how to improve the program’s outreach to states receiving fewer awards, in the 2000 reauthorization of the program, Congress established the Federal and State Technology Partnership Program to help strengthen the technological competitiveness of small businesses, especially in those states that receive fewer SBIR grants.

Clarification on commercialization and other SBIR goals. Finally, in response to our continuing concern that clarification was needed on the relative emphasis that agencies should give to a company’s commercialization record and SBIR’s other goals when evaluating proposals, in 2000 the Congress required companies applying for a second phase award to include a commercialization plan with their SBIR proposals. This requirement partially addressed our concern. Moreover, in the spring of 2001, SBA initiated efforts to respond to our recommendation to develop standard criteria for measuring commercial and other outcomes of the SBIR program, such as uniform measures of sales and developmental funding, and incorporate these criteria into its Tech-Net database. Specifically, SBA began implementing a reporting system to measure the program’s commercialization success. In fiscal year 2002, SBA further enhanced the reporting system to include commercialization results that would help establish an initial baseline rate of commercialization. In addition, small business firms participating in the SBIR program are required to provide information annually on sales and investments associated with their SBIR projects.

Assessing the Performance of the SBIR Program Remains a Challenge

One issue that continues to remain somewhat unresolved after almost two decades of program implementation is how to assess the performance of the SBIR program. As the program has matured, the Congress has emphasized the potential for commercialization as an important criterion in awarding funds and the commercialization of a product as a measure of success for the program. However, in 1999, we reported that the program’s other goals also remain important to the agencies. By itself, according to some program managers, limited commercialization may not signal “failure” because a company may have achieved other goals, such as innovation or responsiveness to an agency’s research needs. We identified a variety of reasons why assessing the performance of the SBIR program has remained a challenge.

First, because the authorizing legislation and SBA’s policy directives do not define the role of the company’s commercialization record in determining commercial potential and the relative importance of the program’s goals, different approaches have emerged in agencies’ evaluations of proposals. As a result, the relative weight that should be given to the program’s goals when evaluating proposals remains unclear. Innovation and responsiveness to an agency’s needs, for example, may compete with the achievement of commercialization. In the view of many program managers, innovation involves a willingness to undertake R&D with a higher element of risk and a greater chance that it may not lead to a commercial product; responsiveness to an agency’s needs involves R&D that may be aimed at special niches with limited commercial potential. Striking the right balance between achieving commercial sales and encouraging new, unproven technologies is, according to the program managers, one of the key ingredients in the program’s overall success.

Second, we found that it has been difficult to find practical ways to define and measure the SBIR program’s goals in order to evaluate proposals. For example, the authorizing legislation lacks a clear definition of “commercialization,” and agencies sometimes differed on its meaning. This absence of a definition makes it more difficult to determine when a frequent winner is “failing” to achieve a sufficient level of commercialization and how to include this information in an agency’s review of the company’s proposal. Similarly, efforts to define and measure technological innovation, which was one of the program’s original goals, have posed a challenge. Although definitions vary, there is widespread agreement that technological innovation is a complex process, particularly in the development of sophisticated modern technologies.

Finally, we reported that as the emphasis on commercialization had grown, so had concerns that noncommercial successes may not be adequately recognized. For example, program managers identified various projects that met special military or medical equipment needs but that had limited sales potential. These projects would be helpful in reducing the agency’s expenditures and meeting the mission of the agency but may not be appropriately captured in typical measurements of commercialization. In general, we found that program managers valued both noncommercial and commercial successes and feared that the former might be ignored in emphasizing the latter.

To help evaluate the performance of the program, in the 2000 reauthorization of SBIR, Congress required SBA to develop a database that would help the agency collect and maintain in common format necessary program output and outcome information. The database is to include the following information on all phase II awards: (1) revenue from the sale of new products or services resulting from the SBIR funded research, (2) additional investment from any non-SBIR source for further research and development, and (3) any other description of outputs and outcomes of the awards. In addition, the database is to include general information for all applicants not receiving an award including an abstract of the project.

In conclusion, Mr. Chairman, our work has shown that, overall, the SBIR program has been successful in meeting its goals and that the Congress and the agencies have implemented actions to strengthen the program over time. However, an assessment of the program’s results remains a challenge because of the lack of clarity on how much emphasis the program should place on commercialization versus other goals.

For further information, please contact Anu Mittal at (202) 512-3841 or mittala@gao.gov.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.