Small Business Innovation Research: SBA Should Work with Agencies to Improve the Data Available for Program Evaluation
Released: 2011-09-14T00:00:00

Letter

Background

SBA coordinates and oversees the efforts of the 11 agencies currently participating in the SBIR program. SBA coordinates the agencies’ schedules for issuing solicitations—announcements of opportunities for small businesses to apply for awards—and provides access to these solicitations through its Web site. As part of its oversight effort, SBA collects SBIR data from the participating agencies, aggregates the data, and uses the data to, among other things, monitor the program and report to Congress. SBA also provides guidance to participating agencies on the general conduct and operation of the program, which it periodically updates, for example, in response to changes in the program’s authorizing legislation. Under the legislation and SBA’s guidance, agencies have considerable flexibility to design their programs. For example, each agency determines, in consultation with SBA, such items as the number of solicitations to be issued during a fiscal year and the dates applications are due. Agencies also have discretion to determine what type of research to include in their solicitations, how to review applications for technical and scientific merit, which applications to fund, and the size of the award, among other things.

The Small Business Innovation Development Act of 1982 provided for a competitive three-phased SBIR program. In phase I, participating agencies award up to $150,000 for a period of about 6 to 9 months to small businesses to conduct experimental or theoretical R&D. Small businesses whose phase I projects demonstrate scientific and technical merit, in addition to commercial potential, may compete for phase II awards of up to $1 million to continue the R&D for an additional period, normally not to exceed 2 years. Phase I and II award funds may be used for costs related to conducting the research, such as salaries, fringe benefits, equipment, and consulting services, as well as for profits and fees. To be eligible for a phase I or II SBIR award, a business must have 500 or fewer employees, be organized for profit with a place of business in the United States, and operate primarily in the United States or make a significant contribution to the U.S. economy. Generally, a business must also be at least 51 percent owned and controlled by one or more individuals who are U.S. citizens or permanent resident aliens. These eligibility requirements apply at the time that a phase I or II award is made. During phase III, businesses must secure non-SBIR funding to develop the commercial potential of the innovative technologies resulting from their SBIR projects; such funding may come from the private sector, federal agencies, or other sources.

As the program has been reauthorized over the years, legislation has established a number of requirements related to the program’s purposes. For example, the Small Business Research and Development Enhancement Act of 1992 directed SBA to make more information available about the SBIR program, particularly about participation by small businesses owned by disadvantaged individuals and women, and required that agencies increase their outreach to such businesses. In addition, the Small Business Reauthorization Act of 2000 directed that applicants for phase II SBIR awards be required to submit commercialization plans, and it mandated that SBA develop, maintain, and make available to the public a database that contained SBIR award data. In addition, the act required SBA to develop and maintain, by June 2001, a restricted government-use database that would contain award- related data from the public database, as well as additional confidential data that would be accessible only to government agencies and other authorized users. The act stated that this database would be used exclusively for program evaluation—which, as we have noted in past work, involves the systematic collection and analysis of accurate, comparable, and complete data on program results. The act required the government-use database to contain, among other things, data that applicants for phase II awards would be required to supply on the commercialization success of any prior phase II awards, such as data on sales of or additional investment in the technologies funded under the awards. The act further specified that the government-use database would contain annual updates to these data, which phase II award recipients would be requested to voluntarily provide for 5 years after the period covered by the award.

To accomplish this mandate, SBA envisioned expanding an electronic database, known as Tech-Net, which it had developed in the late 1990s, into two sections: a public-use portion and a government-use portion containing commercialization data. The public-use portion of the database has been available since 2000, according to SBA, and it contains such award-related data as the phase of the award, amount of the award, name and location of the business receiving the award, an abstract of the work to be conducted under the award, and whether the business is categorized as owned by disadvantaged individuals or women. In October 2006, however, we reported that some SBIR agencies did not consistently provide or correctly format the awards-related data for several fields in the public-use portion of the database. For example, two of the eight agencies we reviewed had not consistently provided data on whether the businesses receiving the awards were categorized as owned by disadvantaged individuals or women. At that time, we also reported that SBA had not implemented the government-use portion of the database, primarily, according to SBA officials, because of increased security requirements for the database, agency management changes, and budgetary constraints. Additionally, we reported that while five of the agencies we reviewed had systematically collected commercialization data, their data collection efforts differed in ways that made it challenging to evaluate the program across agencies. In August 2009, we testified before Congress that SBA said the database would no longer accept incorrectly formatted awards-related data from participating agencies.

A committee of the National Academy of Sciences’ National Research Council has conducted a series of assessments of the SBIR program, both within and across agencies, as part of a legislatively mandated study. The results were summarized in a single report, in which the committee stated that SBIR is making significant progress in achieving congressional goals. The study concluded that the SBIR program is “sound in concept and effective in practice.” The study also recommended changes that could make the program more effective. Among other things, the study recommended that SBA and participating agencies improve the collection of data that track participation in the SBIR program by businesses owned by disadvantaged individuals and women, develop targeted outreach to such businesses that is based on an analysis of factors that affect their participation, and improve documentation of commercialization success. The National Research Council is now undertaking another round of assessments to provide a second snapshot of the program’s progress in achieving its legislative purposes.

Agencies Addressed SBIR’s Purposes through Solicitations, Technical Assistance, Matching Funds Programs, and Outreach

For fiscal years 2008 through 2011, the five participating agencies we reviewed addressed the SBIR purposes of using small business to meet federal R&D needs and stimulating technological innovation through their solicitations. Agencies also used solicitations, as well as technical assistance or matching funds programs, to address the SBIR purpose of increasing commercialization of innovations derived from federal R&D efforts. To address the remaining program purpose—encouraging participation in technological innovation by small businesses owned by disadvantaged individuals and women—agencies relied mainly on outreach activities aimed at a broader audience.

Agencies Addressed the SBIR Purposes Related to Meeting Federal Research Needs and Stimulating Innovation through Their Solicitations

All of the participating agencies that we reviewed designed the SBIR solicitations that they issued for fiscal years 2008 through 2011 to meet federal R&D or mission needs and stimulate technological innovation. All of these agencies selected research topics for their solicitations that were designed to meet their respective R&D or mission needs and specified that applications would be evaluated on the basis of responsiveness to those topics. The agencies that purchase SBIR-funded technologies for their own use—DOD, DOE, and NASA—tended to select solicitation topics that met specific agency needs for R&D. For example, in fiscal year 2011, DOD solicited applications to develop a fuel cell system capable of converting ethanol into electricity in an efficient, small, lightweight, portable power system. According to the solicitation, such advanced fuel cell systems could provide soldiers power to complement batteries and to charge rechargeable batteries, reducing the number of batteries required for extended time in the field. In contrast, NIH and NSF, which generally do not purchase SBIR-funded technologies, tended to issue solicitations for a broader spectrum of R&D to support their missions of advancing biomedical and other scientific and engineering disciplines. Among the agencies we reviewed, NIH and its components gave applicants the most leeway in addressing agency needs: rather than limiting applications to specific research topics identified in solicitations, NIH and its components usually listed suggested topics and encouraged applicants to propose innovative projects that fit the agency’s mission.

Concerning innovation, each of the agencies included instructions in its SBIR solicitations about the type of information applicants had to provide about the innovativeness of the proposed work. For example, NASA informed phase I and II applicants that a competitive application would describe the proposed innovation relative to state-of-the-art knowledge in the field, among other things. In addition, these agencies explained to applicants how reviewers would consider evidence of the innovativeness of the applicants’ proposed research approaches. For example, in its fiscal year 2010 solicitation, NSF stated that applications would be evaluated, in part, on the basis of whether they reflected state-of-the-art knowledge in the major research activities proposed and whether the work was likely to advance state-of-the-art knowledge.

Agencies Worked to Increase Commercialization of Innovations through Solicitations, Technical Assistance, and Matching Funds Programs

The participating agencies we reviewed addressed the SBIR purpose of increasing commercialization of innovations through solicitations, as well as through technical assistance or matching funds programs.

Solicitations. Of the five agencies we reviewed, all but NIH required in their solicitations for fiscal years 2008 through 2011 that applicants for phase I awards submit a commercialization strategy demonstrating that the applicants had taken steps such as identifying a market for their SBIR technologies, planning to secure financing, and estimating expected future sales. For phase II awards, all of the agencies we reviewed required that applicants submit a commercialization plan. In general, the solicitations we reviewed required that phase II commercialization plans discuss the potential market and competitors; the qualifications of key management and technical personnel; as well as financing, marketing, and manufacturing plans, among other things.

The agencies we reviewed differed in their stated processes for evaluating the commercial potential of applications. For example, DOD guidance to applicants outlined a systematic process for how the agency would consider commercialization potential when evaluating applications submitted by small businesses that had received multiple prior awards. DOD indicated that, under this process, it would assign a commercialization achievement score to applicants that had completed the work for four or more phase II awards from any agency; this score would reflect how the applicants’ commercialization experience compared with historical averages. Applicants whose scores fell within the lowest 20 percent would not be allowed to receive more than half the maximum number of points possible for commercialization potential, which was to be assessed on the basis of several factors, including the commercialization strategy or plan. DOD guidance stated that businesses with fewer than four completed phase II awards would not be affected by the absence of a commercialization achievement score. Although the other four agencies we reviewed did not outline as systematic a process for evaluating past commercialization success as a gauge of commercialization potential, they still indicated that commercialization potential would be taken into account in reviewing applications. For example, DOE’s solicitation instructions encouraged phase I applicants to seek firm commitments for private-sector or non-SBIR federal funding prior to applying for a phase II award. The instructions further stated that phase II applicants that obtained such commitments were more likely to receive full credit for commercialization planning during the evaluation of their applications. In the case of NSF, solicitation instructions stated that proposals are usually reviewed by 3 to 10 outside experts in fields related to the proposal; according to NSF officials, these reviewers have business experience. NSF’s solicitation instructions further stated that the agency would not review applications that lacked sufficient information on commercial potential.

In 2010, two agencies we reviewed also issued SBIR solicitations under new programs that were explicitly oriented toward increasing commercialization. Specifically, in July 2010, DOE launched a program under which it solicited applications for phase III of SBIR, the commercialization phase. DOE documents indicated that the agency would make available approximately $30 million, including funding from the American Recovery and Reinvestment Act (Recovery Act), for phase III awards, which are intended to allow businesses to pursue commercial applications of work performed under phase I and II awards. In addition, NIH’s National Cancer Institute began a program under which it solicited phase I applications to continue development of technologies that have originated in its laboratories, with the goal of advancing these technologies toward commercial products. SBA has designated the use of the SBIR program to encourage commercialization of agencies’ internal research as a best practice on its SBIR Web site.

Technical assistance. All five agencies included in our review provided technical assistance to help award recipients build their capacity to commercialize their technologies. To provide the assistance, the agencies contracted with vendors and consultants who have experience in bringing technologies to market. With the exception of NASA, the agencies supported the technical assistance at least in part through the use of SBIR funds.

In fiscal years 2008 to 2010, DOD, DOE, NIH, and NSF spent SBIR funds on technical assistance for phase I award recipients. Some of the assistance was in the form of interactive training Webinars or online tools directed toward a broad spectrum of SBIR applicants and award recipients. For example, the Navy offered phase I award recipients the use of a software tool, known as WebTRIMS, that helps identify, quantify, and track risks associated with SBIR technology development and covers topics such as contracting strategies, business and transition planning, and manufacturing readiness. Other phase I assistance was more customized. For example, DOE offered phase I award recipients customized technical assistance designed to help them develop a commercialization plan complete with an implementation schedule and suggestions for product design. Similarly, on a first-come, first-served basis, NIH offered phase I award recipients assessments of their SBIR- funded technologies’ likely niche in the existing commercial market, which could help recipients develop commercialization plans for phase II applications. Additionally, NSF offered phase I award recipients personalized mentoring and coaching sessions with an advisor.

According to NSF officials, 92 percent of phase I recipients chose to participate in the technical assistance program in 2010. NASA did not provide technical assistance for phase I award recipients; NASA officials told us they believed technical assistance would have the most utility for phase II NASA award recipients.

During at least a portion of the period we reviewed, all five agencies offered individualized technical assistance for phase II award recipients, although DOE curtailed such assistance in 2010, and NASA discontinued its assistance in 2008. Award recipients were selected for assistance on the basis of factors such as recommendations from SBIR program staff and the award recipients’ potential for rapidly moving their technologies to phase III. The assistance consisted of in-depth training and one-on-one assistance from advisors and industry experts. For example, as part of its Commercialization Pilot Program, the Army assisted selected phase II SBIR award recipients in assessing commercialization potential, developing business plans, and matching their technologies with potential government and industry customers. At DOE, staff could nominate phase II award recipients for assistance in preparing to negotiate business deals, for example, including joint ventures and licensing agreements for use of their technologies. DOE officials told us that in 2010 the agency curtailed its use of SBIR funds for phase II technical assistance, spending such funds on assistance only for award recipients that had specifically budgeted for it in their applications. In 2007 and 2008, NASA partnered with the Navy to pilot a technical assistance program for NASA phase II recipients. The program was designed to help SBIR businesses develop a plan for transitioning to phase III, among other things. In 2007, 17 phase II companies with 19 SBIR projects participated in the program, and in the following year, 19 phase II companies with 20 SBIR projects participated. The program was not renewed for 2009; NASA officials told us that they believed the program was generally successful, but that they preferred to use SBIR funds to make larger awards. NIH offered selected current or past phase II award recipients the opportunity to work one-on-one with an advisor over a 9-month period to develop business plans to commercialize their technologies, as well as to prepare materials to help attract potential investors or partners. Since 2004, almost 700 award recipients have received the assistance, including the 80 award recipients currently participating, according to NIH officials. Finally, NSF offered customized assistance to phase II award recipients through its Innovation Accelerator Initiative. According to NSF officials, through this initiative, award recipients received help in connecting with potential investors and negotiating company acquisitions and mergers. NSF officials told us that, in 2010, approximately 33 percent of NSF’s phase II recipients received this assistance.

In some cases, agencies that we reviewed used non-SBIR funds to broaden the scope of the technical assistance they provided to help award recipients commercialize their technologies. For example, DOD used non-SBIR funds to host its annual Beyond Phase II Conference and Technology Showcase, a 3-day event that features matchmaking sessions with SBIR award recipients and prime contractors. Similarly, the Navy used non-SBIR funds to maintain databases with advanced searching capability to help award recipients identify potential business partners. The Navy also used non-SBIR funds for its Transition Assistance Program, which provides individualized help with commercialization planning, culminating in a conference designed to facilitate interaction with potential business partners. Moreover, in 2011, the National Cancer Institute launched its Regulatory Assistance Program using non-SBIR funds. According to agency officials and information from the agency’s Web site, this program provides SBIR award recipients time with consultants experienced in various regulatory requirements—such as those for anticancer therapies, imaging technologies, and medical devices—to prepare strategies for obtaining regulatory approvals required before the technologies can be commercialized. The National Cancer Institute also used non-SBIR funds to support its Investor Forum, which provides competitively selected SBIR award recipients an opportunity to showcase their technologies and enter into discussions with the biotech investment community. In 2010, 14 award recipients that were selected on the basis of strength of research, impact on cancer, product development, and market potential participated in the forum along with more than 175 potential investors, according to the agency’s Web site.

Matching funds programs. Through matching funds programs, agencies provide additional SBIR funds to award recipients that obtain monetary commitments above certain thresholds from outside investors. SBA has designated matching funds programs as a best practice on its SBIR Web site, and all of the agencies we reviewed except DOE have established such programs. For example, for award recipients that obtain a minimum of $100,000 from an outside investor, NSF will match up to 50 percent of the outside investment for a maximum of $500,000 in NSF matching funds. NASA and NIH officials said that matching funds programs encourage outside investment during the early stages of R&D—a time when many investors are reluctant to invest. In particular, officials at the National Cancer Institute said that matching funds can help attract outside investment because they can be used as leverage to increase investors’ potential returns. DOD and NSF offer matching funds to award recipients at the end of phase I and during phase II, while NASA and the National Cancer Institute offer matching funds during phase II. DOE has not established a matching funds program for its SBIR program. DOE officials told us, however, that they are exploring whether to do so and have held discussions with other SBIR participating agencies about their matching funds programs.

Officials at DOD, NASA, and NIH said they have not collected data to compare the commercialization success of recipients that received matching funds with the success of those that did not. NSF conducted a study to assess the effect of the $18 million in matching funds it invested in fiscal year 2006 for 48 phase II award recipients that had raised a total of $58 million from outside investors. According to NSF officials, results of this study showed that, in the 5 years following the start of these phase II projects, 70 percent of recipients that had received matching funds achieved commercial success compared with a 30 percent success rate for recipients that had not received such funds.

Agencies Generally Encouraged Participation by Disadvantaged Individuals and Women through Outreach Activities Directed toward a Broader Audience

SBA’s guidance states that small businesses owned by disadvantaged individuals and women must compete for SBIR awards on the same basis as all other small businesses. However, to meet requirements for greater outreach to small businesses owned by disadvantaged individuals and women, SBA has encouraged participating SBIR agencies to reach out to such businesses and to develop methods that encourage their participation. SBA has also raised the topic of outreach during recent quarterly meetings of agency SBIR program managers. Officials at all of the agencies we reviewed told us they generally reach out to such businesses through activities directed toward a broader audience, such as by attending SBIR national conferences and industry-sponsored events and by sharing information via Web sites or e-mail lists. Agency officials also noted that they try to accommodate requests for speakers at events sponsored by, or likely to be attended by, small businesses owned by disadvantaged individuals and women—for example, events sponsored by trade organizations for minority- or women-owned businesses. However, officials from some trade organizations for businesses owned by disadvantaged individuals and women told us that the outreach of agencies we reviewed was often ineffective in educating the organizations’ members about the SBIR program.

Of the agencies we reviewed, NIH and NSF have made specific efforts, including the following, to improve their outreach:  For fiscal years 2010 and 2011, NIH developed a goal to increase awareness of its SBIR program among businesses owned by disadvantaged individuals and women, and it outlined specific activities aimed at reaching this goal.  Both NIH and NSF offered various fellowships for postdoctoral research conducted by disadvantaged individuals and women; these fellowships were available to support SBIR projects, as well as other research.

In 2010, NSF assigned a full-time staff member to help it develop a plan to increase participation in SBIR by businesses owned by disadvantaged individuals and women in response to a recommendation from its SBIR advisory committee.  Through a review of academic literature, as well as informal polling of NSF applicants and award recipients, NSF has identified barriers to SBIR participation by small businesses owned by disadvantaged individuals and women, NSF officials told us. These barriers include disparities in the owners’ levels of education and access to capital compared with those of other entrepreneurs. To address identified barriers, NSF is, among other things, establishing partnerships with industry and academia to expose African American, Latino, and other college students to entrepreneurship in scientific and technical fields, according to NSF officials.

Evaluation of the effectiveness of agencies’ outreach efforts is hindered by a lack of accurate and complete data. Although SBA collects data on the number and dollar value of awards to small businesses owned by disadvantaged individuals and women, SBA officials told us that they cannot accurately tabulate data on such awards, particularly awards to women-owned businesses, because of inconsistencies in the data on business ownership. According to the officials, SBA has taken steps to correct the inconsistencies for data submitted after 2006 but has not done so for earlier years. Moreover, SBA does not collect data on the number of applications submitted by businesses owned by disadvantaged individuals and women. As a result, SBA’s data do not allow for an examination of trends in the submission of applications from such businesses, analysis of the percentage of applications from these businesses that lead to awards, or correlation of these trends and percentages with outreach efforts. SBA officials told us in March 2011 that they were considering whether their database should include information on the numbers of applications submitted by these businesses.

Comparable Data Are Not Available across Participating Agencies to Evaluate Progress in Increasing Commercialization of SBIR Technologies

SBA has not yet developed the government-use portion of its database for collecting comparable commercialization data on SBIR technologies, but it is taking steps to do so. In the interim, agencies have, for their own purposes, independently gathered commercialization data that are not comparable; the accuracy of these data is largely unknown. Implementing the government-use portion of the database should improve the comparability of the data. However, programwide evaluation of progress in increasing commercialization may continue to be impaired by long- standing challenges.

SBA Has Not Implemented the Government-Use Portion of Its Database Intended to Collect Comparable Commercialization Data for Program Evaluation but Is Taking Steps to Do So

As of June 2011, SBA had not met the legislative mandate to develop and implement, by June 2001, a government-use database that can provide data on commercialization for evaluating the SBIR program. However, the agency’s efforts to develop such a database recently gained additional prominence and resources. Specifically, SBA linked development of the government-use portion of its database to one of the agency’s high-priority performance goals for fiscal years 2011 and 2012. Additionally, in September 2010, SBA allocated $1.4 million in Recovery Act funds to hire a new contractor to develop the government-use portion’s capacity to accept commercialization data submitted by participating SBIR agencies and award recipients, as well as to make other improvements to the database. For example, SBA said that it has been working with the contractor to consolidate data on previous awards. SBA officials said that past award recipients have been assigned unique identifiers that will be used to track awards issued to those recipients over the lifetime of the SBIR program; unique identifiers are also to be assigned to small businesses newly entering the program. In the future, SBA intends for the unique identifiers to allow agencies to validate business information by comparing it against information in other federal databases such as the Central Contractor Registration database, which contains information on businesses that want to contract with the federal government. SBA officials told us that they expect to implement the government-use portion of SBA’s database by August 2011 and to provide for its basic maintenance and support despite reductions in the agency’s overall budget.

The government-use portion is intended to allow both participating agencies and award recipients to enter commercialization data in a comparable format to assist in program evaluation. SBA officials told us that they have worked with participating agencies to develop common metrics for commercialization data, as well as a standardized data collection instrument that will accommodate the various types of SBIR technologies the agencies fund to meet their different missions. These metrics, which will correspond to fields in the database, include the following: indication of whether an award resulted in a commercialized technology and whether other SBIR awards contributed to commercialization of the technology;  estimated investment (other than SBIR funding);  any patents applied for or received related to the award; and  any initial public offering, merger, or sale of the business that resulted, at least in part, from the award.

SBA officials told us in May 2011 that they plan to implement the metrics and data collection instrument in August 2011.

SBA is requesting that participating agencies voluntarily begin entering historical commercialization data into the government-use database before August 2011. To facilitate this process, SBA is working with its contractor to ensure that historical agency data can be matched to fields in the new database. Nevertheless, officials from SBA and participating SBIR agencies said that some agencies may not enter historical data or may be delayed in doing so because they either did not collect such data or do not have the data in electronic form. For example, NASA officials stated that much of their commercialization data are stored in paper format and expressed doubt that the agency would be able to convert the data into the required format for entering by SBA’s August deadline.

SBA officials also told us that, after the government-use portion of the database is available, some agencies may instruct applicants and award recipients to submit their commercialization data directly into the database. Other agencies, such as DOD, may continue to require applicants and recipients to submit commercialization data directly to the agencies, which would then upload the data into the database. As of May 2011, SBA officials were unsure which approach agencies would take, noting that agencies may wait to see how the database works before making a decision.

Without the Government- Use Portion of the SBA Database, Agencies Have Independently Collected Commercialization Data That Are Not Comparable

In the absence of the government-use portion of SBA’s database, the five participating SBIR agencies we reviewed have independently collected commercialization data that are not comparable. The agencies collected these data using various methods for their own purposes, as summarized in table 1.

In conducting their data collection efforts, agencies differed in the extent to which they asked award recipients to do the following, among other things: Identify the type of customer and the amount of sales or further investment for SBIR-funded technologies. For example, most agencies asked award recipients to report federal and nonfederal sales separately, but NIH and NSF asked award recipients to report combined sales.  Account for indirect sales and nonfinancial indicators of commercialization. NASA, NIH, and NSF asked award recipients to indicate whether an SBIR-funded technology had resulted in licensing agreements with other businesses to sell the technology, while DOD and DOE did not ask that question. NASA further asked award recipients to estimate the financial value of such agreements, while the other agencies did not. Similarly NASA, NIH, and NSF asked award recipients to indicate whether specific SBIR-funded technologies had resulted in patents, while DOD and DOE asked award recipients to report the total number of patents resulting from all their SBIR awards.  Quantify the dollar values of cumulative sales. While most agencies asked award recipients to report a specific dollar amount in cumulative sales resulting from their SBIR-funded technologies over a period of time, NIH asked award recipients to report such sales by choosing among ranges, beginning with “$50,000 or less” and extending to “$50,000,000 or more.” Because NIH has reported cumulative sales in ranges rather than specific dollar amounts, comparing its results with those reported by other agencies is difficult.

While each agency’s data collection efforts resulted in, among other information, estimates of total or average sales of SBIR technologies, differences in the agencies’ data collection efforts make it difficult to compare results across agencies. The following are examples of commercialization data reported by agencies:  DOD estimated that commercialization of SBIR technologies that it funded generated federal and nonfederal sales and non-SBIR funding of $22 billion on a program investment of $11 billion from 2000 through March 2010.  DOE estimated that, from 1986 through 2007, SBIR technologies developed by recipients of phase II awards resulted in a total of $2.4 billion in federal and nonfederal sales and $1.6 billion in non-SBIR investment. On average, award recipients reported receiving more than $3 million in sales related to SBIR-funded technologies. During the same period, DOE reported that it had invested $1.6 billion in phase I and II SBIR awards.  NASA estimated that, as of 2002, SBIR technologies developed by award recipients that received a phase II award from 1983 through 1996 had generated approximately $2.8 billion in federal and nonfederal sales and non-SBIR funding compared with $1.1 billion in SBIR investment from NASA.

In NIH’s 2002 survey, which covered 1992 through 2001, 27 percent of respondents reported an estimated total of $821 million in sales of SBIR technologies; the other respondents did not report any sales. NIH estimated that it invested $2.2 billion in phase I and phase II awards from 1992 through 2001. For the 2008 survey, which covered 2002 through 2006, 33 percent of respondents reported an estimated total of $396 million in federal and nonfederal sales of SBIR technologies. NIH estimated that it invested $2.7 billion in phase I and phase II awards from 2002 through 2006. NIH was the only agency we reviewed that reported sales lower than its SBIR investment for the periods it examined. According to NIH officials, many of the technologies that the agency supports through its SBIR program, such as drugs and medical devices, take longer to commercialize than those funded by other agencies because of the need for extensive clinical testing and regulatory approval.  NSF officials estimated that recipients marking the eighth anniversary of the receipt of their awards from July 2005 through May 2010 had realized a total of $1.05 billion in commercial revenue. NSF estimated that it invested $628 million in SBIR awards during roughly the same period.

Further, with the exception of DOD, agencies we reviewed generally did not take steps to verify commercialization data that they received from award recipients, so the accuracy of the data is largely unknown. As officials from some of the agencies in our review noted, award recipients may have an incentive to overstate their commercialization success in the hope of improving their prospects of receiving future SBIR awards. While SBA has worked with SBIR agencies to identify best practices in other areas of SBIR program management, it has not identified best practices for agencies to use in verifying the accuracy of commercialization data. Without consistent practices for verifying the accuracy of these data, the usefulness of the government-use portion of SBA’s database as a tool for evaluating the SBIR program’s success in increasing commercialization may be limited.

To verify the accuracy of award recipients’ commercialization data, DOD performs an annual review of all projects in its Company Commercialization Database, which contains the commercialization data it gathers from award recipients. This review includes checks to ensure that prior award recipients applying for new awards are not reporting the same project results more than once, substituting the results of one project for that of another, or incorrectly reporting sales to third parties. According to DOD officials, after its 2010 review, the agency sent approximately 300 e-mail queries to applicants whose reported commercialization data were identified as having potential problems. The officials said that applicants that do not respond to such queries are blocked from submitting further applications until concerns related to their commercialization reports are addressed. Even with these verification activities, however, Army officials expressed concern to us about the accuracy of the applicants’ self-reported commercialization data; these officials stated their preference for using data from the Federal Procurement Data System, which contains government information on federal contracts, including sales. Moreover, a Navy official acknowledged the possibility that additional verification activities, such as selective spot visits to SBIR award recipients, could further deter recipients from misrepresenting their commercialization success, although he noted that such activities would compete with other administrative priorities. Similarly, officials from DOE and NIH stated that additional verification activities would be useful but also said that they needed to devote program administration resources to higher priority activities, such as preparing solicitations and supporting review panels for applications.

Implementing the Government-Use Portion of the Database Should Improve the Comparability of Commercialization Data, but Long-standing Challenges May Still Impair Evaluation of Commercialization Progress

SBA’s implementation of the government-use portion of its database should improve the comparability of commercialization data available for programwide evaluation. Nevertheless, long-standing challenges may continue to impair programwide evaluation of progress in increasing commercialization of SBIR-funded technologies. As we reported in October 2006, notable among these challenges is that prior award recipients that are no longer participating in the SBIR program are not required to provide updated commercialization data and may prefer not to do so. For example, DOD indicated in written comments to us that, from 2008 to 2010, 46 percent of nonparticipating prior phase II award recipients did not provide updates despite DOD’s request that they update commercialization data annually after their awards ended. Similarly, in a report on its 2002 survey, NASA observed that many recipients of multiple awards elected not to respond to its survey despite “extensive telephone follow-up” and that many recipients that ultimately responded “would likely have preferred not to.” Some award recipients may be reluctant to provide commercialization data because the data are business-sensitive. SBA officials told us that mechanisms to require or encourage nonparticipating recipients to report their data need to be explored. A NASA official told us that effective incentives to encourage wider voluntary reporting might include publicizing commercial success or giving monetary prizes for success.

The difficulties agencies face in persuading prior award recipients to volunteer commercialization information can be compounded by challenges in maintaining contact with them. Specifically, prior award recipients can change names or personnel, go out of business, or be sold during the 10 or more years that it can take for an SBIR-funded technology to reach the marketplace. In NIH’s 2002 survey of award recipients, for example, the portion of the sample that was “unusable”—a group that consisted primarily of recipients that no longer existed or could not be found—increased from 2 percent in the first year after the end of the award to 52 percent in the tenth year.

Programwide evaluation—particularly efforts to compare commercialization success across agencies—can also be complicated by differences in the time required to commercialize various types of SBIR- funded technologies. Comparing agencies’ commercialization results at a given point in time may not present a true picture of each agency’s success because some agencies fund technologies that are relatively close to being market-ready while others fund technologies that need more extensive development or regulatory approval. Furthermore, as we have previously reported, the SBIR program’s other goals remain important, and comparisons that focus on commercialization may not adequately take into account progress toward these goals. For example, one agency official told us that some SBIR-funded technologies, such as those related to national security, may never have great commercial potential but are important to the agency’s mission.

Conclusions

DOD, DOE, NASA, NIH, and NSF have designed their SBIR solicitations to address the program’s purposes of using small businesses to meet federal R&D needs, stimulating technological innovation, and increasing commercialization of innovations derived from federal R&D efforts, and they have further addressed commercialization by providing technical assistance or matching funds to award recipients. These agencies have also conducted outreach and other activities to address the SBIR purpose of encouraging participation in technological innovation by small businesses owned by disadvantaged individuals and women. However, evaluation of progress in achieving the program’s purposes is impeded by a lack of accurate, comparable, and complete data on program results. For example, it is difficult to evaluate the program’s effectiveness in encouraging small businesses owned by disadvantaged individuals or women to participate in technological innovation because SBA does not collect data on the number of applications submitted by such businesses. It is also difficult to evaluate the program’s effectiveness in increasing commercialization of SBIR-funded technologies because, although agencies participating in the program have gathered commercialization data for their own purposes, comparable data on commercialization are not available across agencies. SBA’s planned implementation of a government-use portion of its database should go some way toward improving the comparability of the commercialization data as they are systematically collected using common metrics. However, the commercialization data that the database is intended to contain are largely self-reported by award recipients that may have an incentive to overstate their commercialization success. DOD has adopted practices for verifying the accuracy of commercialization data it collects from prior award recipients, but most of the participating agencies we reviewed did not verify the accuracy of commercialization data from their prior award recipients, and SBA has not identified best practices for participating agencies to use in doing so. As long as participating agencies do not consistently verify the accuracy of commercialization data, the usefulness of the government-use portion of SBA’s database as a tool for evaluating the SBIR program’s success in increasing commercialization may be limited.

Recommendations for Executive Action

To build upon efforts to implement a government-use database for program evaluation, we recommend that the Administrator of the Small Business Administration work with participating SBIR agencies to take the following two actions: collect data on the number of applications submitted by small businesses owned by disadvantaged individuals and women, and identify best practices for verifying the accuracy of data related to progress in increasing commercialization.

Agency Comments and Our Evaluation

We provided a draft of this report to SBA, the Departments of Defense and Energy, the National Aeronautics and Space Administration, the National Institutes of Health, and the National Science Foundation for review and comment. SBA generally agreed with our findings as well as our recommendations, which it offered an action plan to address. Specifically, with respect to our first recommendation, SBA stated that, beginning in fiscal year 2012, it plans to use its database to collect information from agencies about applicants that did not receive awards— information that could include whether the applicants were small businesses owned by disadvantaged individuals or women. Further, SBA indicated that it plans to hold a workshop in fall 2011 for participating SBIR agencies to share best practices for reaching out to small businesses owned by disadvantaged individuals. According to SBA, the workshop should result in a commitment from agencies to develop baselines for numbers of applications from such businesses. Regarding our second recommendation, SBA indicated that it will seek to identify best practices and methods for verifying the accuracy of commercialization data and will work with agencies toward implementation of those practices and methods. SBA also noted that its effort to collect commercialization data is intended to establish a baseline, against which SBA can review progress in increasing commercialization. SBA’s letter conveying its comments is contained in appendix II.

Among the SBIR participating agencies that we reviewed, DOE and NSF concurred with our recommendations and provided general comments, which are included in appendixes III and IV, respectively. Both DOE and NSF also made technical comments, which we have incorporated into our report as appropriate. In its general comments, DOE stated that it collects information on the number of applications submitted by small businesses owned by disadvantaged individuals and women and is willing to report the data to SBA. DOE further stated that it does not verify commercialization data because of resource limitations—not a belief that verification is of limited value—and it expressed an interest in learning about best practices for verification of these data. In addition, DOE commented that, until universal metrics are identified for measuring the success of SBIR programs across agencies, the compatibility of available data among agencies will remain a secondary concern. NSF stated that it concurs with the underlying goals of our recommendations. Moreover, NSF affirmed its commitment to implementation of a government-use database for program evaluation, collection of data on participation in small business innovation, and identification of best practices for verification of commercialization data. The remaining agencies—DOD, HHS, and NASA—neither agreed nor disagreed with our recommendations but provided technical comments, which we have incorporated into our report as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, Administrators of SBA and NASA, Secretaries of Defense and Energy, Directors of NIH and NSF, and other interested parties. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V.

Appendix I: Scope and Methodology

In conducting this study, we reviewed Small Business Innovation Research (SBIR) program-related activities of the Small Business Administration (SBA) and 5 of the 11 SBIR participating agencies—the Department of Defense (DOD), Department of Energy, National Aeronautics and Space Administration, Department of Health and Human Services’ National Institutes of Health (NIH), and National Science Foundation (NSF). For the two agencies with the largest SBIR budgets— DOD and NIH—we reviewed program activities conducted by the three participating subcomponent agencies with the largest SBIR budgets because some key activities are carried out at that level. Specifically, for DOD, we examined the SBIR programs of the Army, Air Force, and Navy, and for NIH, we examined the programs of the National Institute of Allergy and Infectious Diseases; the National Cancer Institute; and the National Heart, Lung, and Blood Institute. The five participating agencies we reviewed accounted for about 96 percent of the total dollars awarded by the program in fiscal year 2009. We reviewed applicable laws and regulations and literature on the SBIR program, including our prior reports and assessments by a committee of the National Academy of Sciences’ National Research Council. To obtain further context for our review, we attended two national conferences and a National Research Council workshop on the SBIR program, and we interviewed National Research Council staff with program expertise.

More specifically, to determine how participating agencies have addressed the SBIR program’s four overarching purposes when implementing their programs, we reviewed SBA documents and data, including SBA’s policy directive on implementation of the SBIR program, minutes from selected meetings of SBA and SBIR program directors, SBA’s SBIR annual report for fiscal year 2008 (the latest year for which an annual report was available), and data on the dollar value of SBIR awards by participating agencies in fiscal year 2009 (the latest year for which SBA could provide the data). We examined relevant documents from participating agencies for fiscal years 2008 through 2010, and for fiscal year 2011 when possible. Documents we reviewed included solicitations for applications issued by each of these agencies, instructions to applicants, minutes from meetings of SBA and SBIR program directors, performance plans and reports, descriptions of commercialization assistance provided to SBIR awardees, and minutes from meetings of agency advisory committees. In addition, we also identified and interviewed SBIR program officials at each agency and officials responsible for implementing programmatic goals. For these interviews, we asked a standard set of questions to help ensure that we obtained consistent information about the SBIR programs at each of the agencies. We also interviewed inspector general staff at NSF, which facilitated SBIR-related activities conducted by the Council of Inspectors General on Integrity and Efficiency. Finally, we interviewed representatives of trade associations about their views of the SBIR program. We selected the trade associations on the basis of their familiarity with the program, the technologies on which they focus, and whether their membership includes small businesses owned by disadvantaged groups and women. The views of the representatives of these associations cannot be generalized to other associations.

To determine the extent of SBIR program data available to evaluate progress in increasing commercialization of SBIR technologies, we reviewed documents related to SBA’s SBIR database, including terms of work, work schedules, and proposed guidance related to the development of the government-use portion of the database. For the five SBIR participating agencies whose programs we reviewed, we examined documents dating from 2002 through 2011; these documents reflected commercialization data for SBIR award recipients that had received awards from 1983 (the first year in which agencies issued SBIR awards) through 2010. The documents we reviewed included surveys and other data collection instruments that the agencies used to gather commercialization information from award recipients; reports on data collection results, including any information on SBIR award spending during the years corresponding to those covered in each of the commercialization data collection efforts; and anecdotal descriptions of commercialization success. We also reviewed agency solicitations from fiscal years 2008 through 2010—and for fiscal year 2011 when possible— that contained reporting requirements for award recipients. We interviewed officials at SBA and each of the five participating agencies included in our review to obtain information on the specific commercialization metrics they use to monitor the commercialization experience of award recipients, the history of each agency’s data collection efforts, and the agencies’ experience in obtaining such information from current and past award recipients.

We conducted this performance audit from June 2010 to August 2011, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence we obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Small Business Administration

Appendix III: Comments from the Department of Energy

Appendix IV: Comments from the National Science Foundation

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, key contributors to this report include Cheryl Williams, Assistant Director; Antoinette Capaccio; Stephen Carter; Nancy Crothers; Laurie Ellington; Cindy Gilbert; Cynthia Norris; Christine Senteno; and Kiki Theodoropoulos.