Small Business Innovation Research: SBA Should Work with Agencies to Improve the Data Available for Program Evaluation
Published: 2011-08-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Federal agencies with a budget of at least $100 million for research and development (R&D) conducted by others must participate in the Small Business Innovation Research (SBIR) program. SBIR has four purposes: meet federal R&D needs; stimulate technological innovation; increase commercialization (e.g., sales) of innovations based on federal R&D; and encourage participation in innovation by small businesses owned by disadvantaged individuals and women. The Small Business Administration (SBA) oversees the efforts of participating agencies, which make awards to small businesses using SBIR funds. Congress directed SBA to develop a database with commercialization data for government use in evaluating the program. GAO was asked to determine (1) how agencies have addressed SBIR's purposes and (2) the extent of data available to evaluate progress in increasing commercialization. GAO analyzed program documents and interviewed officials at SBA and five agencies that accounted for about 96 percent of SBIR awards.
section_title: What GAO Found
paragraphs: For fiscal years 2008 through 2011, the participating agencies GAO reviewed--the Department of Defense (DOD), the Department of Energy (DOE), the National Aeronautics and Space Administration, the Department of Health and Human Services' National Institutes of Health, and the National Science Foundation (NSF)--addressed SBIR's purposes through solicitations for award applications, technical assistance or matching funds programs, and outreach. In particular, the agencies addressed SBIR purposes related to meeting federal R&D needs and stimulating technological innovation through their solicitations, which included research topics that were designed to meet agencies' respective R&D or mission needs. Agencies also addressed commercialization of innovations through solicitations, as well as through technical assistance for award recipients or through matching funds programs. To provide technical assistance, the agencies contracted with vendors and consultants for help in developing business plans and identifying potential customers for SBIR award recipients, among other things. Agency matching funds programs provided additional SBIR funds to award recipients that obtained commitments from outside investors. Agencies generally addressed the remaining SBIR purpose, encouraging participation by small businesses owned by disadvantaged individuals and women, through outreach activities aimed at a broader audience, such as sharing information on Web sites. However, the effectiveness of these efforts is difficult to evaluate, in part because SBA does not collect data on the number of SBIR applications submitted by such businesses, thus hindering analysis of trends in their submission of applications. Comparable data are not available across participating agencies to evaluate progress in increasing commercialization of SBIR technologies. SBA has not yet expanded an existing database to include commercialization data for program evaluation, but the agency has hired a contractor and allocated funds to develop the expanded database by August 2011. SBA has also worked with participating agencies to develop common metrics for commercialization. In the absence of the expanded database, agencies have independently gathered commercialization data for their own use that are not comparable. In collecting these data, agencies differed in the types of data collection instruments used, dates the instruments were administered, award recipient populations queried, and types of data requested. Furthermore, with the exception of DOD, agencies that GAO reviewed did not generally take steps to verify commercialization data they collected from award recipients, so the accuracy of the data is largely unknown. SBA has worked with SBIR agencies to identify best practices in other areas of program management but has not identified best practices for agencies to use in verifying the accuracy of commercialization data. Implementing the expanded database should improve the comparability of commercialization data available, but a lack of consistent practices for verifying the accuracy of these data may limit their usefulness for programwide evaluation.
section_title: What GAO Recommends
paragraphs: GAO recommends that SBA work with participating agencies to (1) collect data on applications from small businesses owned by disadvantaged individuals and women and (2) identify best practices for verification of commercialization data. SBA, DOE, and NSF generally agreed with these recommendations; the other agencies GAO reviewed neither agreed nor disagreed.

section_title: Letter
paragraphs: Small businesses have been collectively described as an engine for  economic growth and innovation. According to the Small Business  Administration (SBA), such businesses have been responsible for  creating 64 percent of net new jobs over the past 15 years and have  produced 13 times more patents per employee than larger firms. To  assist small businesses in undertaking and obtaining the benefits of  research and development (R&D), Congress first passed legislation  authorizing the Small Business Innovation Research (SBIR) program in  1982. SBA’s Office of Technology administers the program, which  presently has four overarching purposes: to use small businesses to meet  federal R&D needs, stimulate technological innovation, increase  commercialization of innovations derived from federal R&D efforts, and  encourage participation in technological innovation by small businesses  owned by disadvantaged individuals and women. Since the program  was first authorized in 1982, there has been increased congressional  interest in its results, particularly the commercialization aspect.
Every federal agency with a budget of $100 million or more for extramural  R&D—which is conducted by nonfederal employees outside federal  facilities—is required to establish and operate an SBIR program funded  by 2.5 percent of that budget. Eleven agencies currently participate in the  program. In fiscal year 2009 (the most recent year for which SBA data  are available), the 11 agencies reported more than $2.2 billion in SBIR  awards. Although each of these agencies manages its own program, SBA  oversees and coordinates agency efforts by setting policy, collecting  program data, reviewing agency progress, and reporting annually to  Congress.
Since the creation of the SBIR program 29 years ago, it has been  reauthorized, modified, and extended by Congress at various times. The  Small Business Reauthorization Act of 2000 extended the program  through September 30, 2008. This act was succeeded by a series of  temporary extensions, the most recent of which extended the program  through September 30, 2011.
In this context, you asked us to examine the practices of participating  agencies. Accordingly, we agreed to determine (1) how participating  agencies have addressed the SBIR program’s four overarching purposes  when implementing their programs and (2) the extent of SBIR program  data available to evaluate progress in increasing commercialization of  SBIR technologies.
Our review of the program focused on SBA and five agencies that  accounted for about 96 percent of the total dollars awarded by the  program in 2009: the Department of Defense (DOD), the Department of  Energy (DOE), the National Aeronautics and Space Administration  (NASA), the Department of Health and Human Services’ National  Institutes of Health (NIH), and the National Science Foundation (NSF).  At two of these agencies—DOD and NIH—we also reviewed program  activities conducted by the three subcomponent agencies with the largest  SBIR budgets because some key activities are carried out at that level.  Specifically, for DOD, we examined the SBIR programs of the Army, Air  Force, and Navy and for NIH, we examined the programs of the National  Institute of Allergy and Infectious Diseases; the National Cancer Institute;  and the National Heart, Lung, and Blood Institute. In conducting our  review, we analyzed documentation from these agencies for fiscal years  2008 through 2010 and, when possible, for fiscal year 2011. The  documentation we reviewed included, among other things, policy  guidance, solicitations for applications for SBIR awards, and descriptions  of commercialization assistance provided to SBIR awardees. We also  reviewed applicable laws and regulations. In addition, we interviewed  SBIR program officials at the agencies, as well as inspector general staff  at NSF and knowledgeable stakeholders, including staff of the National  Academy of Sciences’ National Research Council and representatives of  trade associations. We selected the trade associations based on their  familiarity with the program, the technologies on which they focus, and  whether their membership includes small businesses owned by  disadvantaged individuals and women. To obtain further context for our  review, we attended two national conferences on the SBIR program, as  well as a workshop conducted by the National Research Council. We  conducted this performance audit from June 2010 to August 2011, in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence we obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. A more detailed  description of the scope of our review and the methods we used is  contained in appendix I.
subsections: section_title: Background
paragraphs: SBA coordinates and oversees the efforts of the 11 agencies currently  participating in the SBIR program. SBA coordinates the agencies’  schedules for issuing solicitations—announcements of opportunities for  small businesses to apply for awards—and provides access to these  solicitations through its Web site. As part of its oversight effort, SBA  collects SBIR data from the participating agencies, aggregates the data,  and uses the data to, among other things, monitor the program and report  to Congress. SBA also provides guidance to participating agencies on the  general conduct and operation of the program, which it periodically  updates, for example, in response to changes in the program’s  authorizing legislation. Under the legislation and SBA’s guidance,  agencies have considerable flexibility to design their programs. For  example, each agency determines, in consultation with SBA, such items  as the number of solicitations to be issued during a fiscal year and the  dates applications are due. Agencies also have discretion to determine  what type of research to include in their solicitations, how to review  applications for technical and scientific merit, which applications to fund,  and the size of the award, among other things.
The Small Business Innovation Development Act of 1982 provided for a  competitive three-phased SBIR program. In phase I, participating  agencies award up to $150,000 for a period of about 6 to 9 months to  small businesses to conduct experimental or theoretical R&D. Small  businesses whose phase I projects demonstrate scientific and technical  merit, in addition to commercial potential, may compete for phase II  awards of up to $1 million to continue the R&D for an additional period,  normally not to exceed 2 years. Phase I and II award funds may be used  for costs related to conducting the research, such as salaries, fringe  benefits, equipment, and consulting services, as well as for profits and  fees. To be eligible for a phase I or II SBIR award, a business must have  500 or fewer employees, be organized for profit with a place of business  in the United States, and operate primarily in the United States or make a  significant contribution to the U.S. economy. Generally, a business must  also be at least 51 percent owned and controlled by one or more  individuals who are U.S. citizens or permanent resident aliens. These  eligibility requirements apply at the time that a phase I or II award is  made. During phase III, businesses must secure non-SBIR funding to  develop the commercial potential of the innovative technologies resulting  from their SBIR projects; such funding may come from the private  sector, federal agencies, or other sources.
As the program has been reauthorized over the years, legislation has  established a number of requirements related to the program’s purposes.  For example, the Small Business Research and Development  Enhancement Act of 1992 directed SBA to make more information  available about the SBIR program, particularly about participation by  small businesses owned by disadvantaged individuals and women, and  required that agencies increase their outreach to such businesses. In  addition, the Small Business Reauthorization Act of 2000 directed that  applicants for phase II SBIR awards be required to submit  commercialization plans, and it mandated that SBA develop, maintain,  and make available to the public a database that contained SBIR award  data. In addition, the act required SBA to develop and maintain, by June  2001, a restricted government-use database that would contain award- related data from the public database, as well as additional confidential  data that would be accessible only to government agencies and other  authorized users. The act stated that this database would be used  exclusively for program evaluation—which, as we have noted in past  work, involves the systematic collection and analysis of accurate,  comparable, and complete data on program results. The act required  the government-use database to contain, among other things, data that  applicants for phase II awards would be required to supply on the  commercialization success of any prior phase II awards, such as data on  sales of or additional investment in the technologies funded under the  awards. The act further specified that the government-use database  would contain annual updates to these data, which phase II award  recipients would be requested to voluntarily provide for 5 years after the  period covered by the award.
To accomplish this mandate, SBA envisioned expanding an electronic  database, known as Tech-Net, which it had developed in the late 1990s,  into two sections: a public-use portion and a government-use portion  containing commercialization data. The public-use portion of the database  has been available since 2000, according to SBA, and it contains such  award-related data as the phase of the award, amount of the award,  name and location of the business receiving the award, an abstract of the  work to be conducted under the award, and whether the business is  categorized as owned by disadvantaged individuals or women. In October  2006, however, we reported that some SBIR agencies did not  consistently provide or correctly format the awards-related data for  several fields in the public-use portion of the database. For example,  two of the eight agencies we reviewed had not consistently provided data  on whether the businesses receiving the awards were categorized as  owned by disadvantaged individuals or women. At that time, we also  reported that SBA had not implemented the government-use portion of  the database, primarily, according to SBA officials, because of increased  security requirements for the database, agency management changes,  and budgetary constraints. Additionally, we reported that while five of the  agencies we reviewed had systematically collected commercialization  data, their data collection efforts differed in ways that made it challenging  to evaluate the program across agencies. In August 2009, we testified  before Congress that SBA said the database would no longer accept  incorrectly formatted awards-related data from participating agencies.
A committee of the National Academy of Sciences’ National Research  Council has conducted a series of assessments of the SBIR program,  both within and across agencies, as part of a legislatively mandated  study. The results were summarized in a single report, in which the  committee stated that SBIR is making significant progress in achieving  congressional goals. The study concluded that the SBIR program is  “sound in concept and effective in practice.” The study also  recommended changes that could make the program more effective.  Among other things, the study recommended that SBA and participating  agencies improve the collection of data that track participation in the SBIR  program by businesses owned by disadvantaged individuals and women,  develop targeted outreach to such businesses that is based on an  analysis of factors that affect their participation, and improve  documentation of commercialization success. The National Research  Council is now undertaking another round of assessments to provide a  second snapshot of the program’s progress in achieving its legislative  purposes.
section_title: Agencies Addressed SBIR’s Purposes through Solicitations, Technical Assistance, Matching Funds Programs, and Outreach
paragraphs: For fiscal years 2008 through 2011, the five participating agencies we  reviewed addressed the SBIR purposes of using small business to meet  federal R&D needs and stimulating technological innovation through their  solicitations. Agencies also used solicitations, as well as technical  assistance or matching funds programs, to address the SBIR purpose of  increasing commercialization of innovations derived from federal R&D  efforts. To address the remaining program purpose—encouraging  participation in technological innovation by small businesses owned by  disadvantaged individuals and women—agencies relied mainly on  outreach activities aimed at a broader audience.
subsections: section_title: Agencies Addressed the SBIR Purposes Related to Meeting Federal Research Needs and Stimulating Innovation through Their Solicitations
paragraphs: All of the participating agencies that we reviewed designed the SBIR  solicitations that they issued for fiscal years 2008 through 2011 to meet  federal R&D or mission needs and stimulate technological innovation.  All of these agencies selected research topics for their solicitations that  were designed to meet their respective R&D or mission needs and  specified that applications would be evaluated on the basis of  responsiveness to those topics. The agencies that purchase SBIR-funded  technologies for their own use—DOD, DOE, and NASA—tended to select  solicitation topics that met specific agency needs for R&D. For example,  in fiscal year 2011, DOD solicited applications to develop a fuel cell  system capable of converting ethanol into electricity in an efficient, small,  lightweight, portable power system. According to the solicitation, such  advanced fuel cell systems could provide soldiers power to complement  batteries and to charge rechargeable batteries, reducing the number of  batteries required for extended time in the field. In contrast, NIH and NSF,  which generally do not purchase SBIR-funded technologies, tended to  issue solicitations for a broader spectrum of R&D to support their  missions of advancing biomedical and other scientific and engineering  disciplines. Among the agencies we reviewed, NIH and its components  gave applicants the most leeway in addressing agency needs: rather than  limiting applications to specific research topics identified in solicitations,  NIH and its components usually listed suggested topics and encouraged  applicants to propose innovative projects that fit the agency’s mission.
Concerning innovation, each of the agencies included instructions in its  SBIR solicitations about the type of information applicants had to provide  about the innovativeness of the proposed work. For example, NASA  informed phase I and II applicants that a competitive application would  describe the proposed innovation relative to state-of-the-art knowledge in  the field, among other things. In addition, these agencies explained to  applicants how reviewers would consider evidence of the innovativeness  of the applicants’ proposed research approaches. For example, in its  fiscal year 2010 solicitation, NSF stated that applications would be  evaluated, in part, on the basis of whether they reflected state-of-the-art  knowledge in the major research activities proposed and whether the  work was likely to advance state-of-the-art knowledge.
section_title: Agencies Worked to Increase Commercialization of Innovations through Solicitations, Technical Assistance, and Matching Funds Programs
paragraphs: The participating agencies we reviewed addressed the SBIR purpose of  increasing commercialization of innovations through solicitations, as well  as through technical assistance or matching funds programs.
Solicitations. Of the five agencies we reviewed, all but NIH required in  their solicitations for fiscal years 2008 through 2011 that applicants for  phase I awards submit a commercialization strategy demonstrating that  the applicants had taken steps such as identifying a market for their SBIR  technologies, planning to secure financing, and estimating expected  future sales. For phase II awards, all of the agencies we reviewed  required that applicants submit a commercialization plan. In general, the  solicitations we reviewed required that phase II commercialization plans  discuss the potential market and competitors; the qualifications of key  management and technical personnel; as well as financing, marketing,  and manufacturing plans, among other things.
The agencies we reviewed differed in their stated processes for  evaluating the commercial potential of applications. For example, DOD  guidance to applicants outlined a systematic process for how the agency  would consider commercialization potential when evaluating applications  submitted by small businesses that had received multiple prior awards.  DOD indicated that, under this process, it would assign a  commercialization achievement score to applicants that had completed  the work for four or more phase II awards from any agency; this score  would reflect how the applicants’ commercialization experience compared  with historical averages. Applicants whose scores fell within the lowest 20  percent would not be allowed to receive more than half the maximum  number of points possible for commercialization potential, which was to  be assessed on the basis of several factors, including the  commercialization strategy or plan. DOD guidance stated that businesses  with fewer than four completed phase II awards would not be affected by  the absence of a commercialization achievement score. Although the  other four agencies we reviewed did not outline as systematic a process  for evaluating past commercialization success as a gauge of  commercialization potential, they still indicated that commercialization  potential would be taken into account in reviewing applications. For  example, DOE’s solicitation instructions encouraged phase I applicants to  seek firm commitments for private-sector or non-SBIR federal funding  prior to applying for a phase II award. The instructions further stated that  phase II applicants that obtained such commitments were more likely to  receive full credit for commercialization planning during the evaluation of  their applications. In the case of NSF, solicitation instructions stated that  proposals are usually reviewed by 3 to 10 outside experts in fields related  to the proposal; according to NSF officials, these reviewers have  business experience. NSF’s solicitation instructions further stated that the  agency would not review applications that lacked sufficient information on  commercial potential.
In 2010, two agencies we reviewed also issued SBIR solicitations under  new programs that were explicitly oriented toward increasing  commercialization. Specifically, in July 2010, DOE launched a program  under which it solicited applications for phase III of SBIR, the  commercialization phase. DOE documents indicated that the agency  would make available approximately $30 million, including funding from  the American Recovery and Reinvestment Act (Recovery Act), for  phase III awards, which are intended to allow businesses to pursue  commercial applications of work performed under phase I and II awards.  In addition, NIH’s National Cancer Institute began a program under which  it solicited phase I applications to continue development of technologies  that have originated in its laboratories, with the goal of advancing these  technologies toward commercial products. SBA has designated the use  of the SBIR program to encourage commercialization of agencies’ internal  research as a best practice on its SBIR Web site.
Technical assistance. All five agencies included in our review provided  technical assistance to help award recipients build their capacity to  commercialize their technologies. To provide the assistance, the agencies  contracted with vendors and consultants who have experience in bringing  technologies to market. With the exception of NASA, the agencies  supported the technical assistance at least in part through the use of  SBIR funds.
In fiscal years 2008 to 2010, DOD, DOE, NIH, and NSF spent SBIR funds  on technical assistance for phase I award recipients. Some of the  assistance was in the form of interactive training Webinars or online tools  directed toward a broad spectrum of SBIR applicants and award  recipients. For example, the Navy offered phase I award recipients the  use of a software tool, known as WebTRIMS, that helps identify, quantify,  and track risks associated with SBIR technology development and covers  topics such as contracting strategies, business and transition planning,  and manufacturing readiness. Other phase I assistance was more  customized. For example, DOE offered phase I award recipients  customized technical assistance designed to help them develop a  commercialization plan complete with an implementation schedule and  suggestions for product design. Similarly, on a first-come, first-served  basis, NIH offered phase I award recipients assessments of their SBIR- funded technologies’ likely niche in the existing commercial market, which  could help recipients develop commercialization plans for phase II  applications. Additionally, NSF offered phase I award recipients  personalized mentoring and coaching sessions with an advisor.
According to NSF officials, 92 percent of phase I recipients chose to  participate in the technical assistance program in 2010. NASA did not  provide technical assistance for phase I award recipients; NASA officials  told us they believed technical assistance would have the most utility for  phase II NASA award recipients.
During at least a portion of the period we reviewed, all five agencies  offered individualized technical assistance for phase II award recipients,  although DOE curtailed such assistance in 2010, and NASA discontinued  its assistance in 2008. Award recipients were selected for assistance on  the basis of factors such as recommendations from SBIR program staff  and the award recipients’ potential for rapidly moving their technologies to  phase III. The assistance consisted of in-depth training and one-on-one  assistance from advisors and industry experts. For example, as part of its  Commercialization Pilot Program, the Army assisted selected phase II  SBIR award recipients in assessing commercialization potential,  developing business plans, and matching their technologies with potential  government and industry customers. At DOE, staff could nominate phase  II award recipients for assistance in preparing to negotiate business  deals, for example, including joint ventures and licensing agreements for  use of their technologies. DOE officials told us that in 2010 the agency  curtailed its use of SBIR funds for phase II technical assistance, spending  such funds on assistance only for award recipients that had specifically  budgeted for it in their applications. In 2007 and 2008, NASA partnered  with the Navy to pilot a technical assistance program for NASA phase II  recipients. The program was designed to help SBIR businesses develop  a plan for transitioning to phase III, among other things. In 2007, 17  phase II companies with 19 SBIR projects participated in the program,  and in the following year, 19 phase II companies with 20 SBIR projects  participated. The program was not renewed for 2009; NASA officials told  us that they believed the program was generally successful, but that they  preferred to use SBIR funds to make larger awards. NIH offered  selected current or past phase II award recipients the opportunity to work  one-on-one with an advisor over a 9-month period to develop business  plans to commercialize their technologies, as well as to prepare materials  to help attract potential investors or partners. Since 2004, almost 700  award recipients have received the assistance, including the 80 award  recipients currently participating, according to NIH officials. Finally, NSF  offered customized assistance to phase II award recipients through its  Innovation Accelerator Initiative. According to NSF officials, through this  initiative, award recipients received help in connecting with potential  investors and negotiating company acquisitions and mergers. NSF  officials told us that, in 2010, approximately 33 percent of NSF’s phase II  recipients received this assistance.
In some cases, agencies that we reviewed used non-SBIR funds to  broaden the scope of the technical assistance they provided to help  award recipients commercialize their technologies. For example, DOD  used non-SBIR funds to host its annual Beyond Phase II Conference and  Technology Showcase, a 3-day event that features matchmaking  sessions with SBIR award recipients and prime contractors. Similarly, the  Navy used non-SBIR funds to maintain databases with advanced  searching capability to help award recipients identify potential business  partners. The Navy also used non-SBIR funds for its Transition  Assistance Program, which provides individualized help with  commercialization planning, culminating in a conference designed to  facilitate interaction with potential business partners. Moreover, in 2011,  the National Cancer Institute launched its Regulatory Assistance Program  using non-SBIR funds. According to agency officials and information from  the agency’s Web site, this program provides SBIR award recipients time  with consultants experienced in various regulatory requirements—such as  those for anticancer therapies, imaging technologies, and medical  devices—to prepare strategies for obtaining regulatory approvals required  before the technologies can be commercialized. The National Cancer  Institute also used non-SBIR funds to support its Investor Forum, which  provides competitively selected SBIR award recipients an opportunity to  showcase their technologies and enter into discussions with the biotech  investment community. In 2010, 14 award recipients that were selected  on the basis of strength of research, impact on cancer, product  development, and market potential participated in the forum along with  more than 175 potential investors, according to the agency’s Web site.
Matching funds programs. Through matching funds programs, agencies  provide additional SBIR funds to award recipients that obtain monetary  commitments above certain thresholds from outside investors. SBA has  designated matching funds programs as a best practice on its SBIR Web  site, and all of the agencies we reviewed except DOE have established  such programs. For example, for award recipients that obtain a  minimum of $100,000 from an outside investor, NSF will match up to   50 percent of the outside investment for a maximum of $500,000 in NSF  matching funds. NASA and NIH officials said that matching funds  programs encourage outside investment during the early stages of  R&D—a time when many investors are reluctant to invest. In particular,  officials at the National Cancer Institute said that matching funds can help  attract outside investment because they can be used as leverage to  increase investors’ potential returns. DOD and NSF offer matching funds  to award recipients at the end of phase I and during phase II, while NASA  and the National Cancer Institute offer matching funds during phase II.  DOE has not established a matching funds program for its SBIR program.  DOE officials told us, however, that they are exploring whether to do so  and have held discussions with other SBIR participating agencies about  their matching funds programs.
Officials at DOD, NASA, and NIH said they have not collected data to  compare the commercialization success of recipients that received  matching funds with the success of those that did not. NSF conducted a  study to assess the effect of the $18 million in matching funds it invested  in fiscal year 2006 for 48 phase II award recipients that had raised a total  of $58 million from outside investors. According to NSF officials, results  of this study showed that, in the 5 years following the start of these phase  II projects, 70 percent of recipients that had received matching funds  achieved commercial success compared with a 30 percent success rate  for recipients that had not received such funds.
section_title: Agencies Generally Encouraged Participation by Disadvantaged Individuals and Women through Outreach Activities Directed toward a Broader Audience
paragraphs: SBA’s guidance states that small businesses owned by disadvantaged  individuals and women must compete for SBIR awards on the same basis  as all other small businesses. However, to meet requirements for greater  outreach to small businesses owned by disadvantaged individuals and  women, SBA has encouraged participating SBIR agencies to reach out to  such businesses and to develop methods that encourage their  participation. SBA has also raised the topic of outreach during recent  quarterly meetings of agency SBIR program managers. Officials at all of  the agencies we reviewed told us they generally reach out to such  businesses through activities directed toward a broader audience, such  as by attending SBIR national conferences and industry-sponsored  events and by sharing information via Web sites or e-mail lists. Agency  officials also noted that they try to accommodate requests for speakers at  events sponsored by, or likely to be attended by, small businesses owned  by disadvantaged individuals and women—for example, events  sponsored by trade organizations for minority- or women-owned  businesses. However, officials from some trade organizations for  businesses owned by disadvantaged individuals and women told us that  the outreach of agencies we reviewed was often ineffective in educating  the organizations’ members about the SBIR program.
Of the agencies we reviewed, NIH and NSF have made specific efforts,  including the following, to improve their outreach:    For fiscal years 2010 and 2011, NIH developed a goal to increase  awareness of its SBIR program among businesses owned by  disadvantaged individuals and women, and it outlined specific  activities aimed at reaching this goal.    Both NIH and NSF offered various fellowships for postdoctoral  research conducted by disadvantaged individuals and women; these  fellowships were available to support SBIR projects, as well as other  research.
In 2010, NSF assigned a full-time staff member to help it develop a  plan to increase participation in SBIR by businesses owned by  disadvantaged individuals and women in response to a  recommendation from its SBIR advisory committee.    Through a review of academic literature, as well as informal polling of  NSF applicants and award recipients, NSF has identified barriers to  SBIR participation by small businesses owned by disadvantaged  individuals and women, NSF officials told us. These barriers include  disparities in the owners’ levels of education and access to capital  compared with those of other entrepreneurs. To address identified  barriers, NSF is, among other things, establishing partnerships with  industry and academia to expose African American, Latino, and other  college students to entrepreneurship in scientific and technical fields,  according to NSF officials.
Evaluation of the effectiveness of agencies’ outreach efforts is hindered  by a lack of accurate and complete data. Although SBA collects data on  the number and dollar value of awards to small businesses owned by  disadvantaged individuals and women, SBA officials told us that they  cannot accurately tabulate data on such awards, particularly awards to  women-owned businesses, because of inconsistencies in the data on  business ownership. According to the officials, SBA has taken steps to  correct the inconsistencies for data submitted after 2006 but has not done  so for earlier years. Moreover, SBA does not collect data on the number  of applications submitted by businesses owned by disadvantaged  individuals and women. As a result, SBA’s data do not allow for an  examination of trends in the submission of applications from such  businesses, analysis of the percentage of applications from these  businesses that lead to awards, or correlation of these trends and  percentages with outreach efforts. SBA officials told us in March 2011  that they were considering whether their database should include  information on the numbers of applications submitted by these  businesses.
section_title: Comparable Data Are Not Available across Participating Agencies to Evaluate Progress in Increasing Commercialization of SBIR Technologies
paragraphs: SBA has not yet developed the government-use portion of its database  for collecting comparable commercialization data on SBIR technologies,  but it is taking steps to do so. In the interim, agencies have, for their own  purposes, independently gathered commercialization data that are not  comparable; the accuracy of these data is largely unknown. Implementing  the government-use portion of the database should improve the  comparability of the data. However, programwide evaluation of progress  in increasing commercialization may continue to be impaired by long- standing challenges.
subsections: section_title: SBA Has Not Implemented the Government-Use Portion of Its Database Intended to Collect Comparable Commercialization Data for Program Evaluation but Is Taking Steps to Do So
paragraphs: As of June 2011, SBA had not met the legislative mandate to develop and  implement, by June 2001, a government-use database that can provide  data on commercialization for evaluating the SBIR program. However,  the agency’s efforts to develop such a database recently gained  additional prominence and resources. Specifically, SBA linked  development of the government-use portion of its database to one of the  agency’s high-priority performance goals for fiscal years 2011 and 2012.  Additionally, in September 2010, SBA allocated $1.4 million in Recovery  Act funds to hire a new contractor to develop the government-use  portion’s capacity to accept commercialization data submitted by  participating SBIR agencies and award recipients, as well as to make  other improvements to the database. For example, SBA said that it has  been working with the contractor to consolidate data on previous awards.  SBA officials said that past award recipients have been assigned unique  identifiers that will be used to track awards issued to those recipients over  the lifetime of the SBIR program; unique identifiers are also to be  assigned to small businesses newly entering the program. In the future,  SBA intends for the unique identifiers to allow agencies to validate  business information by comparing it against information in other federal  databases such as the Central Contractor Registration database, which  contains information on businesses that want to contract with the federal  government. SBA officials told us that they expect to implement the  government-use portion of SBA’s database by August 2011 and to  provide for its basic maintenance and support despite reductions in the  agency’s overall budget.
The government-use portion is intended to allow both participating  agencies and award recipients to enter commercialization data in a  comparable format to assist in program evaluation. SBA officials told us  that they have worked with participating agencies to develop common  metrics for commercialization data, as well as a standardized data  collection instrument that will accommodate the various types of SBIR  technologies the agencies fund to meet their different missions. These  metrics, which will correspond to fields in the database, include the  following:  indication of whether an award resulted in a commercialized  technology and whether other SBIR awards contributed to  commercialization of the technology;    estimated investment (other than SBIR funding);    any patents applied for or received related to the award; and    any initial public offering, merger, or sale of the business that resulted,  at least in part, from the award.
SBA officials told us in May 2011 that they plan to implement the metrics  and data collection instrument in August 2011.
SBA is requesting that participating agencies voluntarily begin entering  historical commercialization data into the government-use database  before August 2011. To facilitate this process, SBA is working with its  contractor to ensure that historical agency data can be matched to fields  in the new database. Nevertheless, officials from SBA and participating  SBIR agencies said that some agencies may not enter historical data or  may be delayed in doing so because they either did not collect such data  or do not have the data in electronic form. For example, NASA officials  stated that much of their commercialization data are stored in paper  format and expressed doubt that the agency would be able to convert the  data into the required format for entering by SBA’s August deadline.
SBA officials also told us that, after the government-use portion of the  database is available, some agencies may instruct applicants and award  recipients to submit their commercialization data directly into the  database. Other agencies, such as DOD, may continue to require  applicants and recipients to submit commercialization data directly to the  agencies, which would then upload the data into the database. As of May  2011, SBA officials were unsure which approach agencies would take,  noting that agencies may wait to see how the database works before  making a decision.
section_title: Without the Government- Use Portion of the SBA Database, Agencies Have Independently Collected Commercialization Data That Are Not Comparable
paragraphs: In the absence of the government-use portion of SBA’s database, the five  participating SBIR agencies we reviewed have independently collected  commercialization data that are not comparable. The agencies collected  these data using various methods for their own purposes, as summarized  in table 1.
In conducting their data collection efforts, agencies differed in the extent  to which they asked award recipients to do the following, among other  things:  Identify the type of customer and the amount of sales or further  investment for SBIR-funded technologies. For example, most  agencies asked award recipients to report federal and nonfederal  sales separately, but NIH and NSF asked award recipients to report  combined sales.    Account for indirect sales and nonfinancial indicators of  commercialization. NASA, NIH, and NSF asked award recipients to  indicate whether an SBIR-funded technology had resulted in licensing  agreements with other businesses to sell the technology, while DOD  and DOE did not ask that question. NASA further asked award  recipients to estimate the financial value of such agreements, while  the other agencies did not. Similarly NASA, NIH, and NSF asked  award recipients to indicate whether specific SBIR-funded  technologies had resulted in patents, while DOD and DOE asked  award recipients to report the total number of patents resulting from all  their SBIR awards.    Quantify the dollar values of cumulative sales. While most agencies  asked award recipients to report a specific dollar amount in  cumulative sales resulting from their SBIR-funded technologies over a  period of time, NIH asked award recipients to report such sales by  choosing among ranges, beginning with “$50,000 or less” and  extending to “$50,000,000 or more.” Because NIH has reported  cumulative sales in ranges rather than specific dollar amounts,  comparing its results with those reported by other agencies is difficult.
While each agency’s data collection efforts resulted in, among other  information, estimates of total or average sales of SBIR technologies,  differences in the agencies’ data collection efforts make it difficult to  compare results across agencies. The following are examples of  commercialization data reported by agencies:    DOD estimated that commercialization of SBIR technologies that it  funded generated federal and nonfederal sales and non-SBIR funding  of $22 billion on a program investment of $11 billion from 2000  through March 2010.    DOE estimated that, from 1986 through 2007, SBIR technologies  developed by recipients of phase II awards resulted in a total of $2.4  billion in federal and nonfederal sales and $1.6 billion in non-SBIR  investment. On average, award recipients reported receiving more  than $3 million in sales related to SBIR-funded technologies. During  the same period, DOE reported that it had invested $1.6 billion in  phase I and II SBIR awards.    NASA estimated that, as of 2002, SBIR technologies developed by  award recipients that received a phase II award from 1983 through  1996 had generated approximately $2.8 billion in federal and  nonfederal sales and non-SBIR funding compared with $1.1 billion in  SBIR investment from NASA.
In NIH’s 2002 survey, which covered 1992 through 2001, 27 percent  of respondents reported an estimated total of $821 million in sales of  SBIR technologies; the other respondents did not report any sales.  NIH estimated that it invested $2.2 billion in phase I and phase II  awards from 1992 through 2001. For the 2008 survey, which covered  2002 through 2006, 33 percent of respondents reported an estimated  total of $396 million in federal and nonfederal sales of SBIR  technologies. NIH estimated that it invested $2.7 billion in phase I  and phase II awards from 2002 through 2006. NIH was the only  agency we reviewed that reported sales lower than its SBIR  investment for the periods it examined. According to NIH officials,  many of the technologies that the agency supports through its SBIR  program, such as drugs and medical devices, take longer to  commercialize than those funded by other agencies because of the  need for extensive clinical testing and regulatory approval.    NSF officials estimated that recipients marking the eighth anniversary  of the receipt of their awards from July 2005 through May 2010 had  realized a total of $1.05 billion in commercial revenue. NSF  estimated that it invested $628 million in SBIR awards during roughly  the same period.
Further, with the exception of DOD, agencies we reviewed generally did  not take steps to verify commercialization data that they received from  award recipients, so the accuracy of the data is largely unknown. As  officials from some of the agencies in our review noted, award recipients  may have an incentive to overstate their commercialization success in the  hope of improving their prospects of receiving future SBIR awards. While  SBA has worked with SBIR agencies to identify best practices in other  areas of SBIR program management, it has not identified best practices  for agencies to use in verifying the accuracy of commercialization data.  Without consistent practices for verifying the accuracy of these data, the  usefulness of the government-use portion of SBA’s database as a tool for  evaluating the SBIR program’s success in increasing commercialization  may be limited.
To verify the accuracy of award recipients’ commercialization data, DOD  performs an annual review of all projects in its Company  Commercialization Database, which contains the commercialization data  it gathers from award recipients. This review includes checks to ensure  that prior award recipients applying for new awards are not reporting the  same project results more than once, substituting the results of one  project for that of another, or incorrectly reporting sales to third parties.  According to DOD officials, after its 2010 review, the agency sent  approximately 300 e-mail queries to applicants whose reported  commercialization data were identified as having potential problems. The  officials said that applicants that do not respond to such queries are  blocked from submitting further applications until concerns related to their  commercialization reports are addressed. Even with these verification  activities, however, Army officials expressed concern to us about the  accuracy of the applicants’ self-reported commercialization data; these  officials stated their preference for using data from the Federal  Procurement Data System, which contains government information on  federal contracts, including sales. Moreover, a Navy official  acknowledged the possibility that additional verification activities, such as  selective spot visits to SBIR award recipients, could further deter  recipients from misrepresenting their commercialization success,  although he noted that such activities would compete with other  administrative priorities. Similarly, officials from DOE and NIH stated that  additional verification activities would be useful but also said that they  needed to devote program administration resources to higher priority  activities, such as preparing solicitations and supporting review panels for  applications.
section_title: Implementing the Government-Use Portion of the Database Should Improve the Comparability of Commercialization Data, but Long-standing Challenges May Still Impair Evaluation of Commercialization Progress
paragraphs: SBA’s implementation of the government-use portion of its database  should improve the comparability of commercialization data available for  programwide evaluation. Nevertheless, long-standing challenges may  continue to impair programwide evaluation of progress in increasing  commercialization of SBIR-funded technologies. As we reported in  October 2006, notable among these challenges is that prior award  recipients that are no longer participating in the SBIR program are not  required to provide updated commercialization data and may prefer not to  do so. For example, DOD indicated in written comments to us that, from  2008 to 2010, 46 percent of nonparticipating prior phase II award  recipients did not provide updates despite DOD’s request that they  update commercialization data annually after their awards ended.  Similarly, in a report on its 2002 survey, NASA observed that many  recipients of multiple awards elected not to respond to its survey despite  “extensive telephone follow-up” and that many recipients that ultimately  responded “would likely have preferred not to.” Some award recipients  may be reluctant to provide commercialization data because the data are  business-sensitive. SBA officials told us that mechanisms to require or  encourage nonparticipating recipients to report their data need to be  explored. A NASA official told us that effective incentives to encourage  wider voluntary reporting might include publicizing commercial success or  giving monetary prizes for success.
The difficulties agencies face in persuading prior award recipients to  volunteer commercialization information can be compounded by  challenges in maintaining contact with them. Specifically, prior award  recipients can change names or personnel, go out of business, or be sold  during the 10 or more years that it can take for an SBIR-funded  technology to reach the marketplace. In NIH’s 2002 survey of award  recipients, for example, the portion of the sample that was “unusable”—a  group that consisted primarily of recipients that no longer existed or could  not be found—increased from 2 percent in the first year after the end of  the award to 52 percent in the tenth year.
Programwide evaluation—particularly efforts to compare  commercialization success across agencies—can also be complicated by  differences in the time required to commercialize various types of SBIR- funded technologies. Comparing agencies’ commercialization results at a  given point in time may not present a true picture of each agency’s  success because some agencies fund technologies that are relatively  close to being market-ready while others fund technologies that need  more extensive development or regulatory approval. Furthermore, as we  have previously reported, the SBIR program’s other goals remain  important, and comparisons that focus on commercialization may not  adequately take into account progress toward these goals. For  example, one agency official told us that some SBIR-funded technologies,  such as those related to national security, may never have great  commercial potential but are important to the agency’s mission.
section_title: Conclusions
paragraphs: DOD, DOE, NASA, NIH, and NSF have designed their SBIR solicitations  to address the program’s purposes of using small businesses to meet  federal R&D needs, stimulating technological innovation, and increasing  commercialization of innovations derived from federal R&D efforts, and  they have further addressed commercialization by providing technical  assistance or matching funds to award recipients. These agencies have  also conducted outreach and other activities to address the SBIR purpose  of encouraging participation in technological innovation by small  businesses owned by disadvantaged individuals and women. However,  evaluation of progress in achieving the program’s purposes is impeded by  a lack of accurate, comparable, and complete data on program results.  For example, it is difficult to evaluate the program’s effectiveness in  encouraging small businesses owned by disadvantaged individuals or  women to participate in technological innovation because SBA does not  collect data on the number of applications submitted by such businesses.  It is also difficult to evaluate the program’s effectiveness in increasing  commercialization of SBIR-funded technologies because, although  agencies participating in the program have gathered commercialization  data for their own purposes, comparable data on commercialization are  not available across agencies. SBA’s planned implementation of a  government-use portion of its database should go some way toward  improving the comparability of the commercialization data as they are  systematically collected using common metrics. However, the  commercialization data that the database is intended to contain are  largely self-reported by award recipients that may have an incentive to  overstate their commercialization success. DOD has adopted practices  for verifying the accuracy of commercialization data it collects from prior  award recipients, but most of the participating agencies we reviewed did  not verify the accuracy of commercialization data from their prior award  recipients, and SBA has not identified best practices for participating  agencies to use in doing so. As long as participating agencies do not  consistently verify the accuracy of commercialization data, the usefulness  of the government-use portion of SBA’s database as a tool for evaluating  the SBIR program’s success in increasing commercialization may be  limited.
section_title: Recommendations for Executive Action
paragraphs: To build upon efforts to implement a government-use database for  program evaluation, we recommend that the Administrator of the Small  Business Administration work with participating SBIR agencies to take the  following two actions:  collect data on the number of applications submitted by small  businesses owned by disadvantaged individuals and women, and  identify best practices for verifying the accuracy of data related to  progress in increasing commercialization.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to SBA, the Departments of Defense  and Energy, the National Aeronautics and Space Administration, the  National Institutes of Health, and the National Science Foundation for  review and comment. SBA generally agreed with our findings as well as  our recommendations, which it offered an action plan to address.  Specifically, with respect to our first recommendation, SBA stated that,  beginning in fiscal year 2012, it plans to use its database to collect  information from agencies about applicants that did not receive awards— information that could include whether the applicants were small  businesses owned by disadvantaged individuals or women. Further, SBA  indicated that it plans to hold a workshop in fall 2011 for participating  SBIR agencies to share best practices for reaching out to small  businesses owned by disadvantaged individuals. According to SBA, the  workshop should result in a commitment from agencies to develop  baselines for numbers of applications from such businesses. Regarding  our second recommendation, SBA indicated that it will seek to identify  best practices and methods for verifying the accuracy of  commercialization data and will work with agencies toward  implementation of those practices and methods. SBA also noted that its  effort to collect commercialization data is intended to establish a baseline,  against which SBA can review progress in increasing commercialization.  SBA’s letter conveying its comments is contained in appendix II.
Among the SBIR participating agencies that we reviewed, DOE and NSF  concurred with our recommendations and provided general comments,  which are included in appendixes III and IV, respectively. Both DOE and  NSF also made technical comments, which we have incorporated into our  report as appropriate. In its general comments, DOE stated that it collects  information on the number of applications submitted by small businesses  owned by disadvantaged individuals and women and is willing to report  the data to SBA. DOE further stated that it does not verify  commercialization data because of resource limitations—not a belief that  verification is of limited value—and it expressed an interest in learning  about best practices for verification of these data. In addition, DOE  commented that, until universal metrics are identified for measuring the  success of SBIR programs across agencies, the compatibility of available  data among agencies will remain a secondary concern. NSF stated that it  concurs with the underlying goals of our recommendations. Moreover,  NSF affirmed its commitment to implementation of a government-use  database for program evaluation, collection of data on participation in  small business innovation, and identification of best practices for  verification of commercialization data. The remaining agencies—DOD,  HHS, and NASA—neither agreed nor disagreed with our  recommendations but provided technical comments, which we have  incorporated into our report as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, Administrators of SBA and NASA, Secretaries  of Defense and Energy, Directors of NIH and NSF, and other interested  parties. In addition, this report will be available at no charge on the GAO  Web site at http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors to this report are listed in  appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: In conducting this study, we reviewed Small Business Innovation  Research (SBIR) program-related activities of the Small Business  Administration (SBA) and 5 of the 11 SBIR participating agencies—the  Department of Defense (DOD), Department of Energy, National  Aeronautics and Space Administration, Department of Health and Human  Services’ National Institutes of Health (NIH), and National Science  Foundation (NSF). For the two agencies with the largest SBIR budgets— DOD and NIH—we reviewed program activities conducted by the three  participating subcomponent agencies with the largest SBIR budgets  because some key activities are carried out at that level. Specifically, for  DOD, we examined the SBIR programs of the Army, Air Force, and Navy,  and for NIH, we examined the programs of the National Institute of Allergy  and Infectious Diseases; the National Cancer Institute; and the National  Heart, Lung, and Blood Institute. The five participating agencies we  reviewed accounted for about 96 percent of the total dollars awarded by  the program in fiscal year 2009. We reviewed applicable laws and  regulations and literature on the SBIR program, including our prior reports  and assessments by a committee of the National Academy of Sciences’  National Research Council. To obtain further context for our review, we  attended two national conferences and a National Research Council  workshop on the SBIR program, and we interviewed National Research  Council staff with program expertise.
More specifically, to determine how participating agencies have  addressed the SBIR program’s four overarching purposes when  implementing their programs, we reviewed SBA documents and data,  including SBA’s policy directive on implementation of the SBIR program,  minutes from selected meetings of SBA and SBIR program directors,  SBA’s SBIR annual report for fiscal year 2008 (the latest year for which  an annual report was available), and data on the dollar value of SBIR  awards by participating agencies in fiscal year 2009 (the latest year for  which SBA could provide the data). We examined relevant documents  from participating agencies for fiscal years 2008 through 2010, and for  fiscal year 2011 when possible. Documents we reviewed included  solicitations for applications issued by each of these agencies,  instructions to applicants, minutes from meetings of SBA and SBIR  program directors, performance plans and reports, descriptions of  commercialization assistance provided to SBIR awardees, and minutes  from meetings of agency advisory committees. In addition, we also  identified and interviewed SBIR program officials at each agency and  officials responsible for implementing programmatic goals. For these  interviews, we asked a standard set of questions to help ensure that we  obtained consistent information about the SBIR programs at each of the  agencies. We also interviewed inspector general staff at NSF, which  facilitated SBIR-related activities conducted by the Council of Inspectors  General on Integrity and Efficiency. Finally, we interviewed  representatives of trade associations about their views of the SBIR  program. We selected the trade associations on the basis of their  familiarity with the program, the technologies on which they focus, and  whether their membership includes small businesses owned by  disadvantaged groups and women. The views of the representatives of  these associations cannot be generalized to other associations.
To determine the extent of SBIR program data available to evaluate  progress in increasing commercialization of SBIR technologies, we  reviewed documents related to SBA’s SBIR database, including terms of  work, work schedules, and proposed guidance related to the development  of the government-use portion of the database. For the five SBIR  participating agencies whose programs we reviewed, we examined  documents dating from 2002 through 2011; these documents reflected  commercialization data for SBIR award recipients that had received  awards from 1983 (the first year in which agencies issued SBIR awards)  through 2010. The documents we reviewed included surveys and other  data collection instruments that the agencies used to gather  commercialization information from award recipients; reports on data  collection results, including any information on SBIR award spending  during the years corresponding to those covered in each of the  commercialization data collection efforts; and anecdotal descriptions of  commercialization success. We also reviewed agency solicitations from  fiscal years 2008 through 2010—and for fiscal year 2011 when possible—  that contained reporting requirements for award recipients. We  interviewed officials at SBA and each of the five participating agencies  included in our review to obtain information on the specific  commercialization metrics they use to monitor the commercialization  experience of award recipients, the history of each agency’s data  collection efforts, and the agencies’ experience in obtaining such  information from current and past award recipients.
We conducted this performance audit from June 2010 to August 2011, in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence we obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Small Business Administration
section_title: Appendix III: Comments from the Department of Energy
section_title: Appendix IV: Comments from the National Science Foundation
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, key contributors to this report  include Cheryl Williams, Assistant Director; Antoinette Capaccio; Stephen  Carter; Nancy Crothers; Laurie Ellington; Cindy Gilbert; Cynthia Norris;  Christine Senteno; and Kiki Theodoropoulos.