Small Business Innovation Research: Agencies Need to Strengthen Efforts to Improve the Completeness, Consistency, and Accuracy of Awards Data
Published: 2006-10-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Small Business Innovation Research (SBIR) program was created to increase the use of small businesses to meet federal research needs and commercialize the results of this research. To monitor the program, the Small Business Administration (SBA) requires participating agencies to provide, in a standard format, specific data on all SBIR awards they make. SBA then compiles these data into a database known as Tech-Net. Congress also required SBA to create, by 2001, a restricted and more comprehensive database that would provide information for government agencies to use in evaluating the program. GAO was asked to identify the (1) types of data that agencies report to SBA for inclusion in the Tech-Net database, (2) extent to which these data are provided in a standard format, (3) extent to which SBA has established the government-use database, and (4) extent to which SBIR agencies have developed and implemented techniques to track commercialization of SBIR projects. GAO reviewed 8 of the 11 agencies participating in SBIR.
section_title: What GAO Found
paragraphs: Federal agencies participating in the SBIR program annually submit over 40 data elements to SBA for each award they make. These data include information on the award, such as value and a descriptive abstract; information on the recipient, such as name and gender; and information about the firm receiving the award, such as number of employees and location. Participating agencies submit most of the information required by SBA, but they are not consistently providing all required data elements, including the number of employees in the firm, and the gender and socio-economic status of the award recipient, resulting in incomplete sections of the database. Agencies stated that this happens because they do not collect all of the information that SBA wants and because SBA's requirements change regularly. Some participating agencies are not submitting SBIR award data in the standard format required by SBA, and although SBA's quality assurance processes correct most of these problems, they do not correct all of them. In 2004 and 2005, about 25 percent of the data provided by five participating agencies did not comply with SBA's format. Formatting inconsistencies occur because the template SBA has provided agencies for reporting data can be edited. According to SBA, identifying and correcting inconsistently formatted data involves considerable resources, therefore the agency has focused its quality assurance efforts only on key data elements needed to track awards; other fields, such as those containing demographic data, are generally not corrected. As a result, comprehensive program evaluations may be limited by the quality of the data in these fields. SBA officials expect this problem to be resolved by fiscal year 2007, when all data will be submitted via an Internet interface that will not allow changes in the format. SBA is 5 years behind schedule in meeting the congressional mandate to implement a restricted government-use database for the SBIR program. SBA had planned to meet this requirement by expanding its Tech-Net database to include a restricted government-use section. SBA officials attributed the delay in meeting the 2001 deadline primarily to increased security requirements needed for the database, agency management changes, and budgetary constraints. SBA officials expect the government-use section of Tech-Net to be operational by October 1, 2006, when safeguards to protect the proprietary commercialization information in the database are in place. Most agencies GAO reviewed systematically gather data on the commercialization success of SBIR-funded projects. Five of these eight agencies regularly survey all awardees to gather information on program participation, including commercial success, and one agency is about to start a similar survey. In contrast, two agencies only gather anecdotal success stories from a small sample of SBIR awardees. SBA and agency officials generally agree that despite their best efforts, obtaining commercialization information from awardees remains a major challenge.

section_title: Letter
paragraphs: Recognizing the potential of small businesses to be a source of significant  innovation, the Congress established the Small Business Innovation  Research (SBIR) program in 1982. From fiscal year 1983 through fiscal  year 2004, federal agencies that participated in the SBIR program awarded  over $17 billion in grants, contracts, or cooperative agreements to over  82,000 projects. The primary goals of the SBIR program are to stimulate  technological innovation, meet federal research and development (R&D)  needs, foster participation by minority and disadvantaged persons in  technological innovation, and increase the commercial success  (commercialization) of innovation that is derived from federally funded  R&D. Because the Congress did not define what constitutes commercial  success of federally funded R&D or how best to measure it, agencies have  used different commercialization outcomes for the SBIR program, such as  the sale of the resulting SBIR-funded product or process, the extent to  which SBIR firms have received non-SBIR funding, or the creation of new  jobs or products. The SBIR program is currently scheduled to expire on  September 30, 2008.
Every federal agency with an R&D budget of $100 million or more is  required to establish and operate an SBIR program funded by 2.5 percent  of the agency’s budget for research conducted by others, called extramural  research. Currently, 11 federal agencies participate in the SBIR program.  Each agency manages its own program, including targeting research areas,  reviewing proposed projects, and making research awards through grants,  contracts, or cooperative agreements. The Small Business Administration  (SBA) plays a central administrative role by, for example, issuing policy  directives to the participating federal agencies, collecting data from  participating agencies on awards and recipients, and reporting program  results annually to the Congress.
Over the last 24 years, the SBIR program has been reauthorized and  modified by the Congress at various times. For example, the Small  Business Research and Development Enhancement Act of 1992 directed  SBA and participating agencies to, among other things, emphasize the goal  of increasing commercialization of research results and to improve the  government’s dissemination of program-related data. As a result, agencies  were required to include commercialization potential as a criterion for  selecting award recipients. During this same period, SBA began to develop  a publicly available database, known as Tech-Net, that contained  information on all awards made through the SBIR program. The Tech-Net  database is intended to be, among other things, an electronic gateway of  technology information and resources for researchers, scientists, and  government officials about federally-funded leading edge technology  research. The Small Business Innovation Research Program  Reauthorization Act of 2000 formalized this database by requiring SBA to  develop, maintain, and make available to the public a searchable, up-to- date, electronic database that contained SBIR award information. The  2000 reauthorization act also required SBA to develop and maintain  another restricted government database that would contain additional  information on commercialization not contained in the public Tech-Net  database, thereby allowing better evaluations of the SBIR program on an  ongoing basis. This database was to be established by mid-2001 and made  available only to government agencies and certain other authorized users.
As the program has evolved over time, congressional direction has  focused, among other things, on the ability of SBIR award recipients to  commercialize the results of their research, as evidenced by the increased  emphasis on commercialization in the 1992 and 2000 reauthorizations of  the program. At various points in the life of the program, we have reported  that the SBIR program has been successful in increasing  commercialization of research results and that agencies have used various  methods to measure the commercial success of the projects they fund.  Largely these methods have consisted of surveys of award recipients to  obtain data on indicators of commercial success and soliciting “success  stories” voluntarily provided by SBIR award recipients. However, we also  reported that these methods provided only “snapshots” of commercial  success and did not allow for a systematic demonstration of changes in  program commercialization rates over time. We reported that the lack of  clarity on how much emphasis agencies should place on  commercialization versus other SBIR program goals had also created  challenges for assessing the program’s results.
In the context of efforts to monitor and evaluate the success of the SBIR  program, you requested that we identify the (1) types of data that  participating SBIR agencies are reporting to SBA for inclusion in the Tech- Net database; (2) extent to which agencies provide data for the Tech-Net  database in a standard format to enable program evaluation; (3) extent to  which SBA has met the mandate to establish, by mid-2001, a government- use database that can be used for program evaluation; and (4) extent to  which participating SBIR agencies have developed and implemented  techniques to track the commercialization success of SBIR projects.
In conducting this study, we reviewed the SBIR-related activities at 8 of  the 11 SBIR participating agencies, which account for over 98 percent of  the total dollars awarded by the program in fiscal year 2005. To determine  the types of data these participating agencies are reporting to SBA and the  extent to which SBA has complied with the requirement to establish a  government-use database that can be used for program evaluation  purposes, we compared the provisions in the Small Business Innovation  Research Program Reauthorization Act of 2000, SBA’s amended Policy  Directive implementing the Act issued in September 2002, and other  guidance with agency SBIR Tech-Net database reports. To determine the  extent to which data for the Tech-Net database are provided in a standard  format to enable program evaluation, we compared the data from  participating agencies with data in SBA’s Tech-Net database for fiscal  years 2004 and 2005, the 2 most recent years for which data were  available. Our data reliability review focused on SBA’s data system and  internal controls, rather than on the systems and internal controls  agencies use to create the data provided to SBA. To assess the reliability of  the data in SBA’s Tech-Net database, we interviewed SBA officials about  the database and reviewed related documentation. We determined that the  data are sufficiently reliable for our purpose, which is to report on SBA’s  efforts to ensure consistency and completeness of the data it receives. We  used GAO’s data reliability guidance to identify key attributes of data  quality that can facilitate program evaluation. We also interviewed SBA  and agency officials to determine the extent to which the government-use  database requirements have been implemented. To determine the extent to  which participating agencies and SBA have developed and implemented  techniques to evaluate commercialization success of SBIR projects, we  reviewed agency documentation and interviewed SBA and agency  officials. Appendix I contains a detailed discussion of the scope and  methodology of our review. We conducted our review from April 2006  through September 2006 in accordance with generally accepted  government auditing standards.
subsections: section_title: Background
paragraphs: The Small Business Innovation Development Act of 1982 established the  SBIR program and identified four program goals: technological innovation,  commercialization of the research results, the use of small businesses to  meet agencies’ R&D needs, and participation in federal R&D by minorities  and disadvantaged persons. The legislation provided for a competitive  three-phased program: phase I to determine the technical and scientific  merit and feasibility of a proposed research idea; phase II to further  develop the idea, taking into account its commercial potential; and phase  III to commercialize the resulting product or process with private or  federal investment but no additional SBIR funding.
Under the SBIR program provisions, federal agencies that have external  R&D budgets of $100 million or more are required to use 2.5 percent of  these budgets to establish and operate an SBIR program. SBA oversees  and coordinates the efforts of the eleven agencies currently participating  in the program. In this capacity, SBA coordinates the participating  agencies’ schedules to announce opportunities for firms to apply for  awards, called a solicitation, and provides access to these solicitations  through its Web site. As part of its oversight effort, SBA collects SBIR data  from the participating agencies, aggregates the data, and uses the data to,  among other things, monitor the program and report annually to the  Congress.
In reauthorizing the SBIR program in 1992, the Congress stated its  intention to expand and improve the program by emphasizing its goal of  increasing private sector commercialization, increasing participation in  federal R&D by small businesses, and improving the government’s  dissemination of program-related information. One of the new provisions  under the 1992 legislation requires agencies, when evaluating phase II  proposals, to consider their commercial potential, including the recipients’  experiences commercializing the results from previous SBIR awards,  commitments accompanying the proposals for developmental funding  from sources other than the SBIR program, and other factors.
The SBIR program was again reauthorized in 2000 by the Small Business  Innovation Research Program Reauthorization Act of 2000. The 2000  legislation directed SBA and participating agencies to, among other things,  expand the scope of publicly available information on specific awards and  to annually report data on their SBIR programs to SBA. The act required  that SBIR phase II award recipients be requested to voluntarily provide  information to the agencies describing the outputs and outcomes of their  SBIR award. The act also required SBA to establish, by mid-2001, a  searchable and up-to-date electronic database available for public use, and  a restricted government-use database. To accomplish this mandate, SBA  envisioned expanding the electronic database, known as Tech-Net, that it  had developed in the late 1990s, into two sections: a public-use section and  a restricted government-use section. The public-use section of the Tech- Net database would provide access to nearly all of the statutorily-required  award information for SBIR awards gathered by the agencies. The public- use section was intended to be an electronic gateway of technology  information and resources for researchers, scientists, and government  officials who are seeking information on potential small business partners,  contractors, or leading edge technology research. The government-use  section would be solely used for program evaluation purposes accessible  only to government agencies and other authorized users, and would  contain commercialization data voluntarily supplied by SBIR recipients  upon completion of their phase II SBIR funding agreement, such as  revenue from the sale of new products or services resulting from the  research undertaken with the award. In addition, applicants for phase II  awards would be required to update information on the commercialization  success of any prior SBIR awards they had received. Currently, SBA has  created and is maintaining the public-use section of the Tech-Net  database, which is available on the Internet to the general public.
section_title: Agencies Provide Most of the Required SBIR-Award Data Elements to SBA, but Some Data Submitted are Incomplete
paragraphs: While federal agencies participating in the SBIR program submit a wide  range of descriptive award information to SBA annually, these agencies  are not consistently providing all of the required information. As outlined  in SBA’s guidance, each year SBIR participating agencies are required to  collect and maintain information from recipients and provide it to SBA so  that it can be included in the Tech-Net database. Specifically, participating  agencies are required to provide over 40 data elements for each SBIR  award they make. These data include award-specific information, such as  the date and amount of the award, an abstract of the project funded by the  award, and a unique tracking number for each award. Participating  agencies are also required to provide data about the award recipient, such  as gender and socio-economic status, and information about the type of  firms that received the awards, such as number of employees and  geographic location. Much of the data collected by participating agencies  are provided by the SBIR applicants at the time they apply for an award.  Agencies provide additional information, such as the grant/contract  number and the dollar amount of the award, after the award is made.
For the most part, all of the eight agencies we reviewed provided the  majority of the SBIR award data requested by SBA, including the program  identification number; company name, address, and contact information;  award year and amount; a unique tracking number that will stay with the  award through both phase I and phase II; and the title and abstract of the  project funded. However, we also determined that some of the agencies  are not providing the full range of information required by SBA. For  example, two of the eight agencies we reviewed had not provided SBA  data on the gender or socio-economic status of SBIR award recipients in  2004 and 2005. Similarly, in 2005, five of the eight agencies failed to  provide data on the number of employees working at the firms that  received the awards. As a result, SBA does not have information on the  number of employees at SBIR awardee firms for about one-third of all the  awards reported by these agencies in 2004 and 2005.
SBA officials acknowledged that agencies do not routinely provide all of  the information requested because either they do not capture the  information in their agency databases or they are not requesting the  information from SBIR applicants. Officials at the participating agencies  also cited other reasons for the incomplete data they provided to SBA. For  example, NIH officials stated that for the past several years, the SBA Tech- Net annual reporting requirements have changed each year. At the end of  calendar year 2003, SBA changed the field description from “minority” to  “socially and economically disadvantaged small business.” According to  NIH officials, because the SBIR information is captured by the agency at  the time the application is submitted/received, there is a lag in time  between when application data is input into the NIH database and when  the agency receives new SBA data field requirements. According to these  officials, responding to SBA’s changes in field names presents significant  challenges to NIH for collecting the data needed to complete the Tech-Net  reports, especially if the requirements change annually. As a result, these  officials stated that complying with the changing requirements  necessitates significant NIH resources and efforts to research and identify  the information needed for the new data fields, not all of which can be  provided. In commenting on a draft of this report, SBA stated that it has  only requested minor clarifications of data requirements and has not made  frequent changes as stated by the agencies. SBA believes that this may  have been caused by lack of clear communication to the agencies.
The agencies also noted that data for certain Tech-Net fields will be absent  from their reports to SBA if the data fields do not exist in the NIH  application or in its awards database. Similarly, USDA officials stated that  although they try to keep their records as up-to-date as possible, problems  occur when company or contact information changes and the SBIR  recipient fails to provide updated information to the agency, such as the e- mail address for the central contact person. Additionally, like NIH, USDA  officials stated that certain information requested by SBA is not collected  by their agency. For example, as long as SBIR applicants have certified  that they meet the criterion of being a small business (under 500  employees), they do not ask for nor do they record information on the  number of employees in the firm.
section_title: Agencies Generally Comply with SBA’s Formatting Guidance, but Key Data on SBIR Award Recipients May Be Inconsistent or Inaccurate
paragraphs: Participating agencies are providing some data that do not comply with  SBA’s formatting guidance, and while some of these inconsistencies are  corrected by SBA’s quality assurance processes, others are not. As a result,  some data elements in the Tech-Net database may be inconsistent or  inaccurate thereby, compromising the value of these data for program  evaluation. SBA’s quality assurance efforts focus on obtaining complete  and accurate data for those fields essential to tracking specific awards,  such as the tracking number and award amount, rather than on those  fields that contain demographic information about the award recipient.  Because the data contained in the public-use section of Tech-Net will be  incorporated into the government-use section of the database, inaccurate  data in one section of the database will be replicated in the other, and  these inaccurate data will limit evaluations of the SBIR program.
Both SBA and agency officials acknowledge that SBA has worked with  participating agencies since 2000 to help ensure greater consistency in the  formatting of the SBIR award data reported by agencies each year. To  assist agencies in formatting the data and to minimize the number of  inconsistencies in the data reported to SBA, SBA has taken a number of  steps to improve the data formatting process. Specifically, SBA provided  all of the participating agencies specific guidance on its requirements for  the data, including its preferred program application for submitting the  data, the length of each data field, and whether data should be entered as  numbers, characters, or a mix of both. SBA has also included discussions  of the Tech-Net database as an agenda item at its quarterly meetings with  SBIR program managers from each of the participating agencies.  Additionally, SBA developed a reporting template for agencies to use that  includes the required data fields and instructions for appropriate data  entry.
Under the current process, participating agencies aggregate all of the data  provided by SBIR recipients in their award applications with additional  information on the award amounts and submit the combined data to SBA  by March 15 of each year. SBA then electronically checks the data to  locate and reformat inconsistencies, and adds the data to the Tech-Net  database. SBA officials told us that when they detect inconsistencies in  data fields essential to tracking a specific award, such as the award  tracking number, contact information for the recipient or principal  investigator, or awarding agency, they contact the agency to obtain the  correct information. However, SBA does not currently take steps to ensure  that agency-provided data are accurate and complete. For example, SBA  does not require agencies to submit a random sample of applications so  that it can compare the data submitted by agencies with the original  applicant information to ensure that the submitted data contain all the  relevant application information. Instead, SBA relies on the agencies to  fully report all the required application information on the awards they  make. SBA officials told us that they believe that over the past two years  the quality and consistency of the data received from participating  agencies had greatly improved.
In reviewing the SBIR Tech-Net data that the eight agencies reported to  SBA in 2004 and 2005, we determined that almost a quarter of the data  provided by five agencies was incorrectly formatted for one or more fields.  For example, phone number and award amount fields contained both  characters and numbers and first and last names of principal investigators  were combined into a single word that was used as both the first and last  names. Moreover, we found that agency-provided data on gender and  socio-economic status for over half of the awards reported in 2004 were  incorrectly formatted. SBA officials said that detecting and correcting  some of these formatting errors required a considerable amount of time  and effort, and that their electronic check does not detect all of the errors  that we identified.
The formatting inconsistencies that we identified arise primarily because  the SBA reporting template used by agencies to submit required data can  be edited. Consequently, agencies can and do edit the template. Agencies  can change the names of various data fields in SBA’s template, delete  fields altogether, and enter data as numbers, characters, or both,  regardless of what SBA has specified. One of the data format issues  identified by SBA was the deletion of fields that agencies do not consider  relevant or necessary. For example, DOD is the only agency that uses the  field called “branches” to specify which component of DOD, such as the  Army, made the award; other agencies have deleted this field because it  does not relate to the structure of their agency. Similarly, we found  instances where agencies had entered data on the gender or socio- economic status of the recipient and award amount in a format that  differed from the numerical format specified by SBA. For example, one  agency entered “Y” and “N” rather than “0” and “1.” According to SBA  officials, by fiscal year 2007, this issue should be resolved because  agencies will have to submit their data through an Internet interface that  will contain edit checks and should eliminate many of these problems.
We also determined that inconsistencies or inaccuracies can arise in  certain data fields because SBA interprets the absence of certain data  elements as a negative entry without confirming the accuracy of such an  interpretation with the agency. In other words, if an agency did not  provide information on whether the recipient is a woman or a member of a  socio-economically disadvantaged group, SBA has entered a “no” into the  database. SBA stated that they generally do not contact agencies to obtain  correct information on data elements that are not used to track specific  awards, such as gender or socio-economic status of the recipient.  However, this inaccurate data on the award recipients could limit efforts  to use these fields for comprehensive program evaluation.
Information in the Tech-Net database will be used to populate the  government-use section of the database that SBA is developing (as  discussed below) for the purpose of supporting SBIR program evaluations.  However, SBA has no plans to correct any of the errors or inconsistencies  in the database that relate to the historical data already collected. As a  result, the errors in the existing database will migrate to the government-  use section of the database and we believe will compromise the usefulness  of the government-use database for program evaluation purposes.
section_title: SBA Is Five Years Behind Schedule in Meeting its Obligation to Implement a Government-Use SBIR Database
paragraphs: SBA has not met its obligation to implement a restricted government-use  database that would allow SBIR program evaluation as directed by the  2000 SBIR reauthorization act. As outlined in the legislation, SBA, in  consultation with federal agencies participating in the SBIR program, was  to develop a secure database by June 2001 and maintain it for program  evaluation purposes by the federal government. SBA planned to meet this  requirement by expanding the existing Tech-Net database to include a  restricted government-use section that would be accessible only to  government agencies and other authorized users. In constructing the  government-use section of the database, SBA planned to supplement  existing data already gathered for the public-use section of the Tech-Net  database with information from SBIR recipients and from participating  agencies on commercialization outcomes for phase II SBIR awards.  However, according to SBA officials, the agency has been unable to meet  this requirement, primarily because of increased security and other  information technology project requirements, agency management  changes, and budgetary constraints. SBA’s current goal for having the  government-use section of the Tech-Net database operational is October 1,  2006. In commenting on a draft of this report, SBA modified this date,  stating that they anticipate having the government-use section of the Tech- Net database operational early in fiscal year 2007.
To date, with the help of two contractors, SBA has developed the  framework for importing data into the government-use section of the  Tech-Net database and for an Internet-based interface that would allow  agencies and award recipients to access the database and enter  commercialization information. According to SBA officials, as currently  envisioned, the government-use section of the Tech-Net database will  include the records of all applicants, including those that did not receive  SBIR awards. Participating SBIR agencies will be asked to provide a  unique business identification number, called the Data Universal  Numbering System or DUNS number, for each award recipient,  information about SBIR applicants that were not funded, and any  historical data they have obtained about the commercialization of SBIR  funded technologies. SBA has developed a standardized electronic  commercialization questionnaire to gather data for the government-use  section of the database from applicants and award recipients. Information  that will be captured in the questionnaire will include the number of SBIR  awards the company has received, the number of patents or copyrights  that have resulted from the award, sales revenue realized as a result of the  SBIR award, and sources of additional investment funding. SBA officials  told us that the commercialization questionnaire will become an integral  part of the SBIR application process in the future, and any company  applying for an SBIR award will be required to complete and update the  relevant information on phase II SBIR awards previously received by the  company at the time the application is submitted. According to SBA  officials, applications will not be accepted until this information is  completed, and failure to submit the information may affect an applicant’s  ability to receive an award. In addition, SBIR award recipients will be  requested to voluntarily update the commercialization information in the  government-use section of the Tech-Net database annually for a minimum  period of five years following the completion of their SBIR-funded project.
Although SBA has developed the majority of the functions needed to  populate the government-use section of the Tech-Net database with the  data currently gathered for the public-use section of the database, it can  not be made operational until certain security issues are resolved. For  example, because the government-use section of the database will contain  information from recipients that is considered proprietary and  confidential, SBA needs to ensure that adequate security measures are in  place to prevent unauthorized access to the data. This entails the  successful completion of a series of security and development checks to  ensure that the database system is operating as designed. While SBA  officials expect the government-use section of the database to be  operational by October 1, 2006, they also recognize that additional  enhancements, such as improving the user-friendliness of the interface for  online submission of SBIR data by participating agencies and recipients,  will be needed after the system is made operational. According to SBA  officials, the agency’s priority is to get the government-use section of the  database up and running before considering further improvements to the  database.
section_title: Tracking Data on the Commercialization Success of SBIR Projects Varies Among Agencies and Remains a Challenge
paragraphs: Seven of the eight participating agencies we reviewed have implemented  techniques to track the commercialization success of their SBIR-funded  projects, and the eighth is planning to do so, although the methodological  rigor of these techniques varies significantly. Under the program’s  authorizing legislation and SBA’s implementation guidance, agencies have  been given considerable flexibility to design, monitor, and evaluate the  extent to which their SBIR programs have achieved commercialization  success. For example, while some agencies use more systematic  approaches to gathering data, such as periodically surveying SBIR award  recipients, other agencies are less methodical, choosing instead to follow  up periodically with a relatively small sample of SBIR award recipients.  Regardless of how they track commercialization success, both SBA and  agency officials generally agree that several factors complicate their  efforts to obtain this information.
subsections: section_title: Agencies’ Efforts to Track SBIR Commercialization Success Vary
paragraphs: Of the eight agencies we reviewed, five systematically and periodically  survey SBIR recipients to gather a variety of data on program  participation, including the recipients’ commercialization experiences.  Specifically,    Since 2000, DOD has systematically gathered information electronically on  the commercialization of phase II awards from all phase I and phase II  applicants and award recipients and maintains the information in a  commercialization database. Commercialization outcomes that DOD  monitors include such measures as (1) sales revenue from new products  and non-R&D services resulting from the phase II technology; (2)  additional investment from sources other than the federal SBIR program  in activities that further the development, commercialization, or both of  the phase II technology; (3) whether the phase II technology has been used  in a DOD system or acquisition program, and if so, which system or  program; (4) the number of patents resulting from the contractor’s  participation in the SBIR program; (5) growth in the number of employees  at the firm; and (6) whether the firm has completed an initial public  offering of stock resulting, in part, from the phase II project. DOD uses the  accumulated data to assign a commercialization score to applicants that  have received four or more prior SBIR awards based on a comparison of  their commercialization experience with the average experience of other  comparable applicants and uses the score to help select proposed projects  for funding. In addition, recipients of phase II awards are required to  update the information one year after the start of the project, at the  completion of the project, and subsequently when the recipient submits a  new SBIR application to DOD. Firms that do not submit a new SBIR  application are asked to voluntarily provide updates on an annual basis  after the completion of their phase II project. According to DOD officials,  66 percent of phase II award recipients updated their commercialization  information when they submitted a new application, 11 percent provided  the information without submitting a new application, and 23 percent did  not update their information.
For over 23 years, DOE has conducted an annual survey of SBIR phase II  recipients, both active and inactive, to track the commercialization  success of its SBIR-funded projects. The survey requests recipients to (1)  list all products and services derived from their SBIR projects; (2) report  on sales, phase III investment related to these products and service, or  both; and (3) identify which phase II projects contributed to the  development of the products and services. According to DOE,  approximately 90 percent of its phase II recipients respond to these annual  surveys.
NASA has systematically gathered information on the commercialization  of SBIR awards through annual surveys of phase II award recipients from  1997 to 2002. In these surveys, NASA obtained data on various commercial  outcomes, such as sales to nongovernment markets of the SBIR-funded  research results, procurement of the research results by NASA or other  federal agencies, cumulative private capital funding, royalty and licensing  revenue from nongovernment sources based on the SBIR-funded research  results, creation of new business ventures based on the SBIR-funded  research results, and number of patents and patent applications resulting  from these awards. According to NASA, approximately 91 percent of its  phase II award recipients responded to these annual surveys.
NIH also surveys SBIR award recipients to gather commercialization  information. Specifically, in 2002, NIH conducted a comprehensive  evaluation of its SBIR program. As part of this evaluation, NIH surveyed  recipients of phase II awards between 1992 and 2001 to obtain data  relating to the range of SBIR program goals, including commercialization  of research results. According to NIH officials, the 2002 survey results  formed a baseline that NIH staff could use to systematically monitor and  evaluate the program. In 2004 and 2005, NIH again contacted award  recipients to update the information obtained in the 2002 survey.  Commercialization outcomes tracked by NIH include (1) sales realized for  a product or service that resulted from the SBIR-funded research; (2)  status of the Food and Drug Administration (FDA) approval process for  the SBIR-funded research results; (3) receipt of FDA approval for SBIR- funded research results; and (4) receipt of additional non-SBIR funding or  capital.
Since about 1998, NSF has collected historical commercialization  information from all phase II award recipients at the time an SBIR  application is submitted. This information is used in the proposal review  process to help select proposed projects for funding. In 2005, NSF  developed an annual survey of phase II award recipients that will be used  to gather information three, five, and eight years following their awards.  Specific outcomes on which NSF will gather data include sales revenue  based on the SBIR-funded research results, growth of overall company  sales and employment, receipt of additional non-SBIR funding, and patents  related to the SBIR funded research.
In contrast, two of the remaining three agencies we reviewed have focused  their efforts on gathering anecdotal success stories and conducting  periodic follow-up with a relatively smaller sample of SBIR award  recipients. For example, over the past 7 years, EPA has contacted all  phase II award recipients after their projects end to learn about their  commercial successes. Based on these contacts, officials estimate that  approximately 25 percent of the phase II projects funded by EPA have  been commercialized. EPA defines “success” as the receipt of more than  $300,000 in revenue from sales of the SBIR-funded project, an amount  greater than the SBIR funds awarded by EPA. Similarly, USDA has  periodically contacted a sample of about 20 to 25 percent of award  recipients to obtain information about sales of their SBIR-funded research  results. USDA last surveyed its phase II award recipients in 1997. USDA  publishes the success stories on its Web site and in an agency newsletter.  About 2,500 people receive the newsletter and USDA makes copies of the  success stories available at SBIR conferences. According to USDA  officials, in the future they hope to gather data more systematically and  conduct site visits to the SBIR firms. The eighth agency we reviewed,  NIST, has recently developed a Web-based survey that it plans to send in  2006 to all of its SBIR-award recipients.
section_title: Tracking the Commercialization Success of the Overall SBIR Program Remains a Challenge
paragraphs: Although each of the eight agencies we reviewed has implemented or  plans to implement a method for gathering commercialization data, agency  officials identified several factors that complicate their efforts. First,  agency officials stated that it is difficult to track commercialization  because it can take years before companies achieve commercial success.  For example, USDA officials stated that, even over the short term, the  effort to contact past award recipients consumes considerable effort.  During this time, companies may move, change names, start a new  business, or be purchased by other firms, all of which make it difficult for  the agencies to track and link companies to the original SBIR awards.
Second, because the authorizing legislation lacks a clear definition of what  constitutes “commercialization” success, agencies not only differed on the  types of commercialization outcomes they captured, but also in how they  tracked commercial success. SBA officials acknowledged that its guidance  has provided considerable latitude to agencies on this issue in light of the  wide range of industries represented by the participating agencies.  Commercialization outcomes captured by the participating agencies  included sales revenue based on the SBIR-funded research results, receipt  of additional non-SBIR funding to further develop the research results,  marketing activities ongoing or completed, and public offering of company  stock. However, not all agencies are tracking all of these outcomes;  therefore, assessing overall commercial success of the SBIR program  across the various agencies remains a challenge.
Third, agency officials stated that SBIR award recipients may be reluctant  to provide information related to their trade and business operations,  which they consider proprietary and sensitive. Companies are often not  willing to provide comprehensive data on their sales and particularly the  investments they receive due to competitive concerns. Finally, agency  officials told us that past recipients have no incentive to voluntarily  complete commercialization surveys and update the information on their  commercial experience unless they are applying for a new SBIR award. As  a result, they do not expect that a large percentage of recipients will  complete the information needed for the government-use section of the  Tech-Net database. Agency officials believe that despite their best efforts,  the data needed to conduct evaluations of the SBIR program are likely to  be incomplete.
section_title: Conclusions
paragraphs: In the last 5 years, SBA has been unable to meet the congressional  directive to develop a government-use database that would provide better  information on the SBIR program and allow for program evaluation.  Although it has established a public-use Tech-Net database and has  worked with participating agencies to achieve greater consistency in the  data submitted for the database, the quality of the data remains a concern.  The steps on which SBA relies to ensure that data are complete and  accurate are inadequate and it has no plans to correct errors or supply  missing data associated with the historical data already in the database.  We believe that unless necessary controls are established and  implemented to ensure the completeness, consistency, and accuracy of the  SBIR data reported to SBA by participating agencies, the government-use  section of the Tech-Net database, which depends on the public-use Tech- Net database, will be of limited use for program evaluation purposes when  it becomes operational.
section_title: Recommendation for Executive Action
paragraphs: We recommend that the Administrator, SBA, and the SBIR participating  agencies work together to strengthen efforts to ensure that the data  collected for SBA’s Tech-Net database are complete, consistent, and  accurate.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided SBA and the eight SBIR participating agencies included in  this review a draft of this report for their review and comment. SBA and  the eight agencies generally agreed with the report’s findings and SBA and  five of the eight agencies also stated their concurrence with the  recommendation. Three agencies—EPA, NIST, and NSF—did not indicate  whether they agreed or disagreed with the recommendation.
In addition, SBA stated that it was concerned that our conclusions did not  reflect the fact that it plans to have the government-use section of the  Tech-Net database operational by early fiscal year 2007. We have not  modified our conclusions because the fact remains that SBA has not met  the congressional directive to establish a government-use database during  the last five years. Moreover, throughout this review, SBA officials told us  that the database would be operational by October 1, 2006. However, in its  official comments, the agency has again modified this date to some time  early in fiscal year 2007. SBA also provided us with technical comments  that we have incorporated as appropriate. SBA’s letter is included in  appendix II and the letters that we received from DOD, NASA, and NIH are  included in appendix III through V.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to interested  congressional committees; the Administrators of the Environmental  Protection Agency, National Aeronautics and Space Administration, and  Small Business Administration; the Directors of the National Institutes of  Health, National Institute of Standards and Technology, and National  Science Foundation; the Secretaries of Agriculture, Defense, and Energy;  and other interested parties. We will also make copies available to others  upon request. In addition, the report will be available at no charge on the  GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-3841 or mittala@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made contributions to this report are listed  in appendix VI.
section_title: Appendix I: Objectives, Scope and Methods
paragraphs: Our objectives for this review were to identify (1) the types of data that  participating Small Business Innovation Research (SBIR) program  agencies are reporting to the Small Business Administration (SBA) for  inclusion in the Tech-Net database, (2) the extent to which agencies  provide data for the Tech-Net database in a standard format to enable  program evaluation, (3) the extent to which SBA has met the mandate to  establish by early 2001 a government-use database that can be used for  program evaluation, and (4) the extent to which participating SBIR  agencies have developed and implemented techniques to track the  commercialization success of SBIR projects.
In conducting this study, we reviewed SBA and the SBIR-related activities  of 8 of the 11 SBIR participating agencies—Department of Defense (DOD),  Department of Energy (DOE), Environmental Protection Agency (EPA),  National Aeronautics and Space Administration (NASA), National  Institutes of Health (NIH), National Institute of Standards and Technology  (NIST), National Science Foundation (NSF), and Department of  Agriculture (USDA). These eight agencies account for over 98 percent of  the total dollars awarded by the program in fiscal year 2005. To determine  the types of data these participating agencies are reporting to SBA and the  extent to which SBA has complied with the requirement to establish a  government-use database that can be used for program evaluation  purposes, we compared the provisions in the Small Business Innovation  Research Program Reauthorization Act of 2000, relevant legislative  histories, SBA’s Policy Directive implementing the act issued in September  2002, and other guidance with agency SBIR Tech-Net database reports. We  identified and interviewed SBIR program officials at each agency and  officials responsible for submitting program data to SBA. For these  interviews, we used a protocol guide to obtain information on program  operations, data reporting, data quality, and the Tech-Net database,  including development of the government-use section of the database. We  also reviewed documentation from each of the eight agencies and SBA  regarding data elements and formatting instructions.
To determine the extent to which data for the Tech-Net database are  provided in a standard format, enabling program evaluation, we compared  data provided to SBA by the eight participating agencies with data in  SBA’s Tech-Net database for fiscal years 2004 and 2005, the 2 most recent  years for which data were available. The agency data included information  about the award itself, such as the award amount, a descriptive abstract of  the project, and a unique tracking number; information about the award  recipient, such as gender and socio-economic status; and information  about the type of firm that received the award, such as number of  employees and geographic location. To assess the reliability of the data  provided by the agencies, we reviewed SBA’s data system, internal  controls, and related guidance, rather than the systems and internal  controls participating agencies use to create the data provided to SBA. To  assess the reliability of the data in SBA’s Tech-Net database, we  interviewed SBA officials about the database and reviewed related  documentation. We also conducted tests of the data themselves. We used  analytic software to compare the data provided by participating agencies  with the data maintained by SBA. We focused our review on data fields  considered critical by SBA officials, such as the awarding agency, the date  of award, the award recipient, the amount of the award, and the purpose  of the award. We also reviewed data fields related to SBIR program goals,  such as gender and socio-economic status of the recipient, and data on the  number of employees at the recipient firm. We used GAO’s data reliability  guidance to identify key attributes of data quality that facilitate program  evaluation. These attributes include completeness, accuracy, and  consistency in format. Finally, we reviewed internal quality control  procedures. We determined that the data are sufficiently reliable for our  purpose, which is to report on SBA’s efforts to ensure consistency and  completeness of the data it receives.
To determine the extent to which the government-use database  requirements have been implemented, we interviewed SBA officials, and  reviewed related documentation, such as minutes from meetings of SBA  and SBIR program directors, and overviews of the existing and planned  data systems. In addition, we attended a demonstration of the proposed  Internet interface for the government-use section and interviewed the  current contractor assisting SBA about implementation progress. We also  reviewed a contractor-prepared analysis of the functional and data  requirements for the integration of the public- and government-use  sections of the Tech-Net database. At each of the eight participating  agencies, we interviewed SBIR officials regarding the extent to which SBA  had consulted them in the development of the government-use database.
To determine the extent to which participating agencies have developed  and implemented techniques to evaluate commercialization success of  SBIR projects, we reviewed agencies’ documentation about their  commercialization assistance and monitoring efforts. Specifically, we  reviewed surveys that agencies had administered to award recipients,  resulting reports on survey results, and anecdotal descriptions of  commercialization success. We also reviewed provisions in SBIR  legislation, relevant legislative histories, and SBA’s Policy Directive  regarding commercialization of SBIR-funded projects, as well as past GAO  reports. In addition, we interviewed officials at each of the eight  participating agencies to obtain information on the specific  commercialization outcomes they monitor, the history of each agency’s  data collection efforts, and the agencies’ experience in obtaining such  information from current and past award recipients.
We conducted our work from April 2006 through September 2006 in  accordance with generally accepted government auditing standards.
section_title: Appendix II: Comments from the Small Business Administration
paragraphs: The following are GAO’s comments on the Small Business  Administration’s (SBA) letter dated October 6, 2006.
subsections: section_title: GAO Comments
paragraphs: 1.  We revised footnote 2 to more clearly reflect the 11 SBIR participating  agencies.  2.  We revised the text to clarify this statement.  3.  We deleted reference to SBA in footnote 7.  4.  We added SBA’s position to the report.  5.  We added SBA’s position to the report. However, because SBA has not  met the congressional deadline to develop a government-use database,  we did not modify our conclusions.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Cheryl Williams, Assistant  Director; Bernice H. Dawson; Vondalee R. Hunt; and Marcus L. Oliver made  key contributions to this report. Also contributing to this report were                    Nancy Crothers, Grant Mallie, Gregory Marchand, and Rebecca Shea.