Small Business Innovation Research: Agencies Need to Strengthen Efforts to Improve the Completeness, Consistency, and Accuracy of Awards Data
Released: 2006-11-20T00:00:00

Letter

Background

The Small Business Innovation Development Act of 1982 established the SBIR program and identified four program goals: technological innovation, commercialization of the research results, the use of small businesses to meet agencies’ R&D needs, and participation in federal R&D by minorities and disadvantaged persons. The legislation provided for a competitive three-phased program: phase I to determine the technical and scientific merit and feasibility of a proposed research idea; phase II to further develop the idea, taking into account its commercial potential; and phase III to commercialize the resulting product or process with private or federal investment but no additional SBIR funding.

Under the SBIR program provisions, federal agencies that have external R&D budgets of $100 million or more are required to use 2.5 percent of these budgets to establish and operate an SBIR program. SBA oversees and coordinates the efforts of the eleven agencies currently participating in the program. In this capacity, SBA coordinates the participating agencies’ schedules to announce opportunities for firms to apply for awards, called a solicitation, and provides access to these solicitations through its Web site. As part of its oversight effort, SBA collects SBIR data from the participating agencies, aggregates the data, and uses the data to, among other things, monitor the program and report annually to the Congress.

In reauthorizing the SBIR program in 1992, the Congress stated its intention to expand and improve the program by emphasizing its goal of increasing private sector commercialization, increasing participation in federal R&D by small businesses, and improving the government’s dissemination of program-related information. One of the new provisions under the 1992 legislation requires agencies, when evaluating phase II proposals, to consider their commercial potential, including the recipients’ experiences commercializing the results from previous SBIR awards, commitments accompanying the proposals for developmental funding from sources other than the SBIR program, and other factors.

The SBIR program was again reauthorized in 2000 by the Small Business Innovation Research Program Reauthorization Act of 2000. The 2000 legislation directed SBA and participating agencies to, among other things, expand the scope of publicly available information on specific awards and to annually report data on their SBIR programs to SBA. The act required that SBIR phase II award recipients be requested to voluntarily provide information to the agencies describing the outputs and outcomes of their SBIR award. The act also required SBA to establish, by mid-2001, a searchable and up-to-date electronic database available for public use, and a restricted government-use database. To accomplish this mandate, SBA envisioned expanding the electronic database, known as Tech-Net, that it had developed in the late 1990s, into two sections: a public-use section and a restricted government-use section. The public-use section of the Tech- Net database would provide access to nearly all of the statutorily-required award information for SBIR awards gathered by the agencies. The public- use section was intended to be an electronic gateway of technology information and resources for researchers, scientists, and government officials who are seeking information on potential small business partners, contractors, or leading edge technology research. The government-use section would be solely used for program evaluation purposes accessible only to government agencies and other authorized users, and would contain commercialization data voluntarily supplied by SBIR recipients upon completion of their phase II SBIR funding agreement, such as revenue from the sale of new products or services resulting from the research undertaken with the award. In addition, applicants for phase II awards would be required to update information on the commercialization success of any prior SBIR awards they had received. Currently, SBA has created and is maintaining the public-use section of the Tech-Net database, which is available on the Internet to the general public.

Agencies Provide Most of the Required SBIR-Award Data Elements to SBA, but Some Data Submitted are Incomplete

While federal agencies participating in the SBIR program submit a wide range of descriptive award information to SBA annually, these agencies are not consistently providing all of the required information. As outlined in SBA’s guidance, each year SBIR participating agencies are required to collect and maintain information from recipients and provide it to SBA so that it can be included in the Tech-Net database. Specifically, participating agencies are required to provide over 40 data elements for each SBIR award they make. These data include award-specific information, such as the date and amount of the award, an abstract of the project funded by the award, and a unique tracking number for each award. Participating agencies are also required to provide data about the award recipient, such as gender and socio-economic status, and information about the type of firms that received the awards, such as number of employees and geographic location. Much of the data collected by participating agencies are provided by the SBIR applicants at the time they apply for an award. Agencies provide additional information, such as the grant/contract number and the dollar amount of the award, after the award is made.

For the most part, all of the eight agencies we reviewed provided the majority of the SBIR award data requested by SBA, including the program identification number; company name, address, and contact information; award year and amount; a unique tracking number that will stay with the award through both phase I and phase II; and the title and abstract of the project funded. However, we also determined that some of the agencies are not providing the full range of information required by SBA. For example, two of the eight agencies we reviewed had not provided SBA data on the gender or socio-economic status of SBIR award recipients in 2004 and 2005. Similarly, in 2005, five of the eight agencies failed to provide data on the number of employees working at the firms that received the awards. As a result, SBA does not have information on the number of employees at SBIR awardee firms for about one-third of all the awards reported by these agencies in 2004 and 2005.

SBA officials acknowledged that agencies do not routinely provide all of the information requested because either they do not capture the information in their agency databases or they are not requesting the information from SBIR applicants. Officials at the participating agencies also cited other reasons for the incomplete data they provided to SBA. For example, NIH officials stated that for the past several years, the SBA Tech- Net annual reporting requirements have changed each year. At the end of calendar year 2003, SBA changed the field description from “minority” to “socially and economically disadvantaged small business.” According to NIH officials, because the SBIR information is captured by the agency at the time the application is submitted/received, there is a lag in time between when application data is input into the NIH database and when the agency receives new SBA data field requirements. According to these officials, responding to SBA’s changes in field names presents significant challenges to NIH for collecting the data needed to complete the Tech-Net reports, especially if the requirements change annually. As a result, these officials stated that complying with the changing requirements necessitates significant NIH resources and efforts to research and identify the information needed for the new data fields, not all of which can be provided. In commenting on a draft of this report, SBA stated that it has only requested minor clarifications of data requirements and has not made frequent changes as stated by the agencies. SBA believes that this may have been caused by lack of clear communication to the agencies.

The agencies also noted that data for certain Tech-Net fields will be absent from their reports to SBA if the data fields do not exist in the NIH application or in its awards database. Similarly, USDA officials stated that although they try to keep their records as up-to-date as possible, problems occur when company or contact information changes and the SBIR recipient fails to provide updated information to the agency, such as the e- mail address for the central contact person. Additionally, like NIH, USDA officials stated that certain information requested by SBA is not collected by their agency. For example, as long as SBIR applicants have certified that they meet the criterion of being a small business (under 500 employees), they do not ask for nor do they record information on the number of employees in the firm.

Agencies Generally Comply with SBA’s Formatting Guidance, but Key Data on SBIR Award Recipients May Be Inconsistent or Inaccurate

Participating agencies are providing some data that do not comply with SBA’s formatting guidance, and while some of these inconsistencies are corrected by SBA’s quality assurance processes, others are not. As a result, some data elements in the Tech-Net database may be inconsistent or inaccurate thereby, compromising the value of these data for program evaluation. SBA’s quality assurance efforts focus on obtaining complete and accurate data for those fields essential to tracking specific awards, such as the tracking number and award amount, rather than on those fields that contain demographic information about the award recipient. Because the data contained in the public-use section of Tech-Net will be incorporated into the government-use section of the database, inaccurate data in one section of the database will be replicated in the other, and these inaccurate data will limit evaluations of the SBIR program.

Both SBA and agency officials acknowledge that SBA has worked with participating agencies since 2000 to help ensure greater consistency in the formatting of the SBIR award data reported by agencies each year. To assist agencies in formatting the data and to minimize the number of inconsistencies in the data reported to SBA, SBA has taken a number of steps to improve the data formatting process. Specifically, SBA provided all of the participating agencies specific guidance on its requirements for the data, including its preferred program application for submitting the data, the length of each data field, and whether data should be entered as numbers, characters, or a mix of both. SBA has also included discussions of the Tech-Net database as an agenda item at its quarterly meetings with SBIR program managers from each of the participating agencies. Additionally, SBA developed a reporting template for agencies to use that includes the required data fields and instructions for appropriate data entry.

Under the current process, participating agencies aggregate all of the data provided by SBIR recipients in their award applications with additional information on the award amounts and submit the combined data to SBA by March 15 of each year. SBA then electronically checks the data to locate and reformat inconsistencies, and adds the data to the Tech-Net database. SBA officials told us that when they detect inconsistencies in data fields essential to tracking a specific award, such as the award tracking number, contact information for the recipient or principal investigator, or awarding agency, they contact the agency to obtain the correct information. However, SBA does not currently take steps to ensure that agency-provided data are accurate and complete. For example, SBA does not require agencies to submit a random sample of applications so that it can compare the data submitted by agencies with the original applicant information to ensure that the submitted data contain all the relevant application information. Instead, SBA relies on the agencies to fully report all the required application information on the awards they make. SBA officials told us that they believe that over the past two years the quality and consistency of the data received from participating agencies had greatly improved.

In reviewing the SBIR Tech-Net data that the eight agencies reported to SBA in 2004 and 2005, we determined that almost a quarter of the data provided by five agencies was incorrectly formatted for one or more fields. For example, phone number and award amount fields contained both characters and numbers and first and last names of principal investigators were combined into a single word that was used as both the first and last names. Moreover, we found that agency-provided data on gender and socio-economic status for over half of the awards reported in 2004 were incorrectly formatted. SBA officials said that detecting and correcting some of these formatting errors required a considerable amount of time and effort, and that their electronic check does not detect all of the errors that we identified.

The formatting inconsistencies that we identified arise primarily because the SBA reporting template used by agencies to submit required data can be edited. Consequently, agencies can and do edit the template. Agencies can change the names of various data fields in SBA’s template, delete fields altogether, and enter data as numbers, characters, or both, regardless of what SBA has specified. One of the data format issues identified by SBA was the deletion of fields that agencies do not consider relevant or necessary. For example, DOD is the only agency that uses the field called “branches” to specify which component of DOD, such as the Army, made the award; other agencies have deleted this field because it does not relate to the structure of their agency. Similarly, we found instances where agencies had entered data on the gender or socio- economic status of the recipient and award amount in a format that differed from the numerical format specified by SBA. For example, one agency entered “Y” and “N” rather than “0” and “1.” According to SBA officials, by fiscal year 2007, this issue should be resolved because agencies will have to submit their data through an Internet interface that will contain edit checks and should eliminate many of these problems.

We also determined that inconsistencies or inaccuracies can arise in certain data fields because SBA interprets the absence of certain data elements as a negative entry without confirming the accuracy of such an interpretation with the agency. In other words, if an agency did not provide information on whether the recipient is a woman or a member of a socio-economically disadvantaged group, SBA has entered a “no” into the database. SBA stated that they generally do not contact agencies to obtain correct information on data elements that are not used to track specific awards, such as gender or socio-economic status of the recipient. However, this inaccurate data on the award recipients could limit efforts to use these fields for comprehensive program evaluation.

Information in the Tech-Net database will be used to populate the government-use section of the database that SBA is developing (as discussed below) for the purpose of supporting SBIR program evaluations. However, SBA has no plans to correct any of the errors or inconsistencies in the database that relate to the historical data already collected. As a result, the errors in the existing database will migrate to the government- use section of the database and we believe will compromise the usefulness of the government-use database for program evaluation purposes.

SBA Is Five Years Behind Schedule in Meeting its Obligation to Implement a Government-Use SBIR Database

SBA has not met its obligation to implement a restricted government-use database that would allow SBIR program evaluation as directed by the 2000 SBIR reauthorization act. As outlined in the legislation, SBA, in consultation with federal agencies participating in the SBIR program, was to develop a secure database by June 2001 and maintain it for program evaluation purposes by the federal government. SBA planned to meet this requirement by expanding the existing Tech-Net database to include a restricted government-use section that would be accessible only to government agencies and other authorized users. In constructing the government-use section of the database, SBA planned to supplement existing data already gathered for the public-use section of the Tech-Net database with information from SBIR recipients and from participating agencies on commercialization outcomes for phase II SBIR awards. However, according to SBA officials, the agency has been unable to meet this requirement, primarily because of increased security and other information technology project requirements, agency management changes, and budgetary constraints. SBA’s current goal for having the government-use section of the Tech-Net database operational is October 1, 2006. In commenting on a draft of this report, SBA modified this date, stating that they anticipate having the government-use section of the Tech- Net database operational early in fiscal year 2007.

To date, with the help of two contractors, SBA has developed the framework for importing data into the government-use section of the Tech-Net database and for an Internet-based interface that would allow agencies and award recipients to access the database and enter commercialization information. According to SBA officials, as currently envisioned, the government-use section of the Tech-Net database will include the records of all applicants, including those that did not receive SBIR awards. Participating SBIR agencies will be asked to provide a unique business identification number, called the Data Universal Numbering System or DUNS number, for each award recipient, information about SBIR applicants that were not funded, and any historical data they have obtained about the commercialization of SBIR funded technologies. SBA has developed a standardized electronic commercialization questionnaire to gather data for the government-use section of the database from applicants and award recipients. Information that will be captured in the questionnaire will include the number of SBIR awards the company has received, the number of patents or copyrights that have resulted from the award, sales revenue realized as a result of the SBIR award, and sources of additional investment funding. SBA officials told us that the commercialization questionnaire will become an integral part of the SBIR application process in the future, and any company applying for an SBIR award will be required to complete and update the relevant information on phase II SBIR awards previously received by the company at the time the application is submitted. According to SBA officials, applications will not be accepted until this information is completed, and failure to submit the information may affect an applicant’s ability to receive an award. In addition, SBIR award recipients will be requested to voluntarily update the commercialization information in the government-use section of the Tech-Net database annually for a minimum period of five years following the completion of their SBIR-funded project.

Although SBA has developed the majority of the functions needed to populate the government-use section of the Tech-Net database with the data currently gathered for the public-use section of the database, it can not be made operational until certain security issues are resolved. For example, because the government-use section of the database will contain information from recipients that is considered proprietary and confidential, SBA needs to ensure that adequate security measures are in place to prevent unauthorized access to the data. This entails the successful completion of a series of security and development checks to ensure that the database system is operating as designed. While SBA officials expect the government-use section of the database to be operational by October 1, 2006, they also recognize that additional enhancements, such as improving the user-friendliness of the interface for online submission of SBIR data by participating agencies and recipients, will be needed after the system is made operational. According to SBA officials, the agency’s priority is to get the government-use section of the database up and running before considering further improvements to the database.

Tracking Data on the Commercialization Success of SBIR Projects Varies Among Agencies and Remains a Challenge

Seven of the eight participating agencies we reviewed have implemented techniques to track the commercialization success of their SBIR-funded projects, and the eighth is planning to do so, although the methodological rigor of these techniques varies significantly. Under the program’s authorizing legislation and SBA’s implementation guidance, agencies have been given considerable flexibility to design, monitor, and evaluate the extent to which their SBIR programs have achieved commercialization success. For example, while some agencies use more systematic approaches to gathering data, such as periodically surveying SBIR award recipients, other agencies are less methodical, choosing instead to follow up periodically with a relatively small sample of SBIR award recipients. Regardless of how they track commercialization success, both SBA and agency officials generally agree that several factors complicate their efforts to obtain this information.

Agencies’ Efforts to Track SBIR Commercialization Success Vary

Of the eight agencies we reviewed, five systematically and periodically survey SBIR recipients to gather a variety of data on program participation, including the recipients’ commercialization experiences. Specifically, Since 2000, DOD has systematically gathered information electronically on the commercialization of phase II awards from all phase I and phase II applicants and award recipients and maintains the information in a commercialization database. Commercialization outcomes that DOD monitors include such measures as (1) sales revenue from new products and non-R&D services resulting from the phase II technology; (2) additional investment from sources other than the federal SBIR program in activities that further the development, commercialization, or both of the phase II technology; (3) whether the phase II technology has been used in a DOD system or acquisition program, and if so, which system or program; (4) the number of patents resulting from the contractor’s participation in the SBIR program; (5) growth in the number of employees at the firm; and (6) whether the firm has completed an initial public offering of stock resulting, in part, from the phase II project. DOD uses the accumulated data to assign a commercialization score to applicants that have received four or more prior SBIR awards based on a comparison of their commercialization experience with the average experience of other comparable applicants and uses the score to help select proposed projects for funding. In addition, recipients of phase II awards are required to update the information one year after the start of the project, at the completion of the project, and subsequently when the recipient submits a new SBIR application to DOD. Firms that do not submit a new SBIR application are asked to voluntarily provide updates on an annual basis after the completion of their phase II project. According to DOD officials, 66 percent of phase II award recipients updated their commercialization information when they submitted a new application, 11 percent provided the information without submitting a new application, and 23 percent did not update their information.

For over 23 years, DOE has conducted an annual survey of SBIR phase II recipients, both active and inactive, to track the commercialization success of its SBIR-funded projects. The survey requests recipients to (1) list all products and services derived from their SBIR projects; (2) report on sales, phase III investment related to these products and service, or both; and (3) identify which phase II projects contributed to the development of the products and services. According to DOE, approximately 90 percent of its phase II recipients respond to these annual surveys.

NASA has systematically gathered information on the commercialization of SBIR awards through annual surveys of phase II award recipients from 1997 to 2002. In these surveys, NASA obtained data on various commercial outcomes, such as sales to nongovernment markets of the SBIR-funded research results, procurement of the research results by NASA or other federal agencies, cumulative private capital funding, royalty and licensing revenue from nongovernment sources based on the SBIR-funded research results, creation of new business ventures based on the SBIR-funded research results, and number of patents and patent applications resulting from these awards. According to NASA, approximately 91 percent of its phase II award recipients responded to these annual surveys.

NIH also surveys SBIR award recipients to gather commercialization information. Specifically, in 2002, NIH conducted a comprehensive evaluation of its SBIR program. As part of this evaluation, NIH surveyed recipients of phase II awards between 1992 and 2001 to obtain data relating to the range of SBIR program goals, including commercialization of research results. According to NIH officials, the 2002 survey results formed a baseline that NIH staff could use to systematically monitor and evaluate the program. In 2004 and 2005, NIH again contacted award recipients to update the information obtained in the 2002 survey. Commercialization outcomes tracked by NIH include (1) sales realized for a product or service that resulted from the SBIR-funded research; (2) status of the Food and Drug Administration (FDA) approval process for the SBIR-funded research results; (3) receipt of FDA approval for SBIR- funded research results; and (4) receipt of additional non-SBIR funding or capital.

Since about 1998, NSF has collected historical commercialization information from all phase II award recipients at the time an SBIR application is submitted. This information is used in the proposal review process to help select proposed projects for funding. In 2005, NSF developed an annual survey of phase II award recipients that will be used to gather information three, five, and eight years following their awards. Specific outcomes on which NSF will gather data include sales revenue based on the SBIR-funded research results, growth of overall company sales and employment, receipt of additional non-SBIR funding, and patents related to the SBIR funded research.

In contrast, two of the remaining three agencies we reviewed have focused their efforts on gathering anecdotal success stories and conducting periodic follow-up with a relatively smaller sample of SBIR award recipients. For example, over the past 7 years, EPA has contacted all phase II award recipients after their projects end to learn about their commercial successes. Based on these contacts, officials estimate that approximately 25 percent of the phase II projects funded by EPA have been commercialized. EPA defines “success” as the receipt of more than $300,000 in revenue from sales of the SBIR-funded project, an amount greater than the SBIR funds awarded by EPA. Similarly, USDA has periodically contacted a sample of about 20 to 25 percent of award recipients to obtain information about sales of their SBIR-funded research results. USDA last surveyed its phase II award recipients in 1997. USDA publishes the success stories on its Web site and in an agency newsletter. About 2,500 people receive the newsletter and USDA makes copies of the success stories available at SBIR conferences. According to USDA officials, in the future they hope to gather data more systematically and conduct site visits to the SBIR firms. The eighth agency we reviewed, NIST, has recently developed a Web-based survey that it plans to send in 2006 to all of its SBIR-award recipients.

Tracking the Commercialization Success of the Overall SBIR Program Remains a Challenge

Although each of the eight agencies we reviewed has implemented or plans to implement a method for gathering commercialization data, agency officials identified several factors that complicate their efforts. First, agency officials stated that it is difficult to track commercialization because it can take years before companies achieve commercial success. For example, USDA officials stated that, even over the short term, the effort to contact past award recipients consumes considerable effort. During this time, companies may move, change names, start a new business, or be purchased by other firms, all of which make it difficult for the agencies to track and link companies to the original SBIR awards.

Second, because the authorizing legislation lacks a clear definition of what constitutes “commercialization” success, agencies not only differed on the types of commercialization outcomes they captured, but also in how they tracked commercial success. SBA officials acknowledged that its guidance has provided considerable latitude to agencies on this issue in light of the wide range of industries represented by the participating agencies. Commercialization outcomes captured by the participating agencies included sales revenue based on the SBIR-funded research results, receipt of additional non-SBIR funding to further develop the research results, marketing activities ongoing or completed, and public offering of company stock. However, not all agencies are tracking all of these outcomes; therefore, assessing overall commercial success of the SBIR program across the various agencies remains a challenge.

Third, agency officials stated that SBIR award recipients may be reluctant to provide information related to their trade and business operations, which they consider proprietary and sensitive. Companies are often not willing to provide comprehensive data on their sales and particularly the investments they receive due to competitive concerns. Finally, agency officials told us that past recipients have no incentive to voluntarily complete commercialization surveys and update the information on their commercial experience unless they are applying for a new SBIR award. As a result, they do not expect that a large percentage of recipients will complete the information needed for the government-use section of the Tech-Net database. Agency officials believe that despite their best efforts, the data needed to conduct evaluations of the SBIR program are likely to be incomplete.

Conclusions

In the last 5 years, SBA has been unable to meet the congressional directive to develop a government-use database that would provide better information on the SBIR program and allow for program evaluation. Although it has established a public-use Tech-Net database and has worked with participating agencies to achieve greater consistency in the data submitted for the database, the quality of the data remains a concern. The steps on which SBA relies to ensure that data are complete and accurate are inadequate and it has no plans to correct errors or supply missing data associated with the historical data already in the database. We believe that unless necessary controls are established and implemented to ensure the completeness, consistency, and accuracy of the SBIR data reported to SBA by participating agencies, the government-use section of the Tech-Net database, which depends on the public-use Tech- Net database, will be of limited use for program evaluation purposes when it becomes operational.

Recommendation for Executive Action

We recommend that the Administrator, SBA, and the SBIR participating agencies work together to strengthen efforts to ensure that the data collected for SBA’s Tech-Net database are complete, consistent, and accurate.

Agency Comments and Our Evaluation

We provided SBA and the eight SBIR participating agencies included in this review a draft of this report for their review and comment. SBA and the eight agencies generally agreed with the report’s findings and SBA and five of the eight agencies also stated their concurrence with the recommendation. Three agencies—EPA, NIST, and NSF—did not indicate whether they agreed or disagreed with the recommendation.

In addition, SBA stated that it was concerned that our conclusions did not reflect the fact that it plans to have the government-use section of the Tech-Net database operational by early fiscal year 2007. We have not modified our conclusions because the fact remains that SBA has not met the congressional directive to establish a government-use database during the last five years. Moreover, throughout this review, SBA officials told us that the database would be operational by October 1, 2006. However, in its official comments, the agency has again modified this date to some time early in fiscal year 2007. SBA also provided us with technical comments that we have incorporated as appropriate. SBA’s letter is included in appendix II and the letters that we received from DOD, NASA, and NIH are included in appendix III through V.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to interested congressional committees; the Administrators of the Environmental Protection Agency, National Aeronautics and Space Administration, and Small Business Administration; the Directors of the National Institutes of Health, National Institute of Standards and Technology, and National Science Foundation; the Secretaries of Agriculture, Defense, and Energy; and other interested parties. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3841 or mittala@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope and Methods

Our objectives for this review were to identify (1) the types of data that participating Small Business Innovation Research (SBIR) program agencies are reporting to the Small Business Administration (SBA) for inclusion in the Tech-Net database, (2) the extent to which agencies provide data for the Tech-Net database in a standard format to enable program evaluation, (3) the extent to which SBA has met the mandate to establish by early 2001 a government-use database that can be used for program evaluation, and (4) the extent to which participating SBIR agencies have developed and implemented techniques to track the commercialization success of SBIR projects.

In conducting this study, we reviewed SBA and the SBIR-related activities of 8 of the 11 SBIR participating agencies—Department of Defense (DOD), Department of Energy (DOE), Environmental Protection Agency (EPA), National Aeronautics and Space Administration (NASA), National Institutes of Health (NIH), National Institute of Standards and Technology (NIST), National Science Foundation (NSF), and Department of Agriculture (USDA). These eight agencies account for over 98 percent of the total dollars awarded by the program in fiscal year 2005. To determine the types of data these participating agencies are reporting to SBA and the extent to which SBA has complied with the requirement to establish a government-use database that can be used for program evaluation purposes, we compared the provisions in the Small Business Innovation Research Program Reauthorization Act of 2000, relevant legislative histories, SBA’s Policy Directive implementing the act issued in September 2002, and other guidance with agency SBIR Tech-Net database reports. We identified and interviewed SBIR program officials at each agency and officials responsible for submitting program data to SBA. For these interviews, we used a protocol guide to obtain information on program operations, data reporting, data quality, and the Tech-Net database, including development of the government-use section of the database. We also reviewed documentation from each of the eight agencies and SBA regarding data elements and formatting instructions.

To determine the extent to which data for the Tech-Net database are provided in a standard format, enabling program evaluation, we compared data provided to SBA by the eight participating agencies with data in SBA’s Tech-Net database for fiscal years 2004 and 2005, the 2 most recent years for which data were available. The agency data included information about the award itself, such as the award amount, a descriptive abstract of the project, and a unique tracking number; information about the award recipient, such as gender and socio-economic status; and information about the type of firm that received the award, such as number of employees and geographic location. To assess the reliability of the data provided by the agencies, we reviewed SBA’s data system, internal controls, and related guidance, rather than the systems and internal controls participating agencies use to create the data provided to SBA. To assess the reliability of the data in SBA’s Tech-Net database, we interviewed SBA officials about the database and reviewed related documentation. We also conducted tests of the data themselves. We used analytic software to compare the data provided by participating agencies with the data maintained by SBA. We focused our review on data fields considered critical by SBA officials, such as the awarding agency, the date of award, the award recipient, the amount of the award, and the purpose of the award. We also reviewed data fields related to SBIR program goals, such as gender and socio-economic status of the recipient, and data on the number of employees at the recipient firm. We used GAO’s data reliability guidance to identify key attributes of data quality that facilitate program evaluation. These attributes include completeness, accuracy, and consistency in format. Finally, we reviewed internal quality control procedures. We determined that the data are sufficiently reliable for our purpose, which is to report on SBA’s efforts to ensure consistency and completeness of the data it receives.

To determine the extent to which the government-use database requirements have been implemented, we interviewed SBA officials, and reviewed related documentation, such as minutes from meetings of SBA and SBIR program directors, and overviews of the existing and planned data systems. In addition, we attended a demonstration of the proposed Internet interface for the government-use section and interviewed the current contractor assisting SBA about implementation progress. We also reviewed a contractor-prepared analysis of the functional and data requirements for the integration of the public- and government-use sections of the Tech-Net database. At each of the eight participating agencies, we interviewed SBIR officials regarding the extent to which SBA had consulted them in the development of the government-use database.

To determine the extent to which participating agencies have developed and implemented techniques to evaluate commercialization success of SBIR projects, we reviewed agencies’ documentation about their commercialization assistance and monitoring efforts. Specifically, we reviewed surveys that agencies had administered to award recipients, resulting reports on survey results, and anecdotal descriptions of commercialization success. We also reviewed provisions in SBIR legislation, relevant legislative histories, and SBA’s Policy Directive regarding commercialization of SBIR-funded projects, as well as past GAO reports. In addition, we interviewed officials at each of the eight participating agencies to obtain information on the specific commercialization outcomes they monitor, the history of each agency’s data collection efforts, and the agencies’ experience in obtaining such information from current and past award recipients.

We conducted our work from April 2006 through September 2006 in accordance with generally accepted government auditing standards.

Appendix II: Comments from the Small Business Administration

The following are GAO’s comments on the Small Business Administration’s (SBA) letter dated October 6, 2006.

GAO Comments

1. We revised footnote 2 to more clearly reflect the 11 SBIR participating agencies. 2. We revised the text to clarify this statement. 3. We deleted reference to SBA in footnote 7. 4. We added SBA’s position to the report. 5. We added SBA’s position to the report. However, because SBA has not met the congressional deadline to develop a government-use database, we did not modify our conclusions.

Appendix III: Comments from the Department of Defense

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Cheryl Williams, Assistant Director; Bernice H. Dawson; Vondalee R. Hunt; and Marcus L. Oliver made key contributions to this report. Also contributing to this report were Nancy Crothers, Grant Mallie, Gregory Marchand, and Rebecca Shea.