Small Business Innovation Research: Observations on Agencies' Data Collection and Eligibility Determination Efforts
Released: 2009-08-06T00:00:00

Letter

Successes and Challenges of the SBIR Program

Our reviews of the SBIR program between 1985 and 1999 found numerous examples of program successes such as the following: Funding high-quality research. Throughout the life of the program, awards have been based on technical merit and are generally of good quality.

Encouraging widespread competition. The SBIR program successfully attracts many qualified companies, has had a high level of competition, consistently has had a high number of first-time participants, and attracts hundreds of new companies annually.

Providing effective outreach. SBIR agencies consistently reach out to foster participation by women-owned or socially and economically disadvantaged small businesses by participating in regional small business conferences and workshops targeting these types of small businesses.

Increasing successful commercialization. At various points in the life of the program we have reported that SBIR has succeeded in increasing private sector commercialization of innovations.

Helping to serve mission needs. SBIR has helped serve agencies’ missions and R&D needs, although we found that agencies differ in the emphasis they place on funding research to support their mission versus more generalized research.

Our reviews of the SBIR program during that time have also identified a number of areas of weakness that, over time, have been either fully or partially addressed by the Congress in reauthorizing the program or by the agencies themselves. For example, Duplicate funding. In 1995, we identified duplicate funding for similar, or even identical, research projects by more than one agency. A few companies received funding for the same proposals two, three, and even five times before agencies became aware of the duplication. Contributing factors included the fraudulent evasion of disclosure by companies applying for awards, the lack of a consistent definition for key terms such as “similar research,” and the lack of interagency sharing of data on awards. To address these concerns, we recommended that SBA take three actions: (1) determine if the certification form needed to be improved and make any necessary revisions, (2) develop definitions and guidelines for what constitutes “duplicative” research, and (3) provide interagency access to current information regarding SBIR awards In response to our recommendations, SBA strengthened the language agencies use in their application packages to clearly warn applicants about the illegality of entering into multiple agreements for essentially the same effort. In addition, SBA planned to develop Internet capabilities to provide SBIR data access for all of the agencies.

Inconsistent interpretations of extramural research budgets. In 1998, we found that while agency officials adhered to SBIR’s program and statutory funding requirements, they used differing interpretations of how to calculate their “extramural research budgets.” As a result, some agencies were inappropriately including or excluding some types of expenses. We recommended that SBA provide additional guidance on how participating agencies were to calculate their extramural research budgets. The Congress addressed this program weakness in 2000, when it required that the agencies report annually to SBA on the methods used to calculate their extramural research budgets.

Geographical concentration of awards. In 1999, in response to congressional concerns about the geographical concentration of SBIR awards, we reported that companies in a small number of states, especially California and Massachusetts, had submitted the most proposals and won the majority of awards. The distribution of awards generally followed the pattern of distribution of non-SBIR expenditures for R&D, venture capital investments, and academic research funds. We reported that some agencies had undertaken efforts to broaden the geographic distribution of awards. In the 2000 reauthorization of the program, the Congress directed the SBA Administrator to establish the Federal and State Technology (FAST) Partnership Program to help strengthen the technological competitiveness of small businesses, especially in those states that receive fewer SBIR grants. The FAST Program was not reauthorized when it expired in 2005. In 2006 when we looked at the geographical concentration of awards made by DOD and NIH, we found that while a firm in every state received at least one SBIR award from both agencies, SBIR awards continued to be concentrated in a handful of states and about one third of awards had been made to firms in California and Massachusetts.

Clarification on commercialization and other SBIR goals. Finally, in 2000, the Congress directed the SBA Administrator to require companies applying for a phase II award to include a commercialization plan with their SBIR proposals. This addressed our continuing concern that clarification was needed on the relative emphasis that agencies should give to a company’s commercialization record and SBIR’s other goals when evaluating proposals. In addition, in 2001, SBA initiated efforts to develop standard criteria for measuring commercial and other outcomes of the SBIR program and incorporate these criteria into its Tech-Net database. In fiscal year 2002, SBA further enhanced the reporting system to include commercialization results that would help establish an initial baseline rate of commercialization. In addition, small business firms participating in the SBIR program are required to provide information annually on sales and investments associated with their SBIR projects.

SBIR Tech-Net Database Limitations

Many of the solutions cited above to improve and strengthen the SBIR program relied to some extent on the collection of data or the establishment of a government-use database, so that SBA and participating agencies could share information and enhance their efforts to monitor and evaluate the program. However, in 2006, we reported that SBA was 5 years behind schedule in complying with the congressional mandate to develop a government database that could facilitate agencies’ monitoring and evaluation of the program. We also reported that the information SBA was collecting for the database was incomplete and inconsistent, thereby limiting its usefulness for program evaluations. Specifically, we identified the following concerns with SBA’s data-gathering efforts: SBA had not met its obligation to implement a restricted government-use database that would allow SBIR program evaluation as directed by the 2000 SBIR reauthorization act. As outlined in the legislation, SBA, in consultation with federal agencies participating in the SBIR program, was to develop a secure database by June 2001 and maintain it for program evaluation purposes by the federal government and certain other entities. SBA planned to meet this requirement by expanding the existing Tech-Net database to include a restricted government-use section that would be accessible only to government agencies and other authorized users. In constructing the government-use section of the database, SBA planned to supplement data already gathered for the public-use section of the Tech- Net database with information from SBIR recipients and from participating agencies on commercialization outcomes for phase II SBIR awards. However, according to SBA officials, the agency was unable to meet the statutory requirement, primarily because of increased security and other information technology project requirements, agency management changes, and budgetary constraints. When we reported on this lack of compliance with the database mandate, SBA told us that it anticipated having the government-use section of the Tech-Net database operational early in fiscal year 2007. However, according to an SBA official, the database became operational in October 2008, and agencies have begun to provide data on their SBIR programs using the Internet.

While federal agencies participating in the SBIR program submitted a wide range of descriptive award information to SBA annually, these agencies did not consistently provide all of the required data elements. As outlined in SBA’s policy directive, each year, SBIR participating agencies are required to collect and maintain information from recipients and provide it to SBA so that it can be included in the Tech-Net database. Specifically, the policy directive established over 40 data elements for participating agencies to report for each SBIR award they make; a number of these elements are required. These data include award-specific information, such as the date and amount of the award, an abstract of the project funded by the award, and a unique tracking number for each award. Participating agencies are also required to provide data about the award recipient, such as gender and socio-economic status, and information about the type of firms that received the awards, such as the number of employees and geographic location. Much of the data participating agencies collected are provided by the SBIR applicants when they apply for an award. Agencies provide additional information, such as the grant/contract number and the dollar amount of the award, after the award is made. For the most part, all of the agencies we reviewed in 2006 provided the majority of the data elements outlined in the policy directive. However, some of the agencies were not providing the full range of required data elements. As a result, SBA did not have complete information on the characteristics of all SBIR awards made by the agencies. SBA officials told us that agencies did not routinely provide all of the data elements outlined in the policy directive because either they did not capture the information in their agency databases or they were not requesting the information from the SBIR applicants. Officials at the participating agencies cited additional reasons for the incomplete data they provided to SBA. For example, some officials noted that SBA’s Tech- Net annual reporting requirements often change and others said that if the company or contact information changes and the SBIR recipient fails to provide updated information to the agency, the agency cannot provide this information to SBA.

Participating agencies were providing some data that are inconsistent with SBA’s formatting guidance, and while some of these inconsistencies were corrected by SBA’s quality assurance processes, others were not. In 2006, we determined that almost a quarter of the data provided by five of the eight agencies we reviewed was incorrectly formatted for one or more fields in the Tech-Net database. As a result, we concluded that these inconsistent or inaccurate data elements compromised the value of the database for program evaluation purposes. SBA’s quality assurance efforts focus on obtaining complete and accurate data for those fields essential to tracking specific awards, such as the tracking number and award amount, rather than on those fields that contain demographic information about the award recipient. We found that SBA electronically checked the data submitted by the participating agencies to locate and reformat inconsistencies, but it did not take steps to ensure that all agency-provided data were accurate and complete. We also determined that inconsistencies or inaccuracies could arise in certain data fields because SBA interpreted the absence of certain data elements as a negative entry without confirming the accuracy of such an interpretation with the agency. As we reported in 2006, such inaccuracies and inconsistencies were a concern because information in the Tech-Net database would be used to populate the government-use section of the database that SBA was developing (as discussed above) to support SBIR program evaluations. However, at the time of our review, SBA had no plans to correct any of the errors or inconsistencies in the database that related to the historical data already collected. As a result, we concluded that the errors in the existing database would migrate to the government-use section of the database and would compromise the usefulness of the government-use database for program evaluation and monitoring purposes.

To address the concerns that we identified with regard to the quality of the data that SBA was collecting for the Tech-Net database, we recommended in our 2006 report that SBA work with the participating agencies to strengthen the completeness, accuracy, and consistency of its data collection efforts. According to an SBA official, the database is currently operational and agencies have entered data for fiscal years 2007 and 2008 over the Internet. Moreover, according to this official, the system is set up in such a way that it does not accept incorrectly formatted data.

Agencies Focus on Select Awardee Eligibility Criteria

In 2006, we also found that SBA and some participating agencies focused on a few select criteria for determining applicants’ eligibility for SBIR awards. Specifically, we reviewed DOD’s, NIH’s, and SBA’s processes to determine eligibility of applicants for the SBIR program and found that they focused largely on three SBIR criteria in their eligibility reviews— ownership, size in terms of the number of employees, and for-profit status of SBIR applicants. Although agency officials also told us that they consider information on the full range of criteria, such as whether the principal investigator is employed primarily by the applying firm, and the extent to which work on the project will be performed by others.

Moreover, we found that both NIH and DOD largely relied on applicants to self-certify that they met all of the SBIR eligibility criteria as part of their SBIR applications. For example, at NIH, applicants certified that they met the eligibility criteria by completing a verification statement when NIH notified them that their application had been selected for funding but before NIH made the award. The verification statement directs applicants to respond to a series of questions relating to for-profit status, ownership, number of employees, where the work would be performed, and the primary employment of the principal investigator, among others. Similarly, DOD’s cover sheet for each SBIR application directs applicants to certify that they met the program’s eligibility criteria. NIH and DOD would not fund applications if the questions on their agency’s verification statement or cover sheet were not answered. Both NIH and DOD also warned applicants of the civil and criminal penalties for making false, fictitious, or fraudulent statements. In some cases the agencies made additional efforts to ensure the accuracy of the information applicants provided when they observed certain discrepancies in the applications.

In 2006, we reported that when officials at the agencies had unresolved concerns about the accuracy of an applicant’s eligibility information, they referred the matter to SBA to make an eligibility determination. We found that when SBA received a letter from the agency detailing its concerns, SBA officials contacted the applicants and asked them to re-certify their eligibility status and might request additional documentation on the criteria of concern. Upon making a determination of eligibility, SBA then notified the official at the inquiring agency, and the applicant, of its decision.

Although, SBA made the information about firms it found ineligible publicly available on its Web site so that all participating agencies and the public could access the information, we found that it did not consistently include information on the Web site identifying whether or not the determination was for the SBIR program. An SBA official told us the agency planned to include such information on its Web site more systematically before the end of fiscal year 2006. Once the agencies received information about applicants’ eligibility they also had different approaches for retaining and sharing this information. For example, while both NIH and DOD noted the determination of ineligibility in the applicant’s file, NIH also centrally tracked ineligible firms and made this information available to all of its institutes and centers that make SBIR awards. In contrast, DOD did not have a centralized process to share the information across its awarding components, although DOD officials told us it was common practice for awarding components to share such information electronically.

In conclusion, Mr. Chairman, while the SBIR program is generally recognized as a successful program that has encouraged innovation and helped federal agencies achieve their R&D goals, it has continued to suffer from some long-standing evaluation and monitoring issues that are made more difficult because of a lack of accurate, reliable, and comprehensive information on SBIR applicants and awards. The Congress recognized the need for a comprehensive database in 2000 when it mandated that SBA develop a government-use database. Although SBA did not meet its statutorily mandated deadline of June 2001, the database has been operational since October 2008, and contains limited new information but may also contain inaccurate historical data.

Mr. Chairman, this concludes my prepared statement. I would be happy to respond to any questions that you or other members of the Committee may have.

GAO Contact and Staff Acknowledgments

For further information about this statement, please contact me at (202) 512-3841 or at daltonp@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Vondalee Hunt, Anu Mittal, and Cheryl Williams also made key contributions to this statement.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.