Federal Research: Preliminary Information on the Small Business Technology Transfer Program
Published: 1996-01-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a legislative requirement, GAO reviewed the implementation of the Small Business Technology Transfer (STTR) Program, focusing on: (1) the quality and commercial potential of STTR research; (2) how agencies address potential conflicts of interest from the involvement of federally funded research and development (R&D) centers in STTR; and (3) the need for STTR in light of its similarity to the Small Business Innovation Research (SBIR) Program.
section_title: What GAO Found
paragraphs: GAO found that: (1) federal agencies rated the quality and commercial potential of STTR proposals favorably in 1994, but technical experts were concerned about the commercial potential of STTR research because of the newness of the program; (2) agencies have taken actions to avoid potential conflicts of interest arising from R&D centers' involvement in STTR, such as restricting centers' participation, preventing R&D centers from using confidential information in preparing STTR proposals, and ensuring centers' use of outside peer reviews; (3) although agencies have differing views regarding the effects of STTR on SBIR, there is no evidence of competition between STTR and SBIR for quality proposals; (4) the need for STTR is unclear due to its similarity to SBIR, and it will take years to comprehensively evaluate STTR effectiveness; and (5) Congress believes that STTR addresses a core problem in national economic competitiveness and SBIR does not provide a direct mechanism for technology transfer.

section_title: Letter
paragraphs: This report was mandated by the Small Business Research and Development Enhancement Act of 1992. The report focuses on the implementation of the Small Business Technology Transfer (STTR) Pilot Program, which was established by the act. In particular, the report discusses (1) the quality and commercial potential of the STTR Program’s research as shown by technical evaluations of the winning proposals in the first year of the STTR Program, (2) how agencies addressed potential conflicts of interest resulting from the involvement of federally funded research and development centers in the program, and (3) agencies’ views on the effects of and need for the STTR Program in view of its close similarity to the Small Business Innovation Research Program.
Please contact me at (202) 512-3841 if you or your staff have any questions. Major contributors to this report are listed in appendix I.
subsections: section_title: Introduction
paragraphs: The Small Business Research and Development Enhancement Act of 1992 established the Small Business Technology Transfer (STTR) Pilot Program and authorized it for 3 years, beginning in fiscal year 1994. The act also reauthorized the Small Business Innovation Research (SBIR) Program, which was established in 1982. The STTR Program was modeled closely on the SBIR Program. Both of the programs, for example, share the same four major goals:
Stimulating technological innovation.
Using small business to meet federal research and development (R&D) needs.
Fostering and encouraging participation by minorities and disadvantaged persons in technological innovation. Increasing private-sector commercialization of innovations derived from federal R&D.
The two programs include many of the same agencies, and the same agency offices administer the programs.
In spite of the numerous points in common, the two programs differ in one important respect. The STTR Program requires a small business to collaborate with a U.S. nonprofit research institution, such as a university; a contractor-operated, federally funded research and development center; or other entity, in order to be eligible for an award. This collaboration is permitted under the SBIR Program, but it is not mandatory.
subsections: section_title: The Administration of the STTR Program
paragraphs: Agency participation in and funding for the STTR Program followed the general approach established for the SBIR Program. Agencies having a budget of more than $1 billion annually in fiscal years 1994, 1995, or 1996 for external R&D are authorized to set aside a percentage of this amount for the STTR Program. The act sets the percentage at not less than 0.05 percent in fiscal year 1994, not less than 0.1 percent in fiscal year 1995, and not less than 0.15 percent in fiscal year 1996. Five agencies—DOD, DOE, NASA, NIH, and NSF—currently participate in the program. In the SBIR Program, by contrast, the same five agencies and six additional agencies with smaller budgets for external R&D were authorized to set aside a significantly higher percentage and amount of money. When the STTR Program reaches its highest authorized funding percentage in fiscal year 1996, it will receive about $60 million, according to Small Business Administration (SBA) calculations; when the SBIR Program reaches its maximum funding percentage in fiscal year 1997, it will receive about $1 billion.
As with SBIR, the legislation required agencies to solicit proposals for R&D projects. The solicitation lists and describes the topics to be addressed by company proposals and invites companies to submit proposals for consideration. Each agency is responsible for targeting research areas and administering its own STTR funding agreements. Such agreements include a contract, grant, or cooperative agreement entered into between a federal agency and a small business for the performance of experimental, developmental, or research work funded in whole or in part by the federal government.
The legislation also required SBA to issue a policy directive for the general conduct of STTR Programs within federal agencies. The directive was to include such features as simplified, standardized, and timely SBIR solicitations; a simplified, standardized funding process; and minimization of the regulatory burden for small businesses participating in the program. The STTR policy directive was issued in August 1993 and remains in effect. Federal agencies are also required to report key data to SBA.
To be eligible for an STTR award, small businesses must be independently owned and operated, other than the dominant firms in the field in which they are proposing to carry out STTR projects, organized for profit, the employer of 500 or fewer employees (including its affiliates), and at least 51 percent owned by U.S. citizens or lawfully admitted permanent resident aliens.
The law established a three-phase structure for the program. The first phase, not to exceed 1 year, is designed to determine the scientific, technical, and commercial merit and feasibility of a proposed idea. The second phase, not to exceed 2 years, is designed to further develop the idea. The statute established $100,000 and $500,000 as the general limits for phases I and II awards, respectively. The third phase is somewhat more flexible and difficult to define. In general, this phase is expected to result in commercialization or further continuation of R&D. No STTR funding is provided for phase III. Unlike phases I and II, phase III has no general time limits. In addition, phase III can include not only federal non-STTR funds but private-sector funds.
For selection of phase I proposal awards, SBA’s 1993 policy directive stated that agency criteria shall give primary consideration to scientific and technical merit along with the potential for commercialization. According to the directive, funding for phase II shall be based upon the results of phase I and the scientific and technical merit and commercial potential of the phase II proposal.
section_title: The Rationale for the STTR Program
paragraphs: Given the similarities between the SBIR and STTR Programs, the question of the rationale for the STTR Program naturally presents itself. In the 1992 report cited above, the Chairman of the House Committee on Small Business provided two basic answers to this question when the STTR Program was under consideration. The report states that the program addresses a core problem in U.S. economic competitiveness and that the existing SBIR Program does not provide a direct mechanism for technology transfer.
In discussing the first point, the report noted that the nation’s research institutions—its universities, federal laboratories, and nonprofit research institutions—contain enormous scientific and technical resources. The report also noted that the institutions employ one in four scientists and engineers in the nation and perform nearly $40 billion in R&D each year, or one-quarter of all R&D conducted in the United States. In addition, the report stated that perhaps the core of the U.S. competitiveness problem is the inability to translate its worldwide leadership [in science and engineering] into technology and commercial applications that benefit the economy. The report concluded that what is needed is an effective, systematic “technology transfer” mechanism to move new knowledge from the research institution to industry, where it can be exploited for the national good.
In discussing the second point, the report noted that SBIR has turned out to be a remarkably effective mechanism for commercializing ideas in the small business community. However, according to the report, SBIR is less effective at fostering commercialization of ideas that originate in universities, federal laboratories, and nonprofit research institutions. STTR would provide a strong incentive for small companies and researchers at universities, federal laboratories, and nonprofit research institutions to find each other and work together because the only way they can access STTR funding is by collaborating.
Thus, STTR was envisioned primarily as a technology transfer program, in which promising concepts originating in the nonprofit research community would move toward commercialization. In accomplishing this objective, researchers in nonprofit research institutions would ally themselves with small businesses and play a greater role in STTR projects than they could play in SBIR projects.
section_title: Awards Made During the First Year of the STTR Program
paragraphs: Table 1.1 summarizes basic information about the phase I awards made during the first year (fiscal year 1994) of the STTR Program.
As shown in the table, agencies made 206 phase I awards in the first year of the program, with DOD accounting for about half of them. Among the small business affiliates, universities constituted the majority (78 percent) of the research partners. DOE was the only agency in which the awardees formed partnerships with research and development centers in a majority of cases.
Among other significant points about awardees during this period, according to SBA officials, almost two-thirds of the awards went to companies that had previously received awards from the SBIR Program. In addition, the officials told us that the proposed allocation of award money between the small businesses and the research institutions showed the former receiving about 61 percent of the funding and the latter receiving about 39 percent.
section_title: Objectives, Scope, and Methodology
paragraphs: In addressing the quality and commercial potential of STTR research, we obtained and reviewed all of the agencies’ technical evaluations for 206 proposals that received phase I awards on the basis of fiscal year 1994 solicitations. The technical evaluations were prepared by experts in federal agencies or the private sector, if an agency relied on outside peer review. The evaluations played the primary role in determining which proposals were selected for funding. The number of evaluations for each award ranged from one to as many as six.
We reviewed the technical evaluations of project proposals because they were the only systematic source of information on the quality and commercial potential of the research. At this time, the actual results of these awards cannot be assessed. We restricted our work to the 206 awards because they were the only complete set of awards made under the pilot program at the time that we conducted our review. The results of the first round of the phase II award process were not available for all agencies until late in 1995; in fact, only a few phase II awards were made during fiscal year 1994.
Because our conclusions about the quality of research and commercial potential were drawn from these technical evaluations of proposed projects, we gave additional attention to the evaluation process used by each of the agencies. In our view, greater confidence could be placed in evaluations resulting from a thorough and well-documented evaluation process. In particular, we noted the number of reviewers per proposal, the number of proposals per reviewer, and the level of analysis resulting from the reviews. Our findings were agency-specific for DOD, NIH, NASA, NSF, and DOE. In addition to reviewing the technical evaluations, we obtained further information about the evaluation process from discussions with program officials in each of the STTR agencies.
To address compliance by agencies and research and development centers with a requirement of the act to avoid conflicts of interest, we obtained relevant documents from and conducted interviews with STTR Program officials. The act required each federal agency with an STTR Program to develop, in consultation with the Office of Federal Procurement Policy and the Office of Government Ethics, procedures to ensure that research and development centers (1) are free from organizational conflicts of interests relative to the STTR Program; (2) do not use privileged information gained through work performed for an STTR agency or private access to STTR agency personnel in the development of an STTR proposal; and (3) use outside peer review, as appropriate.
To address the effect, if any, of STTR on SBIR and other agency R&D, we interviewed STTR Program officials at SBA and the five agencies that have STTR Programs.As agreed with the Committees, we also looked at the need for the STTR Program. The STTR Program, as mentioned earlier, is modeled closely on the SBIR Program. The issue then arises whether there is a need for two separate programs. In addressing this concern, we reviewed a 1992 report by the House Committee on Small Business. The report helped us to identify questions that are relevant in determining the need for the STTR Program.
We performed our audit work between June and December 1995 in accord with generally accepted government auditing standards.
section_title: Evaluations of STTR Proposals for Research Quality and Commercial Potential Were Favorable, but the Review Process Varied Greatly
paragraphs: Agencies generally rated the quality of the proposed research and commercial potential in STTR proposals highly. For example, DOE rated the quality of research in all 21 of its winning proposals as being among the top 10 percent of all research in the agency. Evaluations of the commercial potential were also favorable but somewhat more cautious. For example, in some cases there were concerns about the cost of the product that might result or the limited size of its potential market. Such reservations were understandable in view of the newness of the program and the innovation or risk associated with many of the proposed projects. The evaluation process, upon which these findings depended, varied greatly. In agencies other than DOD, the selection relied on input and consensus among several (generally three or four) technical reviewers. In DOD, by contrast, a single reviewer was frequently responsible for the technical evaluation.
subsections: section_title: Evaluation Processes and Their Results
paragraphs: Table 2.1 provides a brief overview of the evaluation processes used by the five agencies and the agencies’ assessment of the research proposals.
The following sections summarize (1) the agencies’ evaluation processes and (2) the quality of research and commercial potential identified in the technical evaluations.
subsections: section_title: NIH’s Evaluation Process and Its Results
paragraphs: In its first (fiscal year 1994) solicitation for STTR proposals, NIH identified seven review criteria, including the soundness and technical merit of the proposed approach and commercial potential, which were used to select proposals for funding. These criteria were not given specific weight in the evaluation process.
In the process of evaluation, NIH assigned two reviewers (who provided written evaluations) and two additional readers to each proposal. A peer review panel consisting of 10 or more experts was convened. The four reviewers and readers began the discussion of each proposal by presenting a specific, numerical score to the panel. The two reviewers also presented their written statements. The panel then discussed the proposal in further detail to assist in the development of a final summary statement. Each member of the panel also provided a score, which ranged between 100 for the best and 500 for the worst. The level of quality was defined as follows:
100 to 150: Outstanding.
150 to 200: Excellent.
200 to 250: Very good.
250 to 350: Good.
350 to 500: Acceptable.
For the 48 winning proposals, the average score was 165 (or “excellent”). Specifically, 14 proposals were judged outstanding, 31 excellent, 2 very good, and only 1 good. There were none in the acceptable category. This overall result compared favorably with NIH’s SBIR Program, in which the average score for all SBIR phase I projects with awards in fiscal year 1994 was 187. About 94 percent of STTR’s awards were judged as outstanding or excellent, which compared very favorably with SBIR’s 66 percent.
In contrast to the practice at other agencies, no separate score was assigned for the quality of research and commercial potential. The Chief of the Technology and Applied Sciences Section said that the importance of individual criteria may vary from one proposal to another. In his view, an overall score allows for greater flexibility than the sum of a series of specific scores, each of which represents a fixed percentage of the total potential score. The Chief added, however, that NIH has organized a committee to evaluate its entire scoring system.
Although the panels did not provide scores for each criterion, they provided detailed written analyses of the proposals. The summaries for NIH’s winning proposals include, among other points, a resume of the research, a critique of strengths and shortcomings in terms of research, and an assessment of its commercial potential.
NIH’s evaluations of the quality of research were generally very favorable. One evaluation, for example, described a proposal as addressing a well worthwhile problem involving a new method of enhancing the power of X-ray tubes. The evaluation stated that the proposed approach was sound, had high technical merit, and was supported by the theoretical calculations presented. Statements about commercial potential were similarly favorable but sometimes contained expressions of concern about costliness or other potential drawbacks. In one instance, involving a new approach to X-ray mammography, the evaluation found that the proposed approach likely will result in an expensive technology, but prospects are good for a result that improves on current technology and should enjoy a very high market potential. In another case, the commercial potential for a product that would help prevent geriatric wandering was described as good if the cost of hardware and system operation could be kept low. The evaluation raised concerns about the potential costliness and sophistication of the system.
section_title: DOD’s Evaluation Process and Its Results
paragraphs: DOD identified four criteria in making its phase I awards, including (1) the soundness and technical merit of the proposed approach and (2) the potential for commercial (government or private sector) application and the benefits expected to accrue from this commercialization. DOD specified that each of the four criteria was worth 25 points. Only one DOD agency, the Ballistic Missile Defense Organization (BMDO), developed a different set of basic criteria and provided no quantitative scores.
DOD officials pointed out that there was a great deal of variation from one DOD agency to another in the evaluation process. In particular, the agencies varied in the number of reviews that they require to make an award. In the Navy, Air Force, and Advanced Research Projects Agency (ARPA), 47 of the winning proposals received only one technical review; in BMDO, by contrast, the number ranged between two and six. Several DOD program managers noted that the number and quality of the reviews were time-dependent and that time pressures may lead to fewer or less thorough reviews. In fact, of the 47 winning proposals with only one review, 37 were limited in the analysis used to support the award.
Generally speaking, the DOD evaluations rated the quality of research as favorable for most of the winning proposals; as with NIH, the evaluations of commercial potential were frequently positive but accompanied by caveats.
subsections: section_title: The Navy
paragraphs: Our work identified a number of concerns which, in combination, may weaken the reliability of the Navy’s evaluation process. These concerns included the limited number of reviewers per proposal, the heavy reliance on only a few reviewers for all of the proposals, and the brevity of their evaluations. The process itself resulted in very favorable findings about the quality of research and the commercial potential.
In general, the Navy’s evaluation process and its conclusions depended primarily on the views of only one person. Of the 22 awards, 18 were made on the basis of a single review. Two of the remaining four received two evaluations and the other two received three evaluations.
In addition to the large preponderance of single reviews, the Navy relied heavily on relatively few individuals for its evaluations. For example, one individual in the Office of Naval Research performed seven of the evaluations for winning proposals. The Navy’s program manager told us that this reviewer was also responsible for evaluating 13 unsuccessful proposals.
As a third concern, comments on most of the winning proposals were typically very brief. Single word comments such as “superb” and “outstanding” were used to justify scores on winning proposals that received only one review. In the case of one proposal, for example, “excellent,” “very adequate,” and “top notch” sufficed for the award of nearly perfect scores in three of the four individual categories; a brief technical comment (less than a sentence) addressed the soundness and merit of the approach. No other written narrative or corroborating review was available to support the scores.
We asked the Navy’s SBIR/STTR Program manager whether the conclusions and scores recorded on the worksheets resulted from previous discussions with other reviewers and represented a consensus by more than the one reviewer signing the worksheet. The program manager told us that the reviewer was presenting only his own assessment; no consensus was involved. However, he added that generally two additional officials, usually he and one other person, checked the individual evaluations. In no cases were changes made as a result of these subsequent checks.
In our view, the absence of detail and a broader consensus may limit the reliability of the results. Other DOD and civilian agencies used more than one reviewer per proposal and arrived at a consensus. The Navy program manager said that, although the number of technical reviews may remain limited, he has already taken steps to improve the thoroughness of the review(s) that the proposals receive. These steps include a greater amount of analysis in the technical review(s) and greater involvement in the subsequent review process by higher level scientists within the Navy.
We found that 10 of the 22 winning proposals received perfect scores (25 out of 25 points) for soundness and merit of approach. The lowest score was a 20 (whose meaning was not defined). Generally speaking, the Navy reviews rated the commercial potential of the winning proposals favorably. Ten of the proposals also received perfect scores regarding the potential for and benefit of commercialization. The lowest score in this area was a 15; the reviewer’s main concern was the relatively high risk of the venture.
section_title: The Army
paragraphs: Each of the Army’s proposals received three reviews. A core group of six scientists at the Army’s Research and Technology Integration Office, which moves Army-funded technology from universities to the marketplace, oversaw the process. Each of the Army’s proposals was assigned to a staff scientist with relevant expertise at the Army Research Office; this scientist was responsible for evaluating assigned proposals. After evaluating a proposal, it was given to one or more Army scientists. The evaluations were returned to the original reviewer, who would identify and resolve any discrepancies and then assign a numerical score to each of the four DOD criteria on a scale of zero to 25.
The Army also defined scoring guidelines that assigned specific meanings to numerical scores to keep evaluators consistent with each other and ensure that low scores would be assigned if deserved. The guidelines provide clear definitions to differentiate the various levels of value in the proposals. Soundness and technical merit, for instance, were defined as follows:
Nobel-quality research—25 points.
Cutting-edge research; good chance of major breakthrough—20 points.
Should lead to important product improvements—15 points.
Should lead to important but widely anticipated product improvements—10 points.
Should lead to minor product improvements, at best—5 points.
Unsound and/or of negligible technical merit—0 points.
Commercial potential was defined as follows:
Potential to create a major industry—25 points.
Potential to redirect a major industry—20 points.
Potential to permit small business to grow to medium size, over time and with expected follow-ons—15 points.
Potential to launch a useful new product or product line with a significant market (private and/or government)—10 points.
Potential to sustain a small business for a limited time—5 points.
Little commercialization potential—0 points.
The scores for soundness and technical merit of the approach ranged between 17 and 23 for the 21 winning proposals. Thus, according to the Army’s definitions, Army reviewers regarded all of the winning proposals as leading at the very least to important product improvements. Thirteen of the awards scored 20 or better, meaning that they were regarded as cutting-edge research with a good chance of a major breakthrough. Overall, the average score of the winning proposals was 19.
For commercial potential, the scores ranged between 12 and 23. At the very least, Army reviewers regarded all but two of the winning proposals as having the potential to permit a small business to grow to medium size. Thirteen of the awards scored 20 or better, meaning that they had the potential to redirect a major industry. The award that scored 23 was midway between the potential to redirect a major industry and the potential to found a major industry. Overall, the average score was about 19.
According to the Army’s STTR Program manager, STTR proposals are unquestionably higher in quality than SBIR because researchers from generally outstanding institutions were willing to include their names in proposals only if the research itself was excellent; by requiring the involvement of research institutions, STTR helped to improve the quality of research. In addition, the small size of the STTR Program helped to foster quality; the proposals were more “scrubbed” in the selection process. The Army made only 21 phase I awards out of 350 proposals in the first year of the program.
section_title: The Air Force
paragraphs: Two laboratories (Wright Laboratory and AFOSR) accounted for all the technical evaluations and awards for 31 proposals in the Air Force’s STTR Program. Wright Laboratory evaluated 18 winning proposals, and AFOSR evaluated the other 13.
The program manager at the Aeronautical Systems Center, which includes Wright Laboratory at the Wright-Patterson Air Force Base, said that generally only one person performed the technical evaluation. There was only one technical review for each of 13 of the laboratory’s 18 winning proposals; two or three reviewers evaluated the other winning proposals. The program manager added that each proposal received subsequent review at three successively higher levels by the branch, division, and directorate. In some cases, the technical reviews were thorough critiques in which the individual reviewer carefully documented the basis for the scores assigned to the proposals. In other cases, there was very little documentation to support the findings of the evaluation.
Reviewers rated the soundness and technical merit of all 18 winning proposals as “excellent.” In addition, all but 1 of the 18 were excellent for commercial potential.
The Director of Plans at AFOSR—the second laboratory involved in the STTR Program—said that the laboratory’s policy is to have concurrent review for each proposal. The technical managers were expected to perform an evaluation and also send it to other reviewers. Then, the managers were supposed to distill the overall findings in their reviews and submit the results to higher officials for funding decisions.
AFOSR officials told us that between three and six reviewers participated in the laboratory’s evaluations. For example, one of the technical managers, who had signed 4 of the 11 single evaluation sheets, described his approach to the evaluation process. The manager said that he had personally reviewed 23 proposals in his technical area (packaging for high temperature electronics), eliminated 18, and focused on 5 with the greatest merit. Reviews were then obtained from five other experts for each of these five proposals. Four of the five proposals received awards. According to the technical manager, the conclusions for each proposal represented the consensus of six concurrent reviewers (including himself).
AFOSR’s conclusions about the scientific/technical quality and the commercial potential of its research were very favorable. For example, the technical manager for the four awards involving packaging for high temperature electronics told us that his projects had made unusually rapid progress toward commercialization. The technical manager said that a total of eight automotive companies, aircraft companies, and geothermal well-drilling companies expressed strong interest in the technologies being developed.
BMDO devised its own review form, which asked five questions: (1) How good is the science and engineering? (2) How is the innovation exciting? (3) Are the principal investigator, firm, and research institution qualified? (4) How will BMDO benefit after phase II? (5) How will phase I get to a phase II decision?
None of these criteria referred to the potential for commercialization, which was one of the criteria stated in DOD’s solicitation. BMDO’s STTR Program manager told us that he had omitted a reference to commercialization because, in his view, the reviewers possess technical expertise but are not in a position to evaluate commercial potential in the private sector. The manager also said that he preferred to emphasize innovation rather than commercial potential in selecting phase I awards. He added that his goal was to have four concurrent evaluations for each proposal. In actual practice the number varied from two to six.
In reviewing the technical evaluations of the 15 winning proposals, we found no scores assigned to the individual criteria nor was an overall score provided. The program manager told us that he preferred not to rely on numerical scores. In his view, numerical scores assigned to widely different technologies would not be comparable. In addition, the manager noted that relatively few proposals had the same raters, and differences in the strictness of the ratings might arise.
Although no scores were assigned, there was a wide range of qualitative judgments regarding the merits of the winning proposals. For one proposal with three reviewers, the first reviewer stated that the approach has been known for many years and was a high school science fair demonstration in the mid-1960s. The second reviewer described the proposal as intriguing science and engineering. The third considered the proposal as innovative with an excellent work plan to prove the concept.
Another winning proposal with six reviewers showed an equally wide range of comments on the quality of the science and engineering. Negative comments included: (1) The need for the R&D was not documented; at least 10 laboratories were working on it. (2) The proposal did not provide enough information to determine the quality. Among the six reviews, the only positive comment was that both the science and engineering were excellent. The other three reviewers simply summarized the approach without addressing the question asked of them: How good is the science and engineering? The proposal received four recommendations against funding and two in favor.
In reconciling these divergent views, the program manager told us that he made use of a panel of experts and encouraged an open debate on the merits of a proposal. Each proposal that received at least one recommendation for funding by a reviewer was forwarded to a panel that included the original reviewers and additional experts. On average, about 6 experts were included in each panel, but the number ranged from 3 to 15. According to the program manager, the panel generally enabled him to make a final decision on funding or rejecting each proposal, but in some cases he remained uncertain and obtained one further review from a recognized expert in the field.
The absence of private-sector commercialization as a criterion, the absence of numerical scores, and the diversity of opinions expressed in the evaluations made it difficult to draw conclusions about the research quality and the commercial potential of BMDO’s awards.
ARPA used a 20-point scale for rating its proposals. The soundness and technical merit were worth 8 points; the other three DOD criteria were worth 4 points apiece. ARPA required one technical evaluation with two further reviews of that evaluation. The program manager for each scientific area in ARPA’s solicitation was responsible for providing a written technical evaluation. Supervisors of the program managers reviewed these evaluations and prioritized the proposals. Then the STTR Program manager made selections mainly on the basis of the priority ranking and program balance.
Of the 16 winning proposals, 14 received perfect scores and 2 received scores of 19. For soundness and technical merit, all 16 proposals received the maximum score of 8. For commercial potential, 14 of the projects received perfect scores; the other 2 were rated “very good.”
The narrative portion of the review consisted of brief comments on (1) the overall technical evaluation and justification for selection or rejection and (2) the qualifications of the nonprofit partner and adequacy of contribution to the research project. The comments provided little information on the quality of the research or commercial potential.
Generally, the comments consisted of only one or two sentences. For example, one reviewer noted that there is considerable commercial impact for the technology under consideration as well as a significant military payoff, but the reviewer did not elaborate on this view.
section_title: NASA’s Evaluation Process and Its Results
paragraphs: NASA identified four criteria for evaluating proposals and assigned them significantly different weights. These criteria included scientific/technical merit (worth 20 percent of the total score), the anticipated commercial applications of the technology (worth 40 percent), and two other criteria (with a combined value of 40 percent). The great emphasis placed upon commercial applications in the evaluation process was unique among the agencies.
NASA developed a clear statement of its selection procedures. According to this statement, proposals were to be evaluated by a review team consisting of three members—one from academia, one from the private sector, and one from government. Each reviewer was to independently review the proposals. The NASA technical manager at the field center was required to resolve differences and obtain a consensus on the merit of the proposal. In a further document providing “Guidelines for Evaluators,” NASA stated that a scoring range of 90 to 100 percent should be interpreted as equivalent in quality to the top 10 percent of all NASA proposals for comparable R&D. A score between 80 and 89 percent signified an above average proposal. Of the 21 winning proposals, 11 were considered above average, and 8 were judged as being among the top 10 percent of all NASA proposals for comparable R&D.
section_title: NSF’s Evaluation Process and Its Results
paragraphs: NSF identified five criteria to which approximately equal consideration was given. These criteria included (1) the scientific/engineering quality and (2) the potential for commercial applications and the success of past commercialization efforts.
After an initial screening to eliminate any proposals not responsive to the solicitation, NSF required three concurrent reviews of the remaining proposals. The reviewers rated each of the above criteria on a five-point scale (from poor to excellent). A “very good” score was 20, and “excellent” (or maximum) score was 25. The reviewers then presented their results to a panel, which developed a summary for each proposal.
Of the 11 winning proposals, the quality of the research and its commercial potential were consistently rated as favorable. Among the 33 reviews of the 11 projects, all but one found the quality to range between very good and excellent. For commercial potential, the majority of the reviews found the quality to be excellent. Only 1 of the 33 reviews evaluated the quality as merely “good.”
section_title: DOE’s Evaluation Process and Its Results
paragraphs: DOE identified five criteria with approximately equal consideration given to each of them. These criteria included (1) the scientific/technical quality of the research and (2) the anticipated technical and/or economic benefits of the proposed research, if successful, with special emphasis on the likelihood that the project will attract further funding for product or process development after the STTR support expires.
DOE’s evaluation process consisted of two steps. First, each of three reviewers provided written comments addressing the five criteria but did not assign a specific score to the criteria. Second, technical managers reviewed the three evaluations and quantified the results.
Of the 21 winning proposals, 16 received perfect scores. The other five received perfect scores on four of the five criteria. The managers expressed unanimous agreement about the quality of the research in particular. According to DOE’s definition of quality as used in evaluating STTR proposals, a rating of “outstanding” for scientific/technical merit indicated that the proposal was comparable to the top 10 percent of projects in DOE. All 21 of the winners were rated as outstanding.
While DOE did not specifically evaluate proposals for commercial potential, it did evaluate anticipated technical and/or economic benefits. Twenty of the 21 proposals received outstanding ratings; only 1 was rated in the next lower category as “significant.” Many of the evaluations on which the managers based their conclusions contained favorable, but qualified, statements about commercial potential. For example, one of the reviewers wrote that, if the research is successful, the technology will be rapidly commercialized.
One reason for the excellent results, as noted by DOE’s STTR Program manager, was the unusually large number of proposals in relation to the number of awards available. DOE received 487 proposals, meaning that less than 5 percent of the applicants were successful. In the manager’s view, the program was excessively competitive. As a result, he took two steps to reduce the number of applicants. First, a notice was included in the second solicitation to alert applicants to the situation. The notice pointed out the ratio (1 award to 23 proposals) in the first year and concluded that only those applications with the highest scientific/technical quality would be competitive. Second, because broad topics tend to attract more proposals, the topics were narrowed in the new solicitation to reduce the number of proposals and improve the award/proposal ratio from 1 in 23 to about 1 in 10.
section_title: Conclusions
paragraphs: In general, technical evaluations of STTR proposals showed favorable views of the quality of proposed research and commercial potential. For research quality, the evaluations (1) awarded perfect scores to many proposals, (2) rated proposals among the top 10 percent of research in certain agencies, (3) described some proposals as “cutting edge,” and (4) generally found the quality to be excellent. For commercial potential, the evaluations arrived at similarly favorable conclusions, although in some cases they were somewhat more cautious because of the newness of the program or the risk associated with the proposals.
The evaluation processes varied greatly, ranging from several technical reviewers to only one per proposal and from detailed critiques to evaluation sheets that provided no analysis in support of the ratings. Almost half (47) of DOD’s winning proposals received only one review; in 37 cases, the reviews were too brief to support the findings. Because of their limited analysis and lack of a broader consensus, these instances tended to reduce our confidence in the reliability of the results.
section_title: Agencies Have Taken Steps to Avoid Conflicts of Interest
paragraphs: The five federal agencies with STTR Programs have taken steps to avoid potential problems relating to conflict of interest with federally funded research and development centers. In addition, DOD and DOE, which accounted for 29 of the 32 awards involving centers during the first year of the program, have taken steps to prevent centers from using privileged information in preparing STTR proposals.
subsections: section_title: Agencies Have Taken Steps to Avoid Potential Problems Resulting From R&D Center Involvement
paragraphs: The legislation establishing the STTR Program required agencies to develop procedures to ensure that R&D centers are free from organizational conflicts of interest. Such conflicts might arise, for example, if a center formed a partnership with a company submitting an STTR proposal and helped a federal agency judge the merits of its own and other proposals. A second requirement directs each agency to develop procedures ensuring that the centers use outside peer review, as appropriate. Under the STTR Program, however, the agencies, not the centers, are responsible for decisions regarding peer review; accordingly, we have focused on what the agencies have done. In addition, the legislation required agencies to ensure that the centers do not use privileged information gained through work performed for an STTR agency or private access to STTR agency personnel in the development of an STTR proposal.
subsections: section_title: Agencies Have Taken Steps to Avoid Conflicts of Interest
paragraphs: In general, the five agencies with STTR Programs have taken steps to prevent conflicts of interest from occurring. DOD, DOE, and NIH have specific policies intended to prevent such conflicts while NASA and NSF have more general procedures to prevent such conflicts of interest from arising.
DOD’s Director of Defense Research and Engineering issued a memorandum in mid-1994 providing policy guidance on research and development centers’ participation with industry in STTR and similar programs. The memorandum concludes that, if a center requests authorization to participate in such programs, the mission of the particular center and the potential for conflict of interest must be primary considerations in the decision process.
As a result of DOD’s policy, only two R&D centers are currently approved research partners for its STTR awardees. The Air Force had to rescind some awards because the proposed research partners (certain DOD laboratories) were ineligible to participate. According to DOD’s STTR Program director, future proposals will be evaluated on a case-by-case basis to ensure that conflicts of interest do not occur.
DOE has a policy addressing conflict of interest for the STTR Program. According to the policy, DOE staff members should neither request nor receive assistance from personnel in research institutions (that are eligible to participate in the STTR Program) in the preparation of technical topics for the STTR solicitation. In addition, no person affiliated with a research institution may serve as a reviewer of a grant application that names that research institution as a participant. Furthermore, no one affiliated with a research institution may assist technical managers with the DOE review, evaluation, and selection process for phase I grant applications in a particular scientific area if that research institution is a participant on any grant application submitted to that scientific area.
NIH has adopted a certification procedure to avoid conflicts of interest. NIH’s solicitation for proposals requires the applicant to certify that it “(1) is free from organizational conflicts of interests relative to the STTR Program, (2) did not use privileged information gained through work performed for an STTR agency...and (3) used outside peer review, as appropriate, to evaluate the proposed project and its performance therein.”
NASA relied on a peer review process for its proposals to avoid conflicts of interest. As stated earlier, NASA’s review teams consisted of three members—one from academia, one from the private sector, and one from the government. According to NASA’s STTR Program manager, none of the reviewers were connected with a federally funded R&D center. The manager also mentioned that NASA’s reviewers certified that they have no conflict of interest in their evaluations. The manager said that the only federally funded R&D center directly associated with NASA on a regular basis is the Jet Propulsion Laboratory. The manager said that, although the laboratory was included as a research partner in several proposals by small businesses, it was not involved in reviewing those proposals or administering the program; in addition, no special information was provided to this center.
NSF also relied on peer review procedures to avoid conflicts of interest. Its “Proposal and Award Manual” provides guidance on the use of peer review and states that each NSF program has one primary method for peer review which represents the minimum evaluation received by proposals in that program. Each of its STTR proposals received three reviews. NSF’s STTR Program manager told us that, in fiscal year 1994, none of NSF’s 11 awards involved a small business in a partnership with a research and development center.
In general, four of the five agencies with STTR programs used peer review in evaluating STTR proposals. NIH, NASA, NSF, and DOE relied on specified numbers of outside reviewers. NIH assigned four technical reviewers to each proposal and provided additional input through its peer review panels. NSF, NASA, and DOE used three. DOD was the only agency that relied mainly on technical expertise within the Department rather than on outside reviewers. This approach followed DOD’s usual policy in evaluating proposals.
section_title: DOD and DOE Have Taken Steps to Prevent Research and Development Centers From Using Privileged Information
paragraphs: DOD and DOE, which accounted for 29 of the 32 awards involving R&D centers as research partners, have policies to prevent centers from using privileged information. DOD’s policy of carefully restricting participation by its own laboratories helped in preventing the centers from using inside knowledge in preparing proposals. As mentioned earlier, only two of DOD’s centers were eligible to participate in the program. The exclusion of the other centers, which would be the main sources of privileged information regarding DOD’s research needs, avoided the potential problem raised by this issue.
DOE’s policy prohibits agency staff members from requesting or receiving assistance from personnel in research institutions (that are eligible to participate in the STTR Program) in the preparation of technical topics for the STTR solicitation. This policy is intended to prevent research institutions from using their expertise to influence DOE’s choice of STTR solicitation topics. Otherwise, research institutions could acquire a significant advantage by designing topics to match their expertise and then preparing a proposal in the same area.
section_title: Differing Views on the Effect of and Need for the STTR Program
paragraphs: Agency officials expressed differing views regarding the effect of STTR on SBIR and other agency R&D. Furthermore, none of the officials indicated any specifically negative effects such as a competition between the two programs for quality proposals. However, conclusive information concerning the effect, if any, of STTR on SBIR and other agency R&D was not available because of the program’s newness and smallness. The similarities between the STTR and SBIR Programs raise a broader issue about the need for the STTR Program. The rationale for the program, which points to certain weaknesses in SBIR and potential strengths in STTR, suggests some additional questions that point more directly toward evaluating the need for STTR.
subsections: section_title: Agency Officials Noted No Negative Effects of STTR on SBIR or Other Federal R&D
paragraphs: The legislation establishing the STTR Program required us to assess the effects of STTR on the SBIR Program and other agency R&D. Our discussions with agency officials provided no evidence to suggest that STTR was competing for quality proposals with SBIR or reducing the quality of agency R&D in general. Instead, agency officials expressed differing views regarding the effect of STTR on SBIR and other agency R&D. A few officials noted some potentially beneficial effects. Others said that sufficient data were not yet available to determine the effect, if any, of STTR on SBIR or other agency R&D.
SBA officials contended that STTR was too small and too new a program to have any real effect on SBIR or on the broader range of agency research at the present time. The officials pointed out that the program represented only 0.05 percent of each agency’s external R&D budget during its first year and that it was only 1 year old.
NIH’s STTR Program manager told us that one of the main potential effects of the program is that universities can have a greater role than under SBIR. However, the manager noted that, in an SBIR survey undertaken by NIH several years ago, collaboration between small businesses and universities was already evident in well over half of the SBIR projects.
In contrast to the view that STTR’s effect was very limited, the Army’s STTR Program manager said that STTR was influencing SBIR in a beneficial way. In his opinion, STTR is becoming known through national conferences and other channels. As a result, small businesses are realizing that they have more credibility and chance of winning an award by collaborating with a university or other research institution. The manager believes that the STTR Program has already led to more collaboration in SBIR. In general, according to the program manager, STTR is a promising program that may be as successful as the SBIR Program. The manager also said that STTR will influence agencies whose research has traditionally involved the university community to a lesser extent. In DOD, for example, the program manager believes that STTR’s impact will be greater than in certain other agencies (such as NIH) where the research program has been closely tied to the universities.
DOE’s STTR Program manager said that he has tried to make the topics in the STTR solicitation somewhat different from SBIR. Regarding the effect of STTR on DOE’s research program, he said that STTR projects, if successful, will help to meet the agency’s R&D needs by contributing to areas of particular interest to the agency. However, the manager added that no conclusions can be drawn in the first year of the program because the data are too limited.
section_title: Three Questions Are Relevant in Determining the Need for the STTR Program
paragraphs: The stated rationale for the STTR Program and the mandatory collaboration suggest three additional questions that are relevant in determining the need for the STTR Program: (1) Is the technology originating primarily in the research institution as envisioned in the rationale for the program or is it originating in the small business? (2) Is the mandatory collaboration between the small business and the research institution effective in transferring the technology to the market place? (3) Can the SBIR Program accomplish the same objective without the collaboration required by the STTR Program?
subsections: section_title: Is the Technology Originating Primarily in the Research Institution?
paragraphs: The technology may originate with the research institution, the small business, or a combination of the two. In the STTR Program, the assumption is that the research institution will be the primary originator of the new concept. However, data to determine the extent to which research institutions are providing the core technologies are not currently available. Neither SBA nor the agencies have collected this information. DOE’s SBIR/STTR Program manager said that he would like to know whether the companies or the research institutions were drafting the proposals, but such information is not available.
The relative roles of the research institution and the small business as the source of the technology bear directly on the need for the STTR Program. If a high percentage of the ideas are originating with small businesses rather than with research institutions, this finding would raise questions about the need for the program. On the other hand, if a high percentage of ideas are originating with research institutions, this finding would suggest that the program was achieving the first step in moving ideas from research institutions to small businesses.
section_title: Is the Collaboration Effective in Transferring Ideas From the Research Institution to the Market Place?
paragraphs: If the program is effective in moving ideas from research institutions to small businesses, then the next logical question concerns whether their collaboration is effective in moving them to the market place. This question can be approached from two directions: (1) Short-term views of how well the collaboration is working in general and (2) long-term data on actual commercialization.
Information on how well the collaboration is working can be obtained in the near future. In particular, it would be useful to know how the small businesses rated the contribution made by their research partners to the research effort. Since most of the companies had not completed even the first phase of their STTR award at the time of this report, such information was not available, but it will be obtainable in the next year or two.
Information on actual commercial outcomes will require a greater amount of time before it can be obtained. Generally, 5 to 9 years are needed to turn an initial concept into a marketable product. Thus, it may be several years before the commercial effectiveness of the program can be evaluated.
section_title: Can the SBIR Program Accomplish the Same Objective Without STTR’s Mandatory Collaboration?
paragraphs: Because one important difference between the two programs is that STTR makes a small business/research institution collaboration mandatory, the question arises whether the SBIR Program could accomplish the objective of transferring technology from research institutions to the private sector without the mandatory collaboration. The rationale for the STTR Program tends to assume that such collaborations were relatively rare in the SBIR Program. However, as noted above, NIH’s Program manager told us that, in an SBIR survey undertaken by NIH several years ago, collaboration between small businesses and universities was already evident in well over half of NIH’s SBIR projects. By contrast, the Army’s program manager believed that STTR’s impact will be greater in the Army than in agencies such as NIH because the Army SBIR Program has had a lesser degree of involvement with universities and other research institutions in the past. Given the apparent variation from one agency to another and the lack of current data, no definite conclusion can be drawn at present concerning the need for STTR in forging new collaborations.
Certain proposals may suggest a need for the STTR program. For example, one NIH proposal involved a relationship between the developer of a new type of microscope and a company with the experience and capability to commercialize the instrument. The principal investigator with the company told us that STTR led to a partnership between his company and the research institution that would not have existed otherwise. According to the investigator, without STTR, his company would have been reluctant to devote a lot of time to a technology that it did not have rights to. In such a situation, according to the investigator, the company would have provided specific components of the microscope to the research institution, let the institution develop the product, and then pursued a license to market it. Instead, under STTR, the company developed an agreement with the research institution which led to the current partnership.
However, other cases may suggest the opposite. For example, the president of a company that has participated in both the SBIR and STTR Programs said that, in his experience, the SBIR Program gave companies a greater advantage in dealing with research institutions as potential partners. Under STTR, according to this official, the research institution has “veto power,” but under SBIR, a company is in a better negotiating position if it wants research to be done. In addition, according to the official, the STTR Program will not be able to alter the research-oriented outlook of the universities in a more commercial direction. In general, the official said that almost all STTR research could be accomplished through SBIR.
section_title: Major Contributors to This Report
subsections: section_title: Resources, Community, and Economic Development Division, Washington, D.C.
paragraphs: Bernice Steinhardt, Associate Director Robin Nazzaro, Assistant Director Dennis Carroll, Evaluator The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.