United States General Accounting Office

GAO Report to Congressional Committees

January 1996 FEDERAL RESEARCH

Preliminary

Information on the

Small Business

Technology Transfer

Program

GAO/RCED-96-19

GAO United States

General Accounting Office

Washington, D.C. 20548

Resources, Community, and

Economic Development Division

B-270359 

January 24, 1996

The Honorable Christopher S. Bond

Chairman

The Honorable Dale Bumpers

Ranking Minority Member

Committee on Small Business

United States Senate

The Honorable Jan Meyers

Chairman

The Honorable John J. LaFalce

Ranking Minority Member

Committee on Small Business

House of Representatives

This report was mandated by the Small Business Research and Development Enhancement Act

of 1992. The report focuses on the implementation of the Small Business Technology Transfer

(STTR) Pilot Program, which was established by the act. In particular, the report discusses

(1) the quality and commercial potential of the STTR Program’s research as shown by technical

evaluations of the winning proposals in the first year of the STTR Program, (2) how agencies

addressed potential conflicts of interest resulting from the involvement of federally funded

research and development centers in the program, and (3) agencies’ views on the effects of and

need for the STTR Program in view of its close similarity to the Small Business Innovation

Research Program.

Please contact me at (202) 512-3841 if you or your staff have any questions. Major contributors

to this report are listed in appendix I.

Victor S. Rezendes

Director, Energy and

 Science Issues
Executive Summary

Purpose The nation’s research institutions—its universities, federal laboratories,

and nonprofit research institutions—account for about a quarter of all the

scientists and engineers in the United States. In an effort to move new

knowledge from research institutions to industry, the Congress authorized

the Small Business Technology Transfer (STTR) Pilot Program for 3 years,

beginning in fiscal year 1994. The Congress also required that GAO report

on the implementation of the program. For this report, GAO reviewed

(1) the quality and commercial potential of the STTR Program’s research as

shown by technical evaluations of the winning proposals in the first year

of the program, (2) how agencies addressed potential conflicts of interest

resulting from the involvement of federally funded research and

development centers in the program, and (3) agencies’ views on the effects

of and need for STTR in view of its close similarity to the Small Business

Innovation Research (SBIR) Program.

Background As part of the legislation (P. L. 102-564, Oct. 28, 1992) reauthorizing the

SBIR Program, the Congress established the STTR Program to (1) stimulate

technological innovation, (2) use small businesses to meet federal

research and development (R&D) needs, (3) foster and encourage socially

and economically disadvantaged persons’ participation in technological

innovation, and (4) increase the private sector’s commercialization of

innovations derived from federal R&D.

Five agencies participate in the STTR Program, including the Departments

of Defense (DOD), Energy (DOE), and Health and Human Services’ National

Institutes of Health (NIH); the National Aeronautics and Space

Administration (NASA); and the National Science Foundation (NSF). Each

agency manages its own program, while the Small Business Administration

(SBA) plays a central administrative role, issuing policy directives and

annual reports for the program.

The legislation authorized each agency having an external R&D budget in

excess of $1 billion to set aside not less than 0.05 percent of that budget

for the STTR Program in fiscal year 1994, not less than 0.1 percent in fiscal

year 1995, and not less than 0.15 percent in fiscal year 1996. In the STTR

Program’s first year, the agencies expended about $20 million and

estimated that funding will triple to $60 million in the third and last year of

the program.

The STTR Program provides funding for phase I and phase II awards. Work

in phase I is intended to determine the scientific, technical, and

Page 2 GAO/RCED-96-19 Federal Research
Executive Summary

commercial merit and feasibility of ideas; the work is generally not to

exceed 1 year. Work in phase II further develops the proposed ideas and is

generally not to exceed 2 years. The statute generally limits the size of

awards in phases I and II to $100,000 and $500,000, respectively.

The STTR Program is closely modeled on the SBIR Program, which was

established in 1982. The two programs share the same four goals and other

basic features, including participation by many of the same agencies, the

use of a percentage of the external budget for funding, and a three-phase

approach.

However, the two programs differ in one important respect. In order to be

eligible for an STTR award, a small business must collaborate with a

nonprofit research institution, such as a university, a federally funded

research and development center, or other entity. This collaboration is

permitted under the SBIR Program but is not mandatory. This special STTR

requirement, according to a 1992 report,1 was to provide a more effective

mechanism for transferring new knowledge from research institutions to

industry.

The Congress expressed concern about potential conflicts of interest

resulting from the role of research and development centers in the

program. For example, a conflict could arise if a center and a small

business submitted an STTR proposal as partners and, at the same time, the

center helped the agency judge its own and other proposals. As a result,

the Congress required agencies to take steps to avoid these potential

conflicts of interest.

Results in Brief Federal agencies rated the quality and commercial potential of the Small

Business Technology Transfer Program’s winning proposals favorably in

the first year of the program. While the quality of the proposals was rated

highly, technical experts were somewhat cautious about the commercial

potential, possibly because of the newness of the program.

Agencies have taken steps to avoid potential conflicts of interest that

might arise because of the close connection between federal agencies and

research and development centers. In addition, the Departments of

Defense and Energy, which accounted for almost all of the awards

involving such centers, have taken steps to prevent the centers from using

1

H.R. Rep. No. 554, 102nd Cong., 2nd Sess., pt. 1 (1992). The report accompanied H.R. 4400, a

predecessor to the bill (S. 2941) that was enacted.

Page 3 GAO/RCED-96-19 Federal Research
Executive Summary

privileged information in preparing Small Business Technology Transfer

Program proposals.

Agency officials expressed differing views on the Technology Transfer

Program’s effect on the Small Business Innovation Research Program and

other agency research and development, although none of them indicated

any negative effects such as competition between the two programs for

quality proposals. The similarity of the two programs, however, raises an

issue about the need for the new program.

Principal Findings

Evaluations Were

Favorable, but the Review

Process Varied

Evaluations of the STTR Program’s proposals by technical experts showed a

generally high degree of confidence in the quality of the research and

commercial potential. As one example, DOE rated the quality of proposed

research for all of its winning proposals as high as that for those proposals

in the top 10 percent of research in the Department. Evaluations of the

commercial potential were also favorable but occasionally expressed

concern about the cost of the product that might result or the limited size

of its potential market. Such reservations may be due to the newness of

the program and the risk associated with many of the projects.

The evaluation process, upon which these findings depended, varied

greatly. In NIH, NASA, NSF, and DOE, the selection of winning proposals relied

on consensus among several technical reviewers. In DOD, by contrast, a

single reviewer was frequently responsible for the technical evaluation of

proposals. Of DOD’s 105 winning proposals, 47 received only one technical

review that, in 37 cases, provided only limited analysis to support the

ratings.

Agencies Have Taken Steps

to Avoid Conflicts of

Interest

In general, the five agencies have taken steps to prevent conflicts of

interest from occurring as a result of involvement by research and

development centers in the STTR Program. DOD, DOE, and NIH developed

specific policy guidance concerning potential conflicts of interest. NASA

and NSF relied on general procedures in their procurement guidelines to

avoid conflicts of interest. DOD and DOE have taken steps to prevent

research and development centers from gaining an unfair advantage in

preparing proposals. DOD has carefully restricted participation by its own

Page 4 GAO/RCED-96-19 Federal Research
Executive Summary

laboratories, and DOE has prohibited agency officials from requesting or

receiving assistance from personnel in research institutions in the

preparation of research topics. These two agencies accounted for all but 3

of the 32 awards involving research and development centers as research

partners during the first year of the pilot program. NIH, NASA, NSF, and DOE

also made use of outside peer review; by contrast, DOD tended to rely on

technical expertise within the Department.

Views Differ on Effects of

and Need for the Program

Agency officials expressed differing views on the effect of the STTR

Program on the agencies’ other R&D, including the SBIR Program. However,

the agencies provided no evidence at this early point in the program to

suggest that the STTR Program was competing for quality proposals with

the Innovation Research Program or reducing the quality of the agencies’

R&D in general. Instead, some officials noted some potentially beneficial

effects such as greater collaboration between small businesses and

research institutions in the Innovation Research Program.

The similarity of the two programs, however, raises an issue about the

need for the pilot program. One way to assess the need for the program is

to determine its effectiveness in transferring technology from research

institutions to the marketplace. Such information will not be ascertainable

for at least several years because of the time needed to turn an initial

concept into a marketable technology.

Agency Comments STTR program managers and other officials from DOD, DOE, NASA, NIH, NSF,

and SBA reviewed a draft of this report and provided oral comments. In

general, these officials regarded the report as accurate and balanced. DOD

officials said that even a single technical evaluation of a proposal may be

adequate because of the reviewer’s level of expertise. Agency officials also

provided minor technical corrections that have been incorporated.

Page 5 GAO/RCED-96-19 Federal Research
Contents

Executive Summary 2

Chapter 1 

Introduction

8

The Administration of the STTR Program 8

The Rationale for the STTR Program 10

Awards Made During the First Year of the STTR Program 11

Objectives, Scope, and Methodology 11

Chapter 2 

Evaluations of STTR

Proposals for

Research Quality and

Commercial Potential

Were Favorable, but

the Review Process

Varied Greatly

14

Evaluation Processes and Their Results 14

Conclusions 26

Chapter 3 

Agencies Have Taken

Steps to Avoid

Conflicts of Interest

27

Agencies Have Taken Steps to Avoid Potential Problems

Resulting From R&D Center Involvement

27

Chapter 4 

Differing Views on the

Effect of and Need for

the STTR Program

30

Agency Officials Noted No Negative Effects of STTR on SBIR or

Other Federal R&D

30

Three Questions Are Relevant in Determining the Need for the

STTR Program

31

Appendix Appendix I: Major Contributors to This Report 34

Tables Table 1.1: An Overview of STTR Awards, Fiscal Year 1994 11

Table 2.1: An Overview of the STTR Evaluation Process, Quality

of Proposed Research, and Commercial Potential, Fiscal Year

1994

15

Page 6 GAO/RCED-96-19 Federal Research
Contents

Abbreviations

AFOSR Air Force Office of Scientific Research

ARPA Advanced Research Projects Agency

BMDO Ballistic Missile Defense Organization

DOD Department of Defense

DOE Department of Energy

FAR Federal Acquisition Regulation

GAO General Accounting Office

NASA National Aeronautics and Space Administration

NIH National Institutes of Health

NSF National Science Foundation

ONR Office of Naval Research

R&D research and development

SBA Small Business Administration

SBIR Small Business Innovation Research Program

STTR Small Business Technology Transfer Pilot Program

Page 7 GAO/RCED-96-19 Federal Research
Chapter 1 

Introduction

The Small Business Research and Development Enhancement Act of 1992

established the Small Business Technology Transfer (STTR) Pilot Program

and authorized it for 3 years, beginning in fiscal year 1994. The act also

reauthorized the Small Business Innovation Research (SBIR) Program,

which was established in 1982. The STTR Program was modeled closely on

the SBIR Program. Both of the programs, for example, share the same four

major goals:

• Stimulating technological innovation.

• Using small business to meet federal research and development (R&D)

needs.

• Fostering and encouraging participation by minorities and disadvantaged

persons in technological innovation.

• Increasing private-sector commercialization of innovations derived from

federal R&D.

The two programs include many of the same agencies, and the same

agency offices administer the programs.

In spite of the numerous points in common, the two programs differ in one

important respect. The STTR Program requires a small business to

collaborate with a U.S. nonprofit research institution, such as a university;

a contractor-operated, federally funded research and development center;

or other entity, in order to be eligible for an award. This collaboration is

permitted under the SBIR Program, but it is not mandatory.

The Administration of

the STTR Program

Agency participation in and funding for the STTR Program followed the

general approach established for the SBIR Program. Agencies having a

budget of more than $1 billion annually in fiscal years 1994, 1995, or 1996

for external R&D are authorized to set aside a percentage of this amount for

the STTR Program. The act sets the percentage at not less than 0.05 percent

in fiscal year 1994, not less than 0.1 percent in fiscal year 1995, and not less

than 0.15 percent in fiscal year 1996. Five agencies—DOD, DOE, NASA, NIH,

and NSF—currently participate in the program. In the SBIR Program, by

contrast, the same five agencies and six additional agencies with smaller

budgets for external R&D were authorized to set aside a significantly higher

percentage and amount of money. When the STTR Program reaches its

highest authorized funding percentage in fiscal year 1996, it will receive

about $60 million, according to Small Business Administration (SBA)

calculations; when the SBIR Program reaches its maximum funding

percentage in fiscal year 1997, it will receive about $1 billion.

Page 8 GAO/RCED-96-19 Federal Research
Chapter 1 

Introduction

As with SBIR, the legislation required agencies to solicit proposals for R&D

projects. The solicitation lists and describes the topics to be addressed by

company proposals and invites companies to submit proposals for

consideration. Each agency is responsible for targeting research areas and

administering its own STTR funding agreements. Such agreements include a

contract, grant, or cooperative agreement entered into between a federal

agency and a small business for the performance of experimental,

developmental, or research work funded in whole or in part by the federal

government.

The legislation also required SBA to issue a policy directive for the general

conduct of STTR Programs within federal agencies. The directive was to

include such features as simplified, standardized, and timely SBIR

solicitations; a simplified, standardized funding process; and minimization

of the regulatory burden for small businesses participating in the program.

The STTR policy directive was issued in August 1993 and remains in effect.

Federal agencies are also required to report key data to SBA.

To be eligible for an STTR award, small businesses must be

• independently owned and operated,

• other than the dominant firms in the field in which they are proposing to

carry out STTR projects,

• organized for profit,

• the employer of 500 or fewer employees (including its affiliates), and

• at least 51 percent owned by U.S. citizens or lawfully admitted permanent

resident aliens.

The law established a three-phase structure for the program. The first

phase, not to exceed 1 year, is designed to determine the scientific,

technical, and commercial merit and feasibility of a proposed idea. The

second phase, not to exceed 2 years, is designed to further develop the

idea. The statute established $100,000 and $500,000 as the general limits

for phases I and II awards, respectively. The third phase is somewhat more

flexible and difficult to define. In general, this phase is expected to result

in commercialization or further continuation of R&D. No STTR funding is

provided for phase III. Unlike phases I and II, phase III has no general time

limits. In addition, phase III can include not only federal non-STTR funds

but private-sector funds.

For selection of phase I proposal awards, SBA’s 1993 policy directive stated

that agency criteria shall give primary consideration to scientific and

Page 9 GAO/RCED-96-19 Federal Research
Chapter 1 

Introduction

technical merit along with the potential for commercialization. According

to the directive, funding for phase II shall be based upon the results of

phase I and the scientific and technical merit and commercial potential of

the phase II proposal.

The Rationale for the

STTR Program

Given the similarities between the SBIR and STTR Programs, the question of

the rationale for the STTR Program naturally presents itself. In the 1992

report cited above, the Chairman of the House Committee on Small

Business provided two basic answers to this question when the STTR

Program was under consideration. The report states that the program

addresses a core problem in U.S. economic competitiveness and that the

existing SBIR Program does not provide a direct mechanism for technology

transfer.

In discussing the first point, the report noted that the nation’s research

institutions—its universities, federal laboratories, and nonprofit research

institutions—contain enormous scientific and technical resources. The

report also noted that the institutions employ one in four scientists and

engineers in the nation and perform nearly $40 billion in R&D each year, or

one-quarter of all R&D conducted in the United States. In addition, the

report stated that perhaps the core of the U.S. competitiveness problem is

the inability to translate its worldwide leadership [in science and

engineering] into technology and commercial applications that benefit the

economy. The report concluded that what is needed is an effective,

systematic “technology transfer” mechanism to move new knowledge from

the research institution to industry, where it can be exploited for the

national good.

In discussing the second point, the report noted that SBIR has turned out to

be a remarkably effective mechanism for commercializing ideas in the

small business community. However, according to the report, SBIR is less

effective at fostering commercialization of ideas that originate in

universities, federal laboratories, and nonprofit research institutions. STTR

would provide a strong incentive for small companies and researchers at

universities, federal laboratories, and nonprofit research institutions to

find each other and work together because the only way they can access

STTR funding is by collaborating.

Thus, STTR was envisioned primarily as a technology transfer program, in

which promising concepts originating in the nonprofit research

community would move toward commercialization. In accomplishing this

Page 10 GAO/RCED-96-19 Federal Research
Chapter 1 

Introduction

objective, researchers in nonprofit research institutions would ally

themselves with small businesses and play a greater role in STTR projects

than they could play in SBIR projects.

Awards Made During

the First Year of the

STTR Program

Table 1.1 summarizes basic information about the phase I awards made

during the first year (fiscal year 1994) of the STTR Program.

Table 1.1: An Overview of STTR

Awards, Fiscal Year 1994 (Dollars in millions)

Agency

Number of

proposals

Phase I

awards Amount Universities

R&D

centers Other

DOD 911 105 $9.8 85 15 5

NIH 283 48 4.7 41 2 5

NASA 159 21 2.1 17 1 3

NSF 110 11 1.1 11 0 0

DOE 487 21 2.1 6 14 1

Total 1,950 206 $19.8 160 32 14

As shown in the table, agencies made 206 phase I awards in the first year

of the program, with DOD accounting for about half of them. Among the

small business affiliates, universities constituted the majority (78 percent)

of the research partners. DOE was the only agency in which the awardees

formed partnerships with research and development centers in a majority

of cases.

Among other significant points about awardees during this period,

according to SBA officials, almost two-thirds of the awards went to

companies that had previously received awards from the SBIR Program. In

addition, the officials told us that the proposed allocation of award money

between the small businesses and the research institutions showed the

former receiving about 61 percent of the funding and the latter receiving

about 39 percent.

Objectives, Scope,

and Methodology

In addressing the quality and commercial potential of STTR research, we

obtained and reviewed all of the agencies’ technical evaluations for 206

Page 11 GAO/RCED-96-19 Federal Research
Chapter 1 

Introduction

proposals that received phase I awards on the basis of fiscal year 1994

solicitations. The technical evaluations were prepared by experts in

federal agencies or the private sector, if an agency relied on outside peer

review. The evaluations played the primary role in determining which

proposals were selected for funding. The number of evaluations for each

award ranged from one to as many as six.

We reviewed the technical evaluations of project proposals because they

were the only systematic source of information on the quality and

commercial potential of the research. At this time, the actual results of

these awards cannot be assessed. We restricted our work to the 206

awards because they were the only complete set of awards made under

the pilot program at the time that we conducted our review. The results of

the first round of the phase II award process were not available for all

agencies until late in 1995; in fact, only a few phase II awards were made

during fiscal year 1994.

Because our conclusions about the quality of research and commercial

potential were drawn from these technical evaluations of proposed

projects, we gave additional attention to the evaluation process used by

each of the agencies. In our view, greater confidence could be placed in

evaluations resulting from a thorough and well-documented evaluation

process. In particular, we noted the number of reviewers per proposal, the

number of proposals per reviewer, and the level of analysis resulting from

the reviews. Our findings were agency-specific for DOD, NIH, NASA, NSF, and

DOE. In addition to reviewing the technical evaluations, we obtained

further information about the evaluation process from discussions with

program officials in each of the STTR agencies.

To address compliance by agencies and research and development centers

with a requirement of the act to avoid conflicts of interest, we obtained

relevant documents from and conducted interviews with STTR Program

officials. The act required each federal agency with an STTR Program to

develop, in consultation with the Office of Federal Procurement Policy

and the Office of Government Ethics, procedures to ensure that research

and development centers (1) are free from organizational conflicts of

interests relative to the STTR Program; (2) do not use privileged

information gained through work performed for an STTR agency or private

access to STTR agency personnel in the development of an STTR proposal;

and (3) use outside peer review, as appropriate.

Page 12 GAO/RCED-96-19 Federal Research
Chapter 1 

Introduction

To address the effect, if any, of STTR on SBIR and other agency R&D, we

interviewed STTR Program officials at SBA and the five agencies that have

STTR Programs.As agreed with the Committees, we also looked at the need

for the STTR Program. The STTR Program, as mentioned earlier, is modeled

closely on the SBIR Program. The issue then arises whether there is a need

for two separate programs. In addressing this concern, we reviewed a 1992

report by the House Committee on Small Business. The report helped us to

identify questions that are relevant in determining the need for the STTR

Program.

We performed our audit work between June and December 1995 in accord

with generally accepted government auditing standards.

Page 13 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

Agencies generally rated the quality of the proposed research and

commercial potential in STTR proposals highly. For example, DOE rated the

quality of research in all 21 of its winning proposals as being among the

top 10 percent of all research in the agency. Evaluations of the commercial

potential were also favorable but somewhat more cautious. For example,

in some cases there were concerns about the cost of the product that

might result or the limited size of its potential market. Such reservations

were understandable in view of the newness of the program and the

innovation or risk associated with many of the proposed projects. The

evaluation process, upon which these findings depended, varied greatly. In

agencies other than DOD, the selection relied on input and consensus

among several (generally three or four) technical reviewers. In DOD, by

contrast, a single reviewer was frequently responsible for the technical

evaluation.

Evaluation Processes

and Their Results

Table 2.1 provides a brief overview of the evaluation processes used by the

five agencies and the agencies’ assessment of the research proposals.

Page 14 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

Table 2.1: An Overview of the STTR Evaluation Process, Quality of Proposed Research, and Commercial Potential, Fiscal

Year 1994

Agency

Total

awards

Reviews per

proposal

Single

reviews

Quality of

proposed

research

Commercial

potential

NIH 48 4 0 a a

DOD (total) 105 1 to 6 47 highly rated highly rated

Navy 22 1 to 3 18 10 of 22

perfect scores

10 of 22

perfect scores

Army 21 3 0 cutting edge

research

highly rated

Air Force/ Wright Laboratory 18 1 to 3 13 excellent excellent

Air Force Office of Scientific Research 13 3 to 6 0 a a

Ballistic Missile Defense Organization 15 2 to 6 0 a a

Advanced Research Projects Agency 16 1 16 all 16 perfect

scores

14 perfect scores

NASA 21 3 0 above average/

top 10 percent

above average/

top 10 percent

NSF 11 3 0 excellent excellent

DOE 21 3 0 top 10 percent highly rated

aNIH, the Air Force Office of Scientific Research (AFOSR), and the Ballistic Missile Defense

Organization (BMDO) provided only qualitative, not quantitative, evaluations for this criterion. At

NIH, each project was given a score on an overall basis that included research quality and

commercial potential. In general, NIH’s proposals were judged as either outstanding or excellent.

At AFOSR, the research quality and commercial potential were highly rated. At BMDO, it was

difficult to draw conclusions because of the diversity of views expressed in the evaluations.

The following sections summarize (1) the agencies’ evaluation processes

and (2) the quality of research and commercial potential identified in the

technical evaluations.

NIH’s Evaluation Process

and Its Results

In its first (fiscal year 1994) solicitation for STTR proposals, NIH identified

seven review criteria, including the soundness and technical merit of the

proposed approach and commercial potential, which were used to select

proposals for funding. These criteria were not given specific weight in the

evaluation process.

In the process of evaluation, NIH assigned two reviewers (who provided

written evaluations) and two additional readers to each proposal. A peer

review panel consisting of 10 or more experts was convened. The four

reviewers and readers began the discussion of each proposal by

Page 15 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

presenting a specific, numerical score to the panel. The two reviewers also

presented their written statements. The panel then discussed the proposal

in further detail to assist in the development of a final summary statement.

Each member of the panel also provided a score, which ranged between

100 for the best and 500 for the worst. The level of quality was defined as

follows:

• 100 to 150: Outstanding.

• 150 to 200: Excellent.

• 200 to 250: Very good.

• 250 to 350: Good.

• 350 to 500: Acceptable.

For the 48 winning proposals, the average score was 165 (or “excellent”).

Specifically, 14 proposals were judged outstanding, 31 excellent, 2 very

good, and only 1 good. There were none in the acceptable category. This

overall result compared favorably with NIH’s SBIR Program, in which the

average score for all SBIR phase I projects with awards in fiscal year 1994

was 187. About 94 percent of STTR’s awards were judged as outstanding or

excellent, which compared very favorably with SBIR’s 66 percent.

In contrast to the practice at other agencies, no separate score was

assigned for the quality of research and commercial potential. The Chief of

the Technology and Applied Sciences Section said that the importance of

individual criteria may vary from one proposal to another. In his view, an

overall score allows for greater flexibility than the sum of a series of

specific scores, each of which represents a fixed percentage of the total

potential score. The Chief added, however, that NIH has organized a

committee to evaluate its entire scoring system.

Although the panels did not provide scores for each criterion, they

provided detailed written analyses of the proposals. The summaries for

NIH’s winning proposals include, among other points, a resume of the

research, a critique of strengths and shortcomings in terms of research,

and an assessment of its commercial potential.

NIH’s evaluations of the quality of research were generally very favorable.

One evaluation, for example, described a proposal as addressing a well

worthwhile problem involving a new method of enhancing the power of

X-ray tubes. The evaluation stated that the proposed approach was sound,

had high technical merit, and was supported by the theoretical

calculations presented. Statements about commercial potential were

Page 16 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

similarly favorable but sometimes contained expressions of concern about

costliness or other potential drawbacks. In one instance, involving a new

approach to X-ray mammography, the evaluation found that the proposed

approach likely will result in an expensive technology, but prospects are

good for a result that improves on current technology and should enjoy a

very high market potential. In another case, the commercial potential for a

product that would help prevent geriatric wandering was described as

good if the cost of hardware and system operation could be kept low. The

evaluation raised concerns about the potential costliness and

sophistication of the system.

DOD’s Evaluation Process

and Its Results

DOD identified four criteria in making its phase I awards, including (1) the

soundness and technical merit of the proposed approach and (2) the

potential for commercial (government or private sector) application and

the benefits expected to accrue from this commercialization. DOD specified

that each of the four criteria was worth 25 points. Only one DOD agency,

the Ballistic Missile Defense Organization (BMDO), developed a different set

of basic criteria and provided no quantitative scores.

DOD officials pointed out that there was a great deal of variation from one

DOD agency to another in the evaluation process. In particular, the agencies

varied in the number of reviews that they require to make an award. In the

Navy, Air Force, and Advanced Research Projects Agency (ARPA), 47 of the

winning proposals received only one technical review; in BMDO, by

contrast, the number ranged between two and six. Several DOD program

managers noted that the number and quality of the reviews were

time-dependent and that time pressures may lead to fewer or less

thorough reviews. In fact, of the 47 winning proposals with only one

review, 37 were limited in the analysis used to support the award.

Generally speaking, the DOD evaluations rated the quality of research as

favorable for most of the winning proposals; as with NIH, the evaluations of

commercial potential were frequently positive but accompanied by

caveats.

The Navy Our work identified a number of concerns which, in combination, may

weaken the reliability of the Navy’s evaluation process. These concerns

included the limited number of reviewers per proposal, the heavy reliance

on only a few reviewers for all of the proposals, and the brevity of their

evaluations. The process itself resulted in very favorable findings about

the quality of research and the commercial potential.

Page 17 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

In general, the Navy’s evaluation process and its conclusions depended

primarily on the views of only one person. Of the 22 awards, 18 were made

on the basis of a single review. Two of the remaining four received two

evaluations and the other two received three evaluations.

In addition to the large preponderance of single reviews, the Navy relied

heavily on relatively few individuals for its evaluations. For example, one

individual in the Office of Naval Research performed seven of the

evaluations for winning proposals. The Navy’s program manager told us

that this reviewer was also responsible for evaluating 13 unsuccessful

proposals.

As a third concern, comments on most of the winning proposals were

typically very brief. Single word comments such as “superb” and

“outstanding” were used to justify scores on winning proposals that

received only one review. In the case of one proposal, for example,

“excellent,” “very adequate,” and “top notch” sufficed for the award of

nearly perfect scores in three of the four individual categories; a brief

technical comment (less than a sentence) addressed the soundness and

merit of the approach. No other written narrative or corroborating review

was available to support the scores.

We asked the Navy’s SBIR/STTR Program manager whether the conclusions

and scores recorded on the worksheets resulted from previous discussions

with other reviewers and represented a consensus by more than the one

reviewer signing the worksheet. The program manager told us that the

reviewer was presenting only his own assessment; no consensus was

involved. However, he added that generally two additional officials,

usually he and one other person, checked the individual evaluations. In no

cases were changes made as a result of these subsequent checks.

In our view, the absence of detail and a broader consensus may limit the

reliability of the results. Other DOD and civilian agencies used more than

one reviewer per proposal and arrived at a consensus. The Navy program

manager said that, although the number of technical reviews may remain

limited, he has already taken steps to improve the thoroughness of the

review(s) that the proposals receive. These steps include a greater amount

Page 18 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

of analysis in the technical review(s) and greater involvement in the

subsequent review process by higher level scientists within the Navy.2

We found that 10 of the 22 winning proposals received perfect scores (25

out of 25 points) for soundness and merit of approach. The lowest score

was a 20 (whose meaning was not defined). Generally speaking, the Navy

reviews rated the commercial potential of the winning proposals

favorably. Ten of the proposals also received perfect scores regarding the

potential for and benefit of commercialization. The lowest score in this

area was a 15; the reviewer’s main concern was the relatively high risk of

the venture.

The Army Each of the Army’s proposals received three reviews. A core group of six

scientists at the Army’s Research and Technology Integration Office,

which moves Army-funded technology from universities to the

marketplace, oversaw the process. Each of the Army’s proposals was

assigned to a staff scientist with relevant expertise at the Army Research

Office; this scientist was responsible for evaluating assigned proposals.

After evaluating a proposal, it was given to one or more Army scientists.

The evaluations were returned to the original reviewer, who would

identify and resolve any discrepancies and then assign a numerical score

to each of the four DOD criteria on a scale of zero to 25.

The Army also defined scoring guidelines that assigned specific meanings

to numerical scores to keep evaluators consistent with each other and

ensure that low scores would be assigned if deserved. The guidelines

provide clear definitions to differentiate the various levels of value in the

proposals. Soundness and technical merit, for instance, were defined as

follows:

• Nobel-quality research—25 points.

• Cutting-edge research; good chance of major breakthrough—20 points.

• Should lead to important product improvements—15 points.

• Should lead to important but widely anticipated product

improvements—10 points.

• Should lead to minor product improvements, at best—5 points.

• Unsound and/or of negligible technical merit—0 points.

2

During our meeting with DOD officials in December 1995 to discuss our draft report, the Navy’s STTR

program manager noted a key reason for the limited number of technical reviewers used by the Navy

in evaluating STTR proposals. He said he has relied exclusively on scientists within the Office of Naval

Research for technical reviews of STTR proposals. He has not included scientists from other DOD

laboratories because he hopes that these laboratories may eventually be included as research partners

in STTR work; their involvement in the evaluation process would create a conflict of interest with their

potential role in STTR-related research.

Page 19 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

Commercial potential was defined as follows:

• Potential to create a major industry—25 points.

• Potential to redirect a major industry—20 points.

• Potential to permit small business to grow to medium size, over time and

with expected follow-ons—15 points.

• Potential to launch a useful new product or product line with a significant

market (private and/or government)—10 points.

• Potential to sustain a small business for a limited time—5 points.

• Little commercialization potential—0 points.

The scores for soundness and technical merit of the approach ranged

between 17 and 23 for the 21 winning proposals. Thus, according to the

Army’s definitions, Army reviewers regarded all of the winning proposals

as leading at the very least to important product improvements. Thirteen

of the awards scored 20 or better, meaning that they were regarded as

cutting-edge research with a good chance of a major breakthrough.

Overall, the average score of the winning proposals was 19.

For commercial potential, the scores ranged between 12 and 23. At the

very least, Army reviewers regarded all but two of the winning proposals

as having the potential to permit a small business to grow to medium size.

Thirteen of the awards scored 20 or better, meaning that they had the

potential to redirect a major industry. The award that scored 23 was

midway between the potential to redirect a major industry and the

potential to found a major industry. Overall, the average score was about

19.

According to the Army’s STTR Program manager, STTR proposals are

unquestionably higher in quality than SBIR because researchers from

generally outstanding institutions were willing to include their names in

proposals only if the research itself was excellent; by requiring the

involvement of research institutions, STTR helped to improve the quality of

research. In addition, the small size of the STTR Program helped to foster

quality; the proposals were more “scrubbed” in the selection process. The

Army made only 21 phase I awards out of 350 proposals in the first year of

the program.

The Air Force Two laboratories (Wright Laboratory and AFOSR) accounted for all the

technical evaluations and awards for 31 proposals in the Air Force’s STTR

Page 20 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

Program. Wright Laboratory evaluated 18 winning proposals, and AFOSR

evaluated the other 13.

The program manager at the Aeronautical Systems Center, which includes

Wright Laboratory at the Wright-Patterson Air Force Base, said that

generally only one person performed the technical evaluation. There was

only one technical review for each of 13 of the laboratory’s 18 winning

proposals; two or three reviewers evaluated the other winning proposals.

The program manager added that each proposal received subsequent

review at three successively higher levels by the branch, division, and

directorate. In some cases, the technical reviews were thorough critiques

in which the individual reviewer carefully documented the basis for the

scores assigned to the proposals. In other cases, there was very little

documentation to support the findings of the evaluation.

Reviewers rated the soundness and technical merit of all 18 winning

proposals as “excellent.” In addition, all but 1 of the 18 were excellent for

commercial potential.

The Director of Plans at AFOSR—the second laboratory involved in the STTR

Program—said that the laboratory’s policy is to have concurrent review

for each proposal. The technical managers were expected to perform an

evaluation and also send it to other reviewers. Then, the managers were

supposed to distill the overall findings in their reviews and submit the

results to higher officials for funding decisions.

AFOSR officials told us that between three and six reviewers participated in

the laboratory’s evaluations. For example, one of the technical managers,

who had signed 4 of the 11 single evaluation sheets, described his

approach to the evaluation process. The manager said that he had

personally reviewed 23 proposals in his technical area (packaging for high

temperature electronics), eliminated 18, and focused on 5 with the greatest

merit. Reviews were then obtained from five other experts for each of

these five proposals. Four of the five proposals received awards.

According to the technical manager, the conclusions for each proposal

represented the consensus of six concurrent reviewers (including

himself).

AFOSR’s conclusions about the scientific/technical quality and the

commercial potential of its research were very favorable. For example, the

technical manager for the four awards involving packaging for high

temperature electronics told us that his projects had made unusually rapid

Page 21 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

progress toward commercialization. The technical manager said that a

total of eight automotive companies, aircraft companies, and geothermal

well-drilling companies expressed strong interest in the technologies being

developed.

BMDO BMDO devised its own review form, which asked five questions: (1) How

good is the science and engineering? (2) How is the innovation exciting?

(3) Are the principal investigator, firm, and research institution qualified?

(4) How will BMDO benefit after phase II? (5) How will phase I get to a

phase II decision?

None of these criteria referred to the potential for commercialization,

which was one of the criteria stated in DOD’s solicitation. BMDO’s STTR

Program manager told us that he had omitted a reference to

commercialization because, in his view, the reviewers possess technical

expertise but are not in a position to evaluate commercial potential in the

private sector. The manager also said that he preferred to emphasize

innovation rather than commercial potential in selecting phase I awards.

He added that his goal was to have four concurrent evaluations for each

proposal. In actual practice the number varied from two to six.

In reviewing the technical evaluations of the 15 winning proposals, we

found no scores assigned to the individual criteria nor was an overall

score provided. The program manager told us that he preferred not to rely

on numerical scores. In his view, numerical scores assigned to widely

different technologies would not be comparable. In addition, the manager

noted that relatively few proposals had the same raters, and differences in

the strictness of the ratings might arise.

Although no scores were assigned, there was a wide range of qualitative

judgments regarding the merits of the winning proposals. For one proposal

with three reviewers, the first reviewer stated that the approach has been

known for many years and was a high school science fair demonstration in

the mid-1960s. The second reviewer described the proposal as intriguing

science and engineering. The third considered the proposal as innovative

with an excellent work plan to prove the concept.

Another winning proposal with six reviewers showed an equally wide

range of comments on the quality of the science and engineering. Negative

comments included: (1) The need for the R&D was not documented; at least

10 laboratories were working on it. (2) The proposal did not provide

enough information to determine the quality. Among the six reviews, the

Page 22 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

only positive comment was that both the science and engineering were

excellent. The other three reviewers simply summarized the approach

without addressing the question asked of them: How good is the science

and engineering? The proposal received four recommendations against

funding and two in favor.

In reconciling these divergent views, the program manager told us that he

made use of a panel of experts and encouraged an open debate on the

merits of a proposal. Each proposal that received at least one

recommendation for funding by a reviewer was forwarded to a panel that

included the original reviewers and additional experts. On average, about

6 experts were included in each panel, but the number ranged from 3 to

15. According to the program manager, the panel generally enabled him to

make a final decision on funding or rejecting each proposal, but in some

cases he remained uncertain and obtained one further review from a

recognized expert in the field.

The absence of private-sector commercialization as a criterion, the

absence of numerical scores, and the diversity of opinions expressed in

the evaluations made it difficult to draw conclusions about the research

quality and the commercial potential of BMDO’s awards.

ARPA ARPA used a 20-point scale for rating its proposals. The soundness and

technical merit were worth 8 points; the other three DOD criteria were

worth 4 points apiece. ARPA required one technical evaluation with two

further reviews of that evaluation. The program manager for each

scientific area in ARPA’s solicitation was responsible for providing a written

technical evaluation. Supervisors of the program managers reviewed these

evaluations and prioritized the proposals. Then the STTR Program manager

made selections mainly on the basis of the priority ranking and program

balance.

Of the 16 winning proposals, 14 received perfect scores and 2 received

scores of 19. For soundness and technical merit, all 16 proposals received

the maximum score of 8. For commercial potential, 14 of the projects

received perfect scores; the other 2 were rated “very good.”

The narrative portion of the review consisted of brief comments on (1) the

overall technical evaluation and justification for selection or rejection and

(2) the qualifications of the nonprofit partner and adequacy of

contribution to the research project. The comments provided little

information on the quality of the research or commercial potential.

Page 23 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

Generally, the comments consisted of only one or two sentences. For

example, one reviewer noted that there is considerable commercial impact

for the technology under consideration as well as a significant military

payoff, but the reviewer did not elaborate on this view.

NASA’s Evaluation Process

and Its Results

NASA identified four criteria for evaluating proposals and assigned them

significantly different weights. These criteria included scientific/technical

merit (worth 20 percent of the total score), the anticipated commercial

applications of the technology (worth 40 percent), and two other criteria

(with a combined value of 40 percent). The great emphasis placed upon

commercial applications in the evaluation process was unique among the

agencies.

NASA developed a clear statement of its selection procedures. According to

this statement, proposals were to be evaluated by a review team consisting

of three members—one from academia, one from the private sector, and

one from government. Each reviewer was to independently review the

proposals. The NASA technical manager at the field center was required to

resolve differences and obtain a consensus on the merit of the proposal. In

a further document providing “Guidelines for Evaluators,” NASA stated that

a scoring range of 90 to 100 percent should be interpreted as equivalent in

quality to the top 10 percent of all NASA proposals for comparable R&D. A

score between 80 and 89 percent signified an above average proposal. Of

the 21 winning proposals, 11 were considered above average, and 8 were

judged as being among the top 10 percent of all NASA proposals for

comparable R&D.

NSF’s Evaluation Process

and Its Results

NSF identified five criteria to which approximately equal consideration was

given. These criteria included (1) the scientific/engineering quality and

(2) the potential for commercial applications and the success of past

commercialization efforts.

After an initial screening to eliminate any proposals not responsive to the

solicitation, NSF required three concurrent reviews of the remaining

proposals. The reviewers rated each of the above criteria on a five-point

scale (from poor to excellent). A “very good” score was 20, and “excellent”

(or maximum) score was 25. The reviewers then presented their results to

a panel, which developed a summary for each proposal.

Page 24 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

Of the 11 winning proposals, the quality of the research and its commercial

potential were consistently rated as favorable. Among the 33 reviews of

the 11 projects, all but one found the quality to range between very good

and excellent. For commercial potential, the majority of the reviews found

the quality to be excellent. Only 1 of the 33 reviews evaluated the quality

as merely “good.”

DOE’s Evaluation Process

and Its Results

DOE identified five criteria with approximately equal consideration given to

each of them. These criteria included (1) the scientific/technical quality of

the research and (2) the anticipated technical and/or economic benefits of

the proposed research, if successful, with special emphasis on the

likelihood that the project will attract further funding for product or

process development after the STTR support expires.

DOE’s evaluation process consisted of two steps. First, each of three

reviewers provided written comments addressing the five criteria but did

not assign a specific score to the criteria. Second, technical managers

reviewed the three evaluations and quantified the results.

Of the 21 winning proposals, 16 received perfect scores. The other five

received perfect scores on four of the five criteria. The managers

expressed unanimous agreement about the quality of the research in

particular. According to DOE’s definition of quality as used in evaluating

STTR proposals, a rating of “outstanding” for scientific/technical merit

indicated that the proposal was comparable to the top 10 percent of

projects in DOE. All 21 of the winners were rated as outstanding.

While DOE did not specifically evaluate proposals for commercial potential,

it did evaluate anticipated technical and/or economic benefits. Twenty of

the 21 proposals received outstanding ratings; only 1 was rated in the next

lower category as “significant.” Many of the evaluations on which the

managers based their conclusions contained favorable, but qualified,

statements about commercial potential. For example, one of the reviewers

wrote that, if the research is successful, the technology will be rapidly

commercialized.

One reason for the excellent results, as noted by DOE’s STTR Program

manager, was the unusually large number of proposals in relation to the

number of awards available. DOE received 487 proposals, meaning that less

than 5 percent of the applicants were successful. In the manager’s view,

the program was excessively competitive. As a result, he took two steps to

Page 25 GAO/RCED-96-19 Federal Research
Chapter 2 

Evaluations of STTR Proposals for Research

Quality and Commercial Potential Were

Favorable, but the Review Process Varied

Greatly

reduce the number of applicants. First, a notice was included in the

second solicitation to alert applicants to the situation. The notice pointed

out the ratio (1 award to 23 proposals) in the first year and concluded that

only those applications with the highest scientific/technical quality would

be competitive. Second, because broad topics tend to attract more

proposals, the topics were narrowed in the new solicitation to reduce the

number of proposals and improve the award/proposal ratio from 1 in 23 to

about 1 in 10.

Conclusions In general, technical evaluations of STTR proposals showed favorable views

of the quality of proposed research and commercial potential. For

research quality, the evaluations (1) awarded perfect scores to many

proposals, (2) rated proposals among the top 10 percent of research in

certain agencies, (3) described some proposals as “cutting edge,” and

(4) generally found the quality to be excellent. For commercial potential,

the evaluations arrived at similarly favorable conclusions, although in

some cases they were somewhat more cautious because of the newness of

the program or the risk associated with the proposals.

The evaluation processes varied greatly, ranging from several technical

reviewers to only one per proposal and from detailed critiques to

evaluation sheets that provided no analysis in support of the ratings.

Almost half (47) of DOD’s winning proposals received only one review; in

37 cases, the reviews were too brief to support the findings. Because of

their limited analysis and lack of a broader consensus, these instances

tended to reduce our confidence in the reliability of the results.

Page 26 GAO/RCED-96-19 Federal Research
Chapter 3 

Agencies Have Taken Steps to Avoid

Conflicts of Interest

The five federal agencies with STTR Programs have taken steps to avoid

potential problems relating to conflict of interest with federally funded

research and development centers. In addition, DOD and DOE, which

accounted for 29 of the 32 awards involving centers during the first year of

the program, have taken steps to prevent centers from using privileged

information in preparing STTR proposals.

Agencies Have Taken

Steps to Avoid

Potential Problems

Resulting From R&D

Center Involvement

The legislation establishing the STTR Program required agencies to develop

procedures to ensure that R&D centers are free from organizational

conflicts of interest. Such conflicts might arise, for example, if a center

formed a partnership with a company submitting an STTR proposal and

helped a federal agency judge the merits of its own and other proposals. A

second requirement directs each agency to develop procedures ensuring

that the centers use outside peer review, as appropriate. Under the STTR

Program, however, the agencies, not the centers, are responsible for

decisions regarding peer review; accordingly, we have focused on what

the agencies have done. In addition, the legislation required agencies to

ensure that the centers do not use privileged information gained through

work performed for an STTR agency or private access to STTR agency

personnel in the development of an STTR proposal.

Agencies Have Taken Steps

to Avoid Conflicts of

Interest

In general, the five agencies with STTR Programs have taken steps to

prevent conflicts of interest from occurring. DOD, DOE, and NIH have

specific policies intended to prevent such conflicts while NASA and NSF

have more general procedures to prevent such conflicts of interest from

arising.

DOD’s Director of Defense Research and Engineering issued a

memorandum in mid-1994 providing policy guidance on research and

development centers’ participation with industry in STTR and similar

programs. The memorandum concludes that, if a center requests

authorization to participate in such programs, the mission of the particular

center and the potential for conflict of interest must be primary

considerations in the decision process.

As a result of DOD’s policy, only two R&D centers are currently approved

research partners for its STTR awardees. The Air Force had to rescind some

awards because the proposed research partners (certain DOD laboratories)

were ineligible to participate. According to DOD’s STTR Program director,

Page 27 GAO/RCED-96-19 Federal Research
Chapter 3 

Agencies Have Taken Steps to Avoid

Conflicts of Interest

future proposals will be evaluated on a case-by-case basis to ensure that

conflicts of interest do not occur.

DOE has a policy addressing conflict of interest for the STTR Program.

According to the policy, DOE staff members should neither request nor

receive assistance from personnel in research institutions (that are eligible

to participate in the STTR Program) in the preparation of technical topics

for the STTR solicitation. In addition, no person affiliated with a research

institution may serve as a reviewer of a grant application that names that

research institution as a participant. Furthermore, no one affiliated with a

research institution may assist technical managers with the DOE review,

evaluation, and selection process for phase I grant applications in a

particular scientific area if that research institution is a participant on any

grant application submitted to that scientific area.

NIH has adopted a certification procedure to avoid conflicts of interest.

NIH’s solicitation for proposals requires the applicant to certify that it

“(1) is free from organizational conflicts of interests relative to the STTR

Program, (2) did not use privileged information gained through work

performed for an STTR agency...and (3) used outside peer review, as

appropriate, to evaluate the proposed project and its performance

therein.”

NASA relied on a peer review process for its proposals to avoid conflicts of

interest. As stated earlier, NASA’s review teams consisted of three

members—one from academia, one from the private sector, and one from

the government. According to NASA’s STTR Program manager, none of the

reviewers were connected with a federally funded R&D center. The

manager also mentioned that NASA’s reviewers certified that they have no

conflict of interest in their evaluations. The manager said that the only

federally funded R&D center directly associated with NASA on a regular

basis is the Jet Propulsion Laboratory. The manager said that, although the

laboratory was included as a research partner in several proposals by

small businesses, it was not involved in reviewing those proposals or

administering the program; in addition, no special information was

provided to this center.

NSF also relied on peer review procedures to avoid conflicts of interest. Its

“Proposal and Award Manual” provides guidance on the use of peer review

and states that each NSF program has one primary method for peer review

which represents the minimum evaluation received by proposals in that

program. Each of its STTR proposals received three reviews. NSF’s STTR

Page 28 GAO/RCED-96-19 Federal Research
Chapter 3 

Agencies Have Taken Steps to Avoid

Conflicts of Interest

Program manager told us that, in fiscal year 1994, none of NSF’s 11 awards

involved a small business in a partnership with a research and

development center.

In general, four of the five agencies with STTR programs used peer review

in evaluating STTR proposals. NIH, NASA, NSF, and DOE relied on specified

numbers of outside reviewers. NIH assigned four technical reviewers to

each proposal and provided additional input through its peer review

panels. NSF, NASA, and DOE used three. DOD was the only agency that relied

mainly on technical expertise within the Department rather than on

outside reviewers. This approach followed DOD’s usual policy in evaluating

proposals.

DOD and DOE Have Taken

Steps to Prevent Research

and Development Centers

From Using Privileged

Information

DOD and DOE, which accounted for 29 of the 32 awards involving R&D

centers as research partners, have policies to prevent centers from using

privileged information. DOD’s policy of carefully restricting participation by

its own laboratories helped in preventing the centers from using inside

knowledge in preparing proposals. As mentioned earlier, only two of DOD’s

centers were eligible to participate in the program. The exclusion of the

other centers, which would be the main sources of privileged information

regarding DOD’s research needs, avoided the potential problem raised by

this issue.

DOE’s policy prohibits agency staff members from requesting or receiving

assistance from personnel in research institutions (that are eligible to

participate in the STTR Program) in the preparation of technical topics for

the STTR solicitation. This policy is intended to prevent research

institutions from using their expertise to influence DOE’s choice of STTR

solicitation topics. Otherwise, research institutions could acquire a

significant advantage by designing topics to match their expertise and then

preparing a proposal in the same area.

Page 29 GAO/RCED-96-19 Federal Research
Chapter 4 

Differing Views on the Effect of and Need

for the STTR Program

Agency officials expressed differing views regarding the effect of STTR on

SBIR and other agency R&D. Furthermore, none of the officials indicated any

specifically negative effects such as a competition between the two

programs for quality proposals. However, conclusive information

concerning the effect, if any, of STTR on SBIR and other agency R&D was not

available because of the program’s newness and smallness. The

similarities between the STTR and SBIR Programs raise a broader issue

about the need for the STTR Program. The rationale for the program, which

points to certain weaknesses in SBIR and potential strengths in STTR,

suggests some additional questions that point more directly toward

evaluating the need for STTR.

Agency Officials

Noted No Negative

Effects of STTR on

SBIR or Other Federal

R&D

The legislation establishing the STTR Program required us to assess the

effects of STTR on the SBIR Program and other agency R&D. Our discussions

with agency officials provided no evidence to suggest that STTR was

competing for quality proposals with SBIR or reducing the quality of agency

R&D in general. Instead, agency officials expressed differing views

regarding the effect of STTR on SBIR and other agency R&D. A few officials

noted some potentially beneficial effects. Others said that sufficient data

were not yet available to determine the effect, if any, of STTR on SBIR or

other agency R&D.

SBA officials contended that STTR was too small and too new a program to

have any real effect on SBIR or on the broader range of agency research at

the present time. The officials pointed out that the program represented

only 0.05 percent of each agency’s external R&D budget during its first year

and that it was only 1 year old.

NIH’s STTR Program manager told us that one of the main potential effects

of the program is that universities can have a greater role than under SBIR.

However, the manager noted that, in an SBIR survey undertaken by NIH

several years ago, collaboration between small businesses and universities

was already evident in well over half of the SBIR projects.3

In contrast to the view that STTR’s effect was very limited, the Army’s STTR

Program manager said that STTR was influencing SBIR in a beneficial way. In

his opinion, STTR is becoming known through national conferences and

other channels. As a result, small businesses are realizing that they have

more credibility and chance of winning an award by collaborating with a

3

During our meeting with DOD officials to discuss our draft report, DOD’s STTR program director

noted that the STTR Program allows a research institution to play a greater role in collaborating with a

small business than under the SBIR Program.

Page 30 GAO/RCED-96-19 Federal Research
Chapter 4 

Differing Views on the Effect of and Need

for the STTR Program

university or other research institution. The manager believes that the STTR

Program has already led to more collaboration in SBIR. In general,

according to the program manager, STTR is a promising program that may

be as successful as the SBIR Program. The manager also said that STTR will

influence agencies whose research has traditionally involved the

university community to a lesser extent. In DOD, for example, the program

manager believes that STTR’s impact will be greater than in certain other

agencies (such as NIH) where the research program has been closely tied to

the universities.

DOE’s STTR Program manager said that he has tried to make the topics in

the STTR solicitation somewhat different from SBIR. Regarding the effect of

STTR on DOE’s research program, he said that STTR projects, if successful,

will help to meet the agency’s R&D needs by contributing to areas of

particular interest to the agency. However, the manager added that no

conclusions can be drawn in the first year of the program because the data

are too limited.

Three Questions Are

Relevant in

Determining the Need

for the STTR Program

The stated rationale for the STTR Program and the mandatory collaboration

suggest three additional questions that are relevant in determining the

need for the STTR Program: (1) Is the technology originating primarily in

the research institution as envisioned in the rationale for the program or is

it originating in the small business? (2) Is the mandatory collaboration

between the small business and the research institution effective in

transferring the technology to the market place? (3) Can the SBIR Program

accomplish the same objective without the collaboration required by the

STTR Program?

Is the Technology

Originating Primarily in the

Research Institution?

The technology may originate with the research institution, the small

business, or a combination of the two. In the STTR Program, the

assumption is that the research institution will be the primary originator of

the new concept. However, data to determine the extent to which research

institutions are providing the core technologies are not currently available.

Neither SBA nor the agencies have collected this information. DOE’s

SBIR/STTR Program manager said that he would like to know whether the

companies or the research institutions were drafting the proposals, but

such information is not available.

The relative roles of the research institution and the small business as the

source of the technology bear directly on the need for the STTR Program. If

Page 31 GAO/RCED-96-19 Federal Research
Chapter 4 

Differing Views on the Effect of and Need

for the STTR Program

a high percentage of the ideas are originating with small businesses rather

than with research institutions, this finding would raise questions about

the need for the program. On the other hand, if a high percentage of ideas

are originating with research institutions, this finding would suggest that

the program was achieving the first step in moving ideas from research

institutions to small businesses.

Is the Collaboration

Effective in Transferring

Ideas From the Research

Institution to the Market

Place?

If the program is effective in moving ideas from research institutions to

small businesses, then the next logical question concerns whether their

collaboration is effective in moving them to the market place. This

question can be approached from two directions: (1) Short-term views of

how well the collaboration is working in general and (2) long-term data on

actual commercialization.

Information on how well the collaboration is working can be obtained in

the near future. In particular, it would be useful to know how the small

businesses rated the contribution made by their research partners to the

research effort. Since most of the companies had not completed even the

first phase of their STTR award at the time of this report, such information

was not available, but it will be obtainable in the next year or two.

Information on actual commercial outcomes will require a greater amount

of time before it can be obtained. Generally, 5 to 9 years are needed to turn

an initial concept into a marketable product. Thus, it may be several years

before the commercial effectiveness of the program can be evaluated.

Can the SBIR Program

Accomplish the Same

Objective Without STTR’s

Mandatory Collaboration?

Because one important difference between the two programs is that STTR

makes a small business/research institution collaboration mandatory, the

question arises whether the SBIR Program could accomplish the objective

of transferring technology from research institutions to the private sector

without the mandatory collaboration. The rationale for the STTR Program

tends to assume that such collaborations were relatively rare in the SBIR

Program. However, as noted above, NIH’s Program manager told us that, in

an SBIR survey undertaken by NIH several years ago, collaboration between

small businesses and universities was already evident in well over half of

NIH’s SBIR projects. By contrast, the Army’s program manager believed that

STTR’s impact will be greater in the Army than in agencies such as NIH

because the Army SBIR Program has had a lesser degree of involvement

with universities and other research institutions in the past. Given the

apparent variation from one agency to another and the lack of current

Page 32 GAO/RCED-96-19 Federal Research
Chapter 4 

Differing Views on the Effect of and Need

for the STTR Program

data, no definite conclusion can be drawn at present concerning the need

for STTR in forging new collaborations.

Certain proposals may suggest a need for the STTR program. For example,

one NIH proposal involved a relationship between the developer of a new

type of microscope and a company with the experience and capability to

commercialize the instrument. The principal investigator with the

company told us that STTR led to a partnership between his company and

the research institution that would not have existed otherwise. According

to the investigator, without STTR, his company would have been reluctant

to devote a lot of time to a technology that it did not have rights to. In such

a situation, according to the investigator, the company would have

provided specific components of the microscope to the research

institution, let the institution develop the product, and then pursued a

license to market it. Instead, under STTR, the company developed an

agreement with the research institution which led to the current

partnership.

However, other cases may suggest the opposite. For example, the

president of a company that has participated in both the SBIR and STTR

Programs said that, in his experience, the SBIR Program gave companies a

greater advantage in dealing with research institutions as potential

partners. Under STTR, according to this official, the research institution has

“veto power,” but under SBIR, a company is in a better negotiating position

if it wants research to be done. In addition, according to the official, the

STTR Program will not be able to alter the research-oriented outlook of the

universities in a more commercial direction. In general, the official said

that almost all STTR research could be accomplished through SBIR.

Page 33 GAO/RCED-96-19 Federal Research
Appendix I 

Major Contributors to This Report

Resources,

Community, and

Economic

Development

Division, Washington,

D.C.

Bernice Steinhardt, Associate Director

Robin Nazzaro, Assistant Director

Dennis Carroll, Evaluator

(307732) Page 34 GAO/RCED-96-19 Federal Research
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100