United States General Accounting Office

GAO Report to Congressional Requesters

March 1997 MEASURING

PERFORMANCE

Strengths and

Limitations of

Research Indicators

GAO/RCED-97-91

GAO United States

General Accounting Office

Washington, D.C. 20548

B-275241 

March 21, 1997

The Honorable Constance A. Morella

Chairwoman

The Honorable Bart Gordon

Ranking Minority Member

Subcommittee on Technology

Committee on Science

House of Representatives

The Honorable John S. Tanner

House of Representatives

This report responds to your request for information on the indicators used to evaluate the

results of research and development (R&D). The report discusses the relative strengths and

limitations of the input and output indicators used by the federal and private sectors to measure

the results of R&D. The report also provides a historical perspective on research spending.

As agreed with your offices, we plan no further distribution of this report until 30 days from its

date of issue, unless you publicly announce its contents earlier. We will then send copies to

interested parties, and we will also make copies available to others upon request.

If you have any questions, I can be reached at (202) 512-3600. Major contributors to this report

are listed in appendix II.

Allen Li

Associate Director, Energy,

 Resources, and Science Issues
Executive Summary

Purpose American taxpayers invested more than $60 billion in federal funds in

military and civilian research and development (R&D) efforts in 1996. The

private sector invested more than $110 billion that same year. The

technological advancements resulting from these efforts are a critical

factor in improving the productivity of American workers and,

correspondingly, the nation’s standard of living. However, while the

contribution of R&D to technological advancement is widely recognized,

there is no widely accepted method of measuring the results of that

research.

To facilitate discussions of the adequacy of the funding and of the results

of the R&D, the Subcommittee on Technology, House Committee on

Science, asked GAO to evaluate the various indicators that are used to

measure the results of R&D. Specifically, this report discusses the strengths

and limitations of the input and output indicators used by the federal and

private sectors to measure the results of R&D. This report also provides a

historical perspective on spending for research.

Background The commitment to reduce the federal deficit is forcing the Congress to

reexamine the value of programs across the federal government. Although

scientific research is often considered to be intrinsically valuable to

society, there is pressure on all federal agencies, including science

agencies, to demonstrate that they are making effective use of the

taxpayers’ dollars. This greater emphasis on results is evident in the

passage of the Government Performance and Results Act of 1993 (GPRA).

The act fundamentally seeks to shift the focus of federal management and

accountability from a preoccupation with staffing, activity levels, and

tasks completed to a focus on results—that is, the real difference that

federal programs make in people’s lives.

The experts in research measurement have tried for years to develop

indicators that would provide a measure of the results of R&D. However,

the very nature of the innovative process makes measuring the

performance of science-related projects difficult. For example, a wide

range of factors determine if and when a particular R&D project will result

in commercial or other benefits. It can also take many years for a research

project to achieve results.

Results in Brief The amount of money spent on research and development, the primary

indicator of the investment in research, is useful as a measure of how

Page 2 GAO/RCED-97-91 Measuring Performance
Executive Summary

much research is being performed. Having been refined over many years,

these data are generally available for the research efforts in both the

public and private sectors. However, the level of spending is not a reliable

indicator of the level of results achieved by research.

Unlike the situation with the input measures of research and development,

there is no primary indicator of the outputs. Output indicators include

quantitative analyses of return on investment, patents granted, and other

outputs as well as qualitative assessments based on peer review. The

companies that GAO spoke with collect data on various output indicators

but, in general, make limited use of them in their investment decisions.

Instead, the companies emphasized that research and development

contribute directly to their “bottom line.” Because companies are

profit-oriented, many of the indicators tracked by the private sector

cannot be directly applied to the federal government. Experiences from

pilot efforts made under the Government Performance and Results Act

have reinforced the finding that output measures are highly specific to the

management and mission of each federal agency and that no single

indicator exists to measure the results of research.

Principal Findings

Funding Indicates

Research Activity but Does

Not Measure the Results of

Research

Funding has been used as the primary input indicator for decades.

Whether a policymaker is interested in basic research, applied research, or

development, the amount of money spent in that area is taken as an

indication of how much research is being performed. The major

advantages of using expenditure data as an indicator are that they are

easily understandable, readily available, and have been, in general,

consistently gathered over time. In addition, spending on different projects

in different research areas can be measured according to the same unit,

dollars, making comparisons between projects straightforward.

The amount of funding, however, does not provide a good indication of the

results of research. Companies told GAO that they are focusing more of

their spending on short-term R&D projects than on long-term projects.

However, the impacts of that change in focus are unclear. The reduced

funding levels may not reflect the fact that the R&D efforts are being

performed with greater efficiency. For example, one way in which the

federal government and the private sector have tried to use R&D resources

Page 3 GAO/RCED-97-91 Measuring Performance
Executive Summary

more efficiently and effectively is through consortia with universities or

other companies. By combining their research activities, the companies

attempt to avoid expensive duplication and learn from each other.

R&D Output Indicators

Can Provide Limited

Information About the

Results of R&D

Because of the difficulties in identifying the impacts of research,

decisionmakers have developed quantitative and qualitative indicators as

proxies to assess the results of R&D activity. The strengths and limitations

are evident in both types of indicators. The current quantitative indicators

focus mainly on return on investment, patenting rates, and

bibliometrics—the study of published data. While implying a degree of

precision, these indicators were not originally intended to measure the

long-term results of R&D. Qualitative assessment provides detailed

information, but it relies on the judgments of experts and may be

expensive.

Because of these difficulties, the companies interviewed by GAO stressed

marketplace results rather than R&D output indicators. While varying in the

types of indicators they collect, they emphasized the difficulties in

measuring R&D’s specific contribution to a company’s overall performance.

For example, one company stated that because so many people have been

involved in a product’s evolution, it is difficult to separate the contribution

of the research unit from that of other units. All of the companies

interviewed have increased their expectation that R&D contribute directly

to their profitability, but instead of increasing their efforts at measuring

R&D results, they have shifted the responsibility for R&D decisions to the

business units. However, many of the R&D output measures tracked by the

private sector do not apply directly to the federal government. In

particular, while facing the same increasing cost pressures as the private

sector, the federal government cannot rely on the profit motive to guide its

decisions.

The GPRA requires the executive agencies to develop their annual R&D plans

with suitable performance measures. The Research Roundtable, a group of

federal researchers and managers representing a cross-section of R&D

departments and agencies, warned about the difficulties of quantifying the

results of R&D and the potential for incorrect application with subsequent

harm to scientific endeavors. The Army Research Laboratory, which was

designated a pilot project for performance measurement under the act, has

developed a multifaceted approach using quantitative indicators, peer

review, and customer feedback to evaluate the results of R&D.

Page 4 GAO/RCED-97-91 Measuring Performance
Executive Summary

Recommendations This report contains no recommendations.

Agency Comments Because this report focuses broadly on the R&D of both the federal and

private sectors, and not on the effort of individual agencies, GAO did not

submit a draft of this report to federal agencies for their review and

comment.

Page 5 GAO/RCED-97-91 Measuring Performance
Contents

Executive Summary 2

Chapter 1 

Introduction

8

The Process of Innovation and the Use of R&D Indicators 8

Past Efforts to Evaluate R&D Impacts 10

The Government Performance and Results Act 11

Objectives, Scope, and Methodology 12

Chapter 2 

R&D Spending Data

Provide Some

Information About

Innovative Activity

but Not About R&D

Results

14

R&D Spending Gives an Indication of Innovative Effort 14

R&D Spending Is Not a Good Indicator of R&D Results 15

Spending Patterns Show a Greater Emphasis on Short-Term

Research

17

Chapter 3 

R&D Output

Indicators Can

Provide Limited

Information About

R&D Results

18

Current Indicators Have Strengths and Limitations 18

The Private Sector’s Emphasis on Marketplace Results Limits

Lessons for the Federal Government

22

Federal Science Agencies Are Still Exploring Ways to Measure

the Impacts of R&D

26

Appendixes Appendix I: Historical Perspective on Research Spending 28

Appendix II: Major Contributors to This Report 34

Figure Figure I.1: U.S. R&D Spending, 1953-96 30

Page 6 GAO/RCED-97-91 Measuring Performance
Contents

Abbreviations

ARPA Advanced Research Projects Agency

DOD Department of Defense

GAO General Accounting Office

GPRA Government Performance and Results Act

IRI Industrial Research Institute

NIH National Institutes of Health

NSF National Science Foundation

NASA National Aeronautics and Space Administration

OECD Office of Economic Cooperation and Development

OTA Office of Technology Assessment

R&D research and development

Page 7 GAO/RCED-97-91 Measuring Performance
Chapter 1 

Introduction

Over $180 billion was spent on research and development (R&D) in the

United States in 1996.1

 Most of that amount was spent by industry and the

federal government—$113 billion and $62 billion, respectively; the balance

was spent by universities and other nonprofit organizations. The leading

experts in the study of research indicators agree that R&D has a significant,

positive effect on economic growth and the overall standard of living.

However, because of the complexity of linking the results of R&D to its

economic impacts, there is no widely accepted method of measuring the

results of R&D spending in either industry or the federal government. The

commitment to reduce the federal budget deficit is forcing the Congress

and the executive branch to undertake a basic reexamination of the value

of programs across the federal government. It is also placing pressure on

all federal agencies, including the civilian science agencies, to clearly

demonstrate that they are making effective use of the taxpayers’ dollars.

The Government Performance and Results Act of 1993 (GPRA) provides a

legislative vehicle for the agencies to use as they seek to demonstrate and

improve their effectiveness. Equally important, if successfully

implemented, GPRA should help the Congress make the difficult funding,

policy, and program decisions that the current budget environment

demands.

The Process of

Innovation and the

Use of R&D Indicators

Researchers have developed their own terminology for describing the

process of transforming R&D into economic results. In its simplest form,

the theory underlying both public and private decision-making has been

that innovative activity positively affects economic performance.

Innovation can be thought of as the development and application of a new

product, process, or service. It can include the use of an existing product

in a new application or the development of a new device for an existing

application. Innovation encompasses many activities, including scientific,

technical, and market research; product, process, or service development;

and manufacturing and marketing to the extent that they support the

dissemination and application of the invention. Innovation is a

combination of invention and commercialization. Invention describes the

initial conception of a new product, process, or service, but not the act of

putting it to use. Commercialization refers to the attempt to profit from

innovation through the sale or use of new products, processes, and

services.

1

These figures are based on preliminary 1996 statistics reported by R&D performers to the National

Science Foundation.

Page 8 GAO/RCED-97-91 Measuring Performance
Chapter 1 

Introduction

This description as well as the traditional views of innovation has been

strongly influenced by the linear model of innovation, which says that

innovation proceeds sequentially through the stages of basic research,

applied research, development, manufacturing, and marketing.2

 This

model assumes that basic research serves as the source of innovation and

that new scientific knowledge initiates a chain of events culminating in the

development and sale of a new product, process, or service. In this model,

basic research is the most uncertain part of the process; once basic

research is conducted, innovation and commercialization can proceed.

The model suggests that the firms with the best technology will likely be

the first to market and win the lion’s share of profits.

The simplicity of this model makes it particularly useful in policy

discussions. Other models may be more accurate, but they provide a more

complex explanation of the relationship between science and the

commercialization of new technology. These models, such as the

“chain-linked model,” include feedback loops that allow for interaction

among the different stages of the linear model. These models also reflect

the fact that the ideas for new inventions or changes to existing products

often arise from the recognition of new market opportunities, advances in

manufacturing capabilities, or advances in technology independent of

progress in the underlying science.

Real-world examples show that technological breakthroughs can precede

as well as follow basic research. In many cases, science is not the source

of innovation. The Wright brothers, for example, developed the first

airplane without an understanding of aerodynamic theory, and Chester

Carlson developed the first xerographic copier without a thorough

understanding of photoconductive materials. These inventions have

resulted in considerable research into aerodynamic theory and materials

science, respectively, as scientists and engineers attempted to improve the

original invention.

In studying complex processes or concepts such as innovation, it is not

always possible to measure them directly. As a result, researchers turn to

the use of “indicators.” Indicators point to or illustrate the process or

2

The National Science Foundation uses the following definitions in its resource surveys: Basic research

has as its objective to gain a more comprehensive knowledge or understanding of the subject under

study, without specific applications in mind. Applied research is aimed at gaining knowledge or

understanding to determine the means by which a specific, recognized need may be met. Development

is the systematic use of the knowledge or understanding gained from research directed toward the

production of useful materials, devices, systems, or methods. The Foundation recognizes the

limitations of this classification scheme but continues to use these categories to maintain historical

consistency, among other reasons.

Page 9 GAO/RCED-97-91 Measuring Performance
Chapter 1 

Introduction

concept in question but do not directly measure it. For example, in order

to determine an object’s temperature, one can use a thermometer to

measure it directly. However, when trying to measure something as

complex as the health of a country’s economy, one relies on different

indicators, such as unemployment rates, stock market averages, or trade

balances that do not provide a direct measurement of economic health but

do give an indication of its status. In the case of innovation, various “input”

and “output” indicators are used that are based on the linear model of

innovation. The following chapters of this report are broken down

according to these two sets of indicators. Chapter 2 is concerned with R&D

spending, or expenditure data, which is the most widely used input

indicator of innovation, and chapter 3 focuses on some widely used output

indicators.

Past Efforts to

Evaluate R&D

Impacts

For almost two decades, numerous reports have documented the

difficulties of quantifying the results of R&D. As noted above, the

identification of the economic and social effects of research involves

complex issues of measurement, analysis, and interpretation. The

following efforts may help to give perspective to the present concerns

about measuring R&D results.

• In 1979, we reported on a wide range of factors that make the

measurement of R&D results difficult.3

 The report noted that R&D

expenditures are undertaken for a variety of reasons. Some attempt to

develop new knowledge; others are directed at meeting needs, such as

national defense, for which there is no commercial market; and still others

are directed at lowering the cost of products. Furthermore, some projects

produce revolutions; others produce nothing. Most important, R&D is only

one input into a complex process. Thus, we concluded that there is no

possibility of eliminating the role that judgment plays in the allocation of

federal R&D resources.

• In 1986, the Office of Technology Assessment (OTA) issued a detailed

report that questioned the utility of the effort to quantify R&D returns.4

According to OTA, the fundamental stumbling block to placing an economic

value on federal R&D is that improving productivity or producing an

economic return is not the primary justification for most federal R&D

programs. The report added that the attempts to measure the economic

return to federal R&D are flawed because many of the research outputs,

3

Assessing the “Output” of Federal Commercially Directed R&D (GAO/PAD-79-69, Aug. 1979).

4

Research Funding as an Investment: Can We Measure the Returns? Office of Technology Assessment,

72 pp. (Apr. 1986).

Page 10 GAO/RCED-97-91 Measuring Performance
Chapter 1 

Introduction

such as national defense, cannot be assigned an economic value. The

report also noted that in industry, where one might expect quantitative

techniques to prevail because of the existence of a well-defined economic

objective, OTA found a reliance on subjective judgment and good

communications between R&D, management, and marketing staffs.

• In 1996, responding to concerns about the implementation of GPRA, the

National Science and Technology Council issued a report that stressed the

limited role of quantification in measuring the results of R&D.

5

 It stated that

the insufficiency of quantitative measures per se is one reason why other

sources of evidence, such as merit review of past performance, narrative

discussion, and descriptions of outstanding accomplishments and more

typical levels of achievement, should be included in annual performance

reports. The report concluded that the cornerstone of world-class science

will continue to be merit review with peer evaluation, while the science

community works to develop the measurement tools and other methods

needed to assess the contributions of fundamental science.

• At the international level, the Organization of Economic Cooperation and

Development (OECD) is also grappling with these questions. Beginning with

the first edition of the Standard Practice for Surveys of R&D (the “Frascati

Manual”) in the 1960s, OECD has been developing international frameworks

for the measurement of R&D inputs. In connection with its more recent

effort to develop new output indicators (the “Oslo Manual”), OECD stated

that the new indicators and the underlying statistics usually take two or

more decades to reach general acceptance and regular collection and

publication. The participants at a 1996 OECD conference extensively

discussed the organization’s efforts to improve the quality of innovation

indicators. At this stage, these indicators appear to be most useful in

helping researchers study and describe the process of innovation.

The Government

Performance and

Results Act

In response to questions about the value and effectiveness of federal

programs, GPRA seeks to shift federal agencies’ focus away from such

traditional concerns as staffing, activity levels, and tasks completed

toward a focus on program outcomes—that is, the real difference the

federal program makes in people’s lives. In the GPRA context, an “outcome

measure” assesses the results of a program activity compared to its

intended purpose. An “output measure,” according to GPRA, tabulates,

calculates, or records the level of activity or effort and can be expressed in

a quantitative or qualitative manner. The output indicators discussed in

our report could be considered either outcome measures or output

measures, depending on the context. For example, return-on-investment

5

Assessing Fundamental Science, National Science and Technology Council (July 1996).

Page 11 GAO/RCED-97-91 Measuring Performance
Chapter 1 

Introduction

calculations could be outcome measures in a business context since

gaining the maximum return on investment is the intended purpose of a

business. Patenting rates would be output measures in some businesses

because patents could serve as one measure of the level of activity of a

research unit.

The act recognizes how difficult it is to state the goals and measure the

results of some programs. While the law encourages the use of objective

measures of performance, it authorizes agencies—with the approval of the

Office of Management and Budget—to use alternative, subjective

measures of performance. Also, instead of having GPRA take effect

immediately after its passage in 1993, the Congress allowed for a period of

time for the government to learn how to implement the act. As part of the

learning process, the act called for the use of pilot projects in performance

measurement. Among the approximately 70 agencies or parts of agencies

that participated in pilot projects, one addresses scientific research. Upon

the full implementation of the act, the executive branch agencies are

required to devise plans that are outcome-oriented. The act calls for the

agencies to develop three documents: a strategic plan, an annual

performance plan, and an annual performance report.

Objectives, Scope,

and Methodology

In response to the Subcommittee on Technology’s request, our objective

was to review various indicators that are used to measure the results of

R&D. Specifically, this report discusses the relative strengths and

limitations of the input and output indicators used by the federal and

private sectors to measure the results of R&D as well as the claim that

industry focuses on short-term profitability rather than long-term R&D

needs. This report also provides a historical perspective on research

spending. (See app. I.)

The impacts of innovation are widely studied in the public and private

sectors as well as in academia. Our work relied on a limited number of

experts in each of these sectors to provide us with the prevailing

understanding of and latest developments in the area. As a result, our

review does not provide an exhaustive examination of R&D measures nor

does it answer the question of how R&D should be measured. It does,

however, as agreed with your offices, consist of information on the

strengths and limitations in the use of these indicators as well as anecdotal

information based on interviews with leading R&D companies. We also

reviewed the relevant literature.

Page 12 GAO/RCED-97-91 Measuring Performance
Chapter 1 

Introduction

We interviewed a number of experts, including a former IBM Vice President

for Research and National Science Foundation (NSF) Director, a former

Executive Director of the Manufacturing Forum, a former Associate

Director of the White House Office of Science and Technology Policy, the

Director of the Special Projects Office for the Army Research Laboratory,

and the former Chief Financial Officer for Apple Computer. We also

conducted six teleconferences with company officials who were typically

at the director level. We interviewed representatives of General Electric,

Lucent Technologies (formerly Bell Labs), Dow Chemical,

Eastman-Kodak, IBM, and Microsoft Corporation. These companies spent

from $5 billion to $800 million on R&D in 1995. We chose companies having

among the largest total R&D budgets, in terms of dollars spent, because we

believed that their experiences would be the most relevant to the needs of

the federal government.

Our work also covered the available data on R&D spending and output

indicators provided by the leading data-gathering organizations in this

field—the OECD, NSF, and the Industrial Research Institute (IRI). Our

collaboration with the OECD included participation in an OECD conference

on science and technology indicators. Throughout the course of this work,

we also interviewed the NSF staff responsible for publishing the Science

and Engineering Indicators.

We performed our audit work from June 1996 through February 1997 in

accordance with generally accepted government auditing standards.

Page 13 GAO/RCED-97-91 Measuring Performance
Chapter 2 

R&D Spending Data Provide Some

Information About Innovative Activity but

Not About R&D Results

R&D spending data provide an indication of how much research is being

performed but do not provide a measure of the impacts of that spending.

Spending data have been used for budgeting purposes; however, they have

also been used as an indicator of the level of innovative activity within a

nation or company. The use of R&D spending data as an indicator has a

number of advantages. For example, it reduces the innovation process to a

single figure for the purposes of discussion. In addition, the data-gathering

methods have been refined over many years and are generally reliable over

time. However, the level of spending is not a reliable indicator of the level

of research results. For example, companies told us that they are focusing

more of their spending on short-term R&D projects than on long-term

projects, but the impacts of that change in emphasis are unclear. The use

of spending data is more appropriate for discussions of R&D spending

priorities than of the effectiveness and impacts of R&D spending levels.

R&D Spending Gives

an Indication of

Innovative Effort

Traditionally, R&D expenditures have been taken to indicate the “amount”

of innovative activity that is occurring within a country or a firm. One of

the advantages of using expenditure data in this way is that it simplifies

the discussion of the complex process of innovation to a single unit of

measurement. Another advantage is that the use of dollars as the unit of

measurement enables direct comparisons to be made. In addition, the

gathering of spending data has been refined over many years, increasing

the data’s reliability and relevance to policy-making.

This straightforward rationale—the more R&D spending, the more

innovative activity—is the primary advantage to using expenditure data in

policy discussions. Its simplicity and close ties to the linear model of

innovation allow it to be readily understood by those with little specialized

knowledge, making it appealing in policy discussions. These same

simplifying characteristics may have led to its use in other areas. In some

contexts, countries and companies are categorized according to their

technological sophistication on the basis of their R&D spending levels; little

attention is given to other factors.

Another advantage arises from the common use of “dollars” in the R&D

spending data. This usage enables the spending in different research areas

to be compared according to the same units. These straightforward

comparisons are useful in demonstrating the priorities of the nation at

large. For example, recent U.S. R&D expenditure data show that a

reduction in defense-related R&D was somewhat counterbalanced by an

increase in federal support for civilian R&D programs, including those

Page 14 GAO/RCED-97-91 Measuring Performance
Chapter 2 

R&D Spending Data Provide Some

Information About Innovative Activity but

Not About R&D Results

aimed at improving the diagnosis and treatment of disease, cleaning up the

environment, and enhancing technological competitiveness and economic

prosperity. In addition, converting different foreign currencies to dollars

allows for international comparisons of research. After allowing for the

variation of inflation over time, funding data depict historical patterns of

real expenditures.

The accuracy and policy relevance of spending data are also important

advantages to their use as indicators. The typical source of these data is

the widely cited Science and Engineering Indicators document published

periodically by the National Science Board. The Science Resources

Studies Division of NSF has been gathering data to use in Indicators for

many years, and many improvements have been made in the accuracy of

the data.6

 For example, NSF has sponsored the Survey of Industrial R&D

since 1953 as a source of data for Indicators. Recent improvements to this

survey include selecting samples annually (rather than less frequently) and

increasing the sample size from approximately 14,000 to nearly 24,000

firms. NSF took these steps to account more accurately for the

establishment of R&D-performing entities in the survey universe and to

survey more fully and accurately the R&D performed by nonmanufacturing

firms. In addition, NSF is constantly searching for ways to make Indicators

more responsive to policymakers’ needs. NSF began to make adjustments

in its surveys when it recognized that there was a need to supply more

information on the service sector to policymakers because the survey

historically had focused on the manufacturing industries in which R&D

performance had been heavily concentrated in the past.

R&D Spending Is Not

a Good Indicator of

R&D Results

The use of spending data is limited in its relevance to the impacts of R&D.

There is some correlation between the level of R&D spending and

innovative success. For example, if fewer research projects are performed,

then companies and countries forgo the potential benefits of the research.

However, spending alone does not guarantee innovative success because

many additional factors figure into the innovation process and have

important effects on the resulting outputs. The reality of the process of

innovation is much more complex than expenditure data alone can reveal.

The usefulness of R&D spending data as an indicator of results is limited

because the data measure the amount of resources a firm or a nation

6

Some problems with the data may still exist, however. For example, NSF itself notes that the data on

trends in the private sector’s basic research contain anomalous spending spikes in 1986 and 1991. The

inconsistencies in the data appear to derive from changes made in the NSF survey in those years. For

example, in 1991 the survey was expanded to cover a broader array of nonmanufacturing firms.

Page 15 GAO/RCED-97-91 Measuring Performance
Chapter 2 

R&D Spending Data Provide Some

Information About Innovative Activity but

Not About R&D Results

dedicates to innovation, but not its ability to convert that effort into

successful products, processes, and services. Accounts in the press have

noted that companies are proud of high R&D spending levels because

well-focused R&D generally pays off in long-term revenue growth and

profits. However, more is not always better, as shown by the companies

that ranked among the leaders for R&D spending and disappeared shortly

thereafter. For example, two companies called Xonics and Transitron

Electronic ranked as the top companies in R&D spending per employee for

1984 and 1986, respectively; in the same years that they achieved their top

ranking, one company was forced to file for Chapter 11 protection and the

other company dissolved.

There is great uncertainty in research and development investments. The

processes leading to commercially viable and socially useful technologies

are complex and involve substantial non-R&D factors. One company official

told us that it was impossible to determine which business function was

most important to the success of a new product—research, marketing, or

sales. While there is some correlation between the level of R&D spending

and innovative success, spending alone does not guarantee success. One

official from a high-technology company told us that his company is

constantly evaluating research projects with respect to their targeted

markets and estimating the expected return on investment. However, in

one case the company lost approximately $100 million when the market

would not support a newly developed product at the price the company

expected.

The usefulness of R&D spending indicators is also limited because the way

in which innovative activities are structured and managed can be as

significant as the amounts of resources devoted to them in determining

their outcomes and effects on performance. Those nations or firms with

extremely efficient innovation systems can outperform those that use

greater R&D resources inefficiently. As a former director of NSF pointed out

to us, streamlining research efforts could reduce both bureaucratic and

financial overhead costs and make up for spending reductions.

In addition, falling spending levels may hide the greater efficiencies that

might be realized by leveraging research. One company official told us that

although his company was cutting back on long-term research, it was

relying more on its relationships with federal and university laboratories

for similar work. Collaboration between firms—through joint ventures,

consortia, and contracting—has recently been on the rise as firms attempt

to efficiently distribute risk, pool their resources, and tap into external

Page 16 GAO/RCED-97-91 Measuring Performance
Chapter 2 

R&D Spending Data Provide Some

Information About Innovative Activity but

Not About R&D Results

expertise. The magnitude of alliance formation is difficult to gauge, as are

the implications for the innovation and commercialization of new

technologies in the United States. However, one academic told us that if

the rhetoric about the efficiencies coming from cooperative R&D is true,

then no one should be too disappointed with recent drops in R&D spending.

Spending Patterns

Show a Greater

Emphasis on

Short-Term Research

Federal and private R&D spending patterns reflect the changing conditions

in the country and the world at large. The share of basic research

supported by private sources peaked in the early post-World War II period,

driven by thriving domestic and international markets. However, recent

surveys and anecdotal evidence suggest that increased international

competitive pressures have forced companies to emphasize short-term

development over long-term research. Company officials echoed this shift

and suggested that the focus on short-term research was part of a push

toward more relevancy in their R&D departments. They pointed out that

their business units are determining their research needs to a greater

extent today than they have in the past.

Some companies told us that this strategy could be viewed as

short-sighted. Others defended this strategy. For example, one company

official told us that his company emphasizes short-term research because

“competition is short-term.” He stated that in today’s competitive

environment, once a market is lost, it is gone forever.

Page 17 GAO/RCED-97-91 Measuring Performance
Chapter 3 

R&D Output Indicators Can Provide Limited

Information About R&D Results

Quantitative and qualitative indicators have been developed to evaluate

R&D activities and their results, but both types of indicators have strengths

and limitations. Our interviews with a number of companies showed that

the private sector stresses marketplace results rather than relying on

output indicators. Because of the companies’ profit orientation, many of

the indicators tracked by the private sector cannot be directly applied to

the federal government. In response to the GPRA, the federal science

agencies are exploring new ways to quantify the impacts of research.

However, it is too early to tell whether new performance measures can be

developed and whether they will meet the needs of the Congress.

Current Indicators

Have Strengths and

Limitations

Because of the difficulties in identifying the impacts of research,

decisionmakers in the public and private sectors typically have chosen to

measure outcomes using a variety of proxies. These quantitative and

qualitative indicators have strengths and limitations. To illustrate these

strengths and limitations, we looked at three of the most frequently cited

quantitative indicators: return on investment, patents issued, and

bibliometrics. While these indicators imply a degree of precision, they

were generally not designed to measure the long-term results of R&D

programs, nor are they easily adaptable to such a purpose. Qualitative

assessment provides detailed, descriptive information, but it depends on

subjective judgments and may be costly.

Return on Investment This indicator aims at measuring the sales and profits resulting from

investments in R&D; as such, it addresses one of the fundamental concerns

about the value of such investments. However, a variety of factors, such as

the complexity of the innovation process and its inherently long time

frames, pose serious obstacles to the calculation of these returns. The

literature dealing with return on investment is replete with words of

caution against quantifying R&D results. NSF’s Science and Engineering

Indicators 1996 pointed out that not only is much of this information

unobtainable or ambiguous, but many of the gains from research are

simply monetarily intangible.

Experts on the R&D process have stated frequently that the long time

periods and multiple inputs involved make the task of calculating the

return on basic research especially difficult. Productivity growth may lag

20 years behind the first appearance of research in the scientific

community, and the lag for interindustry effects may be 30 years. A more

serious impediment, however, is the fact that outcomes are often not

Page 18 GAO/RCED-97-91 Measuring Performance
Chapter 3 

R&D Output Indicators Can Provide Limited

Information About R&D Results

directly traceable to specific inputs or may result from a combination of

such inputs. The National Aeronautics and Space Administration (NASA)

and the National Bureau of Standards (now the National Institute for

Standards and Technology) attempted to measure the economic impacts

and benefits of certain of their technologies in the 1970s and early 1980s.

The studies at the Bureau were discontinued, according to staff, because

of serious theoretical and methodological problems. As with the Bureau’s

studies, NASA’s studies evoked serious criticisms and were likewise

discontinued.

Despite the difficulties in calculating return on investment, the leading

researchers in this field agree that R&D offers high private and social

returns in terms of high productivity. One recent survey of 63 studies

found that R&D activity achieves, on average, a 20- to 30-percent annual

return on private (industrial) investments.7

Patents Patents show certain strengths as useful indicators in measuring technical

change and inventive input and output over time. For example, they can

reveal a variety of trends involving ownership and levels of activity in

technical areas. According to NSF, the data concerning ownership show

that the federal share of patents averaged 3.5 percent of the total number

of U.S. patents during 1963 through 1980 but declined thereafter.

In addition, the data concerning a country’s distribution of patents by

technical area have proved to be a reliable indicator of a nation’s

technological strengths as well as its direction in product development.

For example, the three most emphasized U.S. patent categories for

inventors show specific contrasts between U.S. and foreign patents. U.S.

inventors obtained most of their patents in wells, mineral oils, and certain

areas of surgery. Japanese inventors focused their efforts on certain areas

of communications, organic compounds, and refrigeration. Patent activity

can be used to pinpoint potentially important changes. In 1980 through

1987, U.S. inventors led all other foreign inventors in radio and television

patents, but in 1987 the United States lost its front position to Japanese

inventors in this area.

Despite their usefulness as indicators of broad national and international

trends in various industries and areas of research, patents also have

several intrinsic drawbacks. Inconsistency across industries in the number

of patents granted is a major limitation that results from the wide

7

Nadiri, M.I. Innovations and Technological Spillovers (1993).

Page 19 GAO/RCED-97-91 Measuring Performance
Chapter 3 

R&D Output Indicators Can Provide Limited

Information About R&D Results

variations in the propensity to patent inventions. Consequently, according

to NSF, it is not advisable to compare patenting rates between different

technologies or industries. Inconsistency in quality is a second drawback.

The aggregated patent statistics do not distinguish between those that led

to major innovations and those that led to minor improvements.

Incompleteness is a further limitation. Many inventions are not patented at

all, and trade secrecy may become a preferred alternative to patenting.

Another limitation is that patents do not lend themselves to the evaluation

of the most significant results achieved by an R&D program. They can

provide intermediate measures of progress, but they are not usually the

purpose for which the research was undertaken.

In addition, the use of patents as a measure of federal R&D effectiveness

may be hampered by their limited relevance. The 1996 report entitled

Assessing Fundamental Science by the National Science and Technology

Council noted that any use of patent counts should be undertaken only

with a full awareness of their limitations. A recent academic study of

patents and the evaluation of R&D also commented on the extraordinarily

limited applicability of patent evaluation to government-performed R&D.

Both studies pointed to the relatively low level of federal patenting

activity. The academic study noted that most government laboratories are

granted only one or two patents per year, and only a few of them patent

extensively. It concluded that for government laboratories, one may

question the overall wisdom of evaluating public R&D with private

techniques.8

Bibliometrics A third area of effort in developing quantitative measurements involves

bibliometrics, or the study of published data. Bibliometrics counts

citations in an attempt to address questions of productivity, influence, and

the transfer of knowledge. Its most appropriate use is in quantifying the

work of researchers whose results are published. Thus, it may be

especially applicable in areas such as basic research where the results are

more often published than protected by firms. However, its usefulness as a

measure of research results remains somewhat controversial.

We believe that the use of bibliometrics as a source of information on the

quality of the publications or the citations being counted needs to be

8

In its October 11, 1996 issue, Science magazine noted the French government’s announcement that

patent records will form a part of the evaluation of publicly funded researchers. The new proposal,

according to the magazine, is controversial because it might upset the proper balance between basic

and applied research by favoring those who work in fields that are immediately applicable at the

expense of people doing basic research.

Page 20 GAO/RCED-97-91 Measuring Performance
Chapter 3 

R&D Output Indicators Can Provide Limited

Information About R&D Results

approached with caution. Although bibliometric indicators can be

weighted by publication or other quality measure, the frequency of

citation, for example, provides no indication of the level of research

innovation. Another limitation is the problems that arise in

interdisciplinary comparisons of results. Some critics have gone so far as

to say that bibliometric findings should not be used in science policy work

until the problems with the analysis of citations are addressed.

In addition, the relevance of bibliometric analysis to decision-making by

the federal government appears very limited. One expert noted that a

recent comprehensive review of bibliometrics shows the sparsity of

bibliometric studies for evaluations of the impact of research reported by

the federal government. Another pointed out that few federal agencies use

bibliometric analysis as an evaluative tool. One of the few is the National

Institutes of Health (NIH), which uses this method to evaluate the

effectiveness of its different institutes, the comparative productivity of

NIH-sponsored research and similar international programs, and the

responsiveness of NIH’s research programs to their congressional mandate.

Peer Review Recognizing the limitations of quantitative indicators, the National Science

and Technology Council concluded that it makes sense to track relevant

measures but that they cannot supplant the essential element of expert

judgment. Peer review, the most important form of qualitative assessment,

uses technical experts to judge R&D results on the basis of the expert’s

evaluation of the quality of research. However, peer review has serious

shortcomings; it generally depends on criteria that are inherently difficult

to measure and on subjective judgment that is vulnerable to bias.

Peer review has been used extensively in the selection of proposed

research projects. To a lesser extent, it has also been used to evaluate R&D

impacts. Peer review has come to be viewed by some observers as the best

assurance that quality criteria will prevail over social, economic, and

political considerations, while others view it as an element of elitism in

science that tends to discount such concerns as economic significance.

Its major strength is its ability to bring together the leading experts in the

area of concern. Most peer review procedures require a minimum of three

reviewers; if the review involves a more ambitious scope of coverage

(such as an entire agency), dozens of reviewers may be involved. The

process of selecting the peer reviewers varies. One of the chief

responsibilities of the professional staff in science agencies such as NSF

Page 21 GAO/RCED-97-91 Measuring Performance
Chapter 3 

R&D Output Indicators Can Provide Limited

Information About R&D Results

and NIH is to stay in touch with a specialized community of scientists who

are qualified to judge the agency’s activities. However, others rely on

in-house managers who are not active researchers.

The major limitations of peer review are twofold. First, the perception of

quality depends largely on the expertise of the panel members. It is based

on the judgment of experts about a proposal or a set of research-related

results. Generally, a final judgment will depend on the collective weight of

the different opinions. Frequently, a numerical rating scale—such as 1 for

poor through 5 for excellent—is used. Despite the appearance of precision

conferred by a specific number, the numbers represent the best of

sometimes widely differing judgments. Consequently, although peer

review has been a mainstay in judging science for over three centuries,

questions remain about ways of improving it. For instance, according to

one academic, to improve “validity and reliability,” research needs to be

done on the optimal numbers of reviewers and on the advisability of

training people to perform peer reviews.

Second, peer review evaluation of completed or ongoing R&D projects is a

more thorough and expensive process than peer review for the purpose of

selecting proposals for funding. According to one study, the cost of a

2-day, 10-person, face-to-face NSF merit review panel is in the

neighborhood of $20,000. Another study concluded that if this method

were applied annually to all federal research programs, the cost in

reviewer time alone would be enormous. For example, the Army Research

Laboratory has contracted for a peer review of its activities; the contract

calls for a 3-year review directed by the National Research Council at

approximately $650,000 per year.

The Private Sector’s

Emphasis on

Marketplace Results

Limits Lessons for the

Federal Government

The private-sector companies we interviewed varied in terms of the types

of quantitative and qualitative R&D indicators that they collect, but in

general they made limited use of these indicators in their decisions. Many

companies stressed the difficulties involved in measuring the contribution

of R&D to the firm’s overall performance using return on investment,

patents issued, and other R&D output measures. All of the firms mentioned

that they were increasing R&D that contributes directly to the bottom line

of the firm. Thus, they shifted the responsibility for R&D decisions to the

business-unit level so that the R&D would be tied more directly to the

profits of those units. The private sector’s experience offers general

lessons to the federal government in terms of ensuring that the R&D

Page 22 GAO/RCED-97-91 Measuring Performance
Chapter 3 

R&D Output Indicators Can Provide Limited

Information About R&D Results

contributes directly to the mission of the organization, although the

specific output measures do not apply directly to federal R&D efforts.

Companies Cited

Difficulties in Measuring

the Results of R&D

Companies told us that measuring the return on R&D investment is very

difficult. Companies stated that one factor making measurement more

difficult is the long time lag between the research and any revenue that

might be earned. Companies also stated that because so many people have

been involved in a product’s evolution, it is difficult to separate the

contribution of the research unit from that of other units.

Companies also mentioned difficulties with some of the other indicators

that they track. For example, one indicator was alternatively labeled a

“vitality index” or “product innovation metric,” which reflected the share

of the firm’s products that could be considered new products. This

measure provided an indication of how rapidly the company was

incorporating new ideas and research into its products. Several company

officials mentioned that this measure had to be applied carefully because

of the problem of defining a “new” product; some products are completely

different from their predecessors, while others might incorporate cosmetic

changes.

Firms Increased Their

Attention to R&D’s

Contribution to the Bottom

Line

One issue that was mentioned by all of the firms in our discussions was

the increasing emphasis on the relevance of R&D to business needs and the

“bottom line.” These comments came up in a number of contexts. For

example, some firms mentioned that they look for research with a

shorter-term payoff in order to have a greater impact on getting new

products into the marketplace. Other firms cited the increased emphasis

on applied R&D, or the small amount of research that they perform that

could be called basic research. A common element was that these firms

were attempting to reduce what might be called benevolent research:

projects that benefit the industry or the nation but do not have a payoff to

the firm.

However, a number of the firms noted that this emphasis on the bottom

line does not necessarily create the opportunities for breakthrough

products. All of the firms either reserved a certain fraction of their

research funding for these types of projects or developed processes that

made it possible to continue some level of research in areas that might not

be directly aligned with any particular product line. In this context, one

Page 23 GAO/RCED-97-91 Measuring Performance
Chapter 3 

R&D Output Indicators Can Provide Limited

Information About R&D Results

firm mentioned the importance of cooperation with the federal

government and universities in the pursuit of fundamental research.

Firms Have Reduced

Emphasis on R&D Output

Measures

While this increased emphasis on getting the most out of R&D might be

expected to lead to greater efforts to measure the results of research, most

of the firms that we spoke to responded by changing the organization so

that measuring R&D outputs was no longer so important. By shifting the

responsibility for research decisions to the business units in the firm that

make use of the research outputs, the companies have sidestepped the

need for centralized indicators of the quality of research. If the business

units believe that a particular R&D project would increase their profits, the

firm would budget for that R&D. If the business units do not perform up to

expectations in terms of their profitability, the entire unit would be

responsible.

For example, one firm shifted from a policy of centrally directed research

to a policy in which the individual business units make the decisions on

the appropriate research projects. Under the previous arrangement, the

various business units were assessed to support the central R&D efforts;

under the new arrangement, the units pay only for those projects that they

think are valuable. This shift was designed to make the research more

responsive to the needs of the business units, in that these units do not pay

for the research unless they find it useful. This shift also greatly reduced

the emphasis on developing R&D output indicators in the central laboratory

because each of the business units would be reviewed on the basis of its

profitability. The reasoning is that if these units were making poor

decisions on R&D projects, the unit’s overall profitability would decrease.

Private Sector’s Measures

Do Not Directly Apply to

the Federal Government

Our interviews with private-sector firms suggest that many of the R&D

output measures tracked by the private sector do not apply directly to the

federal government. Many of these measures are directly related to the

contribution of the R&D to the bottom line profitability of the firm.

However, federal agencies do not operate in order to make a profit, but to

accomplish a variety of other missions and goals. For example, agencies

conduct R&D to support a variety of missions, such as maintaining national

security or improving citizens’ health. R&D in these areas can contribute

greatly to the quality of life in the United States, even if it has a negative

Page 24 GAO/RCED-97-91 Measuring Performance
Chapter 3 

R&D Output Indicators Can Provide Limited

Information About R&D Results

return on investment.9 Given these very different missions, there is also no

reason to believe that any single measure is appropriate for different

public-sector agencies.

Despite the lack of specific measures that can be translated from the

private to the public sector, there are general lessons to be learned from

the private sector’s experience. Possibly the most important is the

recognition that as the pressures on costs increased at many of the firms

that we interviewed, the firms made significant efforts to ensure that R&D

contributed directly to the bottom line. The federal government likewise

faces increasing pressure on costs such as R&D expenditures but, unlike

the private sector, cannot rely on the marketplace to ensure that the R&D

contributes to the agency’s goals. Federal R&D is undertaken to support a

variety of agency missions, and producing an economic return is not the

primary justification for most federal R&D programs. In fact, the purpose of

federal R&D is to promote research that is socially beneficial but may not

be profitable for the private sector to pursue. Without this competitive

marketplace to ensure the relevance of R&D, the federal agencies will

continue to be challenged to develop better measures of the outputs of

their R&D.

The literature confirms this general finding of our discussions. For

example, in 1995 the National Research Council reported on the results of

a workshop on what the federal government might learn from the

corporate experience in research restructuring and assessment.10 The

Council invited senior corporate research managers from IBM, AT&T, Ford,

and Xerox to discuss their experiences in this area. The report concluded

that developing useful metrics and using them appropriately is a difficult

problem, but it is not impossible. In addition, the participants were not

successful in translating their private-sector experience into specific

lessons about what can be measured and what makes sense to measure for

the federal government.

9

The return-on-investment measure that is appropriate for the federal government would compare the

total “social benefits” to the nation with the costs of the initial investment. Computing this measure

has many of the difficulties of computing the return on investment for private firms, with the added

complications associated with placing a value on national security, improved quality of life, and other

intangible qualities.

10Research Restructuring and Assessment: Can We Apply the Corporate Experience to Government

Agencies?, 72 pp., National Research Council (1995).

Page 25 GAO/RCED-97-91 Measuring Performance
Chapter 3 

R&D Output Indicators Can Provide Limited

Information About R&D Results

Federal Science

Agencies Are Still

Exploring Ways to

Measure the Impacts

of R&D

Our July 1996 testimony on the implementation of GPRA noted that the

Congress recognized that successful implementation will not come quickly

or easily for many agencies.11 To help address the challenges of

“measuring” the results of R&D programs, the Research Roundtable, a

group of federal researchers and managers representing a cross-section of

departments and agencies, has met periodically to share ideas and

approaches for implementing GPRA. The Army Research Laboratory has

also begun to address this issue in a pilot project for performance

measurement under GPRA.

The Research Roundtable has considered the extent to which R&D agencies

can and should adopt a common approach to measuring performance. In

1995, it issued a paper based on 6 months of discussions on the

development of performance measures for research programs. Although

the Roundtable stated that the results of a research program’s

performance can be measured, it cautioned that at the same time, it is

important to recognize the complexity of the cause-effect relationship

between R&D and its results. It added that this complexity makes it difficult

to establish quantifiable measures that consistently measure program

performance. It also noted that such measures create a potential for

incorrect application, which could lead subsequently to a detrimental

effect on scientific endeavors. It warned that quantitative empirical

demonstrations of such cause-effect relationships should not be required

and are often not even possible.

The Army Research Laboratory was designated a pilot project for

performance measurement under GPRA. Of the more than 70 such pilot

projects governmentwide, the laboratory was the only pilot project that

addresses scientific research. As such, it attempted to break new ground

in both the planning and the evaluation of basic and applied research. The

Chief of the Army Research Laboratory’s Special Projects Office, who is

mainly responsible for designing the laboratory’s approach to

implementing GPRA, submitted a case study to the Office of Management

and Budget in 1996.12 The case study outlines an approach that makes use

of “three pillars:” metrics, peer review, and customer feedback.

In the case study, the laboratory identified about 60 metrics, most of which

measure input using fiscal, facilities, and personnel data. Some of the

11Managing for Results: Key Steps and Challenges in Implementing GPRA in Science Agencies

(GAO/T-GGD/RCED-96-214, July 10, 1996).

12Brown, Edward A. “Applying the Principles of the Government Performance and Results Act to the

Research and Development Function” (1996).

Page 26 GAO/RCED-97-91 Measuring Performance
Chapter 3 

R&D Output Indicators Can Provide Limited

Information About R&D Results

metrics, such as tasks completed, patents awarded, and articles published,

measure output; none measure outcome. The laboratory views the

measures as useful tools for understanding the functional health of the

organization and the management of the laboratory but cautions that the

information will not enable it to determine the real quality and impact of

its R&D. The laboratory is relying more heavily on peer review of its

research and on customer surveys for information about quality.

According to the case study, the laboratory has contracted with the

National Research Council of the National Academies of Science and

Engineering to conduct a retrospective peer review of research over a 2-

or 3-year period. The Council is to assemble a Technical Assessment

Board that consists of six panels with 8 to 10 people, each of whom is of

high repute within the technical community. These panels will appraise

the quality of the laboratory’s technical and scientific efforts and, to a

limited extent, productivity.

The case study also described the use of customer feedback. The

laboratory has identified a number of internal and external customers. For

those customers to whom it delivers specific items, it uses a series of

feedback questionnaires to determine their degree of satisfaction on a 1 to

5 scale in terms of the quality, timeliness, and utility of the deliverable. For

those customers who do not receive a specific, identifiable product, the

laboratory is developing a Stakeholders’ Advisory Board of senior

leadership and user representatives to provide first-hand guidance and

feedback.

Page 27 GAO/RCED-97-91 Measuring Performance
Appendix I 

Historical Perspective on Research

Spending

Since World War II, U.S. public and private research and development

(R&D) spending patterns have reflected changing priorities as well as

reactions to the changing national and international economies. Some of

the more prominent events that have shaped spending in both sectors have

been the Cold War and the recent international competitive pressures.

Declines in spending in both sectors have been less frequent than

increases.

Federal Spending In his July 1945 report, Science—The Endless Frontier, Vannevar Bush,

who headed the U.S. R&D effort during World War II, provided the rationale

for federal support of both basic research and research related to national

security, industry, and human health and welfare. His plan contributed to

the legislation adopted in 1950 that established the National Science

Foundation (NSF). By that time, however, the National Institutes of Health

(NIH) already had control over most health-related research; the Office of

Naval Research had taken on a major role in supporting academic

research in the physical sciences; and the new Atomic Energy Commission

had been assigned control of the R&D on nuclear weapons and nuclear

power. NSF’s mission focused on supporting fundamental research and

related educational activities. Its annual budget was less than $10 million

until the late 1950s. In contrast, NIH’s annual budget, which had been less

than $3 million at the end of the war, grew to more than $50 million by

1950.

The scope of federal R&D support grew in the decade after World War II.

Anxiety over the Cold War and the loss in 1949 of the U.S. monopoly in

nuclear weapons led to expanded R&D programs in the Army and in the

newly established Air Force and to a continuing buildup in support for

nuclear weapons R&D in the Atomic Energy Commission. On the civilian

side, R&D programs were established or expanded in fields with direct

practical importance, such as aeronautics technology, water

desalinization, and atmospheric disturbances and weather. The launch of

Sputnik by the Soviet Union in 1957 led to immediate efforts to expand

U.S. R&D, science and engineering education, and technology deployment.

Within months of the Sputnik launch, the National Aeronautics and Space

Administration (NASA) and the Advanced Research Projects Agency (ARPA)

in the Department of Defense (DOD) were established. NASA’s core included

the aeronautics programs of the National Advisory Committee on

Aeronautics and some of the space activities of DOD; ARPA’s purpose was to

enable DOD to conduct advanced R&D to meet military needs.

Page 28 GAO/RCED-97-91 Measuring Performance
Appendix I 

Historical Perspective on Research

Spending

Federal appropriations for R&D and for mathematics and science education

in NSF and other government agencies rose rapidly over the next decade,

often at double-digit rates in real terms. (See fig. I.1.) During the early

1950s, the growth in federal funding for health research slowed

considerably from its fast pace in the immediate postwar years. However,

in the late 1950s federal support for health research accelerated rapidly.

From 1966 to 1975, federal support for nondefense R&D dropped nearly

22 percent in real terms. The costs of the Vietnam War put downward

pressure on nondefense R&D along with other nondefense discretionary

spending. The conclusion of NASA’s Apollo program contributed to the

decline in federal R&D funding during that period, also. Until recently, this

was the only period in which federal funds for R&D were reduced

substantially.

Page 29 GAO/RCED-97-91 Measuring Performance
Appendix I 

Historical Perspective on Research

Spending

Figure I.1: U.S. R&D Spending, 1953-96

1954 1956 1958 1960 1962 1964 1966 1968 1970 1972 1974 1976 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996

0

20

40

60

80

100

120

140

160

Year

Constant 1987 dollars (billions)

Industry

Total U.S. spending

Federal

Source: National Science Foundation, Science Resources Studies Division.

In the 1970s, new R&D-intensive agencies addressed environmental and

energy issues. Both the environmental movement and the energy crisis of

the 1970s, according to some analysts, led to increased public and private

spending on environmental and energy R&D. The Environmental Protection

Agency was established in 1970. In 1977, the Energy Research and

Development Administration and other federal energy-related activities

were combined to form the Department of Energy, which was given major

new responsibilities to fund energy-related R&D.

Page 30 GAO/RCED-97-91 Measuring Performance
Appendix I 

Historical Perspective on Research

Spending

In the 1980s, the federal role in R&D expanded to enable the United States

to compete in a global market. Out of this atmosphere, several programs

were initiated to provide financial and other incentives for industrial R&D

and for industrially related R&D conducted at universities or federal

laboratories.

Two major factors have influenced federal support for R&D funding in the

1990s. These factors have been the efforts to reduce the budget deficit and

the defense drawdown. The Department of Defense and the Department of

Energy, two of the four largest sources of federal R&D support, have had

constant-dollar reductions in R&D obligations during the 1990s. In fiscal

year 1995, Defense accounted for roughly half of all federal R&D

obligations, down from nearly two-thirds of the total in 1986 at the height

of the defense build-up that occurred during the Reagan administration.

At the same time, military-related R&D spending was being curtailed, while

federal investment in selected civilian R&D activities increased, including

support for research aimed at improving health and the environment and

for technology advancement. The Department of Health and Human

Services, which is a distant second to Defense in terms of total R&D

support, had the largest absolute increase—$3 billion—in federal R&D

obligations during the 1990s. The proportion of all U.S. R&D devoted to

health-related projects has been increasing continuously for nearly a

decade. The Commerce Department has registered the largest percentage

increase in federal R&D obligations during the 1990s so far. In addition, the

federal government, which supplies about three-fifths of all funds used to

perform R&D on college and university campuses, has been increasing its

support of academic research continuously since 1982, even after

adjusting for inflation.13

Private-Sector

Funding

During the early post-World War II period, thriving domestic and

international markets supported large profits and the rapid expansion of

R&D in both the central laboratories and the divisional laboratories of large

companies. Central R&D facilities focused increasingly on fundamental

research in many of these large firms, leaving the development and

application of new technologies, as well as the improvement of established

products and processes, to the divisional laboratories. The data on basic

research for 1953 through 1960 are less reliable than those for later years

but suggest that the share of total U.S. basic research financed by industry

13Much of this section is based on Allocating Federal Funds for Science and Technology by the

Committee on Criteria for Federal Support of Research and Development of the National Academy of

Sciences, National Academy Press (1995).

Page 31 GAO/RCED-97-91 Measuring Performance
Appendix I 

Historical Perspective on Research

Spending

during the postwar period may well have been at its peak during the 1950s

and early 1960s.

Severe competitive pressures from foreign firms, increases in the real cost

of capital, and a slowdown in the growth rate of the domestic economy in

the 1970s have been cited as causes for the apparent decline in the returns

to R&D investment during the mid-1970s, and the rate of growth in real

industry expenditures on R&D declined. Industry’s funding of basic

research shrank, and many of the central research facilities of the giant

corporations entered a period of budgetary austerity or cutbacks. After a

resurgence in the early 1980s, the rate of growth in industry-funded R&D

declined in the late 1980s.

Industry’s funding of R&D was generally flat between 1991 and 1994;

several reasons have been cited for the lack of growth in some companies’

R&D programs during this time. These include corporate downsizing,

decentralization (i.e., a shift of R&D activity from corporate laboratories to

individual business units), and increasing collaboration among industrial

firms and with partners in academia, in government, in the nonprofit

sector, and in other countries. The preliminary data for 1995 and 1996

indicate a slight upswing in industrial R&D funding.

According to NSF, the most striking recent trend in industrial R&D

performance has been the increase in the proportion of total R&D funded

by the companies classified as nonmanufacturing industries. Prior to 1983,

nonmanufacturing industries accounted for less than 5 percent of the

industry’s total R&D. That share grew steadily during the next decade so

that in 1993, nonmanufacturing firms represented more than 25 percent of

all industrial R&D performed in the United States. Part of this reported

growth, however, reflects improvements in the NSF’s survey efforts to

measure R&D in nonmanufacturing industries, in addition to actual growth

in service sector R&D spending.

Between 1984 and 1994, some significant changes occurred among the 100

largest publicly held R&D funding companies, although the four leading

firms were the same in both 1984 and 1994. During the decade, the number

of pharmaceutical and computer hardware and software companies

among the largest R&D funders rose. In contrast, the number of large

defense contractors and chemical and petroleum companies among the

largest R&D funders fell.

Page 32 GAO/RCED-97-91 Measuring Performance
Appendix I 

Historical Perspective on Research

Spending

U.S. firms have begun to alter their R&D patterns in response to increasing

competitive pressures. Firms have shifted a greater portion of their R&D

resources away from long-term investments and toward shorter-term

projects. U.S. companies now allocate 22 percent of their R&D spending to

long-term projects when compared with their Japanese counterparts, who

devote 50 percent. Increasingly, firms are emphasizing short-term R&D for

immediate problem-solving or near-term development over basic research;

and basic research is being directed toward the needs of product

development and manufacturing teams. Many central research

laboratories at large companies—such as AT&T, IBM, General Electric,

Kodak, and Xerox—have been downsized and work more closely with

product development divisions. They now receive a larger share of their

operating funds from individual business units rather than general

corporate funds.

Page 33 GAO/RCED-97-91 Measuring Performance
Appendix II 

Major Contributors to This Report

Resources,

Community, and

Economic

Development

Division, Washington,

D.C.

Victor S. Rezendes, Director

Robin M. Nazzaro, Assistant Director

Andrew J. Vogelsang, Evaluator-in-Charge

Dennis Carroll, Senior Evaluator

Office of the Chief

Economist

Loren Yager, Assistant Director

(307738) Page 34 GAO/RCED-97-91 Measuring Performance
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100